What is actually inside a PDF: writing one by hand, 588 bytes at a time

Most people meet the PDF as a sealed object. It arrives, it opens, the text sits where it was put, and that is the end of the relationship. The format has a reputation for being impenetrable, which is odd, because a PDF is mostly plain text you can read in any editor.

The best way to see that is to write one. Not generate one — write one, by typing. What follows is a complete PDF, 588 bytes, that renders in Chrome, Preview, Acrobat and every other reader. There is no library involved and no binary section at all.

The whole file

This is all of it:

%PDF-1.4
1 0 obj
<< /Type /Catalog /Pages 2 0 R >>
endobj
2 0 obj
<< /Type /Pages /Kids [3 0 R] /Count 1 >>
endobj
3 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 80]
   /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>
endobj
4 0 obj
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>
endobj
5 0 obj
<< /Length 42 >>
stream
BT /F1 18 Tf 20 34 Td (Hello, bytes) Tj ET
endstream
endobj
xref
0 6
0000000000 65535 f 
0000000009 00000 n 
0000000058 00000 n 
0000000115 00000 n 
0000000243 00000 n 
0000000313 00000 n 
trailer
<< /Size 6 /Root 1 0 R >>
startxref
405
%%EOF

Save that as hello.pdf and open it. You get a small white page with the words Hello, bytes in Helvetica. That is a real PDF, and you just read the entire specification-conformant contents of it.

Objects, and the arrows between them

A PDF is a set of numbered objects that point at each other. The syntax 3 0 R means "object number 3, generation 0, referenced here" — it is a pointer, and the R stands for reference. The generation number matters only for files that have been edited in place, which we will come back to.

The five objects above form a chain, and the chain is the document:

  • Object 1, the catalog. The root. It says almost nothing itself; it points at the page tree.
  • Object 2, the page tree. Holds the list of pages and how many there are. In a real document this is a tree rather than a list, so a reader can jump to page 400 without walking through 399 others.
  • Object 3, the page. Its size (/MediaBox [0 0 200 80] — 200 by 80 points, about 70 by 28 millimetres), which fonts it can use, and where its drawing instructions live.
  • Object 4, the font. Here it names Helvetica and stops. Helvetica is one of fourteen fonts every conforming reader is required to have, which is why nothing needed to be embedded. This is also the root of a great deal of trouble, and there is a separate article in it.
  • Object 5, the content stream. The only part that draws anything.

The content stream is a tiny programming language

Those 44 bytes are not markup. They are instructions for a stack machine, written in postfix: operands first, operator last.

BT                 begin text
/F1 18 Tf          use font F1 at 18 points
20 34 Td           move to x=20, y=34
(Hello, bytes) Tj  show this string
ET                 end text

The coordinate origin is the bottom left, not the top left, because PDF inherited its coordinate system from PostScript, which inherited it from mathematics rather than from screens. Anyone who has written a PDF generator has been bitten by this at least once.

Notice what the file does not contain: any statement that these characters form a word, a line, a paragraph or a sentence. It says "put these glyphs at this position". Everything above that — word boundaries, reading order, whether this is a heading — is inferred by whatever reads the file afterwards. That single fact explains most of what people find frustrating about PDFs, and we will come back to it.

The cross-reference table, and why a PDF is read backwards

After the objects comes the part that makes PDF a random-access format:

xref
0 6
0000000000 65535 f 
0000000009 00000 n 
0000000058 00000 n 
0000000115 00000 n 
0000000243 00000 n 
0000000313 00000 n 

One line per object, each giving the byte offset of that object from the start of the file, padded to ten digits. Object 1 begins at byte 9. Object 4 begins at byte 243. The first entry is a placeholder for object 0, which never exists; f means free, n means in use.

Then the last three lines, which are the ones a reader looks at first:

trailer
<< /Size 6 /Root 1 0 R >>
startxref
405
%%EOF

A PDF reader opens the file at the end. It seeks to %%EOF, steps back to read startxref, and learns that the cross-reference table begins at byte 405. It jumps there, reads the offsets, learns from the trailer that the root is object 1, jumps to byte 9, and follows the pointers down to the page it actually wants.

This is the whole design goal. A 900-page document does not have to be parsed to show page 700; the reader jumps straight to the bytes it needs. It is also why a PDF truncated by a failed download is usually unopenable while a truncated JPEG still shows its top half — lose the last few hundred bytes of a PDF and you have lost the index to everything else.

Two details that bite, and one mistake I made writing this

Every cross-reference entry is exactly 20 bytes. Ten digits of offset, a space, five digits of generation, a space, one letter, then two more characters of line ending — n or f followed by a space and a newline. The specification is explicit about the width because readers are allowed to seek directly to entry n by multiplying. If you copy the table above into an editor that strips trailing whitespace, every entry becomes 19 bytes and strict readers will reject the file while lenient ones repair it silently. It is a genuinely nasty bug to chase, because the file looks identical.

And I got the stream length wrong. The content stream is introduced by a dictionary declaring how long it is: << /Length 42 >>. My first draft of this file said 44, because I counted the drawing instructions by eye and miscounted by two.

Chrome rendered it anyway. So did Preview. Neither complained, because a reader that hits endstream before the declared length simply stops, and one that overruns usually backtracks to find the keyword. Real-world PDF readers are extraordinarily forgiving, and that tolerance is the reason so many technically invalid PDFs circulate without anyone noticing — until one lands in a stricter pipeline, a print RIP or an archival validator, and fails there instead. The version above is corrected to 42; you can confirm it with grep -abo endstream hello.pdf and a little arithmetic.

That is the honest lesson of writing a PDF by hand: the format is strict, the readers are not, and the gap between those two facts is where most "this file is corrupt" reports come from.

What this explains

Why copied text comes out wrong. The file stores glyph positions, not words. A reader reconstructs words by looking at gaps between glyphs. Justified text, tables, and columns all widen those gaps, which is why pasted text from a two-column paper arrives interleaved and why copying from a table gives you a single run-on line.

Why editing a PDF is hard. There is no sentence to edit. Changing "Hello" to "Goodbye" means editing a drawing instruction, then dealing with the fact that the new word is wider, then re-flowing everything that followed — except nothing "follows" anything, because there are no paragraphs, only coordinates. This is why most PDF editors draw a white box over the old text and put new text on top, and why the ones that genuinely re-flow cost money.

Why PDFs get big. Our file needed no font because Helvetica is assumed. A real document embeds its fonts — often several, often the full character set rather than the subset actually used — and each embedded font is tens to hundreds of kilobytes before a single image is added.

Why file size and page count are barely related. A 40-page text document can be smaller than a 2-page one containing a photograph, because one is a few kilobytes of drawing instructions and the other is a picture.

What the minimal file leaves out

Being straight about the limits of the demonstration: real PDFs differ from this one in four ways that matter.

  • Compression. Content streams are normally wrapped in /Filter /FlateDecode, which is DEFLATE — the same algorithm as ZIP. Open a real PDF in a text editor and the readable structure is still there, but the streams are binary noise.
  • Object streams. Since PDF 1.5 many small objects are packed together into a single compressed stream and the plain xref table is replaced by a cross-reference stream. Same idea, different encoding.
  • Incremental updates. Editing a PDF often appends changes to the end and writes a new cross-reference table pointing back at the old one, rather than rewriting the file. This is why a PDF can grow when you delete something, and why "removed" content can still be sitting in the bytes. It is also why redaction has to flatten the page rather than cover it.
  • Linearization. Files optimised for the web put a second cross-reference table at the front so a browser can render page one before the rest has downloaded.

None of these change the model. They compress and rearrange it.

Check it yourself

Take any PDF on your machine and look at the last 40 bytes:

tail -c 40 yourfile.pdf

You will see startxref, a number, and %%EOF. That number is a byte offset into the same file. Then:

head -c 9 yourfile.pdf

and you will get %PDF-1. and a version digit. Those two probes tell you more about whether a file is healthy than any amount of opening and closing it — a missing or wrong startxref is the single most common reason a PDF will not open, and it is the first thing our own repair tool checks.

If you want to see the structure of a file without a hex editor, drop it on the File Doctor; it reports the version, the object count, whether there is a text layer and whether the cross-reference table agrees with the file, all in your browser.

The one sentence worth keeping

A PDF is a set of numbered objects, an index of where each one starts, and a pointer at the end of the file to that index. Everything else — the compression, the fonts, the encryption, the forms, the tagged accessibility tree — is built on top of those three things. Once the shape is clear, most PDF problems stop being mysterious and start being specific.

Sources

Tools mentioned

Advertisements

← All articles · corrections: editorial policy