PDF (Portable Document Format) is one of the most complex document formats in wide use, combining a page description language, font subsetting, image embedding, encryption, and a cross-reference table for random access into a single container. Parsing PDF content programmatically—extracting text, images, and structural metadata—requires understanding how these layers interact and implementing parsers that can handle the format’s many edge cases and historical revisions.
Building a PDF parser in systems languages like Go and Zig offers distinct advantages: Go provides excellent concurrency primitives for processing multi-page documents in parallel, while Zig’s manual memory control and comptime features allow for zero-allocation parsing paths critical to high-throughput document pipelines. Both languages avoid the overhead of garbage-collected runtimes that can become a bottleneck when processing millions of documents.
In a typical document ingestion pipeline, the PDF parser handles page-level text extraction with positional information, font decoding across CFF, SFNT (TrueType/OpenType), and Type1 formats, embedded image extraction, and text layout reconstruction—converting the PDF’s absolute-positioned text operators into readable, ordered text streams. Both Go and Zig implementations of such parsers are explored in the examples below.
A PDF file is fundamentally a collection of numbered objects connected through cross-references. The file ends with a cross-reference table (xref) and a trailer that points to the document’s root object (the catalog). Objects can be booleans, numbers, strings, names, arrays, dictionaries, or streams. Streams are dictionary-object pairs where the dictionary describes the stream’s properties (length, compression filter) and the stream contains binary data—page content, images, or embedded fonts.
The xref table maps object numbers to byte offsets in the file, enabling random access to any object without reading the entire file sequentially. Modern PDFs may use cross-reference streams (compressed xref data inside a stream object) instead of the traditional plain-text xref table. A robust parser must handle both formats, plus incremental updates where multiple xref sections are chained together.
Page content is described by a sequence of operators in a content stream. Text rendering uses operators like BT (begin text), ET (end text), Tf (set font), Td/Tm (position text), and Tj/TJ (show text). The TJ operator is particularly important—it accepts an array of strings and numeric kerning adjustments, allowing fine-grained character positioning. Extracting readable text requires interpreting these operators in the context of the current graphics state (font, font size, text matrix, character spacing).
PDF positions text using a 3×3 transformation matrix (the “text matrix”) that combines translation, rotation, and scaling. Each Td or Tm operator modifies this matrix. To reconstruct reading order, a parser must track matrix state across all text operators and use the resulting (x, y) positions to sort text fragments into lines and columns.
PDF supports several font formats, each requiring different parsing logic:
PDF fonts use encoding mappings (CMaps) to translate character codes in the content stream to glyph indices in the font program. A ToUnicode CMap provides the reverse mapping from glyph indices to Unicode code points, which is essential for text extraction. When ToUnicode is missing, the parser must fall back to the font’s built-in encoding, standard encodings (WinAnsiEncoding, MacRomanEncoding), or heuristic glyph-name-to-Unicode mapping.
Images in PDF are stored as stream objects with specific color space and compression configurations. Common filters include DCTDecode (JPEG), FlateDecode (zlib/PNG), CCITTFaxDecode (fax compression for scanned documents), and JBIG2Decode. An image stream’s dictionary specifies width, height, bits per component, and color space (DeviceRGB, DeviceCMYK, DeviceGray, ICCBased, or Indexed). Extracting usable images requires decoding the filter chain and converting the color space to a standard format like sRGB.
PDF supports two image embedding methods. XObject images are standalone objects referenced by name from content streams (/Im1 Do), making them easy to locate via the page’s resource dictionary. Inline images are embedded directly in the content stream between BI (begin image) and EI (end image) operators, requiring the parser to handle them during content stream interpretation rather than as a separate extraction pass.
const std = @import("std");
const TokenTag = enum {
integer,
real,
name, // /SomeName
string_lit, // (parenthesized string)
string_hex, // <hex encoded>
array_open, // [
array_close, // ]
dict_open, // <<
dict_close, // >>
keyword, // true, false, null, obj, endobj, stream, etc.
eof,
};
const Token = struct {
tag: TokenTag,
// Slice into the source buffer — no allocations
bytes: []const u8,
};
const PdfScanner = struct {
src: []const u8,
pos: usize,
pub fn init(src: []const u8) PdfScanner {
return .{ .src = src, .pos = 0 };
}
pub fn next(self: *PdfScanner) Token {
self.skipWhitespaceAndComments();
if (self.pos >= self.src.len) return .{ .tag = .eof, .bytes = "" };
const c = self.src[self.pos];
return switch (c) {
'/' => self.scanName(),
'(' => self.scanStringLiteral(),
'[' => self.single(.array_open),
']' => self.single(.array_close),
'<' => self.scanAngleBracket(),
'>' => self.scanDictClose(),
'0'...'9', '-', '+', '.' => self.scanNumber(),
else => self.scanKeyword(),
};
}
fn scanName(self: *PdfScanner) Token {
const start = self.pos;
self.pos += 1; // skip '/'
while (self.pos < self.src.len and !isDelimiter(self.src[self.pos])) {
self.pos += 1;
}
return .{ .tag = .name, .bytes = self.src[start..self.pos] };
}
fn isDelimiter(c: u8) bool {
return switch (c) {
' ', '\t', '\n', '\r', '(', ')', '<', '>',
'[', ']', '{', '}', '/', '%' => true,
else => false,
};
}
// ... remaining scanner methods
};package pdfextract
import (
"fmt"
"math"
"sort"
"sync"
)
// TextFragment represents a positioned piece of text on a page.
type TextFragment struct {
Text string
X, Y float64 // position in PDF user space (origin bottom-left)
Font string
Size float64
}
// ExtractText processes all pages concurrently and returns ordered text.
func ExtractText(doc *Document) ([]string, error) {
pages := doc.Pages()
results := make([]string, len(pages))
errs := make([]error, len(pages))
var wg sync.WaitGroup
// Limit concurrency to avoid excessive memory use on large docs
sem := make(chan struct{}, 8)
for i, page := range pages {
wg.Add(1)
go func(idx int, p *Page) {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
fragments, err := extractPageFragments(p)
if err != nil {
errs[idx] = fmt.Errorf("page %d: %w", idx+1, err)
return
}
results[idx] = reconstructText(fragments)
}(i, page)
}
wg.Wait()
for _, err := range errs {
if err != nil {
return results, err
}
}
return results, nil
}
// reconstructText sorts fragments into reading order and joins them.
func reconstructText(fragments []TextFragment) string {
if len(fragments) == 0 {
return ""
}
// Sort by Y descending (top to bottom), then X ascending (left to right)
sort.Slice(fragments, func(i, j int) bool {
// Group into lines: fragments within 2pt vertical distance
if math.Abs(fragments[i].Y-fragments[j].Y) > 2.0 {
return fragments[i].Y > fragments[j].Y
}
return fragments[i].X < fragments[j].X
})
var result []byte
prevY := fragments[0].Y
prevX := fragments[0].X + estimateWidth(fragments[0])
for i, frag := range fragments {
if i > 0 {
if math.Abs(frag.Y-prevY) > 2.0 {
result = append(result, '\n')
} else if frag.X-prevX > frag.Size*0.3 {
result = append(result, ' ')
}
}
result = append(result, frag.Text...)
prevY = frag.Y
prevX = frag.X + estimateWidth(frag)
}
return string(result)
}In a search and retrieval pipeline, PDF parsing feeds directly into document ingestion. Documents arrive from various sources (uploads, web crawlers, S3 buckets), and the PDF parser extracts text content with positional metadata. This extracted text is then tokenized, indexed, and made available for full-text search. The positional information enables features like highlighting search results at the exact location in the original document.
Many PDFs use multi-column layouts, tables, headers, footers, and sidebars. Naive text extraction (reading operators in stream order) produces garbled output that interleaves columns. A layout-aware parser can implement column detection by analyzing the X-coordinate distribution of text fragments, identifying column boundaries, and processing each column independently before merging them in reading order.
PDF generators often embed only the glyphs used in the document (font subsetting) and may use custom encodings where character code 1 maps to “A”, code 2 to “e”, etc. Without the ToUnicode CMap, these documents produce gibberish on extraction. The parser implements a fallback chain: ToUnicode CMap → font encoding differences → standard encoding → glyph name mapping (via the Adobe Glyph List) → OCR-based recovery as a last resort.
PDF documents can contain thousands of objects, but a text extraction pass only needs a fraction of them. The lazy resolution pattern reads objects on demand using the xref table’s byte offsets rather than parsing the entire file upfront. This is especially important for large documents (hundreds of megabytes) where eager parsing would consume excessive memory.
PDF streams can have multiple compression filters applied in sequence (e.g., /ASCIIHexDecode followed by /FlateDecode). The filter chain pattern wraps each filter as an io.Reader (Go) or a streaming decoder (Zig), composing them so that reading from the outermost reader automatically decompresses through the entire chain:
// Go: composable stream filter chain
func decodeStream(raw io.Reader, filters []string) (io.Reader, error) {
r := raw
for _, filter := range filters {
switch filter {
case "/FlateDecode":
zr, err := zlib.NewReader(r)
if err != nil {
return nil, fmt.Errorf("flate: %w", err)
}
r = zr
case "/ASCIIHexDecode":
r = newASCIIHexDecoder(r)
case "/ASCII85Decode":
r = ascii85.NewDecoder(r)
case "/LZWDecode":
r = lzw.NewReader(r, lzw.MSB, 8)
case "/DCTDecode":
// JPEG data — return as-is for image extraction
return r, nil
default:
return nil, fmt.Errorf("unsupported filter: %s", filter)
}
}
return r, nil
}PDF content streams use q (save) and Q (restore) operators to push and pop the graphics state. The parser maintains a stack of state objects containing the current transformation matrix, font, font size, text matrix, color, and other rendering parameters. This stack-based approach ensures that nested graphical contexts (e.g., form XObjects drawn within a page) correctly inherit and restore state.
%%EOF marker) and follow xref chains backward./ObjStm). Parsers that only look for obj...endobj at the top level will miss objects stored inside these compressed containers.