PDF is a presentation format, not a semantic one. A PDF file describes where to place glyphs on a page—it has no concept of paragraphs, columns, tables, or reading order. Extracting readable text from a PDF requires reconstructing these higher-level structures from raw glyph positions. This is the job of a layout analyzer: it takes a flat list of positioned characters and infers the logical layout—columns, paragraphs, headers, footnotes, tables—so that text can be read in the correct order.
Column detection is the hardest subproblem. Academic papers, legal documents, newspapers, and depositions frequently use multi-column layouts. Naive left-to-right, top-to-bottom extraction produces garbled text that interleaves columns. For example, a document processing library might implement an adaptive layout analyzer that uses gap-bucketing heuristics to detect column boundaries, filters out line-number columns (common in depositions and legal filings), reconstructs tables from aligned cell positions, and toggles deposition-specific heuristics when the document matches certain structural patterns.
Getting layout analysis right is critical for downstream NLP tasks—search indexing, question answering, and summarization all depend on receiving coherent, correctly-ordered text. A layout error that merges two columns produces text that is not just hard to read but semantically wrong, leading to incorrect search results and hallucinated answers.
PDF text extraction starts with a stream of glyphs, each having an (x, y) position, width, height, font name, and font size. The layout analyzer’s first job is to group these glyphs into words (glyphs separated by small horizontal gaps) and then into lines (words at the same vertical position within a tolerance). Lines are the fundamental unit of layout analysis—all column and paragraph detection operates on lines and the gaps between them.
PDF uses a coordinate system where (0, 0) is the bottom-left of the page, with Y increasing upward. Text extraction libraries often flip this to a top-left origin with Y increasing downward to match screen conventions. The layout analyzer must be consistent about which coordinate system it uses—mixing them produces upside-down reading order. All gap calculations and column detection depend on correct Y-axis orientation.
The core column detection algorithm works by analyzing horizontal gaps between text blocks on each line. Within a column, word spacing is small and relatively uniform. Between columns, there is a significantly larger gap. The analyzer collects all inter-word gaps on a page, builds a histogram, and looks for a bimodal distribution: a cluster of small gaps (word spacing) and a cluster of large gaps (column gutters). The boundary between clusters defines the column-gap threshold.
A fixed column-gap threshold fails across different documents—a narrow two-column academic paper has different gutters than a wide three-column newspaper layout. Adaptive thresholding computes the threshold per page (or per region) by analyzing the actual gap distribution. One effective approach uses Otsu’s method adapted for 1D gap histograms: it finds the threshold that minimizes intra-class variance between “word gaps” and “column gaps,” automatically adapting to each document’s spacing.
Legal depositions and court transcripts often have a narrow column of line numbers along the left margin. These numbers (1, 2, 3, … 25) are not part of the main text content but appear as a detected column. The analyzer identifies line-number columns by checking for a narrow column where every entry is a sequential integer. When detected, the column is stripped from the output, and the remaining columns are renumbered.
When the analyzer detects structural markers of a deposition (line-number columns, Q/A formatting, speaker labels like “THE WITNESS:”), it activates deposition-specific heuristics. These include preserving Q/A structure in the output, collapsing speaker labels with their dialogue, and handling the common pattern where an answer wraps across multiple lines with hanging indentation. This mode toggle illustrates a broader pattern: domain-specific document types often need specialized extraction logic layered on top of general layout analysis.
Tables are detected by finding groups of text blocks that are aligned both horizontally (same column positions across rows) and vertically (consistent row spacing). The analyzer identifies table regions by looking for grid-like alignment patterns, then extracts cell contents by position. Horizontal and vertical rules (drawn as PDF path objects, not text) provide additional signals for table boundary detection.
package layout
import (
"math"
"sort"
)
// TextBlock represents a word or text fragment with its position.
type TextBlock struct {
X, Y, W, H float64
Text string
FontSize float64
}
// Line is a horizontal row of text blocks at approximately the same Y.
type Line struct {
Y float64
Blocks []TextBlock
}
// DetectColumns analyzes gaps between text blocks to find column boundaries.
func DetectColumns(lines []Line) []float64 {
// Collect all inter-block gaps.
var gaps []float64
for _, line := range lines {
sort.Slice(line.Blocks, func(i, j int) bool {
return line.Blocks[i].X < line.Blocks[j].X
})
for i := 1; i < len(line.Blocks); i++ {
gap := line.Blocks[i].X - (line.Blocks[i-1].X + line.Blocks[i-1].W)
if gap > 0 {
gaps = append(gaps, gap)
}
}
}
if len(gaps) == 0 {
return nil // Single column or empty page.
}
// Find threshold using Otsu's method on the gap distribution.
threshold := otsuThreshold(gaps)
// Column boundaries are the X positions of gaps exceeding the threshold.
var boundaries []float64
for _, line := range lines {
for i := 1; i < len(line.Blocks); i++ {
gap := line.Blocks[i].X - (line.Blocks[i-1].X + line.Blocks[i-1].W)
if gap >= threshold {
midpoint := line.Blocks[i-1].X + line.Blocks[i-1].W + gap/2
boundaries = append(boundaries, midpoint)
}
}
}
// Cluster nearby boundaries and return centroids.
return clusterBoundaries(boundaries, threshold*0.5)
}
// otsuThreshold finds the optimal threshold that separates two gap classes.
func otsuThreshold(gaps []float64) float64 {
sort.Float64s(gaps)
n := len(gaps)
bestThresh := gaps[n/2] // Fallback to median.
bestVariance := math.Inf(1)
for i := 1; i < n; i++ {
thresh := (gaps[i-1] + gaps[i]) / 2
mean0, mean1 := mean(gaps[:i]), mean(gaps[i:])
w0, w1 := float64(i)/float64(n), float64(n-i)/float64(n)
variance := w0*w1*math.Pow(mean0-mean1, 2)
// Otsu maximizes inter-class variance (equivalent to minimizing intra).
if variance > 0 && 1/variance < bestVariance {
bestVariance = 1 / variance
bestThresh = thresh
}
}
return bestThresh
}// IsLineNumberColumn checks if a column contains only sequential integers.
func IsLineNumberColumn(blocks []TextBlock) bool {
if len(blocks) < 5 {
return false // Need enough lines to confirm the pattern.
}
prevNum := 0
sequential := 0
for _, b := range blocks {
text := strings.TrimSpace(b.Text)
num, err := strconv.Atoi(text)
if err != nil {
return false // Non-numeric content disqualifies.
}
if prevNum > 0 && num == prevNum+1 {
sequential++
}
prevNum = num
}
// At least 80% of entries must be sequential.
return float64(sequential) / float64(len(blocks)-1) >= 0.8
}
// FilterLineNumbers removes line-number columns from the layout.
func FilterLineNumbers(columns []Column) []Column {
var filtered []Column
for _, col := range columns {
if !IsLineNumberColumn(col.Blocks) {
filtered = append(filtered, col)
}
}
return filtered
}Court filings, depositions, and contracts are among the most layout-complex PDFs. They combine multi-column layouts, line numbering, headers/footers, exhibit stamps, and Bates numbers. The A well-designed layout analyzer’s deposition mode can be specifically developed for this domain, enabling accurate text extraction from documents that generic PDF-to-text tools mangle.
Research papers typically use two-column layouts with figures, tables, and equations interspersed. The layout analyzer correctly reads the left column top-to-bottom before the right column, skips figure captions that would interrupt paragraph flow, and reconstructs tables for separate structured extraction.
The quality of text extraction directly impacts search relevance. If a two-column academic paper is extracted as interleaved columns, a search for “gradient descent converges” may fail because “gradient” is on one extracted line and “descent converges” ended up spliced with text from the adjacent column. Correct column detection ensures that search queries match against coherent text passages.
Layout analysis works best as multiple passes over the same data. The first pass extracts raw glyphs and groups them into lines. The second pass analyzes gaps to detect columns. A third pass identifies special regions (tables, headers, footnotes). This separation of concerns makes each pass simpler and testable in isolation.
// AnalyzePage runs the full layout analysis pipeline.
func AnalyzePage(glyphs []Glyph, opts AnalyzerOptions) *PageLayout {
// Pass 1: Group glyphs into words and lines.
lines := GroupIntoLines(GroupIntoWords(glyphs))
// Pass 2: Detect columns via gap analysis.
boundaries := DetectColumns(lines)
columns := AssignToColumns(lines, boundaries)
// Pass 3: Filter special columns and detect tables.
if opts.FilterLineNumbers {
columns = FilterLineNumbers(columns)
}
tables := DetectTables(columns)
// Pass 4: Domain-specific post-processing.
if opts.DepositionMode || DetectDeposition(columns) {
columns = ApplyDepositionHeuristics(columns)
}
return &PageLayout{
Columns: columns,
Tables: tables,
ReadOrder: BuildReadOrder(columns, tables),
}
}When grouping glyphs into words or words into lines, exact coordinate matching fails because PDF renderers introduce sub-pixel positioning variations. All grouping uses tolerance-based comparisons: two glyphs are on the “same line” if their Y coordinates differ by less than half the font height. Tolerance values are derived from the actual font metrics on the page, not hardcoded constants, making the analyzer robust across different document styles.
The analyzer assigns confidence scores to its layout decisions. A page where the gap histogram shows a clear bimodal distribution gets a high confidence for its column count. A page with ambiguous gaps gets a low score, signaling downstream consumers to treat the extracted text with more caution or flag it for human review.