Plain OCR turns page images into text. Layout-aware OCR turns page images into structured document evidence: text lines, polygons, words, character boxes, semantic regions, reading order positions, table rows, table columns, cells, and confidence values. That difference matters for AI pipelines because downstream search, extraction, RAG, and analytics do not only need the words on a page. They need to know which words belong to a header, a table cell, a formula, a caption, or a body paragraph.
Surya is a useful reference for this production pattern because it exposes the pipeline as separate but composable tasks: text-line detection, recognition, layout labeling, reading order detection, table recognition, and LaTeX OCR. The repository documents concrete output contracts, batching knobs, GPU memory costs, benchmark methodology, and failure modes. Those details make it more than a tool README; they show how a modern document AI stack is decomposed into measurable stages.
The reusable lesson is to treat document OCR as a geometry-preserving pipeline. Each stage should preserve page coordinates, page numbers, confidence scores, and model-specific metadata so later stages can reorder, group, evaluate, and repair the extraction. A string-only output may look convenient, but it throws away the signals that make complex pages usable.
Modern document OCR commonly detects text lines before recognizing their contents. Surya returns line polygons, axis-aligned boxes, confidence scores, vertical-line detections, and page image bounds. Line-level regions are a good unit for multilingual pages, rotated text, and downstream layout analysis because they preserve enough geometry without forcing every later step to reason about individual pixels.
Layout analysis labels boxes as structural elements such as captions, formulas, list items, page headers, pictures, section headers, tables, forms, handwriting, and text. Reading order then assigns positions to those boxes so the final text can be consumed in human order rather than raster order. This is the difference between indexing a page and reconstructing a document.
Tables encode relationships. Cell text alone is insufficient unless the pipeline also knows row identifiers, column identifiers, row and column boxes, spans, header flags, and the table index on the page. Surya’s table-recognition output contract exposes these fields explicitly, which makes later conversion to JSON, Markdown, or HTML more reliable than attempting to infer table structure from flat OCR text.
A layout-aware OCR system should keep polygon and bounding-box coordinates attached to every extracted artifact. The page text is just one view over a richer spatial representation.
A robust pipeline separates detection, recognition, layout analysis, reading order, and table structure. That separation makes it possible to tune thresholds, benchmark individual stages, and rerun only the expensive step that failed. It also lets a system fall back to a narrower model, such as running equation OCR only on regions labeled as formulas.
from PIL import Image
from surya.foundation import FoundationPredictor
from surya.detection import DetectionPredictor
from surya.recognition import RecognitionPredictor
from surya.layout import LayoutPredictor
from surya.settings import settings
image = Image.open("paper-page.png").convert("RGB")
foundation = FoundationPredictor()
detection = DetectionPredictor()
recognition = RecognitionPredictor(foundation)
layout = LayoutPredictor(
FoundationPredictor(checkpoint=settings.LAYOUT_MODEL_CHECKPOINT)
)
text_predictions = recognition([image], det_predictor=detection)
layout_predictions = layout([image])The important design point is not the exact class names. It is the handoff: each model consumes page images or detected regions and emits structured artifacts with coordinates. A production service should persist these artifacts rather than collapsing them immediately to text.
Different OCR tasks return different shapes. Recognition emits text lines, characters, words, boxes, and confidence values. Layout analysis emits labeled boxes and ordering positions. Table recognition emits rows, columns, cells, spans, headers, and table indexes. A normalizer can convert these into a shared page model before indexing or extraction.
def normalize_page(page_id, ocr_page, layout_page, table_page):
return {
"page_id": page_id,
"lines": [
{
"text": line["text"],
"polygon": line["polygon"],
"confidence": line["confidence"],
"words": line.get("words", []),
}
for line in ocr_page["text_lines"]
],
"regions": [
{
"label": box["label"],
"bbox": box["bbox"],
"position": box.get("position"),
"alternatives": box.get("top_k", {}),
}
for box in layout_page["bboxes"]
],
"tables": [
{
"rows": table_page.get("rows", []),
"cols": table_page.get("cols", []),
"cells": table_page.get("cells", []),
}
],
}Once layout regions have reading-order positions, extraction code can sort regions before generating Markdown, chunks, or table-aware records. The order should be treated as model output with confidence and validation, not as a trivial top-to-bottom sort. Multi-column scientific papers, newspapers, forms, and slide decks all violate naive raster-order assumptions.
Engineering manuals, scientific papers, textbooks, and regulatory PDFs often contain columns, captions, formulas, and tables. A layout-aware OCR pipeline can preserve section headers, read columns in order, keep formulas separate from prose, and attach page coordinates for citation rendering. This gives retrieval systems cleaner chunks and gives answer systems better provenance.
Financial statements, shipment reports, insurance documents, and inventory exports are frequently scanned or image-based. Table recognition converts visual grids into row and column relationships that a downstream validator can check. Without that structure, an LLM may see nearby numbers but miss which row or column gives them meaning.
Surya’s repository emphasizes broad language coverage for OCR while keeping text detection, layout analysis, and reading order language-independent. That separation is useful for archives that mix scripts or contain pages where layout is predictable but text language varies. The pipeline can share geometric processing while swapping or configuring recognition behavior.
Store OCR artifacts next to the source document or in an object store keyed by document version, page number, model name, and model version. The sidecar should contain text, geometry, confidence values, reading order, table relationships, and image bounds. This makes re-indexing deterministic and lets evaluators compare model versions without rerunning the whole ingest job.
{
"document_id": "manual-042",
"page": 17,
"image_bbox": [0, 0, 1654, 2339],
"models": {
"detector": "surya-detection",
"recognition": "surya-recognition",
"layout": "surya-layout"
},
"text_lines": [],
"layout_regions": [],
"tables": []
}Use layout labels to trigger specialized processors. Formula regions can go to LaTeX OCR. Table regions can go to a table structure model. Picture regions can be captioned or skipped. Headers and footers can be indexed separately or filtered when they repeat across pages.
Surya reports separate evaluations for OCR similarity, line detection coverage, layout precision and recall, reading order accuracy, table row and column intersection, and LaTeX edit distance. That is the right pattern. End-to-end quality is important, but stage-level metrics tell the team whether a bad answer came from recognition, ordering, table structure, or downstream extraction.