Many document classifiers treat a page as plain text. That works for email-like documents, but it loses the strongest signal in forms, invoices, financial statements, lab reports, and scanned business records: where text appears on the page. A heading, subtotal, table column, signature block, or footer can change the document class even when the words overlap across categories.
Layout-aware document classification combines three inputs: recognized words, bounding boxes for those words, and the page image itself. LayoutLMv3 is a representative architecture for this pattern. It uses a Transformer that consumes text tokens, normalized two-dimensional box coordinates, and visual patch embeddings, then fine-tunes a sequence-classification head for a target document taxonomy.
The transferable pattern is a document-processing pipeline, not a single model checkpoint. A production system must run OCR, normalize geometry, align words with boxes, tokenize with truncation rules, train with imbalanced classes in mind, and validate that the model is learning document structure rather than filename or template leakage.
LayoutLMv3 expects word text, word-level bounding boxes, and an RGB page image. Hugging Face’s processor combines image preprocessing with tokenization and box handling. The source tutorial disables built-in OCR and supplies EasyOCR output manually, which is a useful production pattern when the default OCR engine is too slow or not accurate enough for the document domain.
Bounding boxes are normalized to a 0-1000 coordinate range. This makes geometry independent of original image dimensions and allows the model to learn relative placement. The page image is resized into visual patches, commonly 224 by 224 pixels with 16-pixel patches in the base configuration.
For document classification, the model is loaded as LayoutLMv3ForSequenceClassification. The task-specific head maps the document representation to class logits. Unlike token classification, the label belongs to the whole page or document image, so the dataset returns one label per example rather than one label per word.
Layout-aware classification is only as good as the alignment between words, boxes, and image pixels. OCR is not a preprocessing footnote; it defines part of the model’s input tensor.
OCR systems usually return bounding boxes in image pixels. LayoutLMv3 expects coordinates in a fixed range, so every box must be scaled by the page width and height. Boxes also need a consistent order, typically the OCR engine’s reading order or a layout-aware sort over top-left coordinates.
from PIL import Image
from transformers import LayoutLMv3Processor
def scale_box(box, width, height):
x0, y0, x1, y1 = box
return [
int(1000 * x0 / width),
int(1000 * y0 / height),
int(1000 * x1 / width),
int(1000 * y1 / height),
]
image = Image.open("statement.png").convert("RGB")
width, height = image.size
words = [row["word"] for row in ocr_rows]
boxes = [scale_box(row["bounding_box"], width, height) for row in ocr_rows]
encoding = processor(
image,
words,
boxes=boxes,
max_length=512,
padding="max_length",
truncation=True,
return_tensors="pt",
)The source article wraps image paths in a PyTorch dataset. Each example opens the image, reads the OCR JSON sidecar, builds the processor encoding, and attaches the class label. The dataloader then batches tensors such as input_ids, bbox, pixel_values, attention_mask, and labels.
PyTorch Lightning or a standard Trainer can optimize the classifier. The important boundary is that OCR and box normalization should be deterministic and versioned. If the OCR engine changes between training and inference, the model may see a different geometry distribution even for the same page image.
Financial workflows often need to route statement-like documents, transaction summaries, disclosures, and miscellaneous filings to different extraction pipelines. Plain text may include similar words across all categories, but layout differs: table density, heading placement, totals, and section structure provide classification signal.
A document-processing platform can classify each scanned page before choosing the extractor. Forms go to key-value extraction, receipts go to line-item parsing, contracts go to clause segmentation, and unrecognized pages go to manual review. Layout-aware classifiers reduce wasted work by selecting the right downstream processor early.
Large archives contain scanned records with inconsistent templates. A layout-aware model can assign document families before retention, redaction, or review rules run. This is especially useful when document type is encoded by visual conventions rather than a reliable metadata field.
Store OCR output beside each image as a versioned sidecar. The sidecar should include words, bounding boxes, confidence scores, page dimensions, OCR engine name, and engine version. Training and inference can then share the same input contract without rerunning OCR during every epoch.
class DocumentClassificationDataset(torch.utils.data.Dataset):
def __init__(self, image_paths, labels, processor):
self.image_paths = image_paths
self.labels = labels
self.processor = processor
def __getitem__(self, index):
image_path = self.image_paths[index]
image = Image.open(image_path).convert("RGB")
width, height = image.size
with open(image_path.with_suffix(".json")) as f:
rows = json.load(f)
words = [r["word"] for r in rows if r["word"].strip()]
boxes = [scale_box(r["bounding_box"], width, height) for r in rows if r["word"].strip()]
encoding = self.processor(
image,
words,
boxes=boxes,
max_length=512,
padding="max_length",
truncation=True,
return_tensors="pt",
)
item = {k: v.squeeze(0) for k, v in encoding.items()}
item["labels"] = torch.tensor(self.labels[index])
return item
def __len__(self):
return len(self.image_paths)At serving time, the classifier should receive the same artifacts it saw during training: RGB image, OCR words, normalized boxes, and the same label mapping. If documents span multiple pages, classify pages individually first, then aggregate with a document-level rule such as max probability, majority vote, or a learned aggregator.