OCR for Scanned Documents: A Production Pipeline for AI Agents

The expensive mistake in scanned-document OCR is running every page through the biggest model and calling the resulting text “done.” A scan is not one problem. It can be a clean image of printed text, a fax with broken characters, a form whose labels and values only make sense spatially, or a mixed PDF where half the pages already have a good text layer.

An AI agent needs more than readable prose. It needs the page structure, source coordinates, and an honest signal when the scan is too poor to support an action. If OCR silently turns 8 into 3 in a routing number, the JSON can still look plausible and the agent can still continue.

This guide treats OCR as a measured document pipeline: inspect pages, route them, improve only what needs improvement, preserve layout evidence, validate the extracted facts, and escalate ambiguous regions. The examples use open tools where possible and keep the interface independent of any one OCR vendor.

Table of Contents

• Classify pages before running OCR
• Render a stable image input
• Preprocess only the defects you measured
• Recover layout, not just characters
• Hand structured evidence to the agent
• Measure the errors that matter
• Operate OCR as a retryable workflow

Classify pages before running OCR

A PDF can change type halfway through

Document-level routing is too coarse. A 40-page claim packet can begin with born-digital forms, continue with phone photos, and end with faxed handwritten notes. Running the whole file through one mode either wastes work or damages good text.

Probe every page for:

• Number of extractable characters
• Text coverage and ordering
• Image coverage and effective resolution
• Rotation and skew
• Existing OCR text aligned to the image
• Form widgets, vector lines, and table-like geometry

Classify each page as text, scan, hybrid, or unreadable. Keep the signals with the parse artifact so the routing decision can be explained later.

type PageProbe = {
  page: number;
  kind: 'text' | 'scan' | 'hybrid' | 'unreadable';
  textCharacters: number;
  imageCoverage: number;
  estimatedDpi?: number;
  rotation: 0 | 90 | 180 | 270;
  warnings: string[];
};

Trust a text layer only after testing it

Some scanners add invisible OCR text. Its presence does not prove its quality. The layer may be shifted, duplicated, out of reading order, or left over from an earlier version of the page.

Sample the text for replacement characters, implausible symbol rates, repeated lines, and coordinates outside the page. Compare a few rendered regions with their text boxes. If the hidden layer fails those checks, treat the page as a scan and replace it as a derived artifact; never mutate the source PDF silently.

Route by required capability

Choose the least expensive path that preserves what the downstream task needs.

Page and task	First route	Escalate when
Clean printed prose	Tesseract or equivalent OCR	character quality falls below threshold
Dense table or form	layout-aware parser	rows, labels, or cells lose association
Handwriting	handwriting-capable vision model	field confidence or evidence is weak
Existing good text layer	text/layout parser	reading order or table structure fails
Severe fax/photo defects	restoration plus OCR	the restored image still lacks legible evidence

Our local parser benchmark shows why one route is rarely enough: local systems win on different dimensions, and the strongest text-faithfulness method may be weak on charts or semantic formatting.

Make the router return a decision record, not only an engine name:

• Page class and the signals that produced it
• Chosen recognition and layout capability
• Preprocessing recipe, if any
• Quality threshold for accepting the first pass
• Named fallback when the threshold fails
• Maximum attempts before human review

A versioned decision record lets you replay a routing change against old pages. It also stops the fallback policy from disappearing into scattered if statements. When a page escalates, store both attempts; the delta is valuable evaluation data.

Be conservative with unreadable. A low-resolution page can still contain one legible high-value region. Route the page to review with candidate crops rather than discarding the whole document. Conversely, a page that produces many OCR characters is not necessarily readable: noise can generate confident-looking tokens.

Render a stable image input

Fix resolution before tweaking the model

OCR works on pixels. A PDF page measured in points does not tell you whether a six-point footnote has enough pixels to recognize. Render at a known resolution and record the transform from PDF coordinates to image coordinates.

For ordinary printed text, 300 DPI is a sensible starting point. Very small type may benefit from 400 DPI; low-quality fax input cannot regain information merely by being upscaled. Higher resolution also raises memory and latency, so measure rather than defaulting to 600 DPI.

Poppler’s pdftoppm can produce a reproducible page image:

pdftoppm \
  -f 7 -singlefile \
  -r 300 \
  -png \
  input.pdf page-007

Record the renderer, version, DPI, page box, and rotation. Two renderers can rasterize fonts and transparency differently; unexplained renderer drift can look like model drift.

Preserve coordinate transforms

If the OCR engine returns a pixel box (x, y, width, height), normalize it to the page image:

function normalizeBox(box: { x: number; y: number; w: number; h: number }, image: { w: number; h: number }) {
  return {
    x: box.x / image.w,
    y: box.y / image.h,
    w: box.w / image.w,
    h: box.h / image.h,
  };
}

Normalized boxes survive later rendering at another size. Also store the original pixel dimensions and page rotation so a reviewer can reproduce the exact crop.

Detect blank and separator pages early

Blank pages are not OCR failures. Neither are intentionally blank backsides, barcode separators, or image-only cover sheets.

Use image entropy, foreground coverage, and a minimal OCR probe to classify them. Return a page record with kind: "blank" or kind: "separator" instead of an empty success that downstream code may interpret as missing extraction.

Preprocess only the defects you measured

Deskew and rotation are different operations

Rotation fixes pages turned 90, 180, or 270 degrees. Deskew corrects a few degrees of tilt introduced by scanning. Applying the wrong operation can blur a page or crop content near the edge.

OCRmyPDF provides a reproducible baseline for both while preserving a PDF output:

ocrmypdf \
  --rotate-pages \
  --deskew \
  --clean-final \
  --output-type pdf \
  source.pdf derived-searchable.pdf

Keep source.pdf immutable and mark the searchable file as derived. Review OCRmyPDF’s image-processing caveats before enabling aggressive cleaning across a corpus; some filters can remove punctuation, faint handwriting, or thin table rules.

Binarization is not a universal improvement

Thresholding can help gray fax text stand out, but it can erase colored annotations and subtle decimal points. Denoising can remove speckles, but a small speck may be the dot on an i or the decimal in an amount.

Create a preprocessing policy per document class. Run the original and one or two derived variants against the same gold pages. Keep the simplest transform that improves the target metric without creating a new failure class.

For phone photos, fix perspective before OCR. For bleed-through, prefer background normalization over a hard global threshold. For tables, verify that line removal does not merge adjacent cells.

Segment when the page has clear regions

Full-page OCR asks one engine to solve layout detection and recognition at once. Forms and receipts often improve when you detect regions first, crop them with padding, and recognize each region using a suitable page-segmentation mode.

Tesseract exposes segmentation modes through --psm; its official documentation explains the modes and output formats.

# A single uniform block of text.
tesseract crop.png stdout --oem 1 --psm 6 -l eng tsv

# Sparse text at arbitrary positions, useful for some forms.
tesseract form.png stdout --oem 1 --psm 11 -l eng tsv

TSV or hOCR retains coordinates and confidence. Plain text throws away the evidence an agent will need later.

Recover layout, not just characters

Character accuracy can hide structural failure

An OCR engine may recognize every character on a statement and still interleave two columns, attach an amount to the wrong date, or flatten a table into an unusable paragraph.

Treat recognition and layout recovery as separate stages:

1. Detect regions, lines, words, tables, and figures.
2. Recognize text within those regions.
3. Reconstruct reading order and table relationships.
4. Normalize the result into page nodes.

Layout-aware services such as Google Document AI Enterprise OCR expose detected blocks and document structure. Open pipelines can pair OCR with a layout model. The output still needs to be evaluated on your documents; a richer response shape is not evidence of higher correctness.

Normalize to roles and parent-child relationships

A useful output distinguishes headings, paragraphs, tables, rows, cells, headers, and footers. It also keeps parent relationships so a cell belongs to a row and a row belongs to a table.

{
  "id": "cell_p4_r12_c3",
  "page": 4,
  "role": "cell",
  "text": "$1,842.00",
  "bbox": { "x": 0.78, "y": 0.61, "w": 0.13, "h": 0.025 },
  "parent_id": "row_p4_12",
  "confidence": 0.94
}

That structure is more valuable to an agent than a wall of Markdown when the task is to reconcile line items. Markdown remains a useful projection for reading; it should be derived from the nodes, not be the only retained artifact.

Keep headers and footers without polluting retrieval

Repeated headers can help establish document identity, dates, and page numbers. They can also dominate search results if stored like body text.

Label them and let retrieval policy decide when to include them. Do the same for watermarks, marginalia, and footnotes. Deleting them during parse makes later questions impossible; treating them as ordinary paragraphs makes common questions noisy.

Review the normalized structure with checks that do not require a language model:

• Every nonblank page has at least one node or a terminal error
• Node boxes stay inside page bounds after rotation
• Child boxes overlap or sit inside their parent region
• Table rows keep a stable column count or explain spans
• Reading-order edges do not jump repeatedly between columns
• Repeated header and footer nodes carry the correct role
• Text is not duplicated by overlapping OCR and PDF text layers

These assertions will not detect every semantic mistake, but they turn common adapter bugs into visible failures. They are especially useful when a provider changes its response format or coordinate convention.

Keep the raw provider response immutable. A normalized node can be regenerated after an adapter fix; a discarded payload cannot. Store it by hash in restricted object storage and put only the reference on the node.

For table-specific Python approaches and their failure modes, see pdfplumber, Camelot, and Tabula compared.

Hand structured evidence to the agent

Separate OCR artifacts from business fields

The OCR artifact should describe the page. A later extraction step should map relevant nodes into the domain schema.

This separation lets you change the invoice schema without rerunning OCR, or compare two OCR engines while holding the extraction prompt constant. It also reveals where an error entered the pipeline.

With okraPDF, a parse job can request nodes and Markdown while recording workflow metadata:

import { OkraClient } from '@okrapdf/sdk';

const okra = new OkraClient({ apiKey: process.env.OKRA_API_KEY! });
const job = await okra.parse({
  file: './scanned-claim.pdf',
  parser: { id: 'okrapdf', variant: 'auto' },
  outputs: { nodes: true, markdown: true },
  metadata: {
    workflow: 'claim-intake',
    sourceClass: 'mixed-scan',
    policyVersion: 'ocr-route@4',
  },
});

The same pattern works with another provider: normalize its output to your page-node contract and keep the provider payload by immutable reference.

Return evidence with every consequential field

When an extractor maps a field, return the source node, page, box, and snippet beside it. For a human reviewer, render a crop around that box. For an agent, expose a follow-up inspect_evidence tool rather than injecting every page image into context.

type ExtractedField<T> = {
  value: T | null;
  status: 'observed' | 'absent' | 'uncertain';
  evidence: Array<{
    nodeId: string;
    page: number;
    bbox?: { x: number; y: number; w: number; h: number };
    snippet: string;
  }>;
};

“Uncertain” is an outcome, not an exception. The tool should make it impossible for an agent to confuse an unreadable account number with a missing one.

Minimize sensitive text before broad agent access

OCR makes previously inaccessible text searchable. That changes the security boundary.

Apply document ownership and authorization before serving OCR artifacts. Redact or tokenize fields the agent does not need, and keep raw crops behind narrower permissions. Do not place full OCR text in logs. Our secure document redaction guide includes the verification steps needed to prove that sensitive text is gone from a derived PDF.

Measure the errors that matter

Use CER and WER for recognition diagnostics

Character error rate (CER) is edit distance divided by reference characters. Word error rate (WER) applies the same idea to words. Both are useful for comparing recognition on fixed text, especially handwriting and noisy scans.

They are not sufficient for a document workflow. A one-character error in an address may be tolerable; a one-character error in a tax ID or amount may be severe. Our handwriting OCR benchmark publishes CER with the exact dataset and scoring because a percentage without methodology is not actionable.

Add structure and field metrics

Metric	Catches	Misses
CER / WER	recognition substitutions and omissions	table relationships and field meaning
Reading-order score	interleaved columns and misplaced regions	wrong but well-ordered characters
Table structure score	rows, columns, merged cells	business-field semantics
Field exact match	end-to-end extraction errors	severity differences between fields
Evidence precision	citations pointing at the wrong region	uncited low-risk prose
Silent-error rate	accepted incorrect outputs	visible failures routed to review

Track results by document condition: born-digital, office scan, photo, fax, handwriting, and damaged. A single average lets clean pages hide the subset that will page your operations team.

Test the fallback policy, not only each engine

If the production system escalates low-quality Tesseract output to a layout parser, evaluate that combined policy. Report the first-pass acceptance rate, escalation rate, final error rate, latency, cost, and review rate.

Pin the engine versions, preprocessing policy, renderer, prompt, schema, and dataset revision. The ParseBench repository is a good example of making page-level expected output and scoring reproducible, even though you will still need a domain-specific gold set.

Operate OCR as a retryable workflow

Make every stage resumable

Rendering, preprocessing, recognition, layout recovery, extraction, and validation should each write an immutable artifact or a clear terminal error. On retry, resume from the last compatible artifact instead of repeating every page.

Use content hashes for source and derived images. Include the stage version in cache keys. If a 400-page packet fails on page 387, the system should not lose the first 386 pages.

Track resource use per page and per route:

• Render time and peak image size
• Preprocessing time and recipe
• Recognition latency, retries, and provider request ID
• Layout and normalization time
• Input and output tokens for vision stages
• Billed and estimated cost
• Cache hit and fallback reason

These measurements make cost regressions diagnosable. They also reveal when a supposedly cheap first pass escalates so often that it is slower than sending the document directly to the stronger path.

okraPDF parse jobs expose page and chunk progress plus terminal error fields, but the same contract is easy to implement around a local queue. The invariant is more important than the platform: completed means every required page produced an accepted artifact or an explicitly approved exception.

Distinguish retryable failure from bad input

Provider timeouts, rate limits, and transient storage errors are retryable. A corrupt PDF, unsupported encryption, or unreadable page usually is not. Return an error code, a user-facing explanation, and the retry policy separately.

{
  "status": "failed",
  "stage": "recognition",
  "page": 14,
  "code": "page_unreadable",
  "retryable": false,
  "message": "Page 14 does not contain enough legible detail for reliable extraction."
}

Never turn a failed page into an empty page and then mark the document complete. Empty success is one of the easiest ways to create silent downstream errors.

Set the autonomy threshold from risk

The same OCR result can be good enough for search and unsafe for a payment. Define policies by action:

• Search indexing may accept readable text with partial structure.
• Draft summaries require cited passages and page coverage.
• Database writes require schema and domain validation.
• Payments, identity changes, and legal decisions require complete evidence and may always require human approval.

The production checklist is straightforward:

• Page-level classification before routing
• Reproducible rendering with coordinate transforms
• Measured preprocessing rather than a universal filter stack
• Layout-aware nodes plus a readable projection
• Field evidence and explicit uncertainty
• Metrics split by document condition and risk-bearing field
• Resumable stages with honest failure states
• Authorization and redaction applied to OCR artifacts

That pipeline gives an agent something more useful than OCR text: a structured, reviewable account of what the scan actually supports. To compare concrete parser commands, continue with Document Parsing for AI Agents; to design the typed output contract, use the PDF-to-JSON guide.