PDF extraction

Handwritten Text Recognition: A Developer's Guide

A developer's guide to handwritten text recognition (HTR). Learn about models like CRNN+CTC, evaluation metrics, production trade-offs, and PDF integration.

May 12, 2026 14 min read okraPDF

You get handed a batch of scanned PDFs. Some pages are typed. Some have handwritten notes in the margins. A few are basically forms filled out with a ballpoint pen and then faxed twice. Your existing OCR stack handles the printed text, but the handwritten fields come back as noise, partial tokens, or nothing at all.

That's the point where handwritten text recognition stops being an academic curiosity and becomes a product problem.

If you're building PDF ingestion for support workflows, claims processing, archives, field reports, or financial documents with annotations, you need a different mental model from standard OCR. Printed text is consistent. Handwriting isn't. Writers change slant, spacing, pressure, letter shape, and baseline from line to line. Low-quality scans make all of that worse.

Introduction When Standard OCR Is Not Enough
The Core Architecture of Modern HTR Systems
Why CRNN plus CTC still matters
What each stage is doing
Preprocessing and Augmentation for Better Accuracy
Clean the page before you blame the model
Augmentation should mimic failure modes
The Segmentation Bottleneck Acknowledging a Hard Problem
Where pipelines actually break
What works in practice
Evaluating HTR Performance Beyond Raw Accuracy
Accuracy hides the shape of failure
Benchmark on documents that resemble production
Productionizing HTR Deployment Latency and Privacy
Build vs buy is an operations decision
Privacy and fallback design matter more than demos
Integrating HTR into a PDF Workflow
A practical document flow
Example integration pattern

Introduction When Standard OCR Is Not Enough

Standard OCR is great when the document behaves. Clean scan, printed font, decent contrast, predictable layout. Once a document includes cursive notes, filled forms, signatures near text, or inconsistent penmanship, that comfort zone ends.

Handwritten Text Recognition (HTR) is the family of techniques built for that messier input. It's still OCR in the broad sense, but the modeling problem is harder because the shapes are less regular and the spacing is less trustworthy. That's why handwritten text recognition typically lands around 80-95% accuracy, while printed OCR is often 98-99%.

A line-art illustration showing a person looking at two monitors displaying handwritten text recognition results.

That gap matters operationally. A printed invoice with one bad token is annoying. A handwritten date, amount, or account note that gets misread can break downstream parsing, review queues, and audit trails.

A few common examples:

Field forms: technicians write fast, abbreviate heavily, and scan pages from phones.
Historical archives: paper quality is inconsistent, ink fades, and line structure drifts.
Financial and legal documents: typed content gets mixed with margin notes, initials, stamps, and signatures.

Practical rule: If your OCR engine does well on the page body but fails on the handwritten regions, you don't need more regex. You need a handwriting-aware pipeline.

If you're trying to create editable text from image PDFs, it helps to think in layers: page cleanup, text region isolation, handwriting recognition, then post-processing. A general OCR endpoint can still be useful as a first pass, especially for mixed documents. For that kind of baseline workflow, a tool like PDF OCR processing makes sense before you decide whether the handwritten zones need a specialized route.

The Core Architecture of Modern HTR Systems

Most modern handwritten text recognition systems still look like a practical variation of CRNN plus CTC. The names sound academic, but the idea is straightforward. One part turns pixels into features. Another part interprets those features as a sequence. A decoding layer turns probability distributions into text.

A diagram illustrating the six steps of a modern handwritten text recognition system architecture.

Why CRNN plus CTC still matters

The core reason this design stuck is that handwriting doesn't come pre-segmented into neat characters. You usually have an image of a line or word, not a clean list of boxes for each letter. Character boundaries are fuzzy, especially with cursive or connected writing.

Connectionist Temporal Classification, usually shortened to CTC, solves that alignment problem. It lets the model learn the mapping from image sequence to character sequence without explicit segmentation, by summing probabilities over possible alignments. In practice, CTC is commonly paired with recurrent sequence models like Bidirectional LSTMs that read the feature sequence from both directions and emit per-step character probabilities, as described in this HTR architecture guide.

What each stage is doing

A modern pipeline usually has four stages:

Stage	What it does	Why it matters
Transformation	Normalizes skewed or curved handwriting with CNN-based localization	Reduces variation before recognition starts
Feature extraction	Uses a backbone such as ResNet to encode stroke shapes and local patterns	Gives the sequence model better visual signals
Sequence modeling	Runs a BiLSTM over the extracted features	Preserves left and right context
CTC decoding	Converts timestep probabilities into final text	Avoids hand-built character segmentation

That sequence is worth understanding from a builder's perspective.

The CNN front end is your visual parser. It doesn't “read” text the way a human does. It learns useful patterns such as loops, intersections, ascenders, descenders, and stroke density. If the backbone is weak, the whole system struggles because the sequence model receives poor features.

The BiLSTM layer adds context. That matters because many handwritten characters are ambiguous in isolation. A shape might look like “r”, “v”, or a broken “n” depending on the writer. Reading the neighboring shapes often resolves it.

Then there's decoding. CTC gives you a matrix of probabilities over characters plus a blank token for each timestep. Two practical decoding options show up often:

Best path decoding: fast and simple. Pick the top token at each step, then collapse duplicates and blanks.
Beam search decoding: slower, but more forgiving. It keeps several candidate paths alive and can work better when paired with a language model.

CTC is less magical than it sounds. It's a way to stop forcing an impossible annotation task onto your dataset.

One thing worth being opinionated about: don't over-romanticize end-to-end models. They're helpful, but they don't eliminate the need for document-aware preprocessing, sensible line extraction, and post-recognition cleanup. If your pages are badly rotated, compressed, or mixed with stamps and form lines, the model architecture won't save you by itself.

For teams evaluating vendors or open-source stacks, it also helps to look past marketing labels. “AI OCR” can mean anything from a classical recognizer to a multimodal model wrapped around the same old pipeline. What matters is whether the system exposes enough control around image normalization, line handling, decoding behavior, and confidence output. Methodology matters more than branding, and parser evaluation criteria are usually more informative than product copy.

Preprocessing and Augmentation for Better Accuracy

A lot of HTR failures get blamed on the recognizer when the underlying issue is page quality. If the input is tilted, noisy, overcompressed, or badly cropped, the model starts from a losing position.

A hand-drawn diagram illustrating a document cleaning machine process transforming a noisy skewed document into a clean, aligned one.

Clean the page before you blame the model

Three preprocessing steps usually pay for themselves.

First, deskewing. Slight rotation doesn't just make the page ugly. It changes the geometry of strokes and spacing, which can confuse line extraction and sequence modeling.

Second, binarization or contrast normalization. You're trying to make ink stand out from paper, not produce a beautiful image. A recognizer usually benefits from cleaner foreground separation, especially on photocopied or low-light scans.

Third, noise removal. Speckles, punch-hole shadows, page borders, and scan streaks create false features. If your pipeline interprets those artifacts as strokes, decoding gets unstable.

A practical cleanup checklist:

Fix orientation first: rotate before any crop or line-detection step.
Trim margins carefully: remove scanner borders, but don't clip ascenders or descenders.
Normalize contrast: make faint ink legible without blowing out the page.
Suppress background noise: deal with specks, shadows, and compression artifacts.
Preserve stroke detail: aggressive denoising can erase thin pen marks.

The fastest way to improve handwriting recognition is often to improve the image, not the model.

A quick visual walkthrough helps here:

Augmentation should mimic failure modes

Training-time augmentation matters most when it resembles the ugly inputs your users upload.

Random rotation is useful if phone captures come in crooked. Mild scaling helps when forms are scanned at inconsistent sizes. Elastic distortion can help the model tolerate letter-shape variation. Synthetic blur and compression artifacts are useful when documents arrive from messaging apps or legacy scanners.

What usually doesn't work is augmentation that's detached from production reality. If your users submit grayscale scans with faded blue ink, then wild color jitter and dramatic perspective warps won't teach the right lesson.

A good rule is simple: log the failure modes you see in staging, then mirror those conditions in augmentation. HTR gets stronger when your training examples resemble the mess you process.

The Segmentation Bottleneck Acknowledging a Hard Problem

A lot of teams assume recognition is the hard part. In production, segmentation is often where the pipeline subtly falls apart.

You see it when a page contains tight cursive lines, inconsistent spacing, overlapping notes, or form boxes that cut through writing. The recognizer might be good, but if the upstream step hands it the wrong slice of the page, the output is already compromised.

A dual-page spread featuring dense, illegible cursive handwriting overlaid with abstract red ink scribbles and lines.

Where pipelines actually break

Research on HTR preprocessing and segmentation is very clear on one point. Segmentation is an essential step, and a common failure is that connected characters are returned as one sub-image, which then causes recognition problems downstream, as discussed in this survey of HTR pipeline challenges.

That sounds narrow, but the production impact is broad. If two connected letters get merged into one image patch, character-level assumptions stop working. If a word gets split in the wrong place, language-model cleanup can only do so much. If line segmentation clips the top of a tall letter on one line and the bottom of a descending stroke on the next, you've destroyed information before inference begins.

What works in practice

This is the part academic diagrams often skip. There usually isn't one segmentation strategy that wins everywhere. The best choice depends on whether you're processing lined forms, freeform notes, archival manuscripts, or mixed print-plus-handwriting documents.

In practice, teams do better when they treat segmentation as a configurable stage rather than a solved primitive.

For structured forms: anchor around boxes, baselines, and expected regions.
For freeform notes: prioritize line detection before word splitting. Character splitting is often too brittle.
For mixed documents: separate printed and handwritten regions early if you can.
For low-quality scans: accept that some regions should be flagged for review instead of forced through recognition.

If a document is messy, line-level segmentation is usually safer than pretending you can cleanly isolate every character.

Another practical point: don't assume confidence scores will save you from segmentation errors. A recognizer can be confidently wrong if the image crop itself is wrong. That's why debugging HTR usually requires saving intermediate artifacts, especially line crops and region masks. Without those, you're guessing.

Evaluating HTR Performance Beyond Raw Accuracy

“Accuracy” sounds useful, but it hides too much. In handwritten text recognition, you need to understand how the output is wrong, not just whether a dashboard shows a decent aggregate number.

Accuracy hides the shape of failure

Sequence tasks are better evaluated with Character Error Rate (CER) and Word Error Rate (WER).

CER measures the edit distance between predicted text and ground truth at the character level. WER does the same at the word level. Both account for substitutions, insertions, and deletions, which makes them much more informative than a single pass/fail notion of correctness.

A simple comparison:

Metric	What it tells you	Where it helps
Raw accuracy	Whether outputs match exactly	Quick sanity checks
CER	How many character-level edits are needed	Fine-grained transcription quality
WER	How many word-level edits are needed	Workflow impact on downstream parsing

If the model reads acc0unt numbcr instead of account number, raw accuracy says the output failed. CER tells you it was close. WER tells you whether that closeness is still good enough for your extraction logic.

A single wrong character in a name might be tolerable. A single wrong character in an account number usually isn't.

That distinction matters when you evaluate whether HTR is “good enough” for a product. Searchability, analyst review, autofill, and automated decisioning all have different tolerance levels.

Benchmark on documents that resemble production

Benchmark choice matters almost as much as model choice. A system that performs well on curated handwriting datasets may still struggle on annotated bank statements, scanned claim forms, or legal packets with signatures and stamps.

That's why domain-specific evaluation is the better habit. If your target documents are financial, benchmark on financial documents. If your target is support ingestion, test on actual support attachments. Generic handwriting benchmarks are useful for model development, but they don't replace task-specific validation.

A solid evaluation loop usually includes:

Representative samples: use documents that match real upload conditions.
Field-level acceptance criteria: define which fields need near-perfect extraction and which can tolerate review.
Error bucketing: separate recognition mistakes from segmentation mistakes and layout mistakes.
Human review slices: inspect outputs, not just metrics.

For teams that already use parser benchmarks for PDFs, the same idea applies here. Measure on your own document mix, not just on neat examples.

Productionizing HTR Deployment Latency and Privacy

Once HTR leaves a notebook and enters a real app, the decision isn't “which model is smartest.” It's “which system can survive production traffic, ugly documents, and compliance constraints without turning into an operations tax.”

Build vs buy is an operations decision

You usually have three deployment patterns.

Cloud API is the fastest path. You get a hosted endpoint, little infrastructure work, and a simple integration surface. The trade-off is less control over model behavior, document routing, and data residency.

Self-hosted HTR gives you control. You can tune preprocessing, choose decoding behavior, and keep documents inside your environment. The cost is operational drag: model serving, scaling, queueing, retries, observability, and upgrades all become your problem.

Edge or near-edge deployment sounds attractive for latency and privacy, but handwriting models often need more compute and more memory than lightweight edge patterns comfortably allow. It can work for narrow workflows, but it's rarely the default answer.

A side-by-side view:

Option	Best fit	Main downside
Hosted API	Teams that need speed to integration	Less control
Self-hosted	Teams with strict data handling or custom pipelines	More maintenance
Edge deployment	Narrow, latency-sensitive paths	Resource constraints

Latency also has a subtle failure mode. Teams often optimize for average response time, but document systems live and die on tail latency. One weird handwritten page can hold a queue longer than ten clean ones.

Privacy and fallback design matter more than demos

The hardest production issue is rarely “can the model read English cursive.” It's whether the system behaves safely when it encounters handwriting outside its training comfort zone.

A real challenge in production is handwriting variability across writers, languages, and domains. Systems trained on narrow datasets can degrade when they see handwriting from different populations or contexts, and there's still limited operational guidance on handling that well, as noted in this discussion of cross-domain HTR variability.

That affects deployment decisions directly. If your uploads include mixed-language notes, unusual scripts, or annotations from many different people, you need fallback behavior.

A practical production policy usually includes:

Confidence thresholds: low-confidence outputs should trigger review or alternate routing.
Region-level routing: send typed and handwritten areas through different recognition stacks.
Document retention rules: sensitive pages may need redaction or strict storage controls before third-party processing.
Human-in-the-loop exits: not every page should be forced through automation.

Good HTR systems don't just transcribe. They know when to stop and ask for help.

Privacy is part of the architecture, not a legal afterthought. Handwritten pages often contain names, addresses, signatures, account details, and medical or legal notes. If those documents can't leave controlled infrastructure, that requirement should shape your deployment choice from day one.

Integrating HTR into a PDF Workflow

Organizations typically don't need “handwriting AI” as a standalone feature. They need a document workflow that accepts PDFs, stores them reliably, extracts what it can, and returns structured output that the rest of the app can use.

A practical document flow

The cleanest pattern is usually:

Upload the PDF once
Generate a stable file reference or link
Run extraction against that file
Route the result into JSON-based downstream logic
Flag uncertain regions for review

That model works well for support inboxes, claims attachments, internal ops tools, and financial document pipelines. If your team is looking at adjacent workflows, this guide on data extraction for support teams is a useful companion because it frames extraction as an application problem, not just a model problem.

Example integration pattern

A simple okraPDF flow can look like this:

curl -X POST "https://api.okrapdf.com/v1/files" \
  -H "Authorization: Bearer $OKRAPDF_API_KEY" \
  -F "file=@document.pdf"

The response returns a file_id such as doc-.... You then call parsing on the same file. For handwritten or image-only documents, use the OCR parser configured for your account; this example uses llamaparse:

curl -X POST "https://api.okrapdf.com/v1/parse" \
  -H "Authorization: Bearer $OKRAPDF_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "file": { "id": "doc-..." },
    "parser": "llamaparse",
    "pages": "1",
    "outputs": ["json"]
  }'

The parse call returns a job with a status_url. Poll that URL until the job reaches succeeded, then read the parsed JSON from the job result.

In JavaScript, the shape is similar:

const uploadForm = new FormData();
uploadForm.append("file", file);

const uploadResponse = await fetch("https://api.okrapdf.com/v1/files", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.OKRAPDF_API_KEY}`
  },
  body: uploadForm
});

const upload = await uploadResponse.json();

const parseResponse = await fetch("https://api.okrapdf.com/v1/parse", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.OKRAPDF_API_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    file: { id: upload.file_id },
    parser: "llamaparse",
    pages: "1",
    outputs: ["json"]
  })
});
const job = await parseResponse.json();
console.log(job.status_url);

The implementation details vary, but the integration principle stays the same. Keep upload, storage, extraction, and review in one pipeline. Don't make users re-upload the same document for every operation. If you're wiring that kind of document flow into your product, PDF hosting and the /host API is the part to study first.

If you're building PDF workflows that need hosting first and extraction second, okraPDF is worth a look. You can upload once, keep a stable file reference, and build extraction flows on top without adding a second document pipeline.