PDF extraction
Bill of lading OCR benchmark: 24 parsers, 25 BoLs, one leaderboard
24 PDF parsers scored on 25 bills of lading — digital, scanned, and fax-quality. Field-level accuracy and where each parser class breaks.
If you search for a bill-of-lading parsing benchmark, you won’t find one. Hugging Face has no BoL parser model — searching “bill of lading” returns exactly one small text-classification CSV. ParseBench (the LlamaIndex document-parsing benchmark, ~2,000 human-verified pages) contains insurance, finance, and government documents — we checked all 1,006 PDFs, and not one is a logistics document. Every OCR vendor claims to handle bills of lading; nobody has published numbers.
So we measured it: 24 parsers, the same 25 bill-of-lading PDFs, the same extraction schema, the same scoring.
Disclosure: okraPDF is a document-parsing platform and appears in this leaderboard. The methodology below is designed so you can weight that fairly — the same fixed extractor and scoring ran on every system, and where a competitor beats us on a subset, the table says so.
The bill of lading OCR leaderboard
Each score is field-level accuracy across a 21-field BoL schema (BoL number, SCAC, parties, ports, container and seal numbers, freight terms, weights, hazmat UN numbers…), split by document condition: digital (born-PDF), scan (office scanner: 165 dpi, mild skew and noise, no text layer), and fax-hard (115 dpi, heavy skew, dithered, recompressed).
| # | Parser | Class | Overall | Digital | Scan | Fax-hard |
|---|---|---|---|---|---|---|
| 1 | okraPDF (/v1 production) | document parser | 96.9% | 100% | 100% | 84.2% |
| 2 | Mistral OCR | hosted API | 96.6% | 98.5% | 100% | 86.8% |
| 3 | Gemini 3.1 Flash Lite | vision model | 94.9% | 99.0% | 100% | 77.2% |
| 4 | Qwen3-VL 235B | vision model | 94.9% | 100% | 100% | 74.1% |
| 5 | Claude Sonnet 5 | vision model | 94.5% | 100% | 100% | 72.4% |
| 6 | Qwen3-VL 8B | vision model | 93.5% | 98.5% | 100% | 71.1% |
| 7 | LlamaParse | hosted API | 92.1% | 98.5% | 100% | 64.5% |
| 8 | Reducto | hosted API | 90.8% | 98.5% | 100% | 57.9% |
| 9 | GLM-4.6V | vision model | 89.9% | 99.0% | 87.7% | 68.9% |
| 10 | Gemini 3.7 Flash | vision model | 89.8% | 100% | 100% | 48.7% |
| 11 | Mistral Small 3 | vision model | 89.3% | 98.0% | 100% | 51.3% |
| 12 | Llama 4 Maverick | vision model | 89.1% | 96.5% | 100% | 54.4% |
| 13 | Chandra (Datalab) | hosted API | 88.9% | 98.0% | 100% | 49.6% |
| 14 | Kimi K2.5 | vision model | 88.8% | 92.0% | 100% | 64.9% |
| 15 | GPT-5 Mini | vision model | 88.0% | 100% | 100% | 39.5% |
| 16 | Gemini 2.5 Flash | vision model | 87.6% | 92.5% | 87.7% | 74.6% |
| 17 | Claude Haiku 4.5 | vision model | 86.4% | 93.3% | 99.1% | 50.4% |
| 18 | GPT-5.4 Mini | vision model | 85.1% | 99.5% | 100% | 26.3% |
| 19 | Nova 2 Lite | vision model | 84.1% | 94.0% | 97.5% | 39.5% |
| 20 | GPT-4o Mini | vision model | 83.3% | 90.7% | 94.3% | 48.7% |
| 21 | Tesseract 5 | classic OCR | 79.1% | 96.5% | 90.6% | 17.1% |
| 22 | pdftotext (poppler) | text-layer | 52.4% | 100% | 0% | 0% |
| 23 | PyMuPDF | text-layer | 52.4% | 100% | 0% | 0% |
| 24 | pdfplumber | text-layer | 52.1% | 99.5% | 0% | 0% |
(MinerU was registered but excluded — its API rejected our credentials on every call. We’ll add it when we can run it honestly.)
Three findings matter more than any single rank:
1. The tool class decides more than the tool. Every text-layer library scores a perfect-or-near-perfect 100% on digital PDFs and exactly 0% on scans — there is no text layer to read, and they fail silently by returning nothing. If your intake includes scanned BoLs (it does — drivers photograph paperwork, terminals fax it) and your stack is pdfplumber, roughly a third of your documents are producing empty extractions today.
2. Office scans are a solved problem; fax quality is not. Almost everything above Tesseract reads a clean 165-dpi scan at 94–100%. Drop to fax quality and the field spreads from 86.8% (Mistral OCR) down to 17.1% (Tesseract) — with expensive, current-generation models landing anywhere in between. GPT-5.4 Mini scores 99.5% on digital and 26.3% on fax. You cannot infer bad-document performance from clean-document performance.
3. Identifiers are where parsers die. Ranked by difficulty across all 24 systems, the hardest fields are exactly the ones your TMS reconciles on: trailer numbers (72.8%), PRO numbers (76.0%), seal numbers (76.9%), and BoL numbers (79.5%). Ports, vessels, and weights — the fields with linguistic context — are all above 91%.
What the failures look like
Here’s the same fax-quality ocean BoL, ground truth BWCL13743443 / container TRLU1481952 / seal CN3698954:
| System | BoL number | Container | Seal |
|---|---|---|---|
| Qwen3-VL 8B | BWCL13743443 ✓ | TRLU1481952 ✓ | CN3696954 (8→6) |
| LlamaParse | 8WCL13743443 (B→8) | TRLU1481952 ✓ | CN3696954 (8→6) |
| GPT-4o Mini | BWCL3734343 (dropped/shuffled) | TRLU481952 (dropped 1) | — |
| Nova 2 Lite | SHWCL13734443 (invented prefix) | TRLU4811952 (doubled 1) | CNG8089854 |
| Tesseract 5 | — | — | — |
Every error is one or two characters wide. Nothing here looks wrong at a glance; all of it breaks an automated match against a booking or a PO. And it isn’t just bad scans — on clean digital renders, one mid-tier vision model read HLS544686 as HL5544686 (S→5), doubled a digit in two different PRO numbers, and turned “Crossdock” into “Crossdoeck”.
The practical defense: bills of lading are unusually validatable. ISO 6346 container numbers carry a check digit, so a misread container ID can be rejected rather than ingested — every container number in this benchmark’s dataset passes the check-digit algorithm, and a TRLU4811952 that fails it should bounce to review, not into your system of record.
Methodology
The dataset. Published on Hugging Face as sleepyheeler/bol-bench (Apache-2.0) — the PDFs, the field-level ground truth, and the manifest, so you can score your own parser on the same documents. 25 synthetic single-page bills of lading across six layout families: classic ocean liner form, modern sea waybill, VICS-style straight BoL, LTL pro-sheet, hazmat straight BoL, and an NVOCC house bill. 13 digital, 7 office-scan, 5 fax-hard. All shippers, consignees, carriers, and vessels are fictional (no real carrier’s form was copied); ports and UN numbers are real reference data; container numbers are valid ISO 6346. Synthetic docs mean two things: exact ground truth with zero labeling error, and a dataset we can redistribute without rights problems. The trade-off is candor about what it can’t tell you — see caveats.
Two stages, same for everyone. Each parser converts the PDF to markdown/text — that’s the only stage where systems differ. A single fixed extractor model (Gemini 3.1 Flash Lite, temperature 0, identical prompt) then maps every parser’s output to the 21-field schema. This isolates reading quality from extraction-prompt quality: a parser wins because it transcribed the page faithfully, not because it got a better prompt. One honest confound: the extractor model is also a contestant in its vision-model role; the stages are separate (the extractor never sees the image), but we name it anyway.
Scoring. Exact match after normalization for IDs and names (case, punctuation, corporate suffixes), set-F1 for container/seal/UN lists, 0.5% tolerance for weights, date parsing across formats. A field only counts on documents where ground truth has it, so ocean-only fields don’t penalize truck BoLs. Freight-terms aliases (“Freight Prepaid” = “Prepaid”) are normalized — we score reading, not phrasing.
Harness. Everything runs through one CLI in the ParseBench shape — bol-bench run <pipeline> — with pipelines as pluggable adapters: local tools run as subprocesses, vision models through one OpenRouter path, hosted APIs through their own SDKs/endpoints. Single run per document, N=25: directional, not a leaderboard submission.
Caveats
- Synthetic layouts are cleaner than the wild. Real BoL intake includes handwritten annotations, rubber stamps over text, coffee, staples, and 40 years of layout entropy across thousands of forms. A perfect score here means “reads well-formed BoLs reliably,” not “solved logistics documents.” Treat the relative gaps (digital vs. scan vs. fax; identifiers vs. prose fields) as the durable finding.
- N=25, single run. Per-cell differences under ~3 points are noise. The class-level gaps (0% vs. 100% on scans; 17% vs. 87% on fax) are not.
- Vendors were called with defaults. No vendor-specific tuning, no custom OCR settings — the same courtesy (and the same handicap) for everyone, including us.
If you’re parsing bills of lading
The class-level advice falls out of the table: don’t use text-layer extraction on intake you don’t control; assume fax-quality documents will show up and route them to a parser that survives them; and validate every identifier you can — check digits on containers, SCAC lookups on carriers, format checks on PRO numbers — so one-character misreads bounce instead of propagating.
If you’d rather not build that pipeline, it’s our product: the bill of lading parser API returns carrier, consignee, ports, container IDs, and freight terms as typed JSON — or dry-run a BoL in the browser with the PDF-to-JSON extractor before wiring anything.