ParseBench, run locally
Open parsers scored on the official ParseBench harness on our own hardware: five dimensions, one seeded page subset, bootstrap intervals, measured throughput and GPU memory. Rule-based scoring, judge off.
2 systems · 60 seeded pages per dimension · AMD Radeon RX 7900 XT 20 GB · last run September 9, 2026 · ParseBench harness · Local parsing writeup.
Winning result
Quality — 52.4 (PyMuPDF4LLM + RapidOCR)
Value — not scored on this board.
Speed — 0.9s (PyMuPDF4LLM + RapidOCR)
Leaderboard
| # | System | Overall | Tables | Charts | Faithfulness | Formatting | Grounding | Peak GPU | p50 / page |
|---|
| 1 | PyMuPDF4LLM + RapidOCR Text layer + OCR | 52.4 | 63.6 | 0.0 | 75.6 | 56.2 | 66.5 | — | 0.9s |
|---|
| 2 | Docling OSS pipeline | 51.2 | 69.4 | 0.2 | 75.8 | 44.6 | 66.2 | — | 22.0s |
|---|
- Every row is our own run of the official harness on one AMD Radeon RX 7900 XT (20 GB); the published full-corpus scores are on the dedicated page, never in these columns.
- Peak GPU is measured memory above the idle baseline; CPU pipelines show none. p50 / page is the median per-page latency at the run’s concurrency.
Methods and limits
- Harness: run-llama/ParseBench 1.0.3 at commit 1dbb407, unmodified scoring; our pipeline registrations point the stock providers at local servers.
- Pages: a seeded subset (seed 20260908) of the 2,078-page corpus — 60 table, 60 chart, 60 content-faithfulness, 56 semantic-formatting, and 60 visual-grounding pages. Every system sees the same pages.
- Scoring: rule-based only. The optional Claude LLM-judge normalization is off, so charts and formatting grade more strictly than the published leaderboard, which is why published scores appear as a separate reference and never in the ranked columns.
- Hardware: one AMD Radeon RX 7900 XT (20 GB), 32 cores, 123 GB RAM, Ubuntu 26.04, Linux 7.0.0. GPU rows run through vLLM (ROCm) or Ollama; CPU rows run the parser’s own server.
- Weights: the format actually loaded is in each row — bf16 through vLLM, or the 4-bit GGUF a 16 GB laptop would load through Ollama. Quantized rows are not the published bf16 configuration.
- Throughput and memory: pages per minute of wall-clock time at the row’s concurrency, and peak GPU memory above the idle baseline sampled every two seconds. Both are properties of this host. vLLM reserves 90% of the card up front, so its rows report that reservation; Ollama rows report the resident model plus context.
- Qwen rows follow the paper’s protocol: tables, charts, faithfulness, and formatting come from the parse prompt (markdown with HTML tables); visual grounding comes from a second pass with the layout prompt (JSON with 0–1000 bboxes).
- Intervals: bootstrap 95% intervals over pages, 1,000 resamples. Failed pages score zero and stay in the count.
- Overall: the unweighted mean of the five dimensions, the same formula the leaderboard uses; blank when a dimension did not run.