GDP.pdf: why the hardest PDF benchmark isn't really about PDFs

GDP.pdf is Surge AI’s document benchmark: 100 tasks drawn from real professional workflows — the kind of work a paralegal, claims analyst, payroll admin, or project engineer actually gets handed — each anchored to a real PDF across ten domains (Finance, Healthcare, Legal, STEM, Engineering, Construction, Manufacturing, Insurance, Real Estate, HR). It’s published openly under CC-BY-4.0.

The headline result is brutal: every frontier model scores under 30%. Surge reports GPT-5.5 at 25% and Claude Opus 4.8 at 23% task score — models that pass bar exams and medical boards failing three out of four real document tasks.

That gap — models that ace professional exams failing professional paperwork — is the interesting part. Having replicated the benchmark twice on a stratified sample (more on that below), here is what actually makes it hard. It’s mostly not what people assume.

1. The scoring is all-or-nothing, and the rubrics are long

Each task carries a rubric of 3 to 30 binary criteria, and the headline “task score” requires passing every single one. A 19-criterion lease-administration task graded 17/19 — two narrow misses on an otherwise excellent answer — scores exactly the same as a blank page: zero.

This is not academic sadism; it’s how document work is actually judged. A bonus-payout report with one wrong amount doesn’t get partial credit from payroll. The rubric structure encodes that: criteria are tagged with severity (some are explicitly “certain dealbreakers”), implicitness (stated outright vs. must be inferred), and a named failure mode. The metric is unforgiving because the job is.

The practical consequence: mean criteria pass rate can look respectable while task score stays dismal. The frontier models Surge tested pass roughly 77% of individual criteria while completing under 25–30% of tasks. Long rubrics turn small error rates into near-certain task failure — at 95% per-criterion accuracy, a 19-criterion task still fails ~62% of the time.

2. The traps are written down before the model ever runs

GDP.pdf’s most distinctive design choice: failure modes are first-class data. Many criteria are typed “Dodged Bullet” — they encode the specific tempting wrong answer and award the pass only for avoiding it.

These traps are not hypothetical. In our replication runs, strong agentic setups walked into them repeatedly:

The trap	What happened
A floor plan’s 1,200mm window opening reads, at a glance, like a doorway	A frontier model reading the PDF natively placed a door there — in both independent runs
A wiring manual contains two near-identical diagrams for different serial ranges (“AP0803002 & below” vs “AP0803003+“)	The answer cited the wrong-revision diagram alongside the right one — applicability checking failed twice
A benefits rate table has 0–4 years and 5–9 years service tiers with adjacent columns	Both setups initially priced from the wrong tier

This is the part exam-style benchmarks never test. A multiple-choice question has one deceptive distractor written by a test author; a real document has organically misleading structure — revisions, lookalike tables, openings that resemble doors — and the rubric holds you accountable for all of it.

3. The facts live in pixels, not text

A large share of load-bearing values in GDP.pdf documents are not recoverable from a text layer: dimensioned construction drawings, scanned payroll tables, performance-graph data points in a 10-K, figure-bound health statistics. Parsing real-world pages into faithful structure remains an open research problem — page-level VLM parsing benchmarks (e.g. ParseBench, arXiv:2604.08538) still show meaningful error on tables and dense layouts, and our own runs hit a table that survived zero extraction attempts and had to be read off a zoomed page image.

And when extraction degrades even slightly, criterion 1 (all-or-nothing) amplifies it. The cleanest example from our runs: a weekly bonus computed as $8.62 instead of $8.63 — a one-cent OCR drift on one cell — failed the criterion, which failed the task. Penny-level fidelity is the actual bar, because that’s payroll’s actual bar.

4. The documents are long, and the answer is small

Tasks range from a 4-page kitchen plan to a 143-page annual report, with the median answer hinging on one table, one clause, or one figure. That’s a needle-in-haystack problem layered on top of everything above: navigation, reading order, and context-window economics all tax the model before comprehension even starts. Three of the ten documents in our sample exceed 130 pages; single-pass reading of those is already past what most deployed pipelines hand to a model.

5. Even unconstrained agents don’t saturate it

The strongest evidence that GDP.pdf measures something deeper than document access: give a frontier model everything — the raw PDF, vision, a python interpreter, and an iterative agent loop — and it still doesn’t come close to ceiling. In our two replication runs on a stratified 10-task sample (one per domain, blind per-criterion judging), that maximal setup passed 93.8% of criteria and 5 of 10 tasks; an equivalent agent working through a parsing-and-citation toolchain landed at 91.6% and 5 of 10. Both beat Surge’s published single-shot numbers by ~15 points of criteria pass rate — the agent loop is clearly the biggest single lever — and both still failed half the tasks.

What survives in the residual is exactly the hard stuff: tier-logic comprehension, instruction discipline (a “no commentary” requirement failed in both setups), applicability judgment between lookalike sources, and dimension inference beyond printed labels. Access problems are largely solvable; these aren’t, yet.

Why this benchmark is worth taking seriously

Most document benchmarks test extraction (find the value) or QA (answer from clean text). GDP.pdf tests deliverable production under professional grading — and its design choices (per-example rubrics, severity tags, pre-encoded failure modes, real PDFs at real lengths) are the reason scores collapse. It’s hard the way the work is hard.

For anyone building document systems, the transferable lessons are uncomfortable and useful: grade all-or-nothing because your users do; write the tempting wrong answer into your eval before you run it; assume the fact you need is in a figure; and measure your pipeline on penny-level fidelity, not BLEU.

---

Task and rubric details quoted above are from surgeai/GDP.pdf (© Surge AI, CC-BY-4.0). Replication figures are from our own two evaluation runs on a seeded 10-task stratified sample with blind per-criterion LLM judging; both arms used the same model family, differing only in document access.