E.3.3a · Agentic parsing on ParseBench
Can a text-only agent parse charts it cannot see?
A DeepSeek V4 Flash agent (no vision) drives crop, ask-a-VLM, and parse-a-crop tools inside opencode, against the official ParseBench harness. Compared with the single-shot VLM it uses as a tool, and a Gemini Flash anchor, on the same six pages with the same graders.
Snapshot August 16, 2026. Official run-llama/ParseBench harness, chart + table test split, 6 documents, single run, chart LLM-judge normalization off for every arm.
Arms
-
nemotron-parse single-shot (baseline) —
charts 0%, tables GTRM 67%,
paid $0, list-equivalent $0.
The exact VLM the agent gets as a tool. Perfect structure on the tables it paired (GriTS 1.0 on 2/3 docs), but it emits no chart data tables at all — charts score zero by construction.
-
DeepSeek agent + VLM tools (900s budget) (candidate) —
charts 33%, tables GTRM 81%,
paid $0, list-equivalent $0.351.
First wave. Two of three chart pages blew the 900-second budget or stranded their answer in a malformed tool call and fell back to the raw VLM layout; the one chart page the agent finished scored 5/5 rules. Tables: the agent beat its own tool, 0.667 → 0.811.
-
DeepSeek agent + VLM tools (1800s + trace harvest) (candidate) —
charts 43%, tables GTRM 67%,
paid $0, list-equivalent $0.408.
Second wave, one knob changed. Every document produced genuine agent output — charts reached 43% (Gemini anchor: 50%, and both scored 0/8 on the hardest chart). Tables slipped to 0.667 on a single missed table, bracketing run-to-run variance at roughly ±0.15 on N=3.
-
DeepSeek agent, no vision tools (ablation) —
charts 33%, tables GTRM 67%,
paid $0, list-equivalent $0.400.
Ablation: same loop with the crop and VLM tools removed. Reformatting the pre-computed evidence alone reaches 33% on charts — the live vision tools add the last ten points. Same table variance as wave two.
-
Gemini 3 Flash (thinking minimal) (anchor) —
charts 50%, tables GTRM 82%,
paid $0.025, list-equivalent $0.025.
Anchors this six-page sample against the public leaderboard: the same pipeline scores 64.8 charts / 89.9 tables there with judge normalization on.
What this run does not prove
- 6 docs (3 chart PDFs · 23 data-point rules, 3 table PDFs), single run per arm — directional, not a leaderboard claim.
- Chart LLM-judge normalization off for every arm — absolute chart scores read low vs the public leaderboard; cross-arm comparison unaffected.
- DeepSeek + Nemotron ran on NVIDIA NIM at $0 actual; "at list" prices the same tokens at $0.14/M in · $0.28/M out · cache reads 10%.
- Agent runs inside opencode (coding-agent harness) — its system prompt + file re-reads dominate input tokens; a slim loop is the next cost lever.
- Deterministic format shims before grading: LaTeX tabular → HTML (nemotron), markdown pipes → HTML (agents). GTRM/chart metrics only read HTML tables.
- Page 1 only, DPI 150 — matching the stock VLM baselines. Run-to-run table variance ≈ ±0.15 at N=3.