Benchmark adapters#

How public OCR benchmarks become PuffinParse datasets.

ADR-10 says PuffinParse does not author a competing benchmark: it runs every public benchmark through one harness. An adapter is the piece that makes that true — it fetches one upstream benchmark at a pinned revision and rewrites it into benchmark/datasets/<name>/manifest.json. Datasets whose license permits redistribution are vendored into the repository; the rest stay an index plus a sha256, fetched on demand.

Everything lives in benchmark/adapters/:

CLI#

python -m benchmark.adapters --list
python -m benchmark.adapters <name> [--limit N] [--out DIR] [--cache DIR] [--seed N] [--no-download]

Downloads go over HTTPS through whatever proxy the environment configures. Behind the agent proxy, point huggingface_hub at the CA bundle (the adapter sets these if they are unset) — never disable TLS verification:

export REQUESTS_CA_BUNDLE=/root/.ccr/ca-bundle.crt SSL_CERT_FILE=/root/.ccr/ca-bundle.crt

The CLI prints a build summary: documents by kind, committed size, and the adapter's own counters (rules converted, rules skipped by upstream type, documents skipped and why).

Manifest extension#

The manifest stays exactly what benchmark/README.md and docs/SPEC.md §10.2 describe:

{"name", "version", "description", "license",
 "documents": [{"id", "file", "truth", "pages", "category", "tags"}]}

Adapters add optional keys. Every added key is additive and ignored by the current Rust Manifest / ManifestDoc (serde ignores unknown fields by default), so old manifests keep working and new manifests load in an unmodified CLI.

Document-level additions#

truth is always emitted, even for kind: "rules" documents, where it is the empty string. That is not cosmetic: the Rust ManifestDoc declares pub truth: String without #[serde(default)], so a missing key fails to deserialise the whole manifest.

Manifest-level additions#

generator, upstream ({repo_id, revision, url, repo_type}), attribution, notes, and — for combined datasets — sources:

"sources": [{"name", "version", "license", "path", "documents", "manifest_sha256", "attribution"}]

kind: "transcript"#

The existing behaviour: truth is markdown, scored by puffinparse_core::bench with char_similarity / cer / wer / word_f1 / order_score / table_score.

kind: "rules"#

truth is empty and rules points at a JSON list of machine-checkable assertions. One schema covers every rule-based upstream benchmark we care about (ParseBench today, olmOCR-bench next):

{
  "id": "text_dense_baoutou_order_623",
  "type": "present" | "absent" | "order" | "table_cell" | "bag_of_sentences",
  "text": "…",                       // present / absent
  "before": "…", "after": "…",       // order
  "cell": {"row_header": "…", "col_header": "…", "value": "…"},   // table_cell
  "sentences": ["…"],                // bag_of_sentences
  "threshold": 0.8,                  // bag_of_sentences: required pass fraction
  "case_sensitive": false,
  "max_diffs": 0,                    // optional: upstream fuzzy allowance (olmOCR-bench)
  "source": "text_dense__baoutou_order_623"
}

Matching normalisation (scorer v2): the run's normalize, then every space adjacent to punctuation is dropped on both sides, so a tokenised rule such as (this " agreement ") matches the printed (this "Agreement") and vice versa. Words still need their spaces.

case_sensitive is always present and always explicit. source is the upstream rule id, so any score can be pushed back to the publisher's own harness for cross-checking. max_diffs is optional and records how many Levenshtein edits the upstream scorer tolerates. Since scorer v2 the Rust scorer honours it: present / absent use a fuzzy substring search (Sellers' algorithm, like upstream's fuzzysearch.find_near_matches(max_l_dist=max_diffs)), order uses the fuzzy rule above, and table_cell tolerates max_diffs edits in the header matches and the value. 0 or absent means exact matching after normalisation.

A document's rule score is passed / total; a dataset's is the mean over its rule documents. That is deliberately the same shape as an accuracy in [0, 1], so it slots next to char_similarity in the existing Summary.

ParseBench mapping#

Upstream: llamaindex/ParseBench, pinned to commit 2805a1d940f95a203e0ae4b88be9934f7765b3fc, Apache-2.0, 2,078 single-page documents and 169,011 assertions in five JSONL files.

Rule conversion, whole upstream dataset#

Why the skips:

• unexpected_* / too_many_* / *_percent counters are precision-direction assertions: "the output must not contain sentences the page does not have". Checking one needs the complete reference text, and ParseBench never publishes it for the text split — only bags. A bag_of_* compared against an incomplete reference would punish correct output.
• bag_of_digit_percent is a digit histogram, which no present/absent assertion expresses.
• is_header / is_footer assert a string's role on the page, not its presence.
• text_formatting.jsonl asserts style (bold, italic, superscript, LaTeX, heading level). Markdown carries some of this, but our normalisation strips it before scoring, so a rule about emphasis would be unscoreable. A future formatting kind could pick these up.
• chart.jsonl asserts a value read off a chart; layout.jsonl asserts bounding boxes with IoA matching. Neither has a text-similarity or substring analogue.

Net: 138,230 of 169,011 upstream assertions (81.8%) are expressible, covering the table and text_content dimensions — 1,009 of 2,078 pages (48.6%).

Committed subset#

The full conversion would vendor 517 MB of PDFs, so only a curated subset is committed (benchmark/datasets/parsebench/, 5.7 MB total, 4.0 MB of PDFs):

Selection is deterministic for a given --seed: table pages are taken at most one per source document with roughly a third hard; text pages round-robin across all eight ParseBench document-type buckets (simple, ocr, multicolumns, multilang, misc, dense, sparse, handwritting). Documents over 400 KB, over the 11 MB budget, or with more than one page are skipped and counted.

manifest.full.json in the same directory indexes all 1,009 convertible documents with their upstream_path and is not runnable as-is; subset-committed.txt lists the 40 committed ids.

The table-only caveat#

ParseBench's table truth is the page's table, not the page. A parser returns the whole page. Measured on apple_10_k_page1 with reducto/standard:

So: score table-only documents with table_score. Every such document carries the table-only tag precisely so a scorer can pick the right primary metric. merged-cells marks the 18 documents where colspan/rowspan had to be flattened by repeating cells — cell content survives, table structure (ParseBench's GriTS) does not.

Framework reference (base.py)#

class Adapter(ABC):
    name: str            # CLI argument and dataset name
    license: str         # SPDX for the redistributable part
    upstream: Upstream   # repo_id + pinned revision + url
    default_out: str     # repo-relative output directory

    def download(self, cache_dir: Path) -> Path: ...
    def build(self, out_dir: Path, limit: int | None = None, seed: int = 1234) -> Manifest: ...

Shared helpers:

pypdf is optional on purpose. It pulls in cryptography, whose Rust extension can raise a pyo3 PanicException (a BaseException, not an Exception) on a mismatched wheel, so the import is guarded broadly and cached; a broken optional dependency degrades to the byte scan instead of aborting a build.

Adding an adapter#

1. Create benchmark/adapters/<name>.py.
2. Subclass Adapter, set name, license, default_out, description and upstream with a pinned commit — not a branch.
3. Implement download(cache_dir) (fetch only what conversion needs; keep the big media trees for a per-document fetch) and build(out_dir, limit, seed).
4. Decorate the class with @register and import the module from benchmark/adapters/__init__.py.
5. Use Doc / Manifest / Rule and write_manifest / write_rules so output stays byte-stable, and record every skip with self.bump(...) so the summary is honest.
6. If the dataset belongs in the combined benchmark, add a new SOURCES_V<n> and CombinedV<n>Adapter in combined.py (as combined-v3 did for DP-Bench). Never change the sources of an existing combined version: a result is only reproducible against the exact manifest it scored. Write a README.md in the dataset directory with the license, and list the dataset in benchmark/README.md.
7. Add the dataset to crates/puffinparse-core/tests/benchmark_datasets.rs. Every rule must pass against a witness built from its own document, and every transcript must score 1.0 against itself.

Not adaptable into either kind, per the vendor benchmark survey:

• RealDoc-Bench: QA over a parse. It needs an LLM reader, which conflicts with the deterministic-metrics principle, and its source PDFs are not redistributable.
• RealDoc-Bench-Layout: bounding boxes, no text.
• LongExtractBench: schema-driven extraction, a different mode.

The academic benchmark survey covers olmOCR-bench, OmniDocBench, DP-Bench, READoc, the Nanonets IDP leaderboard, Fox, CC-OCR and OCRBench v2. DP-Bench is now adapted (below); READoc, as a long-document track, is the next candidate.

olmOCR-bench mapping#

Upstream: allenai/olmOCR-bench at 54a96a6fb6a2bd3b297e59869491db4d3625b711, ODC-BY-1.0. It has 1,403 single-page PDFs and 7,010 tests, plus 9 explicit baseline tests. The semantics were read from olmocr/bench/tests.py at f7cfe4c22098b154c76b6ec950d1c0a464eecf8d. Every document is kind: rules.

All rule text has whitespace runs collapsed. Upstream's normalize_text does the same on both sides, so nothing is lost, and it matters for table headers annotated across two lines.

What is committed:

• 40 documents, 8 from each split that has convertible tests: 6.0 MB of PDFs and 205 rules.
• manifest.full.json, an index of all 824 convertible documents.
• conversion-stats.json, which records every count.

Documents whose rules are all absent carry the absent-only tag, because an empty parse passes them. Details and caveats: benchmark/datasets/olmocr/README.md.

OmniDocBench mapping#

Upstream: opendatalab/OmniDocBench at aa1ee96d106dbe53d0ae59474d75c6e6d9b53fec. It has 1,651 page images and one annotation JSON with blocks, reading order and text / LaTeX / HTML. Each page becomes a kind: transcript document. The truth is built the way upstream's tools/json2md.py builds it:

• blocks are emitted in reading order, with truncated chains merged;
• titles become # headings, tables are converted from HTML to pipe tables, and display formulas stay as $$…$$;
• headers, footers, page numbers, page footnotes, abandon regions and figures are dropped, because OmniDocBench does not score them either.

Pages with *_mask regions (126) or less than 120 characters of truth (82) are skipped, which leaves 1,443 convertible pages. category is the upstream data_source. Language, layout, subset, special issues, has-table, has-formula and merged-cells are tags.

Licence. The card has no licence and says "research purposes only and not for commercial use". So only manifest.json, with the image sha256 and truth_sha256, is committed. docs/ and truth/ are excluded from version control and produced by python -m benchmark.adapters omnidocbench. Every document carries the fetch-required tag. Details: benchmark/datasets/omnidocbench/README.md.

DP-Bench mapping#

Upstream: upstage/dp-bench at 24702c61a2fb13325534be664653bc6e60250d13 — data and evaluate.py pinned together. 200 single-page PDFs and dataset/reference.json, which lists every layout element of a page (12 categories) in reading order (id), with content.text and, for tables, content.html.

Licence. The card front-matter says license: mit; the repository has no LICENSE file at that revision. The adapter reads the card at the pinned revision and refuses to build if it no longer says MIT. The 20 Upstage-internal pages are covered only by that declaration.

Truth. Each page becomes one kind: transcript document, following what upstream scores:

No rules are emitted: a document has one kind, and the table truth inside the transcript is already scored by table_score and teds_grid.

Selection. category is the page's dominant layout feature, in priority order table > equation > chart > figure > index > list > text. Upstream has 42 / 18 / 43 / 38 / 10 / 9 / 40 pages in those buckets; the committed subset takes 10 / 5 / 6 / 5 / 3 / 4 / 7 (40 pages, 4.5 MB of PDFs, 4 with merged-cell tables). Pages over 600 KB or past the 8 MB budget are skipped and counted. All 200 upstream pages convert (none skipped). Every element category present is a tag (has-table, has-header, has-chart …). The reference has no per-page source (Library of Congress / OER / Upstage) or domain, so neither can be tagged. Counts: benchmark/datasets/dpbench/conversion-stats.json.

Not comparable to published DP-Bench numbers: upstream NID concatenates element text with newlines removed (not replaced by spaces) and compares with rapidfuzz.fuzz.ratio, on text only; PuffinParse scores a markdown transcript with its own normalisation, tables included. Upstream also discards predicted text that lies inside a ground-truth figure or chart region (--filter-by-gt-area). A transcript has no regions, so chart labels a parser transcribes count as extra text here: tesseract/default scores 65 on chart pages and 99 on text pages.

Self-checks#

crates/puffinparse-core/tests/benchmark_datasets.rs keeps the adapters honest:

• olmocr_rules_are_satisfiable builds a witness prediction from each document's own assertions: every present text, every order pair, and each table_cell as a one-row table. Every rule, including the absent ones, must pass, so an adapter cannot emit a rule the scorer can never satisfy. It also requires an empty parse to fail every document that is not absent-only.
• omnidocbench_truth_scores_itself_perfectly scores each locally built truth against itself and checks that the has-table tag agrees with table_score.
• dpbench_truth_scores_itself_perfectly scores each committed truth against itself (1.0, table_score and teds_grid 1.0 on has-table pages, tag agreement) and requires an empty parse to score 0.
• combined_v2_references_resolve / combined_v3_references_resolve check every committed ../ path of combined-v2 / combined-v3.
• Two #[ignore]d helpers:
• PUFFINPARSE_SELFCHECK_RULES_DIR runs the witness check over any directory of rule files. On the full 824-document olmOCR conversion: 3,040 rules, 0 unsatisfiable.
• PUFFINPARSE_SELFCHECK_PRED_DIR scores real extractions, such as pdftotext output.

Licensing#

Rules for any future adapter:

• Record the SPDX id in the manifest license, and per document when a dataset mixes them.
• Carry the publisher's citation in attribution on the manifest and on each document, so a combined manifest never loses it.
• Never vendor bytes whose license does not clearly allow it — index them with upstream_path + sha256 and fetch at run time.
• Keep upstream.revision a commit hash. A result JSON is only reproducible if the input is.

Known gaps in the Rust side#

Both gaps are closed: ManifestDoc carries kind / rules, rule files are hashed into the dataset SHA, kind: rules documents are scored with puffinparse_core::bench::score_rules, and table-only documents are headlined by table_score (summarize_with).