Academic and community OCR / document-parsing benchmarks: a survey for PuffinParse#

Researched 2026-09-24. Companion to the vendor benchmark review, which covers ParseBench, RealDoc-Bench and LongExtractBench. Every licence below was read from the dataset card on Hugging Face (https://huggingface.co/api/datasets/<id> plus the README at the pinned revision) or from the repository's LICENSE file. Revisions are the commits that were current when this was written. olmOCR-bench and OmniDocBench now have adapters (see adapters.md). For the others this page recommends what to do next.

PuffinParse scores two document kinds (SPEC §10): transcript (reference markdown, scored with char/word edit distance, reading order and table_score) and rules (machine-checkable present / absent / order / table_cell / bag_of_sentences assertions). "Maps to" below means into one of those two kinds.

Summary#

Pinned revisions:

1. olmOCR-bench (Allen Institute for AI): adapted#

• Paper: arXiv:2502.18443 ("olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models"). Code: allenai/olmocr (Apache-2.0). Data: allenai/olmOCR-bench.

What it measures. Whether a PDF → markdown converter gets specific, checkable facts right. It does not compare the output against a reference transcript. Each test is a unit test: this sentence is present, this running header is absent, this paragraph precedes that one, this table cell sits under that heading, this equation renders the same as the reference.

Size. 1,403 single-page PDFs and 7,010 tests in seven JSONL files, one per source. The runner also adds one implicit baseline test per PDF. There are 9 explicit baseline tests, so the files hold 7,019 rows.

Licence. The card front-matter says license: odc-by, and the README says the dataset is "licensed under ODC-BY-1.0 … intended for research and educational use in accordance with AI2's Responsible Use Guidelines". ODC-BY allows redistribution with attribution. The underlying pages are third-party (arXiv, Library of Congress, Internet Archive, crawled PDFs), and every test carries the page's origin url. The adapter keeps that URL as source_url on each document.

Ground truth and scorer. One JSON object per test: {pdf, page, id, type, max_diffs, checked, url, …} plus type-specific keys: text, case_sensitive, first_n, last_n (present/absent); before, after (order); cell, up, down, left, right, top_heading, left_heading (table); math (LaTeX). Semantics, read from olmocr/bench/tests.py:

• Both sides go through normalize_text: whitespace collapsed, **/__/*/_ emphasis stripped, NFC, and typographic quotes and dashes folded to ASCII.
• present / absent use rapidfuzz.partial_ratio with threshold 1 - max_diffs/len(text). first_n / last_n restrict the search to the start or end of the output. present is case-sensitive by default.
• order uses fuzzysearch.find_near_matches with max_l_dist = max_diffs and passes if any before match starts before any after match. It is case-sensitive.
• table parses markdown and HTML tables, finds a cell by fuzz.ratio, then checks the immediate up/down/left/right neighbour or the column/row heading.
• math renders both equations with KaTeX and compares the rendered symbol layout.
• baseline fails on empty output, long repeated n-grams, or CJK and emoji characters.

The leaderboard averages pass rates per split, not per document.

Mapping to PuffinParse. Every test is already a rule, so documents are kind: rules:

max_diffs is kept on every converted rule, so a fuzzy scorer can honour it later.

Recommendation. Done. See benchmark/datasets/olmocr/README.md. Two scorer extensions would recover most of what is skipped or approximated: fuzzy matching driven by max_diffs, and up/down/left/right neighbour fields on table_cell. Math is deliberately out of scope until PuffinParse has a formula metric.

2. OmniDocBench (OpenDataLab / Shanghai AI Laboratory): adapted as an index#

• Paper: arXiv:2412.07626. Code: opendatalab/OmniDocBench (Apache-2.0). Data: opendatalab/OmniDocBench, not gated.

What it measures. End-to-end parsing of diverse real pages: text (normalised edit distance), tables (TEDS), display formulas (CDM and edit distance) and reading order (edit distance over block order). It also has layout-detection and single-module tracks.

Size. 1,651 page images (PNG/JPG, over 1 GB; the first 1,000 alone are 1,019 MB) plus OmniDocBench.json (42 MB), which includes a 296-page hard subset added 2026-04-09. Page attributes:

• data_source (10): book 276, PPT2PDF 253, academic_literature 215, exam_paper 193, colorful_textbook 159, newspaper 151, magazine 149, research_report 132, note 118, historical_document 5.
• language: simplified_chinese 765, english 755, en_ch_mixed 116, traditional_chinese 13, other 2.
• layout: single / double / three column, mixed, other.
• special_issue: watermark, fuzzy_scan, colorful_background, table flags, and so on.
• subset: v1.5, table_hard, layout_hard, equation_hard.

Licence. The card has no licence field (cardData is null in the API). The only terms are the Copyright Statement: "The PDFs are collected from public online channels and community user contributions. Content that is not allowed for distribution has been removed. The dataset is for research purposes only and not for commercial use." That grants no redistribution right and forbids commercial use, so PuffinParse, an MIT repository, does not vendor any of it: not the images, and not the truth derived from the annotations. The evaluation code is Apache-2.0.

Ground truth. Per page: page_info (image path, size, attributes) and layout_dets. Each block has a category_type (28 block classes), a polygon, a reading order, and text / latex / html. extra.relation holds truncated links, which join paragraphs split across columns, and parent_son links, which attach captions. Upstream's tools/json2md.py shows how to turn this into markdown.

Mapping to PuffinParse. One transcript document per page. Blocks are emitted in order, with truncated chains merged. Titles become # headings, tables become pipe tables converted from the HTML (merged cells flattened, tagged merged-cells), display formulas stay as $$…$$, and headers, footers, page numbers, page footnotes, abandon regions and figures are dropped. Those are the page furniture OmniDocBench itself does not score. Pages with *_mask regions (126) are skipped because their content is deliberately unannotated. So are pages whose truth is under 120 characters (82). That leaves 1,443 of 1,651 pages convertible. data_source is the category. Language, layout, subset, special issues and has-table / has-formula are tags.

What is lost: TEDS table structure (PuffinParse's table_score compares cell text row by row), CDM formula matching (LaTeX is compared as text), and OmniDocBench's block-level matching, which forgives reading-order differences. PuffinParse's char_similarity over the whole page does not.

Recommendation. Done as a fetch-at-run-time index. See benchmark/datasets/omnidocbench/README.md. If OpenDataLab ever publishes an explicit licence that permits redistribution, the same adapter can commit the files by deleting its .gitignore.

3. DP-Bench (Upstage): adapted#

• Data, inference scripts and evaluate.py all live in HF upstage/dp-bench. There is no paper.

What it measures. Two things:

• NID (normalised indel distance: edit distance without substitutions) over the text elements serialised in reading order. Tables, figures and charts are excluded.
• TEDS / TEDS-S over the 55 tables.

Size. 200 single-page PDFs (36 MB, largest 4.3 MB): 90 from the Library of Congress, 90 from Open Educational Resources, 20 from Upstage internal documents. The documents contain 1,822 layout elements in 12 classes: Paragraph 804, Heading1 194, Footer 168, Caption 154, Header 101, List 91, Chart 67, Footnote 63, Equation 58, Figure 57, Table 55, Index 10.

Licence. The card front-matter says license: mit. The Library of Congress pages are largely public domain and OER pages are openly licensed. The 20 Upstage internal pages are covered only by the MIT declaration, so record that in the dataset README.

Ground truth. dataset/reference.json (1.5 MB) is {<pdf>: {elements: [{category, coordinates, id, page, content: {text, html, markdown}}]}}. Tables are HTML, equations LaTeX, and id is the reading order.

Mapping. A transcript per page: elements in id order, tables as pipe tables via html_table_to_markdown, and headers and footers either dropped (NID excludes nothing but tables, figures and charts, so they could stay) or tagged. This is structurally the same as the OmniDocBench adapter, so it is roughly a day's work, and the whole dataset (36 MB) is small enough that a 40–60 page subset commits in about 5 MB.

Status. Adapted by benchmark/adapters/dpbench.py: headers and footers are kept (NID scores them), figures and charts dropped, 40 pages committed (4.5 MB), folded into combined-v3. See adapters.md.

4. READoc (Institute of Software, CAS)#

• Paper: arXiv:2409.05137. Code: icip-cas/READoc. Data: lazyc/READoc.

What it measures. Realistic document structured extraction: a whole PDF (arXiv papers, GitHub READMEs rendered to PDF) converted to one markdown file. It scores text, headings, tables, formulas and reading order after a standardisation and segmentation step (the "DSE Evaluation Suite").

Size. 2,233 documents: arxiv_ground_truth/ has 1,009 markdown files and github_ground_truth/ has 1,224. The PDFs come as arxiv.zip (1.19 GB), github.zip (0.59 GB) and zenodo.zip (1.45 GB).

Licence. The card says license: mit. The arXiv PDFs keep their authors' licences, most often arXiv's non-exclusive distribution licence, which does not let third parties redistribute. Treat the markdown truth as MIT and the PDFs as fetch-only.

Ground truth. One markdown file per document, e.g. arxiv_ground_truth/0705.4297.md (91 KB).

Mapping. transcript, multi-page, exactly PuffinParse's existing kind, but with documents of 10 to 40 pages. That makes it a latency and cost stress test as much as an accuracy one.

Recommendation. Worth an adapter as a separate long-document track (readoc-arxiv, about 30 documents). Ship the index plus truth and fetch the PDFs from the zips at run time. Selective extraction needs HTTP range reads of the zip central directory; downloading 1.2 GB for 30 documents is wasteful. Keep it out of combined-* because its per-document cost dwarfs the single-page sets.

5. Nanonets IDP leaderboard#

• Site idp-leaderboard.org. Harness NanoNets/idp-leaderboard-benchmarks (MIT). Toolkit NanoNets/docext.

What it measures. It is an aggregator rather than a dataset. It runs olmOCR-bench (from allenai/olmOCR-bench), OmniDocBench (from opendatalab/OmniDocBench) and its own "IDP Core": key-information extraction, VQA, OCR, document classification, long-document processing, table extraction and confidence calibration. IDP Core has 6,406 samples drawn from existing datasets (Nanonets KIE, DocILE, handwritten forms, ChartQA, DocVQA, OCR handwriting and diacritics sets, table-extraction sets), loaded through docext from Hugging Face.

Licence. The harness is MIT. The IDP Core datasets keep their own terms, and several need an agreement or are research-only (DocILE, DocVQA). Not verified dataset by dataset here.

Recommendation. No adapter of its own. Its two page-parsing benchmarks are now PuffinParse adapters, so PuffinParse can report comparable per-benchmark numbers directly. The KIE and table tracks belong with a future extract-mode benchmark and would need a per-dataset licence audit.

6. Fox (UCAS / MEGVII)#

• Data ucaslcl/Fox_benchmark_data, a single focus_benchmark_test.zip.

What it measures. Fine-grained, focus-prompted document understanding in English and Chinese: OCR of a region given a box, a line, or a colour highlight, plus page-level OCR and multi-page variants. Scoring uses edit distance, F1, BLEU and METEOR.

Licence. CC-BY-NC-SA-4.0 (card and tags). The non-commercial and share-alike terms are incompatible with vendoring into an MIT repository.

Recommendation. Skip. The task is prompt-conditioned, and PuffinParse's providers expose no "focus this box" input. Only the plain page-OCR slice would map to transcript, and the licence keeps it fetch-only.

7. CC-OCR (Alibaba / Qwen team)#

• Paper: arXiv:2412.02210. Data wulipc/CC-OCR, the TSV version used by VLMEvalKit.

What it measures. Literacy of large multimodal models across four tracks: multi-scene text reading, multilingual text reading, document parsing (documents, tables, formulas; scored with normalised edit distance and TEDS), and key-information extraction. It has 39 subsets and 7,058 images, of which 41% come from real applications.

Licence. The card front-matter says mit, but the README's licence section says only that "the source code is licensed under the MIT License". The images' own terms are not stated.

Mapping. The doc_parsing subsets give an image and a reference markdown, HTML or LaTeX string, which maps to transcript, with tables via HTML → pipe table.

Recommendation. Possible later, as a fetch-at-run-time index like OmniDocBench, until the data licence is clarified. Low priority: the parsing track is small and mostly overlaps OmniDocBench.

8. OCRBench v2#

• Paper: arXiv:2501.00321. Data ling99/OCRBench_v2 (parquet, 1.1 GB, 10,000 QA pairs). The first OCRBench (echo840/OCRBench) carries no licence on its card.

What it measures. LMM OCR ability as question answering over 31 scenarios and 23 tasks: text recognition, referring, spotting, key-information extraction, parsing, reasoning. Scoring is task-specific (exact match, ANLS, TEDS, IoU, and so on).

Licence. The card says license: mit. The images are collected from many public datasets that keep their own terms.

Recommendation. Skip. It is QA over images, not page transcription, and it conflicts with the "score parse output" design exactly as RealDoc-Bench's QA track does.

What changes in PuffinParse because of this survey#

1. benchmark/adapters/olmocr.py and benchmark/adapters/omnidocbench.py exist, and combined-v2 includes both. combined-v1 is unchanged because it has committed results.
2. Scorer work that would raise fidelity, in priority order:
3. max_diffs fuzzy matching for present, absent, order and table_cell. The field is already in the rule files.
4. Neighbour relations (up, down, left, right) on table_cell.
5. HTML tables in predictions (in progress elsewhere).
6. A structural table metric (TEDS) next to table_score.
7. Next adapter: READoc as a long-document track (DP-Bench is done: combined-v3).