Datalab (Marker)#

Status: docs-only. Everything below comes from the official Datalab documentation and OpenAPI spec (read 2026-09-11), from the open-source Marker renderer that produces the payloads, and from crates/puffinparse-core/src/providers/datalab.rs. No live call has been made — this repository has no Datalab key. crates/puffinparse-core/tests/fixtures/datalab_convert.json is hand-built from the documented shapes. Mark this page verified only after the #[ignore]d live test in providers/datalab.rs passes with a real key. Help verify it: issue #10.

1. Summary#

Datalab is the hosted version of Marker (plus Surya and Chandra), the open-source PDF → markdown pipeline. The response shapes below are Marker's own: markdown, an HTML-carrying block tree (json), pre-chunked blocks (chunks), and a metadata dictionary with page_stats.

2. Models exposed by PuffinParse#

From the rate card at https://www.datalab.to/pricing: Convert — fast / balanced is $4 per 1 000 pages, Convert — accurate is $10 per 1 000 pages. balanced is the default because the docs recommend it ("balance of speed and accuracy (recommended)"); fast suits clean digital PDFs at high throughput, accurate suits scans, dense layouts and complex tables.

Naming note. These names track the documented mode parameter of the current /api/v1/convert endpoint. The older /api/v1/marker endpoint (with use_llm, force_ocr, format_lines) is marked deprecated in the API reference in favour of /convert, /extract, /segment and /agent, so there is no datalab/marker / datalab/marker-llm pair: use_llm no longer exists as a request field, and its role is taken by mode=accurate.

Add-ons that change the bill are not enabled by PuffinParse and must be opted into through provider_options: word_bboxes (+$3/1k pages), extras="table_cell_bboxes" or "list_item_bboxes" (+$6/1k each, word prediction included once), chart_understanding (+$3/1k), infographic (+$4/1k), merge_cross_page (variable compute, ~$0.50/document), and EU processing_location (1.25× usage).

3. Request flow PuffinParse uses#

1. Submit. POST {base}/api/v1/convert, multipart/form-data, header X-API-Key:

Response: {"success": true, "request_id": "…", "request_check_url": "https://www.datalab.to/api/v1/convert/…", "versions": {…}}. 2. Poll. GET the check URL every 2 s, backing off ×1.5 to 10 s, with the same X-API-Key. Done when status == "complete"; a failure can also appear as success == false (with status still "complete") or status == "failed", so all three are terminal for PuffinParse. The returned request_check_url is re-hosted on the configured base URL (path only), so a base_url override or proxy keeps working. 3. Download, when regional. If the poll body carries result_url, the document content lives there instead of inline (EU processing, and any other region that requires it). PuffinParse fetches it without the API key — the signed URL authorises by itself — and merges: downloaded body first, then every non-null field from the poll response on top, because billing and score fields can be updated after the document was stored. 4. Normalise. json gives the block tree (types + polygons), markdown gives Marker's own page rendering.

Where provider_options are merged: the request is a multipart form, so options are flattened into extra text fields exactly like the built-in ones. Strings pass through; booleans become "true"/"false"; numbers are stringified; objects/arrays are serialised as JSON text (which is what additional_config expects); null is skipped. A key PuffinParse already set is replaced, so provider_options={"output_format": "markdown"} really does turn the JSON block tree off (and with it, all Blocks).

4. Response mapping#

Page markdown splitting. With paginate=true Marker writes \n\n{<page_id>} + 48 dashes + \n\n before each page's content (marker/renderers/markdown.py), where page_id is the same 0-based index used in block ids. PuffinParse recognises any {n} + ≥ 8 dashes line, so a custom page_separator still works.

Block types (Marker's vocabulary, marker/schema/__init__.py):

Marker has no Title type: the document title is a SectionHeader rendered as <h1>, which PuffinParse maps to title so the block vocabulary matches the other providers.

ocr mode is derived from parse (TextResponse::from_parse): lines are block text split on newlines, carrying the block's box; words have no geometry. Datalab does have a real word-level product (word_bboxes=true, +$3 per 1 000 pages) but it annotates HTML output with data-bbox / data-confidence spans, which PuffinParse does not request or parse.

5. Errors, status codes, rate limits, timeouts#

Every HTTP error is {"detail": "message"} (or a FastAPI validation array), which Error::from_http picks up:

Job-level failures come back with HTTP 200: {"success": false, "error": "…"}. PuffinParse turns those into a provider error carrying the message and the request_id — except the page concurrency limit, which is enforced during processing rather than at submission ("Page rate limit exceeded. Your team has … pages in flight …") and is classified as rate_limit so a router can back off or fall back.

Limits. 200 MB per file, 7 000 pages per request, 1 hour result retention. Rate limits are per plan: free tier 25 requests/min and 25 concurrent (the API-limits page still says 10/5), Team 400/400. The page-concurrency ceiling is 5 000 pages in flight per team by default.

Timeouts. timeout_secs (default 300) covers submit + polling + the result_url download and caps each request.

6. Gotchas#

• /api/v1/marker is deprecated. The reference points at /convert, /extract, /segment and /agent. use_llm, force_ocr and format_lines are gone; mode (fast/balanced/accurate) replaces them.
• status: "complete" does not mean success. Check success; a failed document is reported as complete with success: false and an error string.
• EU results are not inline. When result_url is present the content is only there, and the download must go out without the X-API-Key header. Keep the non-null poll fields on top of the downloaded body: billing and confidence numbers can change after the document was stored. (Datalab's own Python SDK 0.5.0 does not follow result_url for convert().)
• Results are deleted one hour after processing. There is no way to re-fetch afterwards.
• Page numbers are original, not sequential. With page_range="5-7" the block ids stay /page/5/…, so PuffinParse's pages are 6, 7, 8 — deliberately, so boxes and page numbers still refer to the input document. Usage.pages is page_count, i.e. the pages actually converted.
• page_range is 0-based while PuffinParse's pages is 1-based; for spreadsheets the same field selects sheet indices instead.
• Group blocks carry no content. TableGroup, FigureGroup, ListGroup and PictureGroup have html consisting only of <content-ref src='…'> placeholders. PuffinParse replaces them with their children; a naive client that reads group HTML gets empty blocks.
• Blocks are HTML, pages are markdown. The json output never contains markdown unless you set include_markdown_in_chunks=true. PuffinParse requests both formats so blocks keep their types and boxes while the page text stays Marker's own rendering.
• Caching is on by default. A repeated conversion of the same file can be served from cache and checkpoint_reused: true means the conversion step is not re-billed — good for cost, fatal for latency benchmarks. Pass provider_options={"skip_cache": True} when measuring.
• Spreadsheets bill by cells, not pages (2 500 cells per page capped at $0.60/sheet in simple mode, 500 cells per page in advanced), so Usage.pages × per_page_usd is wrong for .xlsx input.
• Images count as one page each, and multi-page TIFF frames count individually.
• HTML prettifying changes spacing. The default HTML is indented per tag, which browsers render as stray spaces around inline tags (( 23 )); disable_html_prettify=true fixes it. It affects Block.content for blocks kept as HTML.
• merge_cross_page bills a variable compute surcharge (~$0.50/document) that the price table cannot model, and it applies on every run.

7. Useful provider_options passthrough#

# 1. Benchmarking: defeat the result cache so latency and cost are real.
puffinparse.parse("doc.pdf", model="datalab/balanced", provider_options={"skip_cache": True})

# 2. Per-cell and per-list-item boxes (HTML output, +$6 per 1k pages each).
puffinparse.parse("statement.pdf", model="datalab/accurate", include_raw=True,
              provider_options={"extras": "table_cell_bboxes,list_item_bboxes", "word_bboxes": True,
                                "output_format": "json,html", "disable_html_prettify": True})

# 3. Keep running headers and footers, and preserve spreadsheet formatting.
puffinparse.parse("report.pdf", model="datalab/balanced",
              provider_options={"additional_config": {"keep_pageheader_in_output": True,
                                                      "keep_pagefooter_in_output": True,
                                                      "keep_spreadsheet_formatting": True}})

# 4. Stitch tables and paragraphs that continue across page breaks (beta, compute-billed).
puffinparse.parse("annual-report.pdf", model="datalab/accurate",
              provider_options={"merge_cross_page": True})

# 5. Token-efficient markdown for an LLM pipeline, no images or synthetic captions.
puffinparse.parse("doc.pdf", model="datalab/fast",
              provider_options={"token_efficient_markdown": True, "disable_image_extraction": True,
                                "disable_image_captions": True})

# 6. EU data residency (requires file_url or a pre-uploaded datalab:// reference, 1.25x usage).
puffinparse.parse("https://example.com/doc.pdf", model="datalab/balanced",
              provider_options={"processing_location": "eu"})

# 7. Save a checkpoint so a later /extract or /segment call skips re-parsing.
puffinparse.parse("doc.pdf", model="datalab/balanced", provider_options={"save_checkpoint": True})

8. Links#

• Docs home: https://documentation.datalab.to
• Convert API guide: https://documentation.datalab.to/docs/recipes/conversion/conversion-api-overview
• POST /api/v1/convert reference: https://documentation.datalab.to/api-reference/convert-document
• Result polling reference: https://documentation.datalab.to/api-reference/convert-result-check
• OpenAPI spec: https://www.datalab.to/openapi.json
• Error codes: https://documentation.datalab.to/platform/errors
• Limits and rate limiting: https://documentation.datalab.to/docs/common/limits
• Billing (what counts as a page): https://documentation.datalab.to/platform/billing
• Pricing rate card: https://www.datalab.to/pricing
• Marker (the open-source engine, JSON/chunks/metadata shapes): https://github.com/datalab-to/marker
• Machine-readable docs index: https://documentation.datalab.to/llms.txt