Introducing the RCRR Benchmark: Does Meaning Survive Document Conversion?

Every production RAG pipeline begins the same way: a document page goes in, machine-readable text comes out. Almost nobody measures what that conversion breaks. Today we are publishing the RCRR Benchmark (reading-comprehension recovery), which measures document conversion by the only criterion that matters for enterprise AI: after the page becomes Markdown, can an AI still answer the questions a human answers from the original?

We ran 14 document conversion systems (commercial OCR products, open-weight models, and frontier vision-language model (VLM) APIs) over 99 real Japanese IR pages and 1,410 independently verified questions, under identical conditions. Overall scores ranged from 20.2 to 94.6. Every system can read characters. What differs is how much usable meaning reaches the AI.

The problem: characters survive, meaning does not

Japanese IR materials (earnings decks, securities reports) are slide-format pages whose substance lives in dense tables and annotated charts. Character-accuracy metrics cannot see what conversion does to them: every printed token can be transcribed perfectly while the association between a number and its bar is lost. A page can look right and still be wrong.

The value 32 is worthless unless it stays attached to fixed costs, its unit, and its period. RCRR scores exactly that attachment.

One page, one question

Ask a simple question: among the drivers, which item has the largest negative impact? Looking at the page, there is no way to get it wrong: fixed costs, ▲32. But a callout box near the top decomposes a tiny ▲1 step into components including price ▲23, and that is where conversions fail. We found three distinct failure types, and one page like this can produce all of them:

• Omission: the fact never reaches the Markdown. Legacy pipelines simply drop chart content; the Reader answers “no information.”
• Disassociation: the classic layout-OCR failure. Every label and every value survives, but as separate clusters; the Reader stitches the wrong ones together and answers price/MIX ▲23.
• Misattribution: the failure mode of generative conversion, and the most expensive. the structure looks clean, one association is silently wrong, and the AI returns a confident wrong answer instead of a refusal.

By character-accuracy standards, none of these conversions has a problem. RCRR scores all three as what they are: broken meaning.

How RCRR works

• 99 real single-page documents from publicly available TDnet disclosures (50 table-centric, 49 chart-centric), 1,410 verified questions. Every question’s answer is printed on the page.
• A Reader model answers one question per call from the converted Markdown alone, and refuses when the information is absent. An independent Judge scores each answer 0/1/2 against gold, with explicit rules for Japanese financial units.
• The configuration breaks provenance bias: the gold answers were authored with Gemini 3.1 Pro (text and tables) and Fable 5 (charts) assistance, so the Reader is deliberately a third family (GPT-5.4-mini) and the Judge a different family from the Reader (Gemini 3.5 Flash).
• Statistics use 95% cluster-bootstrap confidence intervals, resampling pages rather than questions. Ties are reported as ties.
• Frontier VLM APIs are evaluated on page-image inputs, the practical condition, since real workloads parse 100-plus-page PDFs page by page. OCR products use their native PDF ingestion.
• Before any scoring: a two-model-family fairness audit of every question, a self-containment filter, a no-document contamination check (zero questions answerable from memory), and exclusion of questions requiring geometric estimation.

Results: 14 systems, one yardstick

Nebula Frontier (frontier VLM APIs running inside Ur AI’s conversion pipeline with whole-page conversion and PDF text-layer fusion) scored 94.4, the highest of the document AI products evaluated: statistically tied with the newest frontier VLM APIs, significantly above Azure Document Intelligence by 6.2 points, and the highest text-and-tables score of any system at 94.3. That last number is the pipeline design paying off: fusing the PDF’s embedded text layer during conversion delivers digit-level fidelity on dense financial tables that pixel-only reading cannot match.

Nebula Sovereign (a fine-tuned Qwen3-VL-32B running entirely on customer-managed GPUs) scored 87.3, statistically equivalent to Azure Document Intelligence, with no document ever leaving the customer’s infrastructure. The fine-tuning effect is a measurement, not a claim: the identical benchmark run on the base model scores 81.1, so the tuned weights add +6.2 overall and +8.1 on text and tables.

The market separates on charts. On the 207 questions whose answers are printed inside a chart, legacy pipelines collapse to 17–46, Azure DI drops 22 points from its text score to 69.1, while frontier VLM conversion holds 94–98 and Nebula Frontier holds 94.7. Even self-hosted, Nebula Sovereign’s 77.3 outperforms every document OCR product we evaluated on this axis. If your documents contain charts (and Japanese IR documents are made of them), this is the column that predicts what your AI will actually know.

Built to be challenged

A benchmark published by a vendor whose product appears in the table has an obligation to be paranoid about its own biases. Every question passed a fairness audit by two independent model families against the original page image; a no-document control confirmed zero questions are answerable from model memory; questions requiring geometric estimation were excluded as unfair to every text pipeline. Where the gold answers’ provenance could favor a model family, we disclose it on the affected rows rather than hiding the systems. And the public data release includes every per-question result (both Readers, both Judges, every system), so every number in the report can be recomputed offline, and our own product’s failure cases are published unedited.

What comes next

• Human-authored gold answers. This release verified what VLM-authored questions can verify, which is, by construction, biased toward what VLMs can read. The questions humans can answer but VLMs struggle with are precisely the next capability gap to measure.
• Charts for Sovereign. The 149 chart-attribution failures we extracted define the next fine-tuning cycle, alongside evaluating newer open-weight base models. The target: match the base model on chart mapping first, then exceed 90.
• Beyond printed text. Roughly half of real Japanese IR chart pages print no data labels at all; the benchmark already contains the track to measure geometry estimation, the next capability frontier.