Dr.DocBench: Diagnosing What Document Parsers Still Cannot Preserve

A document is more than a sequence of recognized words. A usable parse must preserve where those words belong: which column comes first, how cells form a table, which caption belongs to which figure, whether content continues across pages, and how specialized notation should be represented. A system may transcribe much of the visible content while losing the document that organized it.

Document-parsing benchmarks have driven substantial progress, but difficult expert documents remain less thoroughly represented than familiar genres and broadly sampled pages. Dense reference layouts, rotated text, borderless tables, mathematical notation, chemical structures, and sheet music test capabilities that page-level text accuracy cannot fully capture.

Dr.DocBench addresses this gap through difficulty-aware document selection, fine-grained structural annotation, and representations designed for expert content. Its purpose is not only to rank parsers, but to identify what each system fails to preserve.

1. Building a Benchmark Around Failure

The failures Dr.DocBench targets are not limited to missing words. They also include lost reading order, broken relationships, malformed tables, and outputs that no longer satisfy the structure required by the document.

1.1 Selecting Difficult Documents Through Parser Disagreement

Dr.DocBench begins with a large multilingual corpus of long-form books organized under the BISAC subject taxonomy. Uniform sampling would allocate much of the annotation effort to documents on which modern parsers already produce similar outputs. The benchmark instead uses disagreement among existing systems as a signal for selecting more challenging source documents.

For each candidate book PDF, PaddleOCR-VL-1.5, MinerU-VLM, and Gemini-3.1-pro-preview independently parse the document. Their outputs are compared at the book level, across the pages of the candidate PDF. Documents with higher disagreement are retained within each subject for annotation, while lower-disagreement documents are excluded.

Parser disagreement is used only to prioritize source documents; it does not label every selected page as difficult. Final evaluation uses human-verified annotations rather than parser outputs.

The selection process produces exactly six source PDFs for each of the 52 BISAC subjects. This balances the number of documents across subjects. The number of annotated pages varies with the selected documents.

1.2 Annotating More Than Words

Selected pages are annotated as structured document objects. The schema records block regions and types, transcribed content, reading order, and relations between blocks.

These relations preserve information that a flat transcript would discard. Captions can be linked to figures, tables, equations, or code blocks. Parent–child links describe local hierarchy, while continuation links connect material that extends across columns or pages.

Expert content also receives an evaluable structured representation. Formulas are normalized in LaTeX, tables are represented in HTML, and music scores are transcribed into MusicXML. Chemical structures and reaction schemes are identified explicitly, while algorithms and pseudocode remain distinct from ordinary prose or code.

Together, these annotations separate several questions that conventional text scoring can conflate: Was the content recognized? Was it ordered correctly? Were its relationships retained? Was it returned in the required format?

1.3 Matching Annotation to Domain Expertise

Parser-generated sparse pre-labels provide an initial scaffold, after which human annotators correct regions, assign block categories, transcribe content, and add reading-order and structural relations.

General document elements are handled by trained annotators. Specialized blocks are routed to people with relevant graduate-level expertise, including chemistry specialists, music-literate annotators, and reviewers comfortable with dense mathematical notation. Most pages are double-annotated and adjudicated, while expert-content blocks receive additional specialist review.

Dr.DocBench at a glance

• 312 source PDFs: six in each BISAC subject

• 4,514 annotated pages

• 52 BISAC subject domains

• 14 written languages: 68.30% of pages are primarily English

• Approximately 70,000 block-level labels

• 21 block categories

• Source documents averaging about 100 pages

The long-form source material supports multi-page evaluation, while page- and block-level annotations make it possible to identify where a parsing failure occurs.

2. Where Document Parsers Break

Dr.DocBench evaluates twelve systems: two specialized document parsers and ten general-purpose vision-language models. The benchmark measures text extraction, reading order, formula recognition, and table reconstruction using metrics suited to each component.

The aggregate results provide a useful starting point. Decomposing those scores reveals where the systems differ.

2.1 No Parser Wins Everywhere

GPT-5.5 records the highest overall score at 61.94. Kimi-K2.5, Claude Opus 4.6, and Gemini 3.1 Pro follow closely, each scoring just above 60.

Those similar totals conceal different strengths. GPT-5.5 leads in text extraction and reading order. Kimi-K2.5 performs best on tables among the general-purpose VLMs, while the specialized MinerU 2.5 achieves the strongest table results overall. The following table summarizes the main evaluation results, placing aggregate and component-level performance side by side. Normalized edit distance is reported for text extraction, formula transcription, and reading order, where lower is better. Formula recognition also uses Character Detection Matching (CDM), while table reconstruction uses Tree-Edit-Distance-based Similarity (TEDS) and TEDS_S; higher is better for CDM, TEDS, and TEDS_S.

No system leads every evaluated component. An overall score can compare average performance, but it cannot describe the kind of information each parser preserves or loses.

2.2 From Difficult Subjects to Structural Attributes

Subject-level results reveal another layer of variation. Narrative and text-dense categories such as Biography & Autobiography, Fiction, and Young Adult Fiction are relatively easier for many systems. Reference is the most consistently difficult subject, appearing among the five lowest-scoring categories for all twelve evaluated systems. Design, Games & Activities, Medical, and Antiques & Collectibles also recur among the more difficult subjects.

These subjects often contain dense lists, tables, figures, equations, and irregular layouts. The subject labels do not by themselves explain difficulty, but they indicate where more specific structural analysis is needed.

The attribute-level results make those differences more concrete. Single-column pages are generally the easiest for reading order, while multi-column layouts are associated with higher ordering error for nearly all systems. Rotated text is often more difficult than normally oriented text, although the pattern and hardest orientation vary by model.

Table results also change with border style and background color. These variations show why recognizing cell content is only one part of reconstructing a table.

2.3 Recovering Content Is Not Recovering the Document

Borderless tables provide a particularly clear example. Kimi-K2.5 records 49.1 TEDS on full-line tables but 21.4 on borderless tables. Doubao scores 8.1 on the borderless subset.

The qualitative results show that these errors are not interchangeable. A system may focus on content from another page in the input window and miss the target table. Another may recover much of the target text but return Markdown or plain text instead of the required HTML, losing row and column structure in the process.

3. More Context Is Not a Structural Cure

Dr.DocBench is built from long-form documents that span roughly 100 pages, and some of their structure extends beyond the boundaries of a single page. Cross-page continuations and ordering cues can make neighboring context relevant, while requiring a parser to keep page-local and cross-page content organized. The borderless-table case above illustrates one form of cross-page confusion, but the broader question is how document parsing changes as more pages enter the input. To study that question, Dr.DocBench varies the sliding window from one to fifteen pages and tracks overall and component-level performance.

3.1 When More Context Helps and When It Does Not

Larger windows do not consistently improve document parsing. GPT-5.5 and Gemini show small gains in text extraction when the input expands from one page to two, but the pattern does not continue uniformly as more pages are added.

The direction and magnitude of change depend on both the system and the metric. Limited neighboring context can help in some settings, but it does not produce a general improvement across the evaluation.

3.2 Reading Order Degrades Most Consistently

Reading order shows the most consistent degradation. As the window grows from one to fifteen pages, Claude Opus 4.6’s reading-order edit distance rises from approximately 0.24 to 0.69, while GPT-5.5 rises from approximately 0.17 to 0.47. Lower values are better.

3.3 More Pages Do Not Guarantee Structural Control

Formula and table metrics vary with the model and window size rather than following a single trend. The specialized parsers remain comparatively stable across the tested windows.

Taken together, the results show that access to more pages does not automatically improve the ordering and structural organization of content across a document.

4. Beyond Text: Score-to-MusicXML Transcription

Multi-page context is one diagnostic dimension. Score-to-MusicXML transcription tests a different boundary of document parsing. Using a one-page input window, Dr.DocBench asks whether a system can preserve specialized visual notation in its required structured representation.

4.1 When Plain Text Is Not Enough

For many document elements, a rough text transcript preserves at least part of the useful content. For sheet music, however, plain text is not an adequate substitute for the notation itself.

Score-to-MusicXML transcription requires two capabilities to work together. A system must visually recognize elements such as clefs, notes, rests, and key and time signatures, then express them in a schema-faithful structured format.

4.2 Establishing a Cross-Document Null

The probe contains six ground-truth MusicXML documents and uses normalized string edit distance over the serialized XML. Lower values indicate greater similarity to the target.

The mean pairwise distance among the six ground-truth files is 0.624. This cross-document null represents the average distance between a target and another real but unrelated score.

4.3 No Tested System Beats the Null Reference

Qwen3.5-Flash records the strongest result at 0.662. Because lower is better, it remains worse than the 0.624 null. No evaluated system achieves a lower edit distance than the null reference.

This six-document, string-level probe evaluates schema-faithful transcription rather than musical understanding more broadly.

5. From a Leaderboard to a Failure Map

The preceding results move from aggregate performance to subjects, structure, context, and expert representations. Together, they show the diagnostic perspective that Dr.DocBench adds to document-parsing evaluation.

5.1 What an Overall Score Cannot Show

An overall score can compare average performance across systems. It cannot explain which information a parser has lost or whether two similar scores reflect the same kinds of failure.

A diagnostic benchmark must therefore look beneath the aggregate. It must separate content recognition from reading order, structural reconstruction, page selection, and compliance with the required representation.

5.2 Turning Errors into a Diagnostic Map

Dr.DocBench supports analysis across subjects, block types, layout attributes, and expert-domain content. Its reported analyses expose failures in reading order and table reconstruction, while the borderless-table case study illustrates page-position and format-compliance failures. The subject- and attribute-level breakdowns further show how performance varies across columns, text rotation, table borders, and backgrounds.

Its annotation schema provides additional localization through block regions, categories, and relations. These include caption links and continuations across columns or pages, enabling more targeted analysis of where document organization is lost.

5.3 A Defined Diagnostic Scope

Dr.DocBench focuses on document parsing, structural reconstruction, and schema-faithful representation. It evaluates whether systems preserve the content and organization encoded in complex expert documents, rather than the full range of downstream document reasoning.

Within that scope, Dr.DocBench turns document parsing from a single ranking into a map of what each system still fails to preserve.