PDF Parsers vs Math Formulas: Building RAG Over Scientific Papers

Formula extraction is the silent failure point in RAG pipelines built over scientific and technical PDFs: a parser that mangles superscripts, drops subscripts, or emits garbled Unicode produces chunks that embed cleanly and retrieve badly. A new ICPR 2026 benchmark of more than 20 PDF parsers puts numbers and a validated methodology behind that failure mode. Vendor capability claims cited below are self-reported and unverified; all benchmark figures come from a single paper on a synthetic corpus, and neither caveat is optional.

Why do math formulas corrupt silently between PDF and vector store?

Formulas corrupt silently because a PDF stores glyphs and positions, not mathematics, so every parser has to reconstruct structure that was never in the file, and every downstream stage of a RAG pipeline is blind to whether that reconstruction succeeded.

A PDF page is a drawing program: place this glyph at these coordinates, in this font, at this size. A fraction is a horizontal line with glyphs above and below it. A superscript is a smaller glyph offset upward. Nothing in the file says “this is x squared.” The parser’s job is to infer the math from geometry, and parsers differ enormously in how much of that inference they attempt. Plain text extractors read glyphs in content-stream order and hope. Layout-aware models try to detect formula regions as objects. OCR-based pipelines add a recognition layer on top, with its own error modes on mathematical typography.

The failure is silent for a specific reason: nothing downstream checks. A mangled formula is still valid UTF-8 text. The chunker splits it like any other paragraph. The embedding model happily vectorizes “x2 + yi = 0” as if it were “x² + yᵢ = 0” or the LaTeX source it should have been. The chunk lands in the vector store, gets indexed, and retrieves. The pipeline reports no error because, from the pipeline’s perspective, none occurred.

What breaks is retrieval quality, and it breaks on exactly the queries the knowledge base exists to answer. An engineer asking about a specific recurrence relation or a researcher querying a bound is searching against embeddings of corrupted text. Semantic similarity is forgiving of prose noise; it is much less forgiving when the load-bearing tokens of a chunk, the formula itself, have been flattened into the wrong characters. Worse, the corruption is heterogeneous. One parser loses superscripts, another garbles Greek letters into Unicode lookalikes, a third drops the formula region entirely and leaves the surrounding prose referring to an equation that is no longer in the chunk. Each failure mode defeats retrieval differently, and none of them trips a smoke test built on clean prose.

This is why parser choice, not embedding model choice, sets the quality ceiling for a math-heavy corpus. Teams routinely benchmark embedding models against each other on retrieval tasks while feeding both candidates the same corrupted chunks. The comparison measures nothing about the actual bottleneck.

What did the ICPR 2026 benchmark actually measure?

The benchmark evaluates more than 20 contemporary PDF parsers on mathematical formula extraction across 100 synthetic documents containing over 2,000 formulas¹, and reports significant performance disparities between parsers, according to the paper, arXiv 2512.09874.

The paper appears in the ICPR 2026 proceedings (Lecture Notes in Computer Science, vol 16813, pp. 93-107, Springer)¹ and received the Best Scientific Paper Award in the Document Analysis and Recognition track, per the arXiv listing. That matters for how much weight to put on the methodology: the evaluation design was peer-reviewed and human-validated, which is more than most “best PDF parser” comparisons circulating in vendor READMEs and listicles can say.

Two scoping facts deserve emphasis before anyone quotes a ranking. First, the corpus is synthetic: 100 generated documents¹, not scanned journal pages, not the arXiv backlog, not your internal technical reports. Synthetic generation gives the authors clean ground truth for every formula, which is what makes rigorous scoring possible, but it also means the documents lack the noise that breaks parsers in production: degraded scans, two-column layouts with floated equations, inline math mixed with display math, scanned annotations. Second, the specific per-parser accuracy figures live in the paper’s tables and are reported from this one benchmark. Any ranking you read downstream, including any implied by this article, is a single-benchmark result on synthetic documents. Treat it as a shortlisting signal, not a verdict.

Which metric should you trust for formula extraction quality?

Trust LLM-as-judge scoring, which correlated with human judgment at Pearson r=0.78 in the paper’s validation study; distrust character-level matching (r=0.34) and treat plain text similarity as close to uninformative (approximately r=0), per the benchmark’s own human study.

This is the finding with the longest shelf life, and it indicts most existing parser comparisons. The authors validated their evaluation metric against human judgment with a study of 250 formula pairs, 750 ratings from 30 evaluators¹, according to the paper. Against that human baseline, the common automatic metrics fell apart. Character-level matching (CDM), which compares extracted formulas character by character against ground truth, correlated at only r=0.34. Generic text similarity, the metric behind most “parser A beats parser B” claims on the internet, correlated at approximately zero.

The mechanism behind the failure is not mysterious. Character-level metrics punish surface differences that humans judge irrelevant and miss structural differences that humans judge fatal. A formula extracted as \frac{1}{2} versus one rendered as 1/2 scores poorly on character matching while a human calls them equivalent. Conversely, a dropped superscript turns x² into x, a tiny character-level distance that changes the mathematics entirely. Text similarity metrics, tuned for prose, are even blunter: they reward parsers for getting the surrounding words right and are nearly insensitive to whether the math survived.

The practical consequence cuts in two directions. First, any existing parser ranking scored by text similarity should be discounted to near zero for math-heavy corpora; the benchmark measured that metric as uncorrelated with what humans actually judge. Second, LLM-as-judge is now a validated, cheap evaluation instrument you can run on your own corpus. The r=0.78 correlation is not perfect, but it is the difference between a usable signal and noise, and it converts formula-extraction quality from an eyeballing exercise into something you can gate on in CI.

How do Docling and Marker compare on license, runtime, and integration?

Docling fits governed or offline deployments that need Python 3.10+ and plug-and-play LangChain or LlamaIndex integration; Marker’s code is Apache-2.0 but its model weights are free only for research, personal use, and startups under $5M funding/revenue³, per Docling’s repository and Marker’s repository.

The benchmark’s per-parser figures are not quoted here, because they are single-source, synthetic-corpus results discussed above. Marker’s README does publish third-party benchmark numbers, but they are automatic category scores, not formula-semantic measurements, so they do not settle formula quality either. What can be compared rigorously is everything around the accuracy number, and for most teams those constraints decide the shortlist before accuracy testing begins.

The license row deserves a slow read, because it is the one most likely to surprise a team late in procurement. Marker’s repository states the code is Apache-2.0, which reads as a green light, but the model weights ship under a modified AI Pubs Open Rail-M license that is free only for research, personal use, and startups under $5M funding/revenue³, according to the Marker README. “Open source” in the README refers to the code. The weights are what actually parse your documents. A company past that threshold building a commercial knowledge base on Marker needs a license conversation before the proof of concept, not after.

Docling’s operational profile is better documented. The project requires Python 3.10 or newer, having dropped 3.9 support in version 2.70.0, per the Docling repository, and its models download once and cache locally, after which conversion runs fully offline, per docling.ai. Marker’s README states the same floor: Python 3.10+ and PyTorch. For air-gapped environments, regulated industries, or any deployment where documents cannot leave the network, that offline guarantee is a hard filter that eliminates most cloud-based parsing APIs before accuracy is even discussed. The plug-and-play integrations with LangChain, LlamaIndex, Crew AI, and Haystack matter for a different reason: they shrink the surface area of glue code between parsing and chunking, which is exactly where formula-corrupting transformations like aggressive text normalization tend to hide.

What do the vendor pages actually promise, and what is measured?

The two vendor pages make different kinds of claims: Docling’s site promises formula detection as a capability with no accuracy figure attached, while Marker’s README publishes third-party benchmark scores whose tests cover math but are scored automatically rather than formula-semantically.

This separation matters because vendor capability lists answer the question “does the parser attempt formulas” and not the question “does it survive them.” Docling’s site states the tool converts PDFs, Office files, HTML, images, and audio into structured data, detecting tables, formulas, reading order, and OCR. That is a vendor-stated capability with no published accuracy figure attached. Detecting a formula region and reconstructing its mathematical content correctly are different tasks with different error rates; a parser can ace the first and fail the second, and the vendor page does not tell you which.

Marker’s README is the more quantified of the two. It reports results on olmocr-bench, a third-party benchmark of 1,403 PDFs³ with tests covering math, tables, multi-column layout, scans, and hard edge cases: Balanced mode scores 76.0% overall and 83.5% on born-digital PDFs³, ahead of MinerU and docling. That is a measured claim on a real corpus, and more than most parser projects publish. It does not settle formula quality. olmocr-bench scores are automatic category results, not the formula-semantic, LLM-judged equivalence the ICPR 2026 instrument validates against human judgment. The paper’s human study is direct evidence about what surface-level automatic metrics do on formulas: text similarity tracked human judgment at approximately zero, character-level matching at r=0.34. A parser can post a strong aggregate score while flattening superscripts a human evaluator would judge fatal, and the math-category figure that would probe this is not quotable from the README’s public page.

The rule for reading any parser marketing from here on is short. A capability claim (“detects formulas”) is a statement about scope. A measured claim requires a metric, a corpus, and a correlation with human judgment. Marker’s olmocr-bench numbers satisfy the first two; on the third, the ICPR human study measured the automatic surface-level alternatives at r=0.34 and below, and anything scored by text similarity alone does not qualify.

How do you build a formula-validation gate before chunks hit the vector store?

Sample math-heavy pages from your own corpus, run your candidate parsers on them, score the extracted formulas with an LLM-as-judge pass against the rendered originals, and block ingestion for any document whose formula score falls below threshold; the benchmark’s r=0.78 correlation with human judgment makes this a calibrated instrument rather than a vibe check.

The gate has four working parts, and none of them are exotic.

Corpus sampling. Pull a representative set of pages containing display math, inline math, and the layout quirks your documents actually have: two-column papers, appendix proofs, scanned figures with equations baked in. The benchmark used synthetic documents for clean ground truth; your gate does not need ground truth if you render the PDF pages to images, because the original rendering is the reference a human would compare against.

Judge construction. Give an LLM the rendered formula (as an image crop or the page region) and the parser’s extracted text, and ask for a graded equivalence judgment: mathematically identical, minor notational difference, or wrong. The paper’s validation is what licenses this approach: an LLM judge built along these lines tracked human evaluators at r=0.78 across 250 formula pairs and 750 ratings¹, while character-matching managed r=0.34 and text similarity approximately zero. You are not inventing a metric; you are reusing a validated one.

Thresholding and blocking. A document whose formulas pass goes to the chunker. A document that fails gets quarantined for an alternate parser, an OCR pass, or manual review. The gate runs at ingestion, which is the only place the failure is still cheap to catch; once a corrupted chunk is embedded, no retrieval-stage trick reliably finds it.

Regression coverage. Pin a small set of known-hard pages as a standing test suite. Parser upgrades are exactly when silent regressions arrive: a new model version changes layout detection, and your ingestion quality shifts without any code diff on your side. Docling’s drop of Python 3.9 in v2.70.0 is a reminder that these projects move on their own schedules, per the repository. A standing formula test suite turns “the parser updated and something feels off” into a failing check with a diff.

What can this benchmark not tell you?

It cannot tell you how parsers perform on your corpus, because the evaluation ran on 100 synthetic documents¹ rather than real scanned scientific papers, and every ranking it produces is a single-study result that may not transfer to production documents with scan noise, complex layouts, or unusual notation.

The synthetic corpus is a deliberate trade, and the authors made it for a good reason: generated documents come with perfect ground truth for every one of the 2,000-plus formulas¹, which is what made the metric-validation study possible at all. But the property that makes the evaluation rigorous is the same property that limits its external validity. Production scientific corpora include scanned pages with skew and bleed-through, decades-old typesetting, handwritten marginalia, and notation conventions that vary by subfield. A parser ranking established on clean synthetic layouts can reshuffle on any of those.

The single-source caveat compounds this. One benchmark, one corpus, one judging configuration. The methodology is validated, the award is real, and the metric findings are likely to hold up because they are mechanistically plausible, but the per-parser performance ordering is the weakest link in the chain. The right way to consume the rankings is as a prior: they tell you which parsers are worth the cost of running through your own validation gate, and they tell you that the parsers at the bottom of a synthetic test are unlikely to be rescued by noisier real-world input.

There is also a list of parsers this article deliberately does not discuss. Names that circulate in every “PDF parsing for RAG” thread, including specialized math-OCR tools, have no evidence in the sources reviewed here. Asserting their strengths or failure modes without the paper’s own table or independent testing would be exactly the kind of unsourced ranking this benchmark exists to displace.

How should you choose a parser for a math-heavy knowledge base?

Shortlist Docling when you need offline operation, a Python 3.10+ runtime, and existing LangChain or LlamaIndex plumbing; shortlist Marker when its weights license fits your organization; then let a validation gate on your own documents make the final call, because the benchmark says the decision variables you can verify are license, runtime, and integration, while the accuracy ranking has to come from your corpus.

The decision decomposes into a hard-filter pass and a measured pass, in that order. Hard filters first, because they are cheap and non-negotiable. If documents cannot leave your network, Docling’s cached-model, fully offline conversion is the documented fit, per its repository. If your organization has more than $5M in funding or revenue³ and plans commercial use, Marker’s modified AI Pubs Open Rail-M weights license is a procurement problem before it is a technical one, per its repository. If your pipeline already runs on LangChain, LlamaIndex, Crew AI, or Haystack, Docling’s plug-and-play integrations reduce the glue layer where normalization bugs live.

Whatever survives the filters goes through the measured pass: the LLM-as-judge gate from the previous section, run on your hardest pages. Budget a day for it. The benchmark’s contribution here is not a ranking to copy but a method to reuse and a warning about which shortcuts not to take. Scoring your candidates by text similarity against expected output is the natural move, and the human study measured that move as essentially uncorrelated with human judgment on formulas, r of approximately zero against r=0.78 for the LLM judge, per the paper.

The deeper shift is in where the RAG quality budget goes. The default allocation spends evaluation effort on embedding models and rerankers while treating ingestion as a solved library choice. For scientific and technical corpora, that allocation is backwards. Ingestion is where the load-bearing content of the document can be destroyed without an error, and the fix is not a better parser purchased on faith but a gate that measures what the parser actually did. The ICPR 2026 benchmark supplies the validated measuring instrument and the evidence that the obvious alternatives do not work. What it does not supply, and what no single study on synthetic documents can, is the answer for your corpus. That part is a day of testing you cannot skip, and the parsers’ own marketing pages are not a substitute for it.

Frequently Asked Questions

Does the ICPR 2026 benchmark’s synthetic corpus limit its applicability to real-world scanned documents?

Yes. The evaluation used 100 generated documents with clean ground truth, lacking production noise like skew, bleed-through, or handwritten marginalia. Consequently, parser rankings may reshuffle on real scientific corpora, so the benchmark serves as a shortlisting prior rather than a definitive performance verdict for noisy inputs.

How does the LLM-as-judge metric differ from character-level matching in detecting formula errors?

LLM-as-judge correlates with human judgment at r=0.78, whereas character-level matching (CDM) only reaches r=0.34. CDM penalizes superficial notation differences (like \frac{1}{2} vs 1/2) while missing fatal structural changes, such as dropped superscripts, which LLM judges correctly identify as mathematical errors.

What specific operational change is required to implement a formula-validation gate in a RAG pipeline?

Teams must render PDF pages to images to serve as the reference for the LLM judge, rather than relying on ground-truth text. This allows the gate to compare the parser’s extracted output against the visual original, enabling the detection of silent corruption that text-based metrics miss.

Why is Marker’s Apache-2.0 code license insufficient for commercial deployments by larger organizations?

The Apache-2.0 license covers only the code, while the model weights are governed by a modified AI Pubs Open Rail-M license. This weights license restricts free use to research, personal use, and startups with under $5M in funding or revenue, requiring a separate commercial license agreement for larger entities.