Skip to main content

Converting technical PDF documents to text

OCR for STEM

2026-05-23 — 2026-11-09

quality 5.6

Wherein Local Language Models Are Fed PDF Contents via Vision-Based Ingestion or Text-Conversion Pipelines, MinerU and Marker 1.10.2 Are Benchmarked for Equation Fidelity, and Double-Converter Diffing Is Performed.

computers are awful
machine learning
neural nets
NLP
UI
resilient tech
Figure 1

I want my local language models to read my PDFs, analyze them, answer questions about them, and so on.

There are two ways to get there.

  1. We can let a vision-capable model look at the rendered page as an image — direct, no conversion.
  2. We can turn the PDF into text first and feed it that.

For cloud pipelines this doesn’t matter; it is all magically taken care of. For local models I tend to be more stingy with my VRAM and I may want to use a pure LLM for my reasoning rather than waste bytes on a multimodal variant.

In that case it may be fiddlier than I’d like. In particular, since I care about getting the mathematics out of the papers I read, I want a high-fidelity converter, and not all converters are that.

This is the Reverse LaTeX problem at scale: not rendering an equation but recovering it from the render.

1 Decision tree

Does the agent (or the model) already handle the PDF well enough? If so, let the agent do it. If not, do we need the text just once (explain this!), or do we want to keep it (copy-paste required, knowledge base building…)?

To read the PDF once, interactively, let a vision-capable model read the page — fluent, but not for maths to be quoted verbatim.

To keep the text — for a batch, a long document, a RAG index — we convert it. For thousands of PDFs we convert only the pages that need it, on a rented GPU.

To quote the maths — refereeing a paper, say — we convert it twice, with two different converters, and diff the equations, because every converter I have tested gets the occasional equation silently wrong, and they rarely get the same one wrong. If the paper is confidential, there are some leaks to plug first.

2 When the agent already handles it

The easiest thing to do is nothing. Where the agent harness (or the model itself) handles the PDF well enough, this seems fine. Several frontends accept drag-and-dropped PDFs and convert them into something the model can read (markdown or more structured formats), and that does well enough that we stop here. See below for (my best understanding of) the toolchain in each case.

Frontend Native input How
LM Studio ✅ Drag-drop .pdf / .docx / .txt; short docs go fully into context, long ones trigger auto-RAG
Jan ✅ + tool-calling File attachments plus MCP tool-calling can hand the PDF off to whichever extractor we wire up
Osaurus ✅ text-layer Attachments route through a document adapter registry (PDF, XLSX, PPTX, CSV). Scans still need the osaurus-vision plugin or an external converter via a tool/skill.
Qwen-Agent ✅ + RAG Python harness; Assistant(files=[…]) runs a built-in RAG pipeline (DocParser + BM25, no vector DB) over PDF / Office / HTML
MLX Studio ❌ 26+ agentic tools available; invoke a converter via shell.
DwarfStar / pi ⚠️ partial pi’s web UI extracts badly. A community pdf-reader skill is the current workaround.
oMLX ⚠️ text-layer A PDF sent as a file part of a chat request is converted in-process by markitdown, so displayed equations arrive as pseudo-tables. On a 22-page maths paper, Qwen3.8-27B got 4 of 9 theorem statements wrong from it, all plausible-looking.

If that covers the job, we are done. If not — or if we would rather the model saw the page than its extracted text — read on.

3 Reading the page directly

If the agent’s model is vision-capable, the simplest pipeline needs no converter at all. We render the page — or just screenshot it — and pass the image to the model. It sees the two-column layout, the figure, the table, the equations as we do, and for many tasks that is enough: what does this paper say, what is in this plot, summarize section 3. Qwen3.6 and Qwen3.8 read screenshots happily, for example.

For reference, what Claude does as a multimodal model is a hybrid:

  1. extract the text with a run-of-the-mill PDF-to-text extractor (something like markitdown),
  2. also render each page of the PDF as an image, and
  3. slap both into the context, page by page.

This approach has two advantages over conversion. It drops nothing — a chart or a hand-drawn diagram that text extraction would discard is right there in the image — and it needs no setup (assuming our system can render PDFs).

I’m not 100% sure I trust it though; if we never see fiddly stuff like maths and tables as characters, how do we know we got it right? In November 2026 I checked, giving Qwen3.8-27B three pages of an equation-dense paper as images and comparing its transcription against the paper’s LaTeX source. It got every one of the ~17 numeric constants right, which was better than I expected. It also dropped an overline twice, and moved a factor inside a fraction where it changed the meaning of the expression. On a second, 14-page paper it was worse than either specialist converter — 48 of 58 displayed equations exact, against 57 and 54 — mostly by swapping accents and moving conditioning out of subscripts. So the digits survive and the notation and scope do not always; the errors look like valid maths, and nothing flags them. It is a strong option for reading and comprehension but a weak one as a source of record.

A bigger model does better. Qwen3.8-Flash-Next, with the same prompt and thinking off, got 56 of 63 and 54 of 58 displayed equations exact on the two papers, with one meaning-changing error between them — level with the best specialist converter. It costs 75 GiB of memory and 30–55 s a page, so it is a cross-check rather than the first pass.

Moreover, even when this works we might still want conversion of longer documents, since the page images are token-heavy and burn through context fast. Also, since the transcription lives in the chat and dies with it, anything we want to grep, diff, re-embed, or index has to be converted and kept.

4 The converters

So, suppose we want a greppable, reusable text artefact made from a PDF and friendly to an LLM. There are a lot of tools for this, and they vary along a few axes.

Kind
A model is a set of weights that turns a page image into text; we render the pages, prompt it, and stitch its output together ourselves. A toolchain is code that drives other things — a PDF parser, a layout detector, one or more models — and hands back a document. Most converters are both, a toolchain that ships its own models.
Reads
The text layer is what we copy-paste out of a born-digital PDF. It is exact for prose, mangled for maths, and absent from scans. Pixels means rendering the page and recognizing it, which works on scans and recovers maths, but can also invent plausible symbols. Many toolchains read both, prose from the text layer and maths (or whatever the text layer gets wrong) from pixels.
Memory
Peak footprint on my Mac, counting GPU allocations, since that is what competes with a resident LLM for the wired limit.
Languages
How many languages its OCR claims. This only matters for pixels; a text layer is in whatever language the PDF is.
Maths
My coarse verdict for someone who needs the equations: yes (fine as the first converter), cross-check (fine as the second opinion to diff against), no, or untested.
Kind Reads Memory Languages Maths
markitdown toolchain text layer negligible n/a no
MinerU both both 9–11 GiB 109 yes
marker 1.x both both 14 GiB 90+ yes
marker 2 both both 9–16 GiB 90+ no
Docling both both 5 GiB per OCR engine no
Granite-Docling model pixels under 1 GiB English, a few experimental no
mlx-vlm toolchain pixels per model per model n/a
GLM-OCR model pixels 5 GiB 8+ no
DeepSeek-OCR 2 model pixels 6.8 GB weights about 100 untested
Chandra 2 model pixels 10.6 GB weights 90+ untested
Other OCR models model pixels 0.6–4B params various untested
Qwen3.8-27B model pixels 22 GiB 201 for text, no OCR figure no
Qwen3.8-Flash-Next model pixels 75 GiB 201 for text, no OCR figure cross-check

4.1 markitdown

Toolchain · text layer only · negligible memory · no OCR · maths: no

Instant, with nothing to install beyond one package. That is enough for born-digital prose, which is most of what journals and word processors produce. It turns maths into pseudo-tables or nonsense, drops figures, and returns nothing for a scan. It is also what oMLX runs on an attached PDF.

uvx --from 'markitdown[pdf]' markitdown some.pdf

markitdown

4.2 MinerU

Toolchain with its own models · reads both · 9–11 GiB · 109 languages · maths: yes

The most accurate converter I have tried: 118 of 121 displayed equations exact in my test. It also does tables (as HTML), equation numbers and scans, and it ships an MCP server for agents. It is slow to start, since loading its models takes minutes, so it pays off most over a batch. In 3.x, -b vlm-engine reads the whole page with its VLM, and on nine arXiv papers it got about a quarter more inline formulas right than the default backend at the same speed, so for maths I pass it. MinerU 4 (September 2026) is the maintained line and about as accurate; its stateless one-shot command is mineru-kit parse paper.pdf -o out --tier advanced, and its document library sends usage telemetry unless MINERU_TELEMETRY_DISABLED=1.

Install with uv tool install 'mineru[all]'. MinerU, (Wang et al. 2026) 4.0 release notes

4.3 marker 1.x

Toolchain with its own models (small specialist surya recognizers) · reads both · 14 GiB · 90+ languages · maths: yes

Nearly as accurate as MinerU (111 of 121), quicker to start, and it gets different equations wrong, which makes it the second converter for a diff. It is fast on a born-digital text layer and takes minutes when it has to OCR. Pin it, because an unpinned install now gets marker 2: uv tool install marker-pdf==1.10.2, then marker_single paper.pdf. The optional --use_llm pass calls Gemini by default, so I leave it off locally.

marker

4.4 marker 2

Toolchain plus one model (Surya 2.5, 689M, via llama.cpp) · reads both · 9–16 GiB · 90+ languages · maths: no

A different program under the same name: one small VLM replaces marker 1’s specialist recognizers. It is quick (2–10 s a page) and maintained, but at best (--mode fast) it got 84 of 121 displayed equations exact, and --mode balanced invented symbols, so for maths I stay on 1.10.2. It fetches a font on its first run, which fails under a network sandbox.

2.0 release notes

4.5 Docling

Toolchain with its own models (layout, tables, OCR) · reads both · 5 GiB · languages per OCR engine · maths: no

IBM’s converter is the one for structure: tables as structured cells, a layout tree with bounding boxes and reading order, exportable as JSON, YAML, HTML or Markdown. For maths it is weak; --enrich-formula sends equations to a formula model that runs only on CPU on a Mac (30 minutes a paper) and still got just 66 of 121 exact. It is also a front end for many of the newer OCR models (--pipeline vlm --vlm-model …), but its Markdown writer mangles the LaTeX they emit; --debug-vlm-native-output keeps their raw output. For LLM use pass --image-export-mode placeholder, or it inlines every figure as base64.

Install with uv tool install docling. Docling

4.6 Granite-Docling

Model (258M), run by Docling · pixels · under 1 GiB · English, experimental Japanese, Arabic and Chinese · maths: no

Docling’s own end-to-end page model: tiny, about 6 s a page on MLX, and small enough to run alongside anything. It silently corrupts maths, though: on two ICLR papers it turned 6 of 25 \(\log\frac{1}{\delta}\) terms into \(\log\frac{1}{8}\) or \(\log\frac{1}{2}\).

docling --pipeline vlm --vlm-model granite_docling paper.pdf. Granite-Docling, Docling’s VLM benchmarks

4.7 mlx-vlm

Toolchain · pixels · memory and languages depend on the model

How the bare models below run on a Mac. mlx-vlm loads a model from Hugging Face, and we feed it rendered page images (pdftoppm -r 200 -png paper.pdf page) with that model’s own prompt, one page at a time, then stitch the pages together. Layout, figures and cross-page reading order are our problem; that is what the toolchains above add. oMLX bundles a copy.

4.8 GLM-OCR

Model (0.9B, 2.2 GB in bf16) · pixels · 5 GiB · “8+” languages: Chinese, English, French, Spanish, Russian, German, Japanese, Korean · maths: no

Near the top of the OmniDocBench leaderboard and small enough to run anywhere, through mlx-vlm or Docling. In my test it got 67 of 121 displayed equations exact, with substitutions like \(\chi'^2_{2\alpha}\) becoming \(\chi'^2_{2}\) and \(u\) and \(t\) swapped in subscripts.

GLM-OCR

4.9 DeepSeek-OCR 2

Model (3B, 6.8 GB) · pixels · nearly 100 languages (claimed for the first version) · maths: untested; the first version scores 77 on olmOCR-bench arXiv maths

DeepSeek’s document model (Wei, Sun, and Li 2026). The first version scored below marker 1.10 on olmOCR-bench’s arXiv maths, so I have not tried this one. Its predecessor has an 8-bit MLX port and is highly strung: it can loop endlessly on an ordinary page, and it is fussy about the exact prompt.

DeepSeek-OCR 2

4.10 Chandra 2

Model (4B, 10.6 GB), with Datalab’s own toolchain · pixels · 90+ languages · maths: untested; 87 on olmOCR-bench arXiv maths, per Datalab

marker’s big sibling, from the same company, and near the top of olmOCR-bench on scans, handwriting and forms — 89 on its old-scans maths tests, against marker 1.10’s 70. The weights are under a modified OpenRAIL-M licence (free for research, personal use and small startups); Datalab’s API serves a newer Chandra 2.5 whose weights are not released.

Chandra

4.11 Other OCR models

Models, 0.6–4B · pixels · maths: untested

2026 brought a crop of small document models: PaddleOCR-VL-1.6 (0.9B, 109 languages, needs its own layout pipeline; the first version reports 86 on olmOCR-bench arXiv maths), LightOnOCR-3 (0.8B and 4B, Apache 2.0), Unlimited-OCR 2 (3B, for long documents), Nemotron-3.5-Parse (about 650M), and dots.ocr-1.5 (3B, 86 on olmOCR-bench arXiv maths per Datalab, since pulled from Hugging Face). I have not run them. Of their leaderboards, olmOCR-bench’s arXiv maths score is the one worth reading. It passes or fails each formula, and although it ignores fonts, misses rule-drawn marks such as \overline, and tolerates extra material, it fails a wrong, missing or moved symbol. Its test formulas are short fragments, so the pass rates flatter everyone. OmniDocBench’s formula score instead averages a per-formula symbol match, which hides how many equations are wrong: GLM-OCR scores about 97 there, but 67 on olmOCR-bench’s arXiv maths against marker 1.10’s 84, much the gap my test found. Finer orderings disagree: MinerU 2.5.4 scores 77, below marker, the reverse of my test, though that is an older MinerU than the one I ran.

4.12 Two converters and a diff

Clean on a couple of papers turned out not to mean clean. In November 2026 I scored every displayed equation of two arXiv papers against their LaTeX source: 22 pages of bandit theory, and 14 pages of IEEE two-column signal processing. Exact means identical after expanding the papers’ own macros and normalizing away spacing and font wrappers; the meaning-changing errors I checked by eye, for the top five rows only.

paper 1, exact of 63 paper 2, exact of 58 meaning-changing errors font-only errors peak footprint
MinerU 3.3.1 61 57 2 1 10.7 GiB
MinerU 4.1.7, --tier advanced 57 56 2 1 8.5 GiB
Qwen3.8-Flash-Next, page images 56 54 1 2 75 GiB
marker 1.10.2 57 54 3 5 14.0 GiB
Qwen3.8-27B, page images 53 48 7 4 22.5 GiB
marker 2.1.0, --mode fast 43 41 not checked 8.8 GiB
GLM-OCR, raw page output 36 31 not checked 5.3 GiB
Docling 2.148 standard, --enrich-formula 36 30 not checked 5.0 GiB
marker 2.1.0, --mode balanced 36 5 not checked 15.9 GiB
markitdown 0 0 no LaTeX at all

The meaning-changing errors are the plausible kind: MinerU turned \(\sum_{u=1}^{U}\) into \(\sum_{u=1}^{\infty}\), and marker renamed \(H_{\mathrm{BOB}}\) to \(H_{\mathrm{ROR}}\) in the hypothesis of a theorem. MinerU and marker never got the same equation wrong in the same way. So for maths we intend to quote, we convert with both, diff their display equations, and check every disagreement against the rendered page by eye. The two MinerU versions share an error, so they are no use as each other’s cross-check; MinerU plus Flash-Next’s page images is another good pair, as long as the images go one page a request rather than as an attached PDF. That is two papers, so these are anecdotes rather than error rates.

Converters that take prose from the text layer can also pass through raw control bytes, which is how maths fonts encode glyphs with no Unicode mapping (big brackets, angle brackets); MinerU and marker 2 both did, NUL included. Python does not care, but anything that treats the text as a C string stops at the first NUL without complaint, as I found when a 47K-token prompt reached ds4 as 9K tokens. So after conversion we strip the control characters other than newline and tab:

perl -pe 's/[\x00-\x08\x0B\x0C\x0E-\x1F]//g' paper.md > paper.clean.md

5 Local handling

If I am reviewing confidential stuff, it must stay local.

Gotcha: Most of the tools above have a network path somewhere: marker’s --use_llm goes to Gemini or OpenRouter, MinerU 4’s document library sends telemetry and offers remote parsing, markitdown has Azure Document Intelligence endpoints and an LLM-backed OCR plugin, etc. Rather than audit every flag, we can run the converters with outbound network denied, using macOS’s sandbox-exec (deprecated, still works) and a profile that allows only localhost (we still need localhost because MinerU 3.3 spawns and uses a local API server):

(version 1)
(allow default)
(deny network-outbound (remote ip))
(allow network-outbound (remote ip "localhost:*"))
HF_HUB_OFFLINE=1 sandbox-exec -f nonet.sb marker_single paper.pdf --output_dir marker
HF_HUB_OFFLINE=1 sandbox-exec -f nonet.sb mineru -p paper.pdf -o mineru -l en

Both ran clean under it once their weights were downloaded, as did MinerU 4.1 (mineru-kit parse). marker 2 and Docling each fetch something at run time (a font, OCR models), so they need one run with the network on, or --no-ocr for Docling. HF_HUB_OFFLINE only stops them stalling on retries; the sandbox is the guard. MinerU also copies the input PDF into its output directory, so that directory is as confidential as the paper.

Multimodal models can often do this too: Qwen3.8-27B can read the PDFs as page images adequately, and Qwen3.8-Flash-Next reads page images really well.

6 Mass ingestion of academic documents

Suppose I want to OCR a huge corpus of math-heavy PDFs (e.g. my notoriously girth 14,000-PDF Zotero library). I could just batch-process them in the cloud. There is also a stingy option using a hybrid strategy:

  1. If the item has an arXiv id, fetch the TeX source instead of converting anything (1/3 of my PDFs).
  2. If some pages need it, convert those pages only (2/5 of my PDFs, 200,946 pages). “Need it” means maths pages, which we can detect from the fonts on the page, or pages without text layers, or failing that, pages that use words like “Theorem”, or “Proof.”
  3. Otherwise the text layer is enough.

Such triage is imperfect, but this is a numbers game. Users who want to detect garbled tables or multi-column reading order issues might need to work harder.

I build this as an allowlist on the laptop, to prevent accidentally uploading sensitive documents to the rented machine.

The main run took about 15 hours on three A40 VMs— so about USD27 of GPU time. On an NVIDIA card MinerU serves its model through vLLM; an A40 does about 1.5 pages a second, which at US$0.59 an hour is about 10 cents per 1,000 pages. For fun I used these to supplement the index of my patched zotero-mcp.

Result was …OK. MinerU’s default backend, but I think I could do better: A re-run with -b vlm-engine is in progress as I write.

TODO: unpack the arXiv sources into something a model can read; convert new arrivals on the laptop.

7 pdf-ingest skill for agents

When the reader is an agent rather than me, the routing above has to be packaged in such a way that the agent can load it. In the agentskills model, we can set up a skill that dynamically loads the detailed body on demand. Thus, we can put the routing policy in the root skill while each engine’s flags and gotchas can be in per-engine files the skill loads on demand:

.claude/skills/pdf-ingest/
  SKILL.md          # trigger + routing policy + budget disciplines
  engines/
    markitdown.md   # instant text-layer extraction
    docling.md      # Granite VLM, MLX, placeholder images
    marker.md       # maths→LaTeX, MPS, --use_llm wiring
    mineru.md       # the accuracy-maximalist escalation

A converter run from the skill may want an LLM of its own — marker’s --use_llm, say. The agent is already an LLM on a local endpoint, so it’s cheapest to point the converter at that same endpoint rather than load a second model. That looks unsafe — won’t it deadlock, or need the model loaded twice? It works anyway: during tool execution the harness’s model is idle, because the agent loop is generate → dispatch tool → wait. While the agent blocks on the converter, the converter has the endpoint to itself.

That directory is published at danmackinlay/pdf-ingest-skill; its README has the per-harness install steps — Claude Code, Hermes, pi, Osaurus, and Jan.

8 References