PDF ingestion

1 The options

flowchart TD
  A[A PDF I want a local model to read] --> B{Does the client/model<br/>already handle it<br/>well enough?}
  B -->|yes| STOP[Stop — let the client do it]
  B -->|no| C{Read it once,<br/>or keep the text?}
  C -->|read once,<br/>interactively| V[Vision model reads the page<br/>— fluent, but not for<br/>maths to be quoted verbatim]
  C -->|reuse: batch,<br/>long doc, RAG index| D{What is in it?}
  D -->|born-digital<br/>text layer| MIT[markitdown]
  D -->|maths to be used| MRK[marker or MinerU]
  D -->|tables, structure,<br/>JSON for machines| DOC[Docling]
  D -->|scans, photos,<br/>handwriting| OCR[DeepSeek-OCR<br/>or Chandra]

2 When the client already handles it

The easiest method is to do nothing. Where the client (or the model itself) handles the PDF well enough this seems fine. Several frontends accept drag-n-dropped PDFs and convert them into something the model can read, and that does well enough we stop here. See below for (my best understanding of) the toolchain in each case.

If that covers the job, we are done. If not — or if we would rather the model saw the page than its extracted text — the two manual routes follow.

3 Reading the page directly

If the model is vision-capable, the simplest pipeline has no converter in it at all. We render the page — or just screenshot it — and hand the image to the model. It sees the two-column layout, the figure, the table, the equation as we do, and for many tasks that is enough: what does this paper say, what is in this plot, summarize section 3. Qwen3.6 reads screenshots happily, and so does any vision-capable model on the Mac.

For reference, what Claude does as a multimodal model is a hybrid:

1. extract the text with a run of the mill PDF to text extractor (something like markitdown).
2. Additionally render each page of the PDF (!)
3. Slap the extracted text and image per page into the context

The path has two advantages over conversion. It drops nothing — a chart or a hand-drawn diagram that text extraction would discard is right there in the image — and it needs no setup (assuming out system can render PDFs); there is no tool to install, no intermediate file, no second model in RAM beside the one already answering.

I am not 100% sure I trust it though; if we never see fiddly stuff like maths and tables as characters, how do we know that we got it right? It is a strong option for reading and comprehension but a weak one as a source of record.

Two practical limits favour conversion of longer documents. Page images are token-heavy, so a long PDF burns through context fast. Moreover, reading the page leaves no artifact: the transcription lives in the chat and dies with it, so anything we want to grep, diff, re-embed, or index has to be converted and kept.

4 Converting to text

So, suppose we want a greppable, re-usable text artefact made from a PDF. (If we only want to read a page or two and not keep them, there is no converter to pick — we read the page directly.) The right converter depends upon what is in that PDF, and what we want it for:

Commands and gotchas for each are in the tools below; the bake-off is where I ran them over equation-dense papers.

5 The bake-off

I ran the contenders over two equation-dense papers (a free-energy belief-change paper and the V-information ICLR paper; June 2026), reading the PDFs against the conversions. markitdown mangles display maths into pseudo-tables studded with (cid:…) glyph junk — expected, it is a text-layer extractor. Docling/Granite emits plausible LaTeX with token-level substitutions inside it: a dropped leading minus sign in a free-energy decomposition on one paper, and on the other 6 of 25 occurrences of \(\log\frac{1}{\delta}\) silently became \(\log\frac{1}{8}\) or \(\log\frac{1}{2}\) — the same theorem corrupted differently in two places. That is the dangerous failure mode: the output renders cleanly and reads plausibly, so nothing flags the error. marker and MinerU both came through corruption-free — every \(\delta\) intact (26/26), \underbrace annotations, side-annotated inequality chains and fraktur Rademacher complexities all faithful; MinerU additionally preserves equation numbers as \tag{…}. So the refined rule: Docling for structure and prose, marker or MinerU for any equation that will be used rather than skimmed. Scheduling differs, though: marker’s runtime is corpus-dependent (62 s on the paper whose text layer it could reuse, ~8 min on the one that triggered full OCR), while MinerU pays a fixed ~4-minute predictor load and then converted the 24-page test paper in under a minute — so MinerU amortizes over batches, marker wins one-offs.

I also tested Docling’s other architecture, in case Granite’s end-to-end VLM was the wrong tool: the standard pipeline with --enrich-formula, which detects equation regions and hands each to IBM’s specialist CodeFormula model. It does fix the token swaps — 14/14 \(\delta\) intact on the appendix slice — confirming that the silent substitutions are a property of whole-page VLM transcription, not of Docling. But it is not a usable fix: 46 minutes for the 24-page paper (CPU-bound — and ~4.7 min/page on the equation-dense appendix, since per-page cost tracks formula count), and it introduces its own structural errors (the Rademacher subscript \(\mathfrak{R}_{|\mathcal{D}|}\) came out with the absolute-value bars displaced). Both Docling modes also render the predictive family \(\mathcal{V}\) as \(\nu\). So the conclusion stands — marker or MinerU, which are faster and cleaner than either Docling path.

A general vision model reading the pages directly belongs in this comparison but I ran out of enthusiasm. I would not be surprised to see it land beside Docling on the corrupting side, since it is the same whole-page operation, but YMMV.

6 The tools

uvx --from 'markitdown[pdf]' markitdown some.pdf

marker_single ./doc.pdf --use_llm \
  --llm_service marker.services.openai.OpenAIService \
  --openai_base_url http://127.0.0.1:1337/v1 \
  --openai_model qwen3.6-35b-a3b-mxfp4 \
  --openai_api_key unused

7 For agents — the pdf-ingest skill

When the reader is an agent rather than me, the routing above has to be packaged where the agent can load it. The mechanics that decide the design: in the agentskills model, every skill’s one-line description sits permanently in the system prompt and the body loads on trigger — so per-engine skills route at trigger time (the model picks from one-liners, the least reliable dispatch there is), while a single skill routes at instruction time (one trigger, then the body walks the model through the decision). The routing policy is the hard-won part — probe the text layer first, placeholder the images, convert once and read selectively, point refinement calls at the host’s own idle endpoint — so it gets the monolithic treatment, while each engine’s flags and gotchas live in per-engine files the skill loads on demand:

.claude/skills/pdf-ingest/
  SKILL.md          # trigger + routing policy + budget disciplines
  engines/
    markitdown.md   # instant text-layer extraction
    docling.md      # Granite VLM, MLX, placeholder images
    marker.md       # maths→LaTeX, MPS, --use_llm wiring
    mineru.md       # the accuracy-maximalist escalation

That directory is published at danmackinlay/pdf-ingest-skill — projected by git subtree out of this site’s (private) source repo, where it is actually edited and tested, so the public copy cannot fork, only lag. Install: Claude Code, git clone https://github.com/danmackinlay/pdf-ingest-skill ~/.claude/skills/pdf-ingest (or per-project under .claude/skills/); Hermes, the same clone into ~/.hermes/skills/; pi, pi install git:github.com/danmackinlay/pdf-ingest-skill. One key fact makes the wiring safe: during tool execution the harness’s model is idle — the agent loop is generate → dispatch tool → wait — so a converter that wants LLM refinement can call back into the same local endpoint that hosts the agent, with no contention and no second model in RAM.

Osaurus is the awkward middle case, because it plays two roles at once: an agent harness that reads agentskills.io skills natively, and an LLM server listening on localhost:1337 — and this skill fits one role much better than the other. As a harness it can import the skill (Management window → Skills → Import), though its GitHub importer looks for a .claude-plugin/marketplace.json manifest this repo doesn’t ship, so the import path is a local file or zip. The deeper mismatch is execution: Osaurus agents only get a shell inside an isolated Alpine Linux VM (macOS 26+, Apple’s Containerization framework), where Docling loses MLX and marker loses MPS — the recipes still run, just CPU-only, forfeiting the Apple Silicon tuning that justifies half the routing. Better wiring for now uses the server role: run the skill from a harness with a native macOS shell — Claude Code, Hermes, pi — pointed at Osaurus on localhost:1337.

Jan speaks MCP Connectors rather than skills; for it, the official docling-mcp server (LF AI & Data, MIT) covers the Docling path — with one trap: v2.0 defaults to remote conversion against a Docling Serve API, so offline use needs the [local] extra and DOCLING_CONVERSION_MODE=local, or it quietly phones out. MinerU ships its own MCP server too. Nothing pre-existing encodes the routing policy — Osaurus’s plugin registry has no PDF-converter entry at all (checked June 2026) — which is exactly why the skill exists.