Agent harness design

1.1 Where the rest of this lives

Seven pages carry this material between them, split by which layer they are about and which job that layer is doing.

The products that wrap all this — which harness to actually install, and what it can wreck — are in the applied notes. Coding has no harness-theory page of its own, because its harness theory is domain-general: the coding harnesses all sit at one point in this space, and it is the same space everything else sits in. A stranger should read across the harness row.

2.1 Two kinds of executor

A coding executor does work — it edits a file, applies a migration, pushes a commit — and the artefact it leaves behind is the point of running it. A verifier only checks, and nothing outside the check changes. Only the second kind can be run over and over: fifty attempts at a proof cost fifty times the tokens and nothing else, where fifty attempts at a database migration is a different sort of afternoon. That asymmetry is most of why maths loops sample so freely and why nobody fans out an agent with commit access.

What comes back is either raw material for the next attempt or a judgement on the attempt just made — a verdict — and the same tool emits both at different moments. Requena et al. (2026)’s prover works in Lean, whose compiler machine-checks every step and will also compile a proof with declared gaps, reporting what remains to be shown for each. A compile with no gaps is a verdict: theorem proven, stop. A compile with gaps is a to-do list for the next attempt.

3.1 Wiring

Wiring is the “shape” of the call graph — how many steps, in what order, feeding what. There are four things a step can do: chain onto the last one, fan out into copies of itself, route to one of several branches, or recurse into a fresh sub-problem. Only the first two are pure wiring, and they are what the rest of this section is about. The other two are where wiring reaches into the neighbouring axes and are treated there — a route is a dispatch decision given structural form, and a recursion is a sub-agent, which is a context move.

The two pure forms: parallel samples many attempts at once and aggregates them; sequential runs one attempt at a time and feeds each result into the next. Parallel pays when attempts fail independently and a verdict can cheaply pick out the winners; sequential pays when each failure teaches the next attempt something. The two also spend differently: parallel buys attempts with tokens and gets them back in wall-clock, while sequential buys them with wall-clock and a context window that grows every round. Fan-out width is a budget decision as much as an accuracy one. In practice we increasingly use hybrids and compositions of these hybrids (Zhang et al. 2025), which come with vendor names — prompt chaining, routing, evaluator–optimizer and the rest — worth recognizing in documentation.

The parallel half of this is only available where the executor is side-effect-free. Fifty proof attempts are fifty times the tokens; fifty attempts at the same repository are fifty conflicting sets of edits. So the domain, not the designer, decides whether half the design space is on the table at all.

Parallel execution entails an aggregation step — majority vote, best-of-\(n\) against a verifier, filtering out the false answers, LLM-based consensus.

Aggregation is not always available, and whether it is depends on what the branches return. Comparable outputs — a number, a label, a classification, a boxed answer — have a mode, and the mode is a signal. Arguments do not. \(k\) candidate proofs, \(k\) design proposals, \(k\) code-review write-ups are \(k\) different objects, and a set of arguments has no modal element to take. Where the branches return arguments, parallelism still buys candidates, but the aggregation step has to become a judge or a checker rather than a tally — a different and much more expensive component, and one that can be wrong in ways a tally cannot. The fan-out is the cheap part; the thing on the end of it is not. The maths case is where this bites hardest, but it is not a maths-specific problem.

The two forms compose rather than compete. Requena et al. (2026) test single-shot parallel sampling against sequential refinement, and find that adding refinement to a parallel fan-out loop improves sample efficiency — more theorems proven per token spent.

The other thing wiring fixes is how much of the control flow is guaranteed. A hand-written script that branches and retries does the same thing twice; an agent asked to follow a SKILL.md mostly does. That determinism trades against the flexibility that made an agent worth using in the first place, and it is what an agent emitting a script instead of taking turns is buying.

3.2 Verification

Wherever a step judges a candidate, two independent properties of that step are in play: the grade of the judgement — how far it can be trusted — and its unit — whether it is about a whole candidate, one claim inside it, one sub-goal, or one step of the argument.

Grade first, because there is no sense making multiple attempts at something if we cannot choose which attempts are better. It is two questions rather than one, and they have different consequences. Is the verifier complete — does it accept everything that is actually correct? And is it incorruptible — can a wrong candidate talk it round?

Incompleteness costs recall, and nothing else. An incomplete verifier throws away some correct candidates — a computer algebra system (CAS) refuses a pair it cannot simplify — but it cannot be selected against: no amount of sampling teaches a candidate how to be wrong in a way a CAS likes. A corruptible verifier can be. Sample far enough against a judge and we begin selecting for candidates that fool the judge rather than candidates that are right. So incompleteness sets how much of the budget gets wasted, and incorruptibility sets how large the budget can usefully be — which is to say, the second question is the one that licenses sampling wide.

Four combinations, four names:

Against an exact verifier, success keeps climbing with the number of attempts (Pass@k in the benchmark jargon: the chance that at least one of \(k\) samples is the real deal), and prover loops sample in the thousands. Against a soft judge, if we are concerned about reward hacking at inference time, it becomes reasonable to allocate effort into improving the judge instead of widening the search. Nomos is the worked example of taking that option: a judge trained alongside the proposer, and a pairwise tournament where the vote used to be.

The soft grade also has a ceiling no judge clears: wherever aggregating means deciding that two outputs say the same thing, that decision is a semantic equality test, and semantic equality is not in general decidable.

We can usefully combine grades, it turns out: Requena et al. (2026) back the compiler with deterministic checks for the known dodges (which apparently is a thing) and then use an LLM reviewer for the residue — statement-weakening and metaprogramming tricks that compile without proving the theorem. A soft verdict layered over an exact one buys something neither has alone.

Now the unit. A verdict on a whole candidate tells the loop only whether to keep it; a verdict on one claim inside the candidate tells the loop which part to fix, which is what gives a sequential wiring something specific to iterate on. The two properties interact in one direction: a fine unit is only affordable when the grade is cheap and incorruptible, because a fine unit means calling the verifier hundreds of times per candidate, on fragments, and a judge asked to grade a single step answers no better than it grades the whole thing. That is why the finest-grained systems all sit behind a compiler and the coarse ones are where soft verdicts live. Mathematics has climbed further down this axis than anywhere else — the rungs, and which systems sit on them.

3.3 Context

What each call sees, i.e. context engineering. Every call to a model endpoint is a fresh call, so the “history” the model brings to the current conversation is a choice, remade each step. Traditionally that is the system prompt and any skills, some tool definitions, retrieved material, the history so far, and whatever notes the loop has kept. We have considerable scope to vary it.

This breaks in all kinds of interesting ways. Too little context and the model wastes time re-deriving what it worked out an hour ago. Too much context, aside from being expensive, can fail in various complicated ways: we can just run out of context window, attention to the middle of a long context degrades (the lost-in-the-middle effect), and we increase the risk of propagating bad ideas ad infinitum.

The mitigations:

• RAG pulls in knowledge the model does not have in the weights (Mathlib search, repo search, web search) on demand.
• State can be externalised to files the loop re-reads on demand.
• Compaction summarizes older turns when the window fills. Done well, transparent; done badly, the agent loses the thread.
• Sub-agents take a self-contained sub-task to an isolated child — its own conversation history, its own tool budget — and return a summary, so the dirty work (grepping code, searching the web, writing throwaway spikes) happens in a context we then throw away and the parent’s stays lean enough to keep planning.

Claude Code supports sub-agents natively as agent teams; the pi harness leaves them out deliberately, for the reason below.

Requena et al. (2026) ablate this axis against no memory at all, a rolling history of recent attempts, and a self-managed note the model writes for itself. The note wins on both theorems proven and cost.

3.4 Dispatch

We could use a single frontier model for every step, but typically need not. Specialised models cover specialised capabilities, and cheap ones cover the easy steps to keep the cost down. In the hand-built loops the answer is hard-coded. We give the agent a solve() oracle and let it decide when to call it, or wire the specialist into the script ourselves, or pick a fast model for the sub-agents and a slow one for the planner. However, we are free to learn to make this decision also.

3.5 The four, made once

Take a terminal coding agent — Claude Code, OpenCode, any of them — and watch the four answers arrive.

Wiring is sequential, because the executor edits a real repository and each failure teaches the next attempt something anyway. Verification is the test suite, which is empirical. Context is where the actual engineering goes: grep and read on demand rather than a pre-built index, compaction when the window fills, sub-agents for the searching — and each of those three costs some of the record of what the agent did. Dispatch is one frontier model, plus in some products a cheap one screening shell commands before they run.

Only dispatch was free, and it is the cheapest of the four to change — which is roughly the situation any harness designer is in, and the reason the interesting arguments are about the other three.

3.6 How the four interact

Wiring is the primary choice, and verification the primary constraint. Wiring is what a designer actually decides, and the other three answer questions it raises: every wiring needs a verdict of some sort, since parallel needs one to select among attempts and sequential needs one to inform the next attempt and to decide when to stop. But the grade of verdict available bounds both shapes, and the grade is mostly handed to us by the domain rather than chosen. So where the domain fixes what a verdict can be — as mathematics does, emphatically — the dependency runs the other way: the verdict we can get decides the shapes we can afford, and that was settled before we arrived.

Sequential wiring additionally drags in context, because attempt \(n+1\) that remembers nothing of attempts \(1..n\) will make their mistakes again, so the memory design is what stops a sequential loop going in circles. Dispatch is the one that really is orthogonal — whatever the shape, we still choose who fills each slot.

That is why verification and memory keep turning up together in ablations despite looking unrelated: one is what any wiring needs in order to know whether it is getting anywhere, and the other is what the sequential kind needs in order to get anywhere at all.

3.7 Systems as coordinates

Factored this way, harnesses built for different jobs are points in one space.

The coding agents crowd into one corner while the maths agents spread across the space. The always-on assistants answer verification with nothing at all, which is why they suit open-ended errands and are dangerous left alone. Ornith is not a point in the space but a search over it, below.

What these four choices become once the answer is a piece of mathematics is in the reasoning notes; what I built out of them is a six-step path.

4.1 Session memory

Memory does two jobs in this literature. The scratchpad a refinement loop carries between attempts — what failed last round, which lemma does not exist — is a context move at the scale of a few turns, and the maths notes have the numbers on it. This is the other one: what survives a window filling, or a week off.

The simplest answer is compaction, and it degrades on long tasks, because each summary is a lossy pass over the previous summary and the losses compound the same way the per-turn errors below do.¹

Two structural answers, beyond summarizing harder.

Let the human pick what survives. pi’s /tree jumps back to an earlier point in the history and collapses everything since into a précis, so the selection is deliberate and visible rather than automatic and silent.

Move memory out of the main loop. The main agent keeps no notes of its own; a separate writer subagent reads the conversation and commits structured checkpoints to disk. It fires early — well before the window is tight — so the extraction happens while there is still room to think, and before lost-in-the-middle erodes the thing being extracted. As the window fills, the runtime opens a fresh one and rebuilds context from those files, so from the model’s point of view the conversation never breaks. The checkpoints sit on different lifecycles — this session, this project, every project — with a full unindexed trace of every message and tool call underneath as the fallback for whatever the structured layers dropped.

4.2 Multi-turn reliability

A long-horizon run is a chain of individual turns — read context, decide, call a tool, repeat. Each turn accumulates some additional chance of a wrong call: editing the wrong file, passing a test by changing the test… Those per-turn odds compound, so even a low rate dominates a long enough run — at i.i.d. error rate of 2% per turn, a 200-turn task finishes clean only about 2% of the time (\(0.98^{200}\approx0.02\)). A short interactive session is easy to correct — we, the humans, spot the bad turn and fix it. But the economic benefit of agents that they aim to sell us on is doing unattended multi-hour runs. In that case, we really want to juice that error rate down low to top out the famous reliable task-length metric.

A compounding error rate has exactly three attack surfaces. We can make each turn less likely to go wrong, make the chain shorter so there are fewer turns available to go wrong, or catch the wrongness at the end and go round again.

• Buy down the per-turn odds, and spend to do it. Sample several candidate plans in parallel and use a low-temperature judge to pick one, aimed at the planning turns where a wrong decision costs the most downstream. A marginal gain at a large cost, which might still be the right trade on a high-stakes task.
• Shorten the chain by emitting code instead of taking turns. The agent writes a script — spawn an agent, run these in parallel, pipeline those — and a sandbox runs it deterministically. This is the wiring choice again, with branch and retry logic guaranteed by code rather than by a model remembering to follow a SKILL.md, which is that many fewer per-turn decisions available to go wrong. That code-beats-prompt principle for predictable control flow is originally Anthropic’s.
• Catch the failures at the end with a stopping-condition verifier. The user states a done-condition in natural language, and each time the agent tries to stop, an independent model call checks the whole history against it; if the work is not finished, it whinges about the shortcomings and tries again. This is the disciplined form of the Ralph Wiggum loop — the brute-force pattern of relaunching an agent on the same prompt until the job is done.

Note what that last one needs — an independent judgement of whether the work is actually done, stated in natural language and evaluated by a model. That is a soft verdict guarding a long and expensive run, with all the foolability that implies, and the longer the run the more opportunity the agent has had to talk itself into being finished.

4.3 Evolution — learning across sessions

OK, now ultra-long horizons. Not just tasks, but whole projects, or careers. If we come back to the same project regularly and each time the agent has forgotten everything, it feels like we’re leaving performance on the table, re-deriving the same constraints and repeating the same mistakes. The answer is background agents grinding over the agent’s own history, and they want two clocks rather than one. The faster one tends the facts: weekly, it reads past sessions and the memory files, merges and deduplicates, checks that file references still resolve, and compresses the result back down before it sprawls. The slower one harvests process rather than facts: monthly, it hunts for recurring work patterns and solidifies them into reusable skills, CLI commands, and standing procedure documents.

Hermes takes a different approach, learning the user rather than the project, accreting auto-generated skills, FTS5-searchable session history, and a persistent model of our preferences via Honcho’s dialectic user-modelling, on the pitch that the longer it runs the more it knows about us. OpenClaw keeps persistent memory in the same spirit. Either way the library is now writing itself, which is a standing risk.