Agent harness design

2.1 Kinds of executor

An executor is an interpreter for some vocabulary of actions, and the vocabulary is the main thing that distinguishes one executor from another. Some executors offer a fixed menu of typed calls — read_file, search, whatever an MCP server exposes. The model’s instructions invoke one from that menu, passing values that fill in the arguments. More generally, some executors are powered by a general-purpose programming language, such as a shell or a Python interpreter, and the model writes whole programs for those. To my mind the mathematical agents are another kind again: here the agent writes formal proofs rather than programs and the results are checked and analyzed by a Lean compiler or a computer algebra system. The controlled vocabulary option is presumably safer (don’t put dangerous things on the list, simple rules check each), but sacrifices flexibility. A general-purpose language is, by contrast, extremely flexible and arbitrarily dangerous, since code itself is dangerous and hard to monitor.

Which vocabulary to offer is ours to choose, and it is a dispatch decision like any other choice of tool. Two other things differ from one executor to the next — the state its actions land on, and what its return value can tell us about the attempt — but those are set by the task rather than by the interpreter, and they are where the design space starts.

3.1 State

The first given is the state that the executor’s actions land on. A side-effecting executor changes something in the world (writes some code, buys me a flight to Bali…) and thus needs to take into account how its past actions have changed the world. A side-effect-free one has outputs that are pure functions of the inputs. This can be somewhat relative: The same shell could be side-effecting in my working copy of the code but effectively side-effect-free if I run it to see the outputs in a throwaway container. A mathematical proof checker OTOH has no side-effects; different attempts at the same proof do not interfere with one another.

Side-effect-free executions have the useful property of being repeatable and parallelizable: fifty attempts at a proof cost fifty times the tokens and nothing else, where fifty attempts at a database migration is some kind of slow nightmare. That asymmetry is most of why maths loops sample so freely and fan out so much.

3.2 Verifiers

The second given is what, if anything, can judge an attempt. The thing being judged is a candidate — whatever the model has offered as an answer or as progress toward one, a patch, a proof, a plan.

Every execution returns an observation, the output of whatever it did. This might be anything — a directory listing, a stack trace, a compiler error. All of it is information the model can use on its next attempt. Some observations can additionally be read as a verdict, i.e. a judgement on whether the candidate just submitted was “correct” in some well-defined sense, e.g. the tests passed, the proof compiled. A verifier is whatever produces verdicts — a compiler, a test suite, a judge model, me. A Lean compile that fails is a verdict (meaning “not proven”), but it still reports where it failed and which goals were left unsolved at that point. Requena et al. (2026)’s prover leans on this: Lean will compile a proof with declared gaps, reporting what remains to be shown for each, which turns an unfinished attempt into a to-do list for the next one. A compile with no gaps and no errors, by contrast, ends the loop with a verdict of “successful”.

Verifiers differ in grade — how far their verdicts can be trusted — and the best grade on offer is mostly a fact about the domain: a proof has a compiler, a patch has whatever tests someone wrote, and a flight booking has only my opinion of it. There is no sense making multiple attempts at something if we cannot choose which attempts are better, so the grade limits most of what follows. It is two questions rather than one, and they have different consequences. Is the verifier complete — does it accept everything that is actually correct? And is it incorruptible — can a wrong candidate talk it round?

Incompleteness costs recall, and nothing else. An incomplete verifier throws away some correct candidates — a computer algebra system (CAS) refuses a pair it cannot simplify — but it cannot be selected against: no amount of sampling teaches a candidate how to be wrong in a way a CAS likes. A corruptible verifier can be. Sample far enough against a judge and we begin selecting for candidates that fool the judge rather than candidates that are right. So incompleteness sets how much of the budget gets wasted, and incorruptibility sets how large the budget can usefully be — which is to say, the second question is the one that licenses sampling wide.

Four combinations, four names:

Against an exact verifier, success keeps climbing with the number of attempts (Pass@k in the benchmark jargon: the chance that at least one of \(k\) samples is the real deal), and prover loops sample in the thousands. The soft grade has a ceiling no judge clears: wherever aggregating means deciding that two outputs say the same thing, that decision is a semantic equality test, and semantic equality is not in general decidable.

3.3 Wiring

Wiring is the control flow of the harness — how many steps, in what order, feeding what. A step here is one node in that graph, which is not the same unit as a turn: fifty steps can run at once, where turns only ever come one after another. A harness is a program, and its control flow can be anything a program’s can: it branches, retries, hands pieces of the problem to sub-agents. The popular arrangements have vendor names — prompt chaining, routing, evaluator–optimizer and the rest — and surveys catalogue more (Zhang et al. 2025). I do not think there is a useful taxonomy of all that, any more than there is of call graphs in general.

One decision does recur across harnesses, though: how to spend more than one attempt on the same problem. Parallel wiring samples many attempts at once and picks among them; sequential wiring runs one attempt at a time and feeds each result into the next.

Parallel pays when attempts fail independently and a verdict can cheaply pick out the winners; sequential pays when each failure teaches the next attempt something. The two also spend differently: parallel buys attempts with tokens and gets them back in wall-clock, while sequential buys them with wall-clock and a context window that grows every turn. Fan-out width is a budget decision as much as an accuracy one. It can be a resumability decision too, once a runtime is checkpointing: Claude Code’s workflow runtime replays cached results in the order agents started, stopping at the first one that did not finish, so everything launched after it re-runs even if it completed. Stopping mid-fan-out therefore costs less the finer the fan-out is, and “many small agents or one long one” turns out to be a question about interrupted runs as well as about accuracy.

The parallel half of this is only available where the executor is side-effect-free. Fifty proof attempts are fifty times the tokens; fifty attempts at the same repository are fifty conflicting sets of edits. So the domain sets the default, and a designer who wants the parallel half has to pay for the isolation that makes it safe, a decision most people never notice making.

Parallel sampling needs an aggregation step to pick among the attempts — majority vote, best-of-\(n\) against a verifier, filtering out the false answers, LLM-based consensus.

Aggregation is not always available, and whether it is depends on what the branches return. Comparable outputs — a number, a label, a classification, a boxed answer — have a mode, and the mode is a signal. Arguments do not. \(k\) candidate proofs, \(k\) design proposals, \(k\) code-review write-ups are \(k\) different objects, and a set of arguments has no modal element to take. Where the branches return arguments, parallelism still buys candidates, but the aggregation step has to become a judge or a checker rather than a tally — a different and much more expensive component, and one that can be wrong in ways a tally cannot. The fan-out is the cheap part; the thing on the end of it is not, and what that thing can be depends on the verifiers available. The maths case is where this bites hardest, but it is not a maths-specific problem.

The two forms compose rather than compete. Requena et al. (2026) test single-shot parallel sampling against sequential refinement, and find that adding refinement to a parallel fan-out loop improves sample efficiency — more theorems proven per token spent. Fanning out partial attempts, keeping the promising ones and extending those is also the shape of the particle-style sampling tricks, which do it at the scale of tokens and paragraphs rather than whole attempts.

The other thing wiring fixes is how much of the control flow is guaranteed. A hand-written script that branches and retries does the same thing twice; an agent asked to follow a SKILL.md mostly does. That determinism trades against the flexibility that made an agent worth using in the first place, and it is what an agent emitting a script instead of taking turns is buying.

3.4 Context

What each call sees, i.e. context engineering. Every call to a model endpoint is a fresh call, so the “history” the model brings to the current conversation is a choice, remade each turn. Traditionally that is the system prompt and any skills, some tool definitions, retrieved material, the history so far, and whatever notes the loop has kept. We have considerable scope to vary it.

This breaks in all kinds of interesting ways. Too little context and the model wastes time re-deriving what it worked out an hour ago. Too much context, aside from being expensive, can fail in various complicated ways: we can just run out of context window, attention to the middle of a long context degrades (the lost-in-the-middle effect), and we increase the risk of propagating bad ideas ad infinitum.

The common responses are below, grouped under three verbs for convenience. The list is not exhaustive, but most of what this page later calls memory is built out of these moves.

Fetch — leave it on disk, pull it in when wanted.

RAG

pulls in knowledge the model never had in the weights — Mathlib search, repo search, web search.

Externalization

does the same for knowledge the loop produced itself: write the state to a file, keep a pointer in the window.

Shed — cross a window boundary by throwing something away.

Compaction

summarizes the conversation so far, discards the originals, and starts a fresh window with the summary at the front.

Rebuild

discards the window outright and seeds a fresh one from externalized files, so nothing is ever a summary of a summary. Xiaomi’s term, and the move that session memory turns on.

Delegate — spend a different window.

Sub-agents

take a self-contained sub-task to an isolated child — its own conversation history, its own tool budget — and return a summary, so the dirty work (grepping code, searching the web, writing throwaway spikes) happens in a context we then throw away and the parent’s stays lean enough to keep planning. Claude Code supports these natively as agent teams; the pi harness leaves them out deliberately, for the reason below.

As the timescale stretches, the same moves keep turning up, composed differently and run by different actors, which is the long-horizon problem.

Requena et al. (2026) ablate this choice against no memory at all, a rolling history of recent attempts, and a self-managed note the model writes for itself. The note wins on both theorems proven and cost.

3.5 Dispatch

We could use a single frontier model for every step, but typically need not. Specialized models cover specialized capabilities, and cheap ones cover the easy steps to keep the cost down. In the hand-built loops the answer is hard-coded. We give the agent a solve() oracle and let it decide when to call it, or wire the specialist into the script ourselves, or pick a fast model for the sub-agents and a slow one for the planner. Dispatch is often the cheapest of the three to change, since swapping the model behind a step is frequently a configuration edit, where rewiring or reworking the context means changing the harness’s code. DeepSeek Harness is built to make the rest cheap to change too, on paper at least: every capability in it — models, tools, skills, sessions, sandboxes, storage, loops, scheduling — is a plugin chosen in a configuration file. That reaches below the tool the model sees: a capability with interchangeable back ends, such as shell execution, is packaged as a service definition, a provider that implements it, and a tool that consumes it, so local Bash could be swapped for another executor without changing anything on the model’s side. We are also free to learn to make this decision rather than hard-code it.

3.6 How the givens bound the choices

Most of the dependencies I keep running into go through wiring, and both givens bound it. The state decides whether the parallel shape is available at all without paying for isolation. The grade decides how wide a parallel wiring can usefully sample and how fine a unit a sequential one gets to iterate on. So where the domain fixes what a verdict can be — as mathematics does, emphatically — the dependency runs the other way: the verdict we can get decides the shapes we can afford, and that was settled before we arrived.

Among the choices themselves, sequential wiring drags in context, because attempt \(n+1\) that remembers nothing of attempts \(1..n\) will make their mistakes again, so the memory design is what stops a sequential loop going in circles. Dispatch is the most nearly independent of the three — whatever the shape, we still choose who fills each slot — though a router makes dispatch part of the control flow, and a cheaper model in each slot changes how wide we can afford to fan out.

That may be why verifiers and memory keep turning up together in ablations despite looking unrelated: one is what any wiring needs in order to know whether it is getting anywhere, and the other is what the sequential kind needs in order to get anywhere at all.

3.7 Systems compared

Let us compare some harness design philosophies in terms of the three choices, plus the verifier each one works with, whose grade is the task’s doing rather than the designer’s.

The coding harnesses tend towards sequential design because they need to handle side effects in a controlled order, although MiMo Code introduces some weird twists. The maths agents are way less constrained because theorems don’t interact with one another, unless the world is weirder than we thought. This makes them relaxing to design (I designed one for fun). The always-on assistants are not really verifiable in any meaningful sense and thus dangerous left alone. Ornith is kinda ill-defined, learning its own harness. DSH is also ill-defined, the parameters being determined by the mounted plugins.

4.1 Multi-turn reliability

A long-horizon run is a chain of individual turns — read context, decide, call a tool, repeat. Each turn accumulates some additional chance of a wrong call: editing the wrong file, passing a test by changing the test… Those per-turn odds compound, so even a low rate dominates a long enough run — at an i.i.d. error rate of 2% per turn, a 200-turn task finishes clean only about 2% of the time (\(0.98^{200}\approx0.02\)). A short interactive session is easy to correct — we, the humans, spot the bad turn and fix it. But the economic benefit that agents are being sold to us on involves unattended multi-hour runs. In that case, we really want to juice that error rate down to top out the famous reliable task-length metric.

Under that model there are three things to attack. We can make each turn less likely to go wrong, make the chain shorter so there are fewer turns available to go wrong, or catch the wrongness at the end and go round again.

• Buy down the per-turn odds, and spend to do it. Sample several candidate plans in parallel and use a low-temperature judge to pick one, aimed at the planning turns where a wrong decision costs the most downstream. A marginal gain at a large cost, which might still be the right trade on a high-stakes task.
• Shorten the chain by emitting code instead of taking turns. The agent writes a script — spawn an agent, run these in parallel, pipeline those — and a sandbox runs it deterministically. This is the wiring choice again, with branch and retry logic guaranteed by code rather than by a model remembering to follow a SKILL.md, which means many fewer per-turn decisions are available to go wrong. That code-beats-prompt principle for predictable control flow is originally Anthropic’s, and Anthropic now ships the worked version of it too.
• Catch the failures at the end with a stopping-condition verifier. The user states a done-condition in natural language, and each time the agent tries to stop, an independent model call checks the whole history against it; if the work is not finished, it whinges about the shortcomings and tries again. This is the disciplined form of the Ralph Wiggum loop — the brute-force pattern of relaunching an agent on the same prompt until the job is done.

All three ship in MiMo Code, as Max mode, Dynamic Workflow and Goal, and Xiaomi’s write-up is where I got this way of dividing the problem.

Two systems have converged on the second one, down to the name. Xiaomi calls it Dynamic Workflow; Claude Code calls it dynamic workflows, also JavaScript, also agent() and pipeline(), written by the model and executed by a runtime outside the conversation. The docs frame the distinction better than I had: what separates a workflow from a subagent is who holds the plan — a subagent leaves Claude deciding turn by turn with every intermediate result landing in the window, where a script holds the loop, the branching and the intermediates itself, so only the final answer comes back. That makes it a context move as much as a wiring one: emitting code is externalization applied to control flow. It is also where the quality patterns get somewhere to live — adversarial cross-review of findings before they are reported, or drafting a plan from several angles and weighing them — because a script can express “then check this” in a way a prompt can only ask for.

DSH has a smaller-grained relative, Code mode, which exposes the tools through an SDK so the model can do several steps in one TypeScript program instead of one turn per tool call. It is the same bet — code rather than a sequence of turns holds the control flow — made over tool calls, where the two workflow runtimes make it over agents.

Note what the third one needs — an independent judgement of whether the work is actually done, stated in natural language and evaluated by a model. That is a soft verdict guarding a long and expensive run, with all the foolability that implies, and the longer the run the more opportunity the agent has had to talk itself into being finished.

4.2 Memory within a session

One long conversation about one cohesive project — the app I am building, my tax return — outlives its window well before it outlives its usefulness. Session memory is whatever policy we pick over the context moves to carry it across that boundary.

The default policy is compaction and nothing else. We might imagine this could even be beneficial, because the summary might skip over red herrings and false turns in the original, no? In practice many things can go wrong. Compaction degrades on long tasks, because each summary is a lossy compression of the previous summary and the losses compound the same way the per-turn errors do.

Two policies that do better.

Let the human pick what survives. Still compaction, but the cut is chosen rather than automatic. pi’s /tree jumps back to an earlier point in the history and collapses everything after into a précis, making what counts as “currently” relevant context an overt user choice.

Move memory out of the main loop. Externalization and a sub-agent and a rebuild, with compaction dropped entirely. The main agent keeps almost no notes of its own; a separate writer subagent, dispatched by the runtime rather than invoked by the model, reads the conversation and commits structured checkpoints to disk. It fires early — well below the limit — so the extraction happens while there is still room to think, and before lost-in-the-middle erodes the thing being extracted. When the window does fill, the runtime rebuilds from those files instead of summarizing what is in it, so from the model’s gaslit point of view the conversation never breaks.

MiMo Code is the fullest worked example I know of, and Xiaomi document it in enough detail to copy. The writer fires at roughly 20%, 45% and 70% of the context budget, each pass an incremental update to the last rather than a fresh summary of a summary, which is what breaks the compounding-loss chain. Its output sits on four lifecycles: checkpoint.md for this session, MEMORY.md for this project, a global file for this user, and an unindexed SQLite trace of every message and tool call underneath. That bottom layer is the observability hedge, since everything above it is a summary written by the thing we would want to audit. Each structured file has exactly one writer, enforced in code rather than by prompt; the main agent’s one channel is a free-form scratchpad that the writer drains and clears at each checkpoint.

The general form is older than the coding agents and has a better name: Letta’s sleep-time compute, in which the primary agent is deliberately not given the tools to edit its own memory and a background agent gets them instead. OpenClaw cites the same work for its consolidation pass. Qwen Code looks like this and is not, which cost me some confusion: it does run a background extraction agent over the transcript, but it fires once per turn rather than at budget fractions, it harvests durable facts about the user rather than what the agent is in the middle of, and when the window actually fills it compacts like everyone else.

4.3 Learning across sessions

OK, now ultra-long horizons. Not just tasks, but whole projects. Or whole careers, but let’s not get ahead of ourselves.

If we come back to the same project regularly and each time the agent has forgotten everything, it feels like we’re leaving performance on the table, re-deriving the same constraints and repeating the same mistakes. A new session starts empty, so everything it knows it reads off disk, and the interesting question is how anything got written there.

Not by being composed at project scale — by surviving. MiMo’s writer moves an observation from checkpoint.md up into MEMORY.md once it has held steady across several session checkpoints, so the project file is a filter over the session files rather than a separate act of authorship.

What is new at this timescale is rot. A session checkpoint is overwritten every few hours and never lives long enough to go stale, where a project memory accretes for months, its file references decay, and no one in the session loop is looking after it. So we get background agents grinding over the agent’s own history, and they want two clocks rather than one. The faster one tends the facts — MiMo calls it Dream: weekly, it reads past sessions and the memory files, merges and deduplicates, checks that file references still resolve, and compresses the result back down before it sprawls. The slower one harvests process rather than facts — Distill: monthly, it hunts for recurring work patterns and solidifies them into reusable skills, CLI commands, and standing procedure documents.

Only the first of those is still memory. A skill changes what the agent does rather than what it knows, which is why this section is learning and the one above is not — and it is the same move the next one makes on the harness itself.

The dreaming half is sleep-time compute again with the period stretched from minutes to weeks, and enough people have arrived at it independently that they have converged on the name: MiMo, Qwen Code and Letta all ship a dream pass, with Qwen’s gated on a day having passed and five new sessions having accumulated. Hermes points the same machinery at the user rather than the project, accreting auto-generated skills, FTS5-searchable session history, and a persistent model of our preferences via Honcho’s dialectic user-modelling, on the pitch that the longer it runs the more it knows about us. OpenClaw keeps persistent memory in the same spirit, and cites Letta for it. Either way the library is now writing itself, which is a standing risk, and a worse one here than within a session: a bad session checkpoint dies with the session, where a bad skill is loaded into every session after it.