AI agents, applied

Stack vocabulary, MCP, and the products that wrap the models

2025-02-02 — 2026-07-25

Wherein the Reader Is Furnished a Shopping List of Agentic Harnesses, Wherein the Trade-Offs Between Observability, Blast Radius, and Attention Budget Are Weighed, and MCP and Skills Protocols Are Duly Distinguished Before Recommendations Are Ventured.

AI safety
computers are awful together
faster pussycat
language
machine learning
neural nets
NLP
premature optimization
slop
technology
UI

As is appropriate for the subject, this page is slop — albeit slop distilled from my own notes while I installed things and tried to understand how they worked.

Figure 1

We’re using “agentic” AI now — Claude Desktop, OpenClaw, and the rest. How do we do that? Should we do that? What is the least worst way to extract value from The Machine without divulging all our secrets to The Man?

This page is mostly AI slop, i.e. LLM descriptions of my decisions in some recent exploratory projects. However, it is useful, so I am publishing it now rather than waiting for a more polished version.

An agent, here, is “a language model in a loop”: it can call tools — read a file, run a shell command, fetch a URL — see what came back, and pick its next move, going around until it decides the job is done. The difference from a chat window is that it acts on the world rather than only emitting text about it. The loop is the harness — the code deciding how the model gets called and what becomes of its output, sampling parameters included — so an agent is a model plus a harness, and the two halves come apart. The harness design notes keep them apart deliberately, because when an agent gets better it matters which half did it.

This page is the shopping list. What a harness is made of, the four choices that distinguish one from another, and the layer vocabulary the rest of this uses are on the design page, along with the map of which page covers what; anyone who already runs an agent and wants to know why it plateaus wants that page rather than this one.

Most of the disagreements below are about what to trade for what: how much of the agent’s working we can still see afterwards (which the context tricks quietly spend), how large the blast radius is when it does something stupid, and how to spend an attention budget that is both finite and metered.

1 Should we? When an agent is the right tool

Agents are more likely to be helpful when:

  • The task is multi-step and exploratory (debug this, refactor that, find references to X).
  • The intermediate steps have value beyond the final answer (the agent reads files, runs commands, reports what it found).
  • Tool use unlocks capability the model lacks on its own (executing code, querying live data, browser automation).
  • The user has enough executive function to avoid getting stuck in an addictive dependency cycle

Agents are less helpful when:

  • The task is a one-shot transformation (translate this text, summarize this PDF, generate boilerplate).
  • The task is so specific that a hardcoded script is faster and less error-prone (but we could write that script with an agent).
  • The cost of a wrong action is high and approval gates would make the agent slower than doing the task directly.

Granting that we should: first the two protocols everything speaks (MCP and skills), because every product below is described partly in terms of them. Then the products themselves, in three tiers by how much assembly is required — libraries we build a harness on top of, ready-to-run harnesses we point at a model server and start, and always-on assistants that live on a machine somewhere and message us. Then the cross-cutting practicalities that apply whichever we picked: how documents get into it, where the thing runs, and what it can wreck. Which one I would actually install is at the end, where it belongs.

2 Agents I have known and loved

Form, in the table below, is what we actually launch.

System Tier Form Note
pi library ↔︎ ready-to-run Node CLI Minimal enough to build on, complete enough to run as-is
smolagents library Python package The agent writes Python instead of calling named tools
Qwen-Agent library Python package Alibaba’s, tuned for Qwen models, batteries included
LangGraph, CrewAI, … library Python frameworks Industrial; state machines and checkpointing
OpenCode ready-to-run terminal app Any of 75+ providers, big community
Goose ready-to-run desktop app + CLI + API The institutional one; rides a subscription rather than an API key
Open WebUI ready-to-run self-hosted web app A chat UI that grew agentic features later
Claude Code ready-to-run CLI, or hosted in a browser Closed, Anthropic-only
Cowork assistant-ish desktop app Closed commercial; assistant-shaped but not self-hosted
Hermes always-on assistant Python daemon Self-hosted, multi-channel, six sandbox backends
OpenClaw always-on assistant Node daemon, built on pi Self-hosted, ~25 channels, fewer scruples

2.1 What actually differs between them

Products differ mostly in how many of the four design choices they have already made on our behalf. pi is all primitives and almost no features — no built-in MCP, no built-in sub-agents, no built-in plan mode — where Claude Code ships sub-agents, plan mode and agent teams.

Four features recur across everything below. All four are things the harness does to the loop, which is narrower than what it does to the environment the loop runs in: nearly everything here also ships isolation of some kind — Claude Code’s five tiers, Hermes’s six sandbox backends, Osaurus’s Linux VM — and pi is the conspicuous refusal. That side of it is its own question, below.

Sub-agents
The model farms a sub-task out to a child and gets back only a summary. Three things at once: the parent’s context window stays free, the children can run in parallel, and each child can be given its own model, tools and prompt.
Compaction
The harness summarizes the older turns once the window fills, so a long session degrades rather than hitting a wall. This one really is only a context move.
Plan mode
The agent writes down what it intends to do and waits for us to approve the whole plan before it touches anything. Mostly this buys quality rather than safety: correcting a plan is cheap, and correcting finished work is not. Claude Code’s is the version most people have met.
Permission gates
Interrupt action by action instead, on whatever the harness counts as dangerous. The finest-grained of the controls and the least loved: the defaults slide toward security theatre.

The first two buy their window space with observability. A child that reports one paragraph, or a summary of a summary, has thrown away the record we would want when something goes wrong.

Two compatibility details are worth checking before committing to one, because neither shows up in any feature list:

  • The tool-call dialect. Harnesses accept different tool-call response formats (JSON, XML, Qwen xml_function, Mistral [TOOL_CALLS]) with matching parsing logic, so a model fine-tuned to emit its own — the maths specialists especially — will not drive a harness expecting the native tools protocol.
  • How extensions load. Whether the harness reads agentskills.io markdown, TypeScript extensions, Python modules, or nothing at all. This decides whether work done for one harness transfers to the next.

3 Extension protocols

3.1 MCP — Model Context Protocol

MCP is an open protocol for connecting LLM applications (clients, harnesses) to data sources and tools (servers). The harness is the client; an MCP server is a separate process exposing tools, resources and prompts, which can perfectly well be another process on the same laptop talking over stdio.

The point of standardizing this is that neither end needs to know about the other in advance: I run a Zotero MCP server once, and every MCP-speaking client I own can suddenly search my reference library, without any of them shipping Zotero support.

MCP is not strictly necessary nor universal. pi deliberately does not ship it — its position is that an MCP server is just a wrapper around tools that could equally well be exposed as CLI tools with README files (i.e. Skills). Zechner observes that MCP does clog up the context window more.

Some fun MCP servers:

3.2 Skills and agentskills.io

agentskills.io is a second open standard in this space, solving a different problem from MCP: where an MCP server is a live process exposing tools over a protocol, a skill is a plain markdown file of instructions, read into the prompt only when it looks relevant. Skills carry YAML frontmatter describing the capability and its tools, and are either a single name.md file or a directory with SKILL.md plus supporting files.

Skills are relatively transferable. pi, Hermes, Anthropic’s own Skills system, and various community frameworks all consume the same file format; a skill written for Hermes can be dropped into a pi extension directory and Just Work.

The contrast with MCP is when the capability reaches the model — binding time, as a programmer would say. Skills are typically loaded on-demand per session, often based on what the user is asking about. A harness might have hundreds of skills installed but only load three for any given session, keeping context usage low.

4 Build agent harnesses

We can just build our own specialised agent harness; many have, and I’ve done it. It is not rocket science, but there is a long tail of annoying difficulties to solve, so in general we should probably not — and there exist libraries for the common problems, plus basic harnesses one can customize.

4.1 pi

earendil-works/pi (Mario Zechner / badlogic, MIT, TypeScript / Node) is a minimalist harness. Behaviour beyond the loop-and-tool-calls baseline gets added as skills or TypeScript extensions.

Zechner ships it as the Pi Coding Agent, so pi overtly straddles the line between a generic harness and a coding one.

pi’s entire system prompt plus its four tools (read, bash, edit, write) come in under 1000 tokens, on the argument that frontier models are RL-trained enough to already know what a coding agent is, so a 10k-token system prompt buys little. Zechner backs that with a run on Terminal-Bench 2.0 — a standard obstacle course of terminal tasks — where Opus 4.5 under pi places competitively against harnesses carrying far more scaffolding.

The minimalism falls out of two commitments Zechner sets out at length. The first is context engineering — exact control over what enters the model’s context, on the premise that mainstream harnesses inject material behind our backs that never surfaces in the UI and degrades the output. The second is observability — being able to inspect every byte of every exchange, with a documented session format we can post-process.

One agent built on pi is the famously bloated OpenClaw, which uses pi as its agent core and adds interface gateways, persistent memory, and the rest.

Affordances:

  • TypeScript native — agents and tools are TypeScript modules with type-checked tool schemas.
  • OS-agnostic — Node CLI, runs on Mac / Linux / WSL.
  • Parallel tool calls — pi can fire several tool calls in a single turn, so any extension that wants parallelism gets it for free.

4.1.1 Subagents and context bloat

pi ships no subagent implementation at all. Its native answer to context bloat is the /tree command: we jump back to an earlier point in the chat history and pi summarizes everything since, collapsing the intervening turns into a précis — many of a subagent’s goals, but explicit.

Zechner objects to classic subagents on two grounds: they are unobservable, and they could instead be constructed explicitly, as separate sessions with shared file context.

No one else is so austere. The popular pi-subagents reinstates the familiar subagent(prompt, ...) tool, the child running on either fresh context (only its instructions) or forked context (a copy of the parent’s window plus instructions). Daniel Nouri’s pi-submarine is a smaller take — fresh/fork context, named agents as markdown files, nested subagents, resumable runs. Both make the subagent’s conversation observable, which addresses most of Zechner’s objection.

4.1.2 YOLO no guardrails

pi runs in full YOLO mode: unrestricted filesystem access, any command executed with our user privileges, no permission prompts, and no screening of bash commands by a cheap model before they run (which is what Claude Code uses Haiku for). The rationale is that anything more is just security theatre; if we want a boundary, run pi inside a container, which maybe we should do in general.

4.2 smolagents

smolagents (Hugging Face, Apache 2.0, Python) is also minimalist, but for Python.

The distinguishing feature is the CodeAgent paradigm: instead of the model emitting JSON tool calls that the harness parses and executes, the model emits Python code snippets that get executed in a sandbox (it resembles the TIR loop in mathematical agents).

Tool calls become function calls:

from smolagents import CodeAgent, WebSearchTool, InferenceClientModel

model = InferenceClientModel()
agent = CodeAgent(tools=[WebSearchTool()], model=model, stream_outputs=True)

agent.run("How many seconds would it take for a leopard at full speed to run through Pont des Arts?")

HF’s benchmark claim is that this paradigm uses ~30% fewer model steps than JSON tool-calling on difficult agentic benchmarks.

Affordances:

  • Model-agnostic — any HF Hub model via InferenceClientModel, OpenAI / Anthropic / Bedrock via the LiteLLM integration, local execution via transformers or ollama.
  • Tool-agnostic — MCP servers, LangChain tools, and HF Spaces all work as tools.
  • Modality-agnostic — text, vision, video, audio inputs.

Arbitrary Python execution from a language model is at least as risky as it sounds. For real isolation, smolagents supports various managed sandboxes — Modal, E2B, Blaxel — plus Docker for self-hosting.

A ToolCallingAgent is also available alongside CodeAgent for models fine-tuned for the classical JSON paradigm.

Observability is unusually good for free: the loop streams a step panel per thought / execution / observation to the console, agent.logs holds the structured trace afterwards, GradioUI(agent).launch() turns the same object into a web chat that visualizes each step live, and OpenTelemetry traces are one instrumentor call away for anything more serious.

4.3 Qwen-Agent

Qwen-Agent (Alibaba, Apache 2.0, Python) is the agent framework the Qwen team built for their own Qwen models; it is the backend of Qwen Chat. Just like smolagents, it is a Python library. It is not to be confused with the Qwen Code CLI, a different codebase.

There is no desktop app and no standalone CLI. The unit of work is a short Python script that instantiates an agent object — Assistant unless we have a reason otherwise — which we then either drive from a terminal chat loop or pass to WebUI(bot).run() for a Gradio web UI off the same object. It installs like any other PyPI package, with the interesting features behind bracketed extras: uv add "qwen-agent[gui,rag,code_interpreter,mcp]". We extend it through register_tool / BaseTool plus MCP — but agentskills.io skills are unsupported.

Three nifty affordances:

4.3.1 Driving non-Qwen models

Qwen-Agent reputedly works best with Qwen models — the default tool-call template (fncall_prompt_type='nous') is tuned for Qwen3 / Qwen3-Coder / QwQ-32B, and it will drive anything else through an OpenAI-compatible endpoint but might be janky. The setting that decides it: by default, Qwen-Agent parses tool calls out of the raw model text itself; to defer instead to the server’s own native tool-call interface — needed for non-Qwen models, or for Qwen served behind vLLM’s built-in parser — we set use_raw_api: True in llm_cfg’s generate_cfg. Qwen models are well supported on Apple Silicon, so for local use the Qwen-on-Qwen path is likely smooth.

4.4 Heavier orchestration

pi, smolagents, and Qwen-Agent are relatively simple structures, mostly a loop plus some tools. There exists a more industrial category of agent orchestration: LangGraph, CrewAI, AutoGen, LlamaIndex, the role-playing multi-agent frameworks. These do a lot. e.g. LangGraph models the agent as a state machine (nodes, conditional edges, cycles, persistence underneath): a run can checkpoint, survive a process restart, and resume from the last node, optionally pausing for a human.

CrewAI and Swarms scaffold some other interesting design patterns, e.g. several named agents passing work between them.

These feel too heavy for anything I currently need, but they might be interesting for specialist use cases.

5 Ready-to-run harnesses

Between building our own (above) and the always-on assistants (below) is a middle category: finished, generic harnesses that we point at any OpenAI- or Anthropic-compatible server and simply use. Coding is an obvious application but not the only one. pi straddles the line — minimal enough to count as a library to build on, complete enough to run as-is. The code-native tools — Aider, Cline — are in the coding notebook; OpenCode and Goose below I’ll admit as general-purpose agents that happen to be good at code.

5.1 OpenCode

OpenCode (anomalyco/opencode, MIT) is a terminal harness supporting 75+ providers, local models included. It wraps LSP, MCP, and a plugin system, and has a massive community. We point it at any endpoint by adding a custom provider with a baseURL in ~/.config/opencode/opencode.json.

curl -fsSL https://opencode.ai/install | bash
# or: npm install -g opencode-ai
# or: brew install anomalyco/tap/opencode

Pro-tip: the canonical repo is anomalyco/opencode; AFAICT opencode-ai/opencode is name-squatting.

Xiaomi’s MiMo Code is a fork that adds long-horizon memory and a self-improvement layer.

5.2 Goose

Goose (source at aaif-goose/goose, Apache-2.0, Rust) is a grown-up member of the bunch, in the sense that it pays its taxes and goes to meetings. By which I mean, it has been adopted by the Linux Foundation. Unlike the Python and Node harnesses above, it is a native desktop app and a CLI and an embeddable API.

It works with many model providers and 70-odd MCP servers, and notably will ride an existing Claude / ChatGPT / Gemini subscription through the Agent Client Protocol rather than via a metered API key. The ACP runs both ways: Goose can also back an ACP-speaking editor like Zed or JetBrains.

Pointing it at a local server means dropping a JSON file into ~/.config/goose/custom_providers/ with a base_url and a list of models. That list is the part to watch: it is ours to maintain, and Goose never calls /v1/models to check it. So a model renamed on the server side — in a vllm-mlx registry, say — keeps working everywhere else and fails only here, with The model X does not exist. Available models: …. The error at least prints the real names, so the repair is mechanical once we know to look.

It bites twice if we keep more than one server profile, as I do — a full-fat registry and a lean one, same port. Goose cannot tell which is loaded, so the provider file either lists the union and 404s on the models the lean profile does not hold, or lists the intersection and hides the rest. The union is the better failure, on the grounds that a self-diagnosing error beats a model that is quietly not offered. Open WebUI has the opposite behaviour and reads /v1/models, which is why its picker just populates.

Skills are supported through the built-in Summon extension, reading them from ~/.agents/skills/ globally and .agents/skills/ per project, with backward-compatible discovery of .claude/skills/ and friends.

It also supports a Goose-specific format, recipes. A recipe packages a prompt, parameters, tools and extensions into a one-click shareable workflow, with subrecipes for fanning work out across subagents. The difference from a skill is binding time again: a recipe configures the session before it starts, whereas a skill is task know-how consulted mid-session if the request happens to match it.

The docs diffuse the pitch across many pages; the Goose Janitor write-up might be a good central depot.

5.3 Open WebUI

Open WebUI (open-webui/open-webui, open-source, self-hostable) is venerable in chat years, dating all the way back to 2023. If I am not mistaken, it grew out of an attempt to provide a chat UX for any by-the-token chat API in the browser. It may still function as such? At some point the UI grew a genuine agentic harness, speaking MCP natively, executing Python code, and offering built-in document RAG. So it straddles two layers — frontend and harness — which is why we can keep the frontend and swap the harness out from under it. The feature set skews toward tool-using conversation rather than the autonomous, long-horizon, agency-first design of the more recent agents here. It seems especially well-fitted to being an agentic mathematics frontend, with the best equation rendering out of all the options here.

It shows the signs of a long and chaotic evolution, comprising messy strata of Node packages, Python scripts, and weird version requirements. I suspect we don’t want to see how this sausage is made.

5.3.1 Setup

Most walkthroughs bundle an extra copy of ollama for shits and giggles, which is not what I want, having already over-engineered too many token serving options. Every environment variable here is load-bearing:

ENABLE_OLLAMA_API=False WEBUI_AUTH=False DATA_DIR=~/.open-webui \
  uvx --python 3.11 open-webui@latest serve --host 127.0.0.1 --port 8111
  • DATA_DIR is where chats and settings persist; without it, uvx can leave them in a cache directory that evaporates.
  • WEBUI_AUTH=False at first launch skips login for a single-user laptop install, and the choice is permanent per datastore once initialized.
  • ENABLE_OLLAMA_API=False stops the app probing for the Ollama we decided not to run.
  • The UI is served by default promiscuously on 0.0.0.0:8080, so --host 127.0.0.1 restricts it to loopback, which is what we want on a laptop we are not deliberately sharing.

The same variables work as -e flags on the containerized version. That one persists to a volume rather than to DATA_DIR, and always listens on 8080 inside itself, whatever the host-side half of -p says. There is also an alpha desktop app wrapping the same web UI in a native window.

Backends are configured in the UI under User Settings → Admin Settings → Connections → OpenAI:

  • OpenRouter — URL https://openrouter.ai/api/v1 plus an API key. OpenRouter exposes thousands of models, which swamps the model picker and makes page loads crawl; add the handful of model IDs we use to the connection’s Model IDs allowlist and switch on Cache Base Model List (or ENABLE_BASE_MODELS_CACHE=True).
  • A local server — vllm-mlx or oMLX at http://localhost:8000/v1, Osaurus at http://localhost:1337/v1; API key blank (or the --api-key we launched the server with, if we did that). From inside the container, localhost is the container itself — the host’s endpoint is http://host.docker.internal:8000/v1. These servers all implement /v1/models, so the model names auto-detect and populate the picker.

Pro-tip: curl http://localhost:8000/v1/models | jq '.data[].id' shows what the picker will see before we touch the UI.

Fun feature: equation rendering works offline with nothing to configure. Open WebUI bundles KaTeX — the library, the mhchem chemistry extension, the stylesheet, and the maths fonts — into its frontend build.

6 Personal AI assistants

A category of agent product distinct from coding assistants: the always-on, multi-channel personal AI that learns about us over time. These look like:

  • A long-running daemon on infrastructure I own (or rent cheaply — a $5 VPS, a home server, a Modal pod that hibernates when idle).
  • Multi-channel deployment — instead of a desktop window, the agent listens on Telegram, Discord, Slack, WhatsApp, Signal, iMessage, email — simultaneously.
  • Persistent memory across sessions, often with auto-generated skills.
  • Sometimes scheduled cron-style automations: daily reports, nightly backups, weekly audits.

Hermes and OpenClaw below are iconic examples. The Anthropic Cowork agent can be configured to do some of these things and is very capable, so I keep it here as a kind of reference off-the-shelf option.

6.1 Claude Desktop (Cowork, Code, Remote)

The Anthropic bundle has several distinct surfaces under one app.

Cowork is the personal-assistant tab — “Claude Code for non-developers.” Describe a multi-step task, grant Claude access to a folder, walk away while it works, with Office and Chrome integrations built in. The polished commercial option, and closed all the way down.

Within the one bundle, three-and-a-half different execution environments:

  • Cowork (local). Runs on my machine. Code execution goes through a local VM Claude manages; computer-use does not — it is molesting my actual screen, using my actual apps. Within days of launch, Cowork was demonstrated to be vulnerable to prompt injection from web pages it visited. Cool.

    • Variant: Remote Control. Phone or browser drives a local Claude Code session; the agent still runs on my machine, but the phone is now a remote.

  • Claude Code (local CLI). Several isolation tiers: sandboxed Bash (Seatbelt on macOS, bubblewrap on Linux), full process sandbox, dev container, custom container, full VM. The default “sandboxed Bash” tier only sandboxes Bash — other built-in tools (Read, Edit, WebFetch) run unsandboxed in the parent process. Several exploits have been published for that fella too.

  • Claude Code on the web (claude.ai/code). Each session runs in a fresh Anthropic-managed VM with the repo cloned through a credential proxy. Git credentials never enter the sandbox — git auth goes through a proxy with scoped tokens (though it’s unclear to me how credentials for other services are managed — access tokens need to meet the code at some point). Network access is limited by default; can be disabled entirely. Still, the strongest isolation tier Anthropic offers natively.

6.2 Hermes Agent

NousResearch/hermes-agent — FOSS, MIT, Python 3.11 + uv. Nous’s own codebase end-to-end. Designed to run on infrastructure I own (a $5 VPS, a home server, my laptop, a Modal pod), and installed by a setup-hermes.sh that does the whole uv-venv-symlink dance for us.

Distinctive weirdness:

  • Model-agnostic — 15+ providers plus arbitrary OpenAI- or Anthropic-compatible endpoints, switchable mid-session.
  • Multi-channel gateway — Telegram, Discord, Slack, WhatsApp, Signal, email, CLI all from one long-running process.
  • Closed learning loop — auto-generated skills from experience, FTS5 session search, and persistent memory via Honcho, a layer that keeps a running model of me assembled from everything I have said across sessions. Its dialectic endpoint is the interesting bit: rather than retrieving documents, the agent asks Honcho a plain-language question about me (“how should I pitch a technical explanation to this person?”) and Honcho answers from the accumulated profile.
  • Serverless deployment — Modal and Daytona backends with hibernate-on-idle so the agent costs cents between sessions.
  • MCP-native — first class, not bolted on.
  • hermes claw migrate — an explicit migration tool from OpenClaw, the giveaway about who they see as the user base they’re courting.

Hermes ships the agent loop separately from the execution sandbox. Six backends, picked via terminal.backend in config:

Backend Where it runs Isolation Setup
local Host as the user None Default — for testing only
docker Single persistent container Linux namespaces, dropped caps Docker installed
ssh Remote box the user owns Network boundary SSH key + host config
modal Modal cloud VM per task Strongest Modal account
daytona Daytona managed workspace Strong; resumable Daytona API key
singularity HPC-style Apptainer container Namespace isolation without $HOME apptainer installed

A subtle but useful detail: remote backends sync touched files back to the host on teardown into ~/.hermes/cache/remote-syncs/<session-id>/. No need to remember to scp artifacts off the cloud sandbox manually.

Hermes borrows OpenClaw’s DM pairing pattern — unknown senders are ignored until the operator explicitly approves them — and adds an encrypted secret-exchange flow for credentials the agent needs at runtime. The user pastes a secret into pi.dev/secret — yes, that pi; the page encrypts in the browser and sends nothing to the server — gets an encrypted blob, pastes it into chat, and the gateway decrypts it locally with an ephemeral private key, storing the cleartext inside the sandbox without the agent itself ever seeing it.

6.3 OpenClaw

openclaw/openclaw — Peter Steinberger’s personal AI assistant project, built on pi. Same shape as Hermes (multi-channel, persistent, owns its own infrastructure) but more cowboy.

The long-running process is a single gateway daemon installed as launchd (macOS) or systemd (Linux) user service. The gateway routes messages from ~25 channels (WhatsApp, Telegram, Slack, Discord, iMessage, Matrix, WeChat, …) to agent sessions, which can be sandboxed with Docker, SSH, or OpenShell.

Default isolation: the main session (interactive use by the owner) runs tools on the host with no sandbox. Non-main sessions (group chats, automation, external users) get sandboxed if the operator opts in (agents.defaults.sandbox.mode: "non-main"). Default deny list for sandboxed sessions covers browser, canvas, nodes, cron, discord, gateway.

Cool tricks:

  • DM pairing. Unknown senders on any channel get a pairing code and the bot ignores them until openclaw pairing approve adds them to a local allowlist.
  • Companion apps. Optional macOS menu-bar app, paired iOS/Android nodes — useful if someone in the household wants the assistant in their pocket.

No cloud-VM backend; for remote execution OpenClaw points at a box the operator already controls via SSH.

7 Feeding documents in

One thing that is patchy for my (scholarly) purposes across the open-source products is ingesting PDFs, Word docs, spreadsheets, etc. There are three architectures on the table, depending on the harness and model.

  1. Native multimodal model. The model itself ingests the document binary (or rendered pages) and processes text plus charts plus layout in one go. Highest fidelity. Only frontier closed-weight models (Claude, GPT-5.x, Gemini) currently do this well.
  2. Frontend- or harness-side text extraction. The harness runs a PDF→text library and drops the (text) result into context. Loses charts and visual layout but is usually fine for prose-heavy documents.
  3. Agent extracts the content using shell tools. The harness invokes a converter, reads the markdown back, and continues. Composable but more setup.

7.1 Harness-side affordances

Which architecture each harness gives us out of the box:

Harness Architecture Notes
Qwen-Agent Assistant harness-side files=[long_pdf] runs DocParser + BM25 hybrid retrieval, no vector DB; .pdf/.docx/.pptx/.txt/.csv/.xlsx/.html, 1M-token tested
Claude Desktop native multimodal Drag-drop; vision models read the document binary, no extraction step
smolagents CodeAgent agent with shell tools No pipeline, but the agent writes Python (pypdf, pdfplumber, pandas) in its loop as needed
Hermes, pi, OpenClaw none — bring our own No reliable built-in extraction; Hermes’s web_extract handles PDF URLs → markdown, local files via a shell converter or skill

For non-trivial PDFs — maths, scans, complex layout — see PDF ingestion.

8 Long running assistants

The always-on assistants — Hermes, OpenClaw, anything with a gateway — are not launched, they are deployed, and that is a different kind of problem. Something has to stay up while we are asleep, it has to be somewhere with an address, and a run that lasts weeks goes wrong in ways a run that lasts an afternoon does not.

8.1 The processes we have to keep alive

Once an agent stops being a thing we launch and starts being a thing that is always up, it is the stack that gets deployed rather than run — and deployment changes which layers we have to keep breathing.

The model server disappears: the tokens come from an endpoint at the far end of an HTTPS call rather than from anything we installed. The frontend splits in two, into a foreground surface the user types into (short-lived, per session) and a channel surface — Telegram, Slack, Discord, email, a phone — that has to be listening even when nobody is typing. The harness grows a gateway daemon in front of it, holding state, routing messages between channels and sessions, and managing sandboxes; Claude Code without web mode has none, while Hermes and OpenClaw revolve around theirs. And the execution sandbox stops being an implementation detail of the harness and becomes a thing with its own address — local, containerized, or a cloud VM.

Those last three are what have to stay alive.

A deployment that stays up for weeks also meets problems an afternoon’s session does not. Context summaries degrade, per-turn errors compound, and whatever the agent worked out evaporates between sessions — three walls, with separate fixes. Sub-agents, compaction, plan mode and permission gates all help, and past a few hundred turns none of them is enough. Which leaves the question of where to put the thing.

8.2 Hosting the gateway

The gateway has to be a live process for the messaging surfaces to work: Telegram, Discord, Slack and friends need either a long-polling connection (the gateway dials out and waits) or a webhook endpoint (the provider POSTs in), and both mean an always-on process. Sub-agents on Modal or Daytona can hibernate; the parent gateway cannot, not if it expects to receive our next “hey, status?” from the phone. When my laptop sleeps, the bot is dead.

Five options for hosting the always-on gateway:

  • A $4–5/mo VPS (Hetzner, DigitalOcean, OVHCloud). The boring correct answer. Public IP, webhook channels Just Work, Hermes installs in a one-liner under systemd. Six months later we will have forgotten the VPS exists, which is the point.
  • A Raspberry Pi or repurposed laptop at home (~$50–100 one-time, ~$1–2/mo electricity). Tailscale Funnel or Cloudflare Tunnel solves the inbound webhook problem, so we do not forward ports through the home router. Privacy wins; ISP and power outages become our problem.
  • Hermes deployed on Modal in webhook mode. It hibernates between messages and cold-starts on inbound, so bursty personal use costs single-digit dollars a month. The first message after a quiet period waits ~2–5s for that cold start.
  • Existing hardware — a NAS, a Mac mini behind the TV, an Intel NUC. A few watts, plus whatever our tolerance is for “this is the box that holds the assistant; please don’t unplug it”.
  • Not a 24/7 RunPod instance. GPU rental at $0.20–$4/hour is fine if we are also hosting a local model on that box. For a process that only proxies API calls it is waste: spinning a pod up for inference and tearing it down again is a model-server play, not a gateway-hosting one.

For the “I want both the model AND the gateway on the same box” case, the question shifts from “where do I host the gateway” to “where do I host the model” — see running LLMs locally on a Mac and the Australian sovereign-LLM project for that side.

9 What can go wrong

This one applies to every agent above, not only the long-running ones, which is why it sits on its own rather than under any of the tiers.

The vulnerability surface of an agent is unusually broad — it spans the model, the tools it runs, the inputs it reads, and the vendor behind it. If classic computer security is about keeping intruders out of the house, then agent security is more like managing a toddler who might let intruders in, burn it down, or open the door and run into traffic. Four kinds of failure that agents layer on top of all the nonsense of classic computer security, and surely more exist in the wild:

  • The model makes mistakes. It misreads the task and does something destructive in good faith — deletes the wrong directory, force-pushes over someone’s work, pastes a secret into a public channel, emails the draft to the client instead of to us.
  • The code it runs can be very wrong. An agent that installs and runs software in good faith inherits that software’s bugs and side effects. A correct decision to call a tool is still only as safe as the tool itself.
  • Plain English input can be hostile. Web pages, emails, Slack messages, and documents can carry prompt injection that redirects the agent’s tools toward someone else’s goals. This is the famed prompt injection attack.
  • The provider is some weird corporation domiciled elsewhere. Unless we are running the model ourselves, every prompt, file, and pasted secret the agent touches also goes to the token vendor. That makes the vendor a high-value target — a breach there can expose many customers’ data at once. Moreover, they are a party whose interests need not match ours: logs can be retained, scanned for training data, mined by an insider, or handed over under subpoena. Not divulging all our secrets to the Man is not a solved problem; self-hosting relocates that trust under our own roof rather than removing it.

tl;dr An agent wiring together our private data, code execution, and network access can do a lot of damage regardless of why it misbehaves. The job is less to build an impermeable wall than to reduce the rate and blast radius of the agent’s fuckups.

Mitigations are often layered:

  • At the agent boundary. Bounded autonomy: permission gates on destructive tools, plan-mode approval before execution, default-deny pairing for unknown senders, read-only or narrowly-scoped tools, credential proxies so the agent never sees the raw secret, a human in the loop on anything irreversible. These catch good-faith mistakes and the clumsier injections before they fire.
  • At the isolation boundary. Run the execution environment in a sandbox or a container, restrict network egress, keep the environment reclaimable. This on average reduces the damage of a bad action regardless of which failure mode produced it.

None of this is foolproof. A determined agent can still find ways through its own gates, at the behest of an adversarial prompt or of a surfeit of helpful enthusiasm. And real-world sandboxes are routinely escapable and “not especially secure”: isolation is an inconvenience for models but not insurmountable. Further, there is, as often in security, a trade-off between power and convenience. By definition we are using agents because we want them to do powerful things on our behalf, so the more we lock them down, the less useful they become. The defaults tend to be weak, sliding toward “security theatre”: if we need to click the “approve” dialogue box 200 times, how well have we assessed the risks each time? Zechner argues that once an agent can write and run code the lethal trifecta — private data, code execution, network reach — is already in play. So pi ships no guardrails at all and leaves that to you. He would rather configure good sandboxes and monitor through logs and audit trails than put guardrails on the agent itself.

Other things to think about:

  • Default-deny the untrusted path. Ignore external senders until paired (OpenClaw and Hermes both do this for their inputs), and run web-touching sessions in a sandbox that can be thrown away. Claude Cowork’s computer-use mode, for example, used no sandbox at all, and was demonstrated vulnerable within days of launch.
  • Credentials are vulnerable. Attackers and confused agents would be delighted to get their hands on our secrets: API keys, SSH keys, OAuth tokens, the .env file, the cloud credentials. These translate into real resources, the ability to impersonate us and outlast the current session. Once an agent has filesystem and network access, every secret in ~/.ssh, ~/.aws, or a .env is one cat and one POST away, which is why “grant the agent our home folder and hope” is such a weak default. Stronger patterns keep the raw secret out of the agent’s reach in a secret manager, dispensing short-lived, scoped tokens. Claude Code on the web routes git auth through such a proxy, so credentials never enter the sandbox; Hermes’s encrypted secret exchange decrypts on the gateway with an ephemeral key, so the cleartext lands inside the sandbox but the agent and the model never see it.
  • Watch the auto-skill-generation loop. When a harness writes its own skills from past tasks (Hermes; MiMo Code’s Dream/Distill passes), one bad prompt can mint a persistent capability the agent reuses later, unprompted. Audit ~/.hermes/skills/, .mimocode/, or the equivalent periodically.

I’m sure there are more failure modes than these; the point is to be thoughtful about the risks and mitigations, and to accept that some risk is inherent in powerful agents until alignment and capabilities both jointly achieve perfection.

10 Picking one

Building on a library, I would prototype on pi or smolagents and keep it simple unless I had a reason to escalate:

  • TypeScript, want maximal control and observability → pi.
  • Python-native computational work — NumPy/pandas in the loop, cross-vendor flexibility → smolagents, whose CodeAgent paradigm fits when the natural step is “run this snippet” rather than “call this named tool”.
  • Qwen-family models locally, batteries included — long-doc RAG, sandboxed code interpreter, Gradio UI → Qwen-Agent.
  • The job needs deterministic branching → an agent that emits a script for a sandbox to run.
  • Checkpoint-and-resume across process restarts → escalate to LangGraph.

Picking a finished product instead — where OpenClaw is the reference build-on-pi implementation, Hermes the from-scratch Python alternative, and Cowork the closed commercial one:

  • Claude doing things with my files and apps while I work on something else → Cowork, accepting the security caveats and not letting it browse untrusted websites.
  • A coding task shipped to a hosted sandbox where my credentials never leave my laptop → Claude Code on the web.
  • An assistant that messages me on WhatsApp / iMessage / Telegram off my own infrastructure → OpenClaw if channel breadth matters most, Hermes if isolation choice does.
  • The same agent reachable from my phone, on a Modal pod that hibernates between sessions → Hermes with the modal backend.
  • A small group sharing one assistant across Slack and Discord, onboarded by DM pairing → OpenClaw, sandboxing non-main sessions.
  • The assistant backed by my own local LLM → Hermes or OpenClaw, for which a local-server URL is just another provider; Cowork cannot, being locked to Anthropic’s hosted Claude.

Whichever it is, sandbox it before pointing it at anything that matters.

11 Incoming

12 References

Bengio, Cohen, Fornasiere, et al. 2025. “Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?”
Carey, Langlois, Merwijk, et al. 2025. “Incentives for Responsiveness, Instrumental Control and Impact.”
Chen, Dong, Shu, et al. 2023. “AutoAgents: A Framework for Automatic Agent Generation.”
Crutchfield, and Jurgens. 2025. “Agentic Information Theory: Ergodicity and Intrinsic Semantics of Information Processes.”
Everitt, Garbacea, Bellot, et al. 2025. “Evaluating the Goal-Directedness of Large Language Models.”
Goemans, Altman, Dreksler, et al. 2026. “Comprehensive AI Governance Requires Addressing Non-Model Gains.”
Guo, Chen, Wang, et al. 2024. “Large Language Model Based Multi-Agents: A Survey of Progress and Challenges.”
Hammond, Chan, Clifton, et al. 2025. “Multi-Agent Risks from Advanced AI.”
Hyland, Gavenčiak, Costa, et al. 2024. “Free-Energy Equilibria: Toward a Theory of Interactions Between Boundedly-Rational Agents.” In.
Kalai, and Lehrer. 1993. “Rational Learning Leads to Nash Equilibrium.” Econometrica.
Li, Al Kader Hammoud, Itani, et al. 2023. “CAMEL: Communicative Agents for ‘Mind’ Exploration of Large Language Model Society.” In Proceedings of the 37th International Conference on Neural Information Processing Systems. NIPS ’23.
Qu, Dai, Wei, et al. 2025. “Tool Learning with Large Language Models: A Survey.” Front. Comput. Sci.
Schmidgall, Su, Wang, et al. 2025. “Agent Laboratory: Using LLM Agents as Research Assistants.”
Su, Diao, Lu, et al. 2025. “ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration.”
Walters, Kaufmann, Sefas, et al. 2025. “Free Energy Risk Metrics for Systemically Safe AI: Gatekeeping Multi-Agent Study.”
Wu, Bansal, Zhang, et al. 2023. “AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation.”
Yang, Gong, Huang, et al. 2026. “SkillOpt: Executive Strategy for Self-Evolving Agent Skills.”
Zhang, Jiang, Li, et al. 2026. “DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints.”