2026-02-10: Embedded agency, internal models, category theory, Bayes in the wild, peer review
2026-02-10 — 2026-02-10
Over the past five days Dan’s lobbed in three new posts and given seven older ones a good tighten-up — no mucking about. The main thread is how you describe a thinking system without kidding yourself: “embedded agency” just means the thing making choices is stuck inside the world it’s acting in, not sitting outside like a puppet-master. He’s also gone into “category theory” — that’s a very high-level kind of maths for tracking how bits connect and map to each other, like a set of tidy wiring diagrams for ideas. On the clean-up side he’s been back on Bayes “in an open world” (doing odds when you don’t even know all the possible options yet), had another swing at peer review, and mixed in some real-life bits like sleep, London, travel hacks, and that research residency he’s at.
1 Newly published
1.1 Internal model principles
Ever tried steering a ute on corrugations without looking ahead? That’s the vibe here: the internal model principle says if you want to keep something steady, you need a little model of it inside the thing doing the steering. Dan’s been jotting notes on that, plus the “good regulator” idea — the best regulator is, in a real sense, one that carries a model of what it’s regulating. The new rabbit hole is the ε‑transducer: it takes ε‑machines (fancy pattern‑spotters) and turns them into agents that act, learn, and keep revising their bets the Bayes way as new clues roll in. Worth a squiz if you care about AI or control gear that doesn’t just predict the world, but can poke it and still stay on the rails.
1.2 Categorical systems theory
It’s like turning messy systems into neat little boxes with plugs, so you can hook them together without losing your mind — that’s “categorical systems theory”, in plain speak. Dan’s doesn’t know any category theory but he’s persuaded we might use that category-theory gear and using it to talk about how a system deals with its world through those plug-points, and how you can snap small bits into bigger ones without rewriting the whole bloody thing. The juicy bit is the string diagrams: instead of pages of symbols, you draw the wiring, then you can actually do the sums over Bayesian networks by pushing stuff along the strings. He even says what kicked it off — seeing Martin Biehl use these diagrams to make hard probabilistic models feel less like wrestling an octopus. Worth a look if you like models that stay modular when they get big, instead of turning into spaghetti.
1.3 Embedded agency
Ever seen a dog chase her own tail till she stacks it? That’s the sort of weird Dan’s poking at with ‘embedded agency’ — minds stuck inside the world, trying to reason about themselves and each other. He went in expecting down-to-earth stuff, but found it’s mostly about what happens when you pretend the agent has endless compute, AIXI-style, and then let it build whole pretend worlds inside its head. Once you do that, Gödel and Löb pop up everywhere and the logic starts biting its own arse, especially when agents model themselves or other agents doing the same trick. Dan reckons it’s fun and a good place to learn Löb, but he’s not chasing it right now because real AI lives and dies on tight compute budgets, not fairyland infinity.
2 Updated
2.1 London
London’s got a city inside the city, and it’s been getting away with it for about a thousand years — the Square Mile runs on its own strange rules while the rest of London muddles along around it. Dan’s gone down the rabbit hole on how the Poms deal with a growing mega-city: they don’t tidy it up, they just slap new layers on and keep the old little fiefdoms, like patching a leaky tank with more hose clamps. The bit that’ll make you blink is the City flat-out refused to “grow up” and take in the suburbs back in the 1600s, and you can still see the mess today. Then he lightens the mood with properly London-only fun — summoning museum treasure from a warehouse by robot, and poking around Cecil Court for old prints like a bloke who’s lost his car keys in 1780.
2.2 Bayes inference in an open world
Here’s the sneaky bit: if your model’s wrong (and it is), plain Bayes can get more and more sure of the wrong answer as you feed it data. Dan’s banging on about the “M-open” world, which just means the real truth isn’t even in your little set of models, so stop pretending you’ll ever find it. So instead of picking one ‘winner’, you can stack a few models and let out-of-sample checks (LOO with that PSIS shortcut) decide the mix that predicts best. And if Bayes is getting cocky, you can ‘temper’ the likelihood with an η knob — basically turning down how loud the data yells — so it doesn’t lock onto a dud story.
2.3 Improving peer review
Peer review’s meant to be a fair go, but half the time it turns into a messy blue over who said what in the rebuttal. Dan’s big idea is chucking in an AI “umpire” just for the dispute bit — a few fixed back-and-forth turns, it points to the exact lines, gives a confidence call, and the whole chat gets saved so you can check it later. He also leans on a bit of randomness in who reviews what, plus rules that stop easy mate-swaps, so collusion rings and fake reviewer games are harder to pull off. And on the bias stuff, he likes a middle road: keep names hidden while scores go in, then lift the curtain after, so you dodge prestige carry-on without losing the proper conflict checks.
2.4 Sleep
Here’s a weird one: Dan reckons jet lag’s not just “sleep it off” — this Timeshifter app gives you bossy orders about sun, shade, coffee and naps, and it can about drag your body clock into line. After seeing it work, he’s got a lot more respect for light timing, and he’s cranky it’s now harder (and pricier) to get melatonin in Australia.
2.5 Intentional language is ok
Funny thing: we’re rubbish at cold logic, but give it a “who’s breaking the rules?” story and our brains wake up — same trick can help with AIs. Dan reckons saying a model “wants” or “thinks” is a quick way to guess its next move, without pretending it’s got feelings or deserves any special care. Handy tool, just don’t start seeing a person in every bit of code.
2.6 PIBBS x ILIAD Research residency January-February 2026
More on this “singular learning theory” caper — it’s the maths for when your model’s a knotty beast, not a neat straight line, and the usual tidy rules stop working. What’s clever is how it ties together how well a learner behaves out in the wild with the “shortest story” you can tell about the data, plus those old Jeffreys-prior tricks so you’re not kidding yourself just because you picked funny units. They even rope in coding rules like Kraft–McMillan — same vibe as making sure your zips and checksums actually add up before you trust the file. And it loops back to agency and internal models: if something’s meant to act on the world, what’s the little working map it needs inside its head to stay on track?
2.7 Travel hacks
Here’s the sneaky truth Dan’s finally said out loud: half the pain of travel isn’t the flying, it’s the dumb little frictions you forget to plan for. He’s big on dodging SIM card stuffing about — pay a bit more for a travel eSIM if it saves you wasting an hour hunting a shop and mucking round with rego, and he’s even picked one that’s rough round the edges but actually answers you. Then he gets properly practical about long-haul misery: a cheap gel cushion for your bony arse, and instead of those useless neck pillows, you strap your head up like a packed load so you don’t wake up twisted and drooling. He’s also chucked in a straight answer on folding bikes and a quick nudge for jet lag tools, so you’re not just “toughing it out” like a martyr.