Homunculi all the way down

2.1 Interactive POMDPs

Gmytrasiewicz and Doshi’s interactive partially observable Markov decision processes (I-POMDPs) (Gmytrasiewicz and Doshi 2005) provide one formulation of recursive belief. The state space of any one agent is augmented with models of the other agents, which themselves include models of this agent, and so on. A finitely-nested I-POMDP truncates the recursion at level \(k\) — agents at level 0 treat others as noise; level 1 models level-0s; level 2 models level-1s; and so on. This is an operationalization of “reduced rank”: the recursion is cut off, and the depth is tunable.

Suggestively related work in game theory:

• Level-k / cognitive hierarchy models in behavioural game theory (Camerer, Ho, and Chong 2004), where players assume others are reasoning at a lower level than themselves. Humans reportedly cluster at \(k \in \{0,1,2\}\).
• Quantal response equilibrium (McKelvey and Palfrey 1995), where bounded rationality is modelled by stochastic best-response rather than a deeper recursion.
• Epistemic game theory (Dekel and Siniscalchi 2015), which formalizes common knowledge, common belief, and the belief hierarchies above.

2.2 Bayesian Theory of Mind

Baker, Saxe, and Tenenbaum (C. L. Baker et al. 2017; C. Baker, Saxe, and Tenenbaum 2011) formalize human social cognition as inverse planning: observers invert a generative model of rational action to infer the latent goals and beliefs of others. The other-model here is the generative model — typically a small MDP or POMDP parameterized by a utility and belief — and inference is Bayesian. This gives us a concrete posterior over other minds’ rationality and actions that one can compute with and prove things about.

2.3 Machine Theory of Mind

Rabinowitz et al.’s ToMnet (Rabinowitz et al. 2018) is a deep-learning analogue: a meta-learning agent that, from a few observations of a target agent, infers an embedding which predicts the target’s future behaviour. The embedding is the reduced-rank other-model. ToMnet variants have been extended to false-belief tasks and inverse-RL settings (Oguntola et al. 2023).

Oguntola and colleagues push ToMnet toward interpretability with Concept Bottleneck Models (Oguntola, Hughes, and Sycara 2021): the network is forced to predict named mental-state concepts (the opponent believes the door is locked, wants the key) before producing an action, so the recursive belief-state is in principle inspectable by a human overseer. The catch, as ever, is concept leakage — the net routes information around the bottleneck through residual pathways and re-hides the mental states the bottleneck was meant to expose (Margeloiu et al. 2021). The same pathology is plausibly what any \(\Phi\)-probe on a residual stream will face (below).

2.4 Opponent-modelling in multi-agent RL

This also has a game-theoretic shape if we squint at it, but with learning theory sprinkled on top.

• LOLA (Learning with Opponent-Learning Awareness) (Foerster et al. 2018) computes gradients through a model of the opponent’s learning dynamics. This is a differentiable other-model.
• COLA, POLA, M-FOS and successors refine LOLA with higher-order or policy-level models (Willi et al. 2022; Zhao et al. 2022; Lu et al. 2022).
• Opponent shaping more generally treats the other agent as a learnable dynamical system, which is a particular operationalization of “the other is a reduced-rank version of me”.
• Self Other-Modelling (SOM) (Raileanu et al. 2018) takes the last bullet literally: the agent uses its own policy to predict the other agent’s actions, and online-updates a belief over the other’s hidden goal. A single generative model is reconfigured between egocentric and allocentric readings of the same observation. This is computational simulation theory — model the other by running our own machinery with the other’s sensors wired in — and it induces cooperation in imperfect-information resource-gathering without reward shaping or explicit communication. In the budget frame below it is the corner case where the self-model is the other-model and pays for both with the same bits.

2.5 LLMs and emergent theory of mind

The question of whether transformer language models contain an implicit theory of mind has generated a cottage industry (Kosinski 2023; Ullman 2023; Sclar et al. 2023; Gandhi et al. 2023). The answer seems to be that they carry shallow heuristics that look like ToM on canonical tasks and break on adversarial ones. Whatever they do have is, almost by construction, a reduced-rank model: compressed into attention patterns and residual-stream features.

2.6 Mechanized causal graphs

Causal games (Hammond et al. 2023) and mechanized causal Bayesian networks (MacDermott, Everitt, and Belardinelli 2023) supply the graph-theoretic vocabulary for the formalisms above. Each object variable \(V\) acquires a mechanism parent \(\tilde{V}\): if \(V\) is an agent’s decision \(D\), then \(\tilde{D}\) is its decision rule. The strategic structure — who observes what, who chooses what — lives at the mechanism layer, and reasoning about an agent becomes reasoning over a graph of mechanism variables connected by edges of strategic relevance.

Two things matter for us.

One, what Bob carries of Alice has to be a model of Alice’s mechanism, not of Alice’s output. A pure action-predictor short-circuits the recursion: any Alice who outsmarts the predictor falsifies it, and the graph picks up a cycle (MacDermott, Everitt, and Belardinelli 2023). This is a graphical criterion for what counts as a model-of-another-agent in the first place, and it rhymes with the observer-relative construction in Virgo et al. (2025) at internal models. Different formalisms, same shape of constraint — what Bob represents of Alice is a map to mechanism-space, not to action-space.

Two, one rung of mechanism-level modelling is native to the framework, but deeper recursion is not — there is no \(\tilde{\tilde{D}}\). Hammond et al. recover depth by unrolling a mechanized MAID into an extensive-form game, with the familiar exponential blow-up in tree size. In the budget frame below that exponential is the cost of depth in this particular vocabulary — nodes rather than, say, bits.

3.1 World models containing self

A direct ML instantiation is Ha & Schmidhuber’s World Models (Ha and Schmidhuber 2018), where a recurrent latent model predicts both environment dynamics and the consequences of the agent’s own actions. The agent’s policy is trained inside this compressed dreamscape. The self here is a reduced-rank conditional — “what would my controller do, given this latent” — rather than an introspectable entity, but the “compression” part sounds well-posed.

The lineage continues through Dreamer (Hafner et al. 2024), MuZero (Schrittwieser et al. 2020), and the larger world-model-RL programme.

See also world models.

3.2 Active inference and the self as generative model

Active inference (Friston et al. 2017; Parr, Pezzulo, and Friston 2022) treats the agent as a generative model of its own sensorium, including its own actions. Free-energy minimization forces the self-model to be as compressed as is consistent with prediction — a direct rate-distortion pressure. The self here is a probabilistic model with the agent’s own observation-action trajectory as a latent.

3.3 Self-modelling robots

A concrete line: Bongard, Zykov, and Lipson’s Resilient Machines Through Continuous Self-Modeling (Bongard, Zykov, and Lipson 2006) — a quadruped robot that learns a forward model of its own body, then uses it to plan locomotion; when a limb is damaged, the model updates, and the robot recovers. The self-model is an explicit, parameterized, low-rank dynamical system. See the follow-up (Kwiatkowski and Lipson 2019) for differentiable variants.

3.4 Attention Schema Theory

Graziano (Graziano 2013; Graziano et al. 2019) argues that consciousness is the brain’s (incomplete, schematic) model of its own attention. This is explicitly a reduced-rank model: the schema is coarser than the machinery it represents, because representing attention in full would require as much machinery as attention itself. Kaplan, Dolan, and colleagues have attempted to operationalize this in neural-network models (Wilterson and Graziano 2021).

3.5 Schmidhuber and reflective learners

Schmidhuber’s early work on self-referential neural networks (Schmidhuber 1993) and later Gödel machines (Schmidhuber 2003) formalizes learners that inspect and modify their own code, subject to provability constraints. The Gödel-machine construction is where the proof-theoretic aspect of self-modelling seems to cause grief: self-modification is gated by a proof that the modification improves expected utility.

3.6 Predictive coding and hierarchical self

Hierarchical predictive coding architectures (Rao and Ballard 1999; Clark 2013) include top-down predictions that span the whole sensory hierarchy, including proprioceptive and interoceptive signals — i.e., representations of the organism. See predictive coding, again. And harder.

3.7 Reflective LLM agents

A lighter ML lineage treats self-reflection as a control loop over the decoder rather than as a learnt world-model. Reflexion (Shinn et al. 2023) keeps a log of past behaviour, self-critiques, and revised plans in the context window, so the agent adjusts across episodes without any gradient update — verbal reinforcement learning rather than the usual kind. Language Agent Tree Search (Zhou et al. 2024) extends this to a Monte-Carlo tree search over action paths, with an LLM-powered value function and self-reflection pruning unpromising branches. The self-model here is an in-context transcript and the reflection is a prompt — coarse compared with active inference or Dreamer, and with no mechanism for the self-model to outlive the context. I include them as an existence proof for a reduced-rank self-model whose entire cost lives at inference time rather than in training, which is a distinct point on the budget frontier from everything above.

8.1 Depth as a profile

The “depth” axis above reads as a single number \(k\) in level-\(k\) and I-POMDP treatments, but that collapses some structure worth pulling apart. Bob regulates a scene that contains Alice; there is a natural partial ordering on what kind of thing Alice can be inside Bob’s world-model.

0. Alice as noise — a draw from a distribution over behaviours. No agent-shaped variables in Bob’s ontology.
1. Alice as a stateful process — memory, non-Markovian dynamics, possibly a “type” Bob is inferring. Not intentional.
2. Alice as a belief-carrier whose model of Bob is type-0. She has goals, she is optimizing against a world-model, but that world-model contains Bob only as noise.
3. Alice as a belief-carrier whose Bob-model is type-1. She conditions on Bob’s type but does not treat him as intentional.
4. Alice as a belief-carrier whose Bob-model is itself belief-carrying. Recursion; the same classification applies at the next rung.

Scalar \(k\) is the uniform case: every rung up to depth \(k\) is type-4, and the chain terminates in type-0 at the bottom. What the scalar flattens is the profile — a sequence of (ontology type, rank) pairs along the chain, which in general need not be uniform. In particular it throws away asymmetric drop-off: “Alice thinks I am nearly noise” (type-2 terminal) and “Alice thinks I am stateful” (type-3 terminal) are both depth-2 in scalar terms, but play differently. In the first Bob has slack to act unpredictably; in the second Alice is already conditioning on his type, and only type-misrepresentation is exploitable. Budget-constrained agents probably allocate non-uniformly — most of their rank on the first rung or two, terminating early into type 0 or 1. The human plateau at \(k \in \{0,1,2\}\) is at least as consistent with this as with “humans can’t recurse further”, and the shallow-but-lopsided flavour of LLM-ToM failure (Ullman 2023) looks similar.

Two scalars suggest themselves in place of \(k\) and they answer different questions: chain length (the traditional quantity), and total rank spent on reflection, \(\sum_i R_i\) along the chain. The budget frame of this post is more naturally the second; the level-\(k\) literature has mostly used the first.³

Both scalars, and the profile itself, assume the chain is a chain. A self-model is the case where the chain closes instead of terminating, and the profile picks up a cycle — which scalar \(k\) cannot write down at all. The strange-loop section above argues the cycle is not a degenerate case; it is where the “self” part of the programme lives.

Humans, LLMs, and the agents we would actually like to build do not sit at any corner. They are interior points, and the interior is where unwritten papers live.

9.1 Finding \(\Phi\) in a transformer

The synthesis above says a bounded social agent carries a structured latent — call it \(\Phi\) — partitioned into world, self, each other, and reflective bookkeeping. The ToM-in-LLMs literature (Kosinski 2023; Ullman 2023) presently seems to treat these pieces as present-or-absent behavioural properties. If a frontier model has acquired any social competence under finite pretraining compute, there should be detectable structure in \(\Phi\) tracking these pieces — not necessarily a neat partition, because superposition lets a learner share parameters across sparse features, but some signature visible to the right probe.

That is a mechanistic-interpretability problem. Run sparse-autoencoder or dictionary-learning analysis on the residual stream during ToM-style tasks, and look for:

• a low-rank subspace whose ablation selectively breaks co-agent prediction without breaking world-prediction;
• a disjoint self-model subspace whose ablation breaks in-character behaviour but not factual recall;
• approximately no reflective-bookkeeping subspace in base models, and a small one in post-RLHF models — which are rewarded for consistency with a remembered “I”, and so should be under pressure to allocate bits to reflective bookkeeping that a base model is not.

It is also the careenium question (Hofstadter 2008) with an effect size attached — Hofstadter asks whether the causal story of a system is better told over coarse symbols (his simmballs, groan) than over the micro-dynamics implementing them; ablation asks the residual stream the same thing.

If the structure is not there, one of three things is wrong: the framework, our assumption that representational storage is actually binding for this class of model, or our belief that the pretraining signal actually rewards social prediction rather than a confound. Each possibility is informative. If the structure is there, we have an interpretability handle on social cognition specifically, rather than features in general.

9.2 A value-of-reflectivity calculation

The MIRI tradition treats reflectivity as a correctness property: a reasoner that cannot consistently model its own reasoning has a bug, and the whole agent foundations edifice is a search for bug-free constructions. The budget view recasts it as an economic question: when is it worth spending bits on reflective machinery, and when is it better to skip it?

An idea in this space is that reflectivity pays in proportion to the reactivity of the environment — the degree to which other agents, or future versions of ourselves, condition on our internal commitments rather than just our past actions. Non-reactive one-shot stateless POMDPs: reflectivity is dispensable. Program-equilibrium and modal-combat settings (Bárász et al. 2014; Critch 2019): reflectivity must be high, because opponents are literally reading our source. Open-ended self-modification (Gödel-machine style): Löbian obstacles arise exactly because reflectivity has to be preserved while the substrate is mutating underneath it.

A testable version of this, without the formalism: build toy environments in each regime — a non-reactive bandit, a modal-combat tournament, a self-modification sandbox — and train the same architecture across all three. The conjecture predicts that reflective structure (by whatever detector the \(\Phi\) angle supplies) should emerge spontaneously in the second and third, and not the first. That is a scaling claim about spontaneous situational awareness under training pressure, and it falsifies in either direction.

Premakumar et al. (Premakumar et al. 2024) train networks with an auxiliary task of predicting their own internal activations and observe narrower weight distributions and reduced effective complexity — a “self-prediction-as-regularizer” effect. As a bonus the regularized networks are easier for other networks to model. If the effect replicates beyond their setting then reflectivity is paying for itself partly in rank elsewhere, which is the shape of trade-off the budget frame predicts. It also gives an interpretability-friendly story about alignment: agents incentivised to self-predict become more predictable to overseers, for free. The LessWrong writeup makes the alignment spin explicit.

9.3 Collectives are instrumentable

The same budget calculus should scale past individual minds. A firm, team, or agency is a social agent carrying compressed models of itself and its counterparties — self-model embodied in policy and narrative, other-models in competitor dossiers and customer segments, reflectivity in governance and audit. This complements the transformer angle above: organizations are, if anything, easier to instrument than LLMs — meeting notes, decision logs, communication graphs, and promotion criteria are legible in a way residual-stream activations are not. Mapping those proxies to rank, depth, and reflectivity is a separate project, but the framework commits to specific organizational over-allocations (governance theatre, regulatory capture, siloed product teams, academic insularity) being allocation failures of the same kind as the cognitive ones, which gives two independent scales at which to play with it.