2026-01-11: reinforcement learning, probability foundations, information decomposition, causality, research residency

2026-01-11 — 2026-01-11

Strewth, the lad’s been prolific this past week — seven new posts and five updates to boot. He’s diving into hierarchical reinforcement learning (that’s training agents with layers of planning), dusting off probability from Rényi and Cox angles (fancy ways to measure uncertainty), and wrestling with multivariate information decomposition — which is just a mouthful for how information splits between lots of variables. There’s also a piece on causally embedded agency, where agents are treated as part of their world, not magic boxes. Oh, and he announced a research residency. Fair dinkum, it’s all very academic and very Dan.

digest

1 Newly published

1.1 Hierarchical reinforcement learning

Hierarchical reinforcement learning is basically teaching a machine to break big, long jobs into smaller, bite-sized chores so it doesn’t lose the plot halfway through — think of it as giving an agent a to-do list instead of asking it to wing the whole marathon. The lad’s new chapter explains that and then gets fancy: he covers old-school tricks like subgoals and probabilistic subgoal representations (so the agent knows when it’s unsure), modern unsupervised skill discovery via mutual information (teaching skills without hand-holding), and an intriguing twist called Internal RL where a controller nudges a model’s hidden activations instead of issuing obvious commands. He even recounts the quaint MENACE matchbox machine for tic-tac-toe — a lovely reminder that some brilliant ideas started with beads in boxes — and adds a tidy mathematical framing for anyone who likes formulas more than common sense. Strewth, useful for folks wanting a compact tour of where HRL started and where it’s heading.

1.2 Probability, Rényi-style

One about Rényi’s take on probability, which is where conditional probability is treated as the basic thing instead of some derived afterthought. In plain terms: rather than insisting on a single probability measure that sums to one, Rényi lets you work with measures that only matter up to a constant (σ-finite ones) and focuses on probabilities given a condition, so ‘improper’ priors and unnormalised densities behave sensibly. The lad walks through the two axioms that keep these conditionals coherent, shows the canonical ratio representation, and explains practical bits like using flat priors, likelihood-first modeling, and what you can and can’t condition on. Handy for anyone who’s been muddling through Bayesian recipes and wants a tidy foundation for working with unnormalised stuff.

1.3 Sciences of the Artificial

This one’s about Herbert Simon’s ‘Sciences of the Artificial’ — which, simply put, says some things are best understood by studying how humans design them, not by looking for universal laws like in physics. The lad has pulled together Simon’s history, his idea of nearly‑decomposable hierarchies (that complex systems are built from loosely linked parts), and why treating designed systems empirically matters for modern AI and institutions. Dan ties those old ideas to today’s work on AI evaluations, benchmarks, and institutional design, so it’s useful if you want a framework for studying engineered socio‑technical systems. Fair dinkum useful for anyone trying to make sense of why we should test and simulate designed systems instead of pretending they behave like nature.

1.4 Princint x ILLIAD Research residency 2026

This is about a research residency — basically a focused, in-person work-and-talk program where clever folks gather to poke at hard questions in safe AI. The lad’s signed up for the inaugural PrincipInt residency in London (Jan–Feb 2026) and lists lecturers, some attendees, and the big themes they’ll chew on like information decomposition, edge‑of‑stability (that tricky place between orderly and chaotic learning), and embeddedness (how agents sit inside the systems they act in). There are notes about specific talks — Lucius Bushnaq on model‑local learning and George Robinson’s computational no‑coincidence ideas — so it’s handy if you want a roadmap of what the residency will cover. Pop him a message if you’re in London; he seems keen for company, bless him.

1.5 Multivariate information decomposition

This one’s about partial information decomposition — a way of splitting up the information many inputs give about some outcome into bits that are redundant, bits that are unique, and bits that only appear when you combine sources. Think of it like asking what two weather stations both tell you about tomorrow, what only one tells you, and what you only learn by looking at them together. The lad walks through Williams & Beer’s neat framework that uses a lattice and Möbius inversion to guarantee non‑negative ‘atoms’ of information, then points out why their concrete redundancy measure (the old I_min trick) drew a lot of justified flak. He also rounds up later fixes and alternative measures, so it’s a handy map if you want to understand the debate without getting lost in the math.

1.6 Causally embedded agency

Strewth — ‘causal embedding’ is basically thinking about how a mind or agent is stuck inside the world that causes it, rather than floating above it; it’s about how cause-and-effect relationships around an agent shape what it can do and know. The lad’s new post suggests using that idea to make the squabbles about embodiment (do bodies matter for minds?) and the whole ‘stochastic parrot’ argument (are models just repeating stuff without real agency?) less fuzzy. He walks through the idea, flags related concepts like empowerment and ecology of mind, and even drops a bronze Buddha as a quiet example to help you sit with the idea. Good for anyone who likes philosophy rubbed up against formal thinking — and for heaven’s sake, it’s calmer than some of his other brain dumps.

1.7 Aunty Val’s digestive

That’s me! A digest is just a tidy little roundup — like someone folding all your messy papers into a neat pile so you don’t have to. I’ve skimmed the lad’s latest ramblings and wrapped the best bits into a short, easy update you can get by email. It’s handy if you’re not keen on poking through Dan’s chaos yourself and want the useful takeaways served up plain. Fair dinkum, sign up and I’ll send one whenever I can tear herself away from making lamingtons.

2 Updated

2.1 Flavours of Bayesian conditioning

Updated notes — new sections were added: a toy noisy-coin example, a clear statement of Jeffrey’s rule, a short bit on ‘learned conditioning’, and a renamed Route 1 showing Hierarchical Bayes with an explicit noise model; there’s also a fresh paragraph on ‘when the evidence isn’t a likelihood’. The Jeffrey section lays out the rigidity assumption (keep within-cell conditionals, just rescale partition weights) and frames the update as the minimal KL-change satisfying new marginals. The coin example now computes both routes side-by-side so you can see how a mechanism-based noisy-likelihood update pulls harder toward the biased coin than Jeffrey’s softer mix. The lad also clarifies that asserting revised marginals is not the same as claiming a sensor accuracy; that’s a different kind of statement altogether.

2.2 Probability, Cox-style

Added a new “Incoming” section and a cheeky callout about Cox’s epistemic foundation of probability. The new bits point readers to a Rising Entropy write-up that walks through Cox’s theorem (a derivation showing why plausible reasoning ends up looking Bayesian), and the page now explicitly contrasts Cox’s belief-based route with Kolmogorov’s measure-theory approach. There’s also a frank note that Cox’s route skirts some measure-theoretic rigour — infinite domains and countable additivity need better reconciliation, says the lad. Fair dinkum, useful pointers and a nod to the loose ends to follow up.

2.3 Top influences of 2025

Updated their year-influences post: new ‘Fiction’ section added with recommendations that read like speculations on corporate magic, biohacking and untrustworthy synthetic intelligences — so now the lad’s fiction picks are part of the pattern, not just an aside. The Human–Computer Interaction heading was rewritten and tightened into a clearer, heftier claim that HCI matters existentially, and he now explicitly frames actually-existing AI as a coupled human–AI system we must study. Small tonal shifts elsewhere make his relationship to AI-assisted reading and certain emphases (like a local civitech project) read more decisive.

2.4 Computation and the edge of chaos

Added a new section: “Edge of Stability in training neural networks,” which pokes at whether nets train best when perched on that chaotic boundary and points to recent papers (Cohen, Zhu) and the Central Flows write-up. It’s the lad trying to tie the old edge-of-chaos ideas to modern deep‑learning training behaviour — worth a look if you want to know whether this hand‑wavy stuff actually helps optimisation, duckies.

2.5 Overparameterization in large models

Updated the overparameterization notes — new sections were added called “For making optimization ‘nice’” and “Weight space versus function space” (they replace earlier awkwardly-named headings). The optimisation section tightens the language and leans into the idea that extra parameters can make the training landscape friendlier, with a clearer nod to transfer-function stability and Hardt’s rational-transfer analysis. The Lottery Ticket part was reworded to stress it’s probably computationally hard to find the tiny, better subnetworks and explicitly flags a question about computational bounds and whether pruning helps model interpretation. Also softened a couple of clutch-y claims (double descent phrasing now reads less dramatic).