Renormalization for ML
A Bayesian’s dictionary for renormalization-group vocabulary
2026-06-29 — 2026-06-29
Wherein a Vocabulary Mapping Between Renormalization-Group Physics and Bayesian Machine Learning Is Presented, With Depth Identified as the RG Flow Itself and Universality Classes as Basins of Fixed Points.
Drafted by an LLM as a quick orientation aid: a telegraphic dictionary mapping renormalization-group (RG) and QFT vocabulary onto Bayesian-ML concepts. Not yet checked, deduplicated against the rest of the notebook, or rewritten. 🚧TODO🚧
Prompted by Renormalizing interpretability (PIBBSS; Greenspan & Vaintrob), which proposes importing renormalization into neural-network interpretability. Their plan has a fever-dream shape for me, since I am not a physicist, and I don’t know their vocabulary, for all that I recognize some parts of it. It looks like variational approximation of probabilistic estimates (the “momentum-space, perturbative” flavour — literally so (Mehta and Schwab 2014)) and like graph marginalization (the “real-space, block-spin” flavour).
1 Translation table
A physics “model” is a log-density: \(P(\text{config}) = e^{-\beta H}/Z\). Read the energy/action \(H\) as a negative log joint, the partition function \(Z\) as the evidence, and the free energy \(-\tfrac{1}{\beta}\log Z\) as the negative log evidence. Then:
| Renormalization / QFT | Bayesian ML |
|---|---|
| Boltzmann distribution \(e^{-\beta H}/Z\) | posterior (or any model) \(\propto e^{-\text{energy}}\) |
| Hamiltonian / action / energy \(H\) | negative log joint (unnormalized log-density) |
| Partition function \(Z\) | normalizing constant / marginal likelihood (evidence) |
| Free energy \(-\tfrac1\beta\log Z\) | negative log evidence (\(\beta = 1\)) |
| Variational free energy | negative ELBO (\(\mathbb{E}_q[H]-S[q]\)); gap \(=\mathrm{KL}(q\Vert p)\) |
| Mean-field theory | mean-field (factorized-\(q\)) variational inference — same self-consistency equations |
| Saddle point / classical solution | MAP / posterior mode |
| Loop expansion (powers of \(\hbar\)) | expansion around the Laplace approximation (powers of posterior variance) |
| Temperature \(1/\beta\) | likelihood tempering; \(1/n\); observation noise \(\sigma^2\) |
| Integrating out / coarse-graining | marginalization |
| Real-space (block / decimation) RG | variable elimination on an MRF, then project back to a fixed family (\(\approx\) expectation propagation) |
| Effective Hamiltonian | negative log marginal density of the retained variables |
| Couplings \(g\); running couplings \(g(\Lambda)\) | model / natural parameters; effective parameters as a function of resolution |
| Cutoff \(\Lambda\) | resolution / bandwidth / level of detail retained |
| RG flow | trajectory of parameters under iterated marginalize-and-rescale |
| Fixed point | scale-invariant (self-similar) model |
| Gaussian fixed point | Gaussian / GP / Laplace / NNGP limit |
| Relevant operator | parameter direction that grows under coarse-graining (matters macroscopically) |
| Irrelevant operator | direction that shrinks / washes out (microscopic detail) |
| Universality class | basin of attraction of a fixed point (micro-models sharing macro behaviour) |
| Free theory + propagator | Gaussian reference + prior covariance (kernel) |
| Perturbation theory / Feynman diagrams | cumulant / Edgeworth expansion around the Gaussian; diagrams = cumulant bookkeeping |
| Interactions / non-Gaussianity | higher cumulants / non-Gaussian corrections |
| Effective field theory | low-dimensional parametric surrogate for the marginal / predictive |
| Renormalizability | the coarse model stays in a finite-dimensional family across scales (closed assumed-density recursion) |
- The CLT is an RG flow. The map “sum \(n\) iid, recentre, divide by \(\sqrt{n}\)” has the Gaussian as its attracting fixed point; finite-variance laws are its universality class; the Edgeworth/cumulant corrections (skew \(\sim n^{-1/2}\), excess kurtosis \(\sim n^{-1}\)) are the irrelevant operators that decay along the flow; centring is tuning the one relevant direction to criticality. (This is Vaintrob’s Laws of Large Numbers post.)
- Decimation is variable elimination plus EP. Real-space RG sums out a sublattice — exactly variable elimination on a Markov random field — then projects the induced all-order couplings back onto a chosen family (e.g. nearest-neighbour), a moment-matching / assumed-density step. So an RG step is (marginalize → project-to-family), i.e. expectation propagation indexed by scale.
2 Renormalization stuff that I have not seen in ML
Most of the table seems to be relabelling familiar machinery — evidence, negative log evidence, mean-field VI, Laplace, cumulant expansions, Gaussian processes. However, some ideas have no routine Bayesian analogue AFAIK:
- A flow on model space. Iterated coarse-graining traces a trajectory of models across scales, not one approximation at one scale. Variational inference gives the single approximation; RG gives the path and its dynamics.
- Relevant vs irrelevant directions. A Jacobian-eigenvalue (local sensitivity) analysis of that flow at a fixed point: which parameters grow under coarse-graining (survive to the macro-scale) and which decay (microscopic detail).
- Universality. Many micro-models flow to the same macro-model — a principled reason coarse descriptions are robust to microscopic detail.
Plus one mechanical ingredient ordinary Bayes lacks: the rescaling step after each marginalization (the \(\div\sqrt{n}\) below) that makes the iteration a flow with fixed points rather than a sequence of ever-coarser marginals.
3 Neural-network connections
- Infinite width → the Gaussian fixed point: a GP prior over functions (NNGP), neurons non-interacting, propagator = NNGP kernel.
- \(1/\text{width}\) → the coupling constant; finite width → weakly-interacting (perturbative) non-Gaussian corrections.
- Depth → the RG flow itself: each layer is one marginalize-and-rescale step (PDLT’s \(d/w\) is the step size).
- something about Neural tangent kernel
4 Incoming
Singular learning theory — Watanabe’s free energy and the overparametrized (singular) posterior; Vaintrob’s Basics of Bayesian learning is SLT setup.
Progress update: synthetic models of natural data — LessWrong
The Transformer as Renormalization Group Flow
The forward pass through a transformer implements a Kadanoff-Wilson renormalization group flow, coarse-graining microscopic token representations into stable semantic attractors.