Renormalization for ML
A Bayesian’s dictionary for renormalization-group vocabulary
2026-06-29 — 2026-06-29
Wherein a Vocabulary Mapping Between Renormalization-Group Physics and Bayesian Machine Learning Is Presented, With Depth Identified as the RG Flow Itself and Universality Classes as Basins of Fixed Points.
On this page
Drafted by an LLM as a quick orientation aid: a telegraphic dictionary mapping renormalization-group (RG) and QFT vocabulary onto Bayesian-ML concepts. Not yet checked, deduplicated against the rest of the notebook, or rewritten. 🚧TODO🚧
Prompted by Renormalizing interpretability (PIBBSS; Greenspan & Vaintrob), which proposes importing renormalization into neural-network interpretability. Their plan has a fever-dream shape for me, since I am not a physicist, and I don’t know their vocabulary, for all that I recognize some parts of it. It looks like variational approximation of probabilistic estimates (the “momentum-space, perturbative” flavour — literally so (Mehta and Schwab 2014)) and like graph marginalization (the “real-space, block-spin” flavour).
1 Translation table
A physics “model” is a log-density: \(P(\text{config}) = e^{-\beta H}/Z\). Read the energy/action \(H\) as a negative log joint, the partition function \(Z\) as the evidence, and the free energy \(-\tfrac{1}{\beta}\log Z\) as the negative log evidence. Then:
| Renormalization / QFT | Bayesian ML |
|---|---|
| Boltzmann distribution \(e^{-\beta H}/Z\) | posterior (or any model) \(\propto e^{-\text{energy}}\) |
| Hamiltonian / action / energy \(H\) | negative log joint (unnormalized log-density) |
| Partition function \(Z\) | normalizing constant / marginal likelihood (evidence) |
| Free energy \(-\tfrac1\beta\log Z\) | negative log evidence (\(\beta = 1\)) |
| Variational free energy | negative ELBO (\(\mathbb{E}_q[H]-S[q]\)); gap \(=\mathrm{KL}(q\Vert p)\) |
| Mean-field theory | mean-field (factorized-\(q\)) variational inference — same self-consistency equations |
| Saddle point / classical solution | MAP / posterior mode |
| Loop expansion (powers of \(\hbar\)) | expansion around the Laplace approximation (powers of posterior variance) |
| Temperature \(1/\beta\) | likelihood tempering; \(1/n\); observation noise \(\sigma^2\) |
| Integrating out / coarse-graining | marginalization |
| Real-space (block / decimation) RG | variable elimination on an MRF, then project back to a fixed family (\(\approx\) expectation propagation) |
| Effective Hamiltonian | negative log marginal density of the retained variables |
| Couplings \(g\); running couplings \(g(\Lambda)\) | model / natural parameters; effective parameters as a function of resolution |
| Cutoff \(\Lambda\) | resolution / bandwidth / level of detail retained |
| RG flow | trajectory of parameters under iterated marginalize-and-rescale |
| Fixed point | scale-invariant (self-similar) model |
| Gaussian fixed point | Gaussian / GP / Laplace / NNGP limit |
| Relevant operator | parameter direction that grows under coarse-graining (matters macroscopically) |
| Irrelevant operator | direction that shrinks / washes out (microscopic detail) |
| Universality class | basin of attraction of a fixed point (micro-models sharing macro behaviour) |
| Free theory + propagator | Gaussian reference + prior covariance (kernel) |
| Perturbation theory / Feynman diagrams | cumulant / Edgeworth expansion around the Gaussian; diagrams = cumulant bookkeeping |
| Interactions / non-Gaussianity | higher cumulants / non-Gaussian corrections |
| Effective field theory | low-dimensional parametric surrogate for the marginal / predictive |
| Renormalizability | the coarse model stays in a finite-dimensional family across scales (closed assumed-density recursion) |
- The CLT is an RG flow. The map “sum \(n\) iid, recentre, divide by \(\sqrt{n}\)” has the Gaussian as its attracting fixed point; finite-variance laws are its universality class; the Edgeworth/cumulant corrections (skew \(\sim n^{-1/2}\), excess kurtosis \(\sim n^{-1}\)) are the irrelevant operators that decay along the flow; centring is tuning the one relevant direction to criticality. (This is Vaintrob’s Laws of Large Numbers post.)
- Decimation is variable elimination plus EP. Real-space RG sums out a sublattice — exactly variable elimination on a Markov random field — then projects the induced all-order couplings back onto a chosen family (e.g. nearest-neighbour), a moment-matching / assumed-density step. So an RG step is (marginalize → project-to-family), i.e. expectation propagation indexed by scale.
2 Renormalization stuff that I have not seen in ML
Most of the table seems to be relabelling familiar machinery — evidence, negative log evidence, mean-field VI, Laplace, cumulant expansions, Gaussian processes. However, some ideas have no routine Bayesian analogue AFAIK:
- A flow on model space. Iterated coarse-graining traces a trajectory of models across scales, not one approximation at one scale. Variational inference gives the single approximation; RG gives the path and its dynamics.
- Relevant vs irrelevant directions. A Jacobian-eigenvalue (local sensitivity) analysis of that flow at a fixed point: which parameters grow under coarse-graining (survive to the macro-scale) and which decay (microscopic detail).
- Universality. Many micro-models flow to the same macro-model — a principled reason coarse descriptions are robust to microscopic detail.
Plus one mechanical ingredient ordinary Bayes lacks: the rescaling step after each marginalization (the \(\div\sqrt{n}\) below) that makes the iteration a flow with fixed points rather than a sequence of ever-coarser marginals.
3 Neural-network connections
- Infinite width → the Gaussian fixed point: a GP prior over functions (NNGP), neurons non-interacting, propagator = NNGP kernel.
- \(1/\text{width}\) → the coupling constant; finite width → weakly-interacting (perturbative) non-Gaussian corrections.
- Depth → the RG flow itself: each layer is one marginalize-and-rescale step (PDLT’s \(d/w\) is the step size).
- something about Neural tangent kernel
4 Incoming
Singular learning theory — Watanabe’s free energy and the overparametrized (singular) posterior; Vaintrob’s Basics of Bayesian learning is SLT setup.
The Transformer as Renormalization Group Flow
The forward pass through a transformer implements a Kadanoff-Wilson renormalization group flow, coarse-graining microscopic token representations into stable semantic attractors.
