MaxEnt inference
Looks annoyingly like Bayesian inference, but I’m not convinced
2010-12-01 — 2026-01-27
Wherein Constraints Encoding Observations and Prior Knowledge Are Imposed on a Probability Distribution, and the Distribution of Maximum Entropy Subject to Them Is Selected; Connections to Predictive Coding and Optimal Transport Are Noted.
If we think about entropy versus information for long enough, we invent MaxEnt inference.
How about we throw out classical Bayes and do something different? We encode our observations and prior knowledge as constraints on a probability distribution, then select the maximum-entropy distribution that satisfies them.
The rationale is that the maximum-entropy distribution is the ‘least biased’ estimate possible while still satisfying those constraints.
Jaynes and Bretthorst (2003) founded this school of thought. Shore and Johnson (1980) axiomatized it (although I’m told there are significant errors: their conditions don’t suffice to restrict us to the Shannon entropy form).
However, from context cues — the fact people introduced a CamelCase acronym — I deduce there’s more going on here. I first looked into it 10+ years ago for some long forgotten reason, but my interest has been piqued again after ignoring it for ages. Bert de Vries claimed to have actioned MaxEnt as a particularly useful phenomenological idea within the predictive coding theory of mind, which inclined me to return to the original MaxEnt work by Caticha. Caticha’s treatment now appears in textbooks (Caticha 2015, 2008) and review articles (Caticha 2021, 2014).
idk though, this formalism never seems to solve things intuitively. Do I want to maximize entropy subject to constraints, or minimize KL divergence to a prior subject to constraints? Why turn everything into constrained optimization problems?
1 Generalizations
Dixit et al. (2018):
We review here Maximum Caliber (Max Cal), a general variational principle for inferring distributions of paths in dynamical processes and networks. Max Cal is to dynamical trajectories what the principle of Maximum Entropy (Max Ent) is to equilibrium states or stationary populations. In Max Cal, you maximize a path entropy over all possible pathways, subject to dynamical constraints, in order to predict relative path weights. Many well-known relationships of Non-Equilibrium Statistical Physics—such as the Green-Kubo fluctuation-dissipation relations, Onsager’s reciprocal relations, and Prigogine’s Minimum Entropy Production—are limited to near-equilibrium processes. Max Cal is more general. While it can readily derive these results under those limits, Max Cal is also applicable far from equilibrium. We give recent examples of MaxCal as a method of inference about trajectory distributions from limited data, finding reaction coordinates in bio-molecular simulations, and modeling the complex dynamics of non-thermal systems such as gene regulatory networks or the collective firing of neurons. We also survey its basis in principle, and some limitations.
2 Incoming
Belghazi et al. (2021):
We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, and strongly consistent. We present a handful of applications on which MINE can be used to minimize or maximize mutual information. We apply MINE to improve adversarially trained generative models. We also use MINE to implement Information Bottleneck, applying it to supervised classification; our results demonstrate substantial improvement in flexibility and performance in these settings.