Energy based models

Inference with kinda-tractable un-normalized densities

2021-06-07 — 2021-06-07

Wherein inference for undirected graphical models is framed as optimization of an energy function, and local gradient descent toward more probable configurations is depicted.

approximation

Bayes

generative

Monte Carlo

optimization

probabilistic algorithms

probability

statistics

statmech

I don’t actually know what fits under this heading, but it sounds like it is simply inference for undirected graphical models. Or is there something distinct going on?

Figure 1: Descending the local energy gradient to a more probable configuration

1 References

Bardes, Ponce, and LeCun. 2022. “VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning.”

Che, Zhang, Sohl-Dickstein, et al. 2020. “Your GAN Is Secretly an Energy-Based Model and You Should Use Discriminator Driven Latent Sampling.” arXiv:2003.06060 [Cs, Stat].

Clifford. 1990. “Markov random fields in statistics.” In Disorder in Physical Systems: A Volume in Honour of John Hammersley.

Hinton. 2010. “A Practical Guide to Training Restricted Boltzmann Machines.” In Neural Networks: Tricks of the Trade. Lecture Notes in Computer Science 7700.

LeCun, Chopra, Hadsell, et al. 2006. “A Tutorial on Energy-Based Learning.” In Predicting Structured Data.

Montavon, Müller, and Cuturi. 2016. “Wasserstein Training of Restricted Boltzmann Machines.” In Advances in Neural Information Processing Systems 29.

Pollard. 2004. “Hammersley-Clifford Theorem for Markov Random Fields.”

Salakhutdinov. 2015. “Learning Deep Generative Models.” Annual Review of Statistics and Its Application.