Score matching

November 11, 2021 — September 23, 2022

Very famous now thanks to neural diffusions.

Figure 1

1 Basic

TBC

2 Sliced

Sliced Score Matching (Y. Song et al. 2019).

3 Noise Conditional

Yang Song, Generative Modeling by Estimating Gradients of the Data Distribution

4 Incoming

Suggestive connection to thermodynamics (Sohl-Dickstein et al. 2015), score estimators in gradient, and Bregman divergences (Gutmann and Hirayama 2011).

5 References

Dockhorn, Vahdat, and Kreis. 2022. GENIE: Higher-Order Denoising Diffusion Solvers.” In.
Gutmann, and Hirayama. 2011. Bregman Divergence as General Framework to Estimate Unnormalized Statistical Models.” In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence. UAI’11.
Holzschuh, Vegetti, and Thuerey. 2022. “Score Matching via Differentiable Physics.”
Hyvärinen. 2005. Estimation of Non-Normalized Statistical Models by Score Matching.” The Journal of Machine Learning Research.
Lim, Kovachki, Baptista, et al. 2023. Score-Based Diffusion Models in Function Space.”
McAllester. 2023. On the Mathematics of Diffusion Models.”
Sharrock, Simons, Liu, et al. 2022. Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Models.”
Sohl-Dickstein, Weiss, Maheswaranathan, et al. 2015. Deep Unsupervised Learning Using Nonequilibrium Thermodynamics.” arXiv:1503.03585 [Cond-Mat, q-Bio, Stat].
Song, Yang, and Ermon. 2020a. Generative Modeling by Estimating Gradients of the Data Distribution.” In Advances In Neural Information Processing Systems.
———. 2020b. Improved Techniques for Training Score-Based Generative Models.” In Advances In Neural Information Processing Systems.
Song, Yang, Garg, Shi, et al. 2019. Sliced Score Matching: A Scalable Approach to Density and Score Estimation.”
Song, Jiaming, Meng, and Ermon. 2021. Denoising Diffusion Implicit Models.” arXiv:2010.02502 [Cs].
Song, Yang, Sohl-Dickstein, Kingma, et al. 2022. Score-Based Generative Modeling Through Stochastic Differential Equations.” In.
Swersky, Ranzato, Buchman, et al. 2011. “On Autoencoders and Score Matching for Energy Based Models.” In Proceedings of the 28th International Conference on Machine Learning (ICML-11).
Tran, Rossi, Milios, et al. 2021. Model Selection for Bayesian Autoencoders.” In Advances in Neural Information Processing Systems.
Vincent. 2011. A connection between score matching and denoising autoencoders.” Neural Computation.
Zhuang, Abnar, Gu, et al. 2022. Diffusion Probabilistic Fields.” In.