Placeholder.
AFAICS, generative models using score-matching to learn and Langevin MCMC to sample. I am vaguely aware that this oversimplifies a rich and interesting history of convergence of many useful techniques, but not invested enough to perform a reconstruction upon the details.
Training: score matching
Denoising score matching Hyvärinen (2005). See score matching.
Sampling: Langevin dynamics
See Langevin samplers.
Incoming
- What are Diffusion Models?
- Yang Song, Generative Modeling by Estimating Gradients of the Data Distribution
- Diffusion models are autoencoders – Sander Dieleman
- Denoising Diffusion-based Generative Modeling: Foundations and Applications
- Tutorial on Denoising Diffusion-based Generative Modeling: Foundations and Applications
- What's the score? – Review of latest Score Based Generative Modeling papers.
Suggestive connection to thermodynamics (Sohl-Dickstein et al. 2015).
References
Anderson, Brian D. O. 1982. “Reverse-Time Diffusion Equation Models.” Stochastic Processes and Their Applications 12 (3): 313–26.
Dhariwal, Prafulla, and Alex Nichol. 2021. “Diffusion Models Beat GANs on Image Synthesis.” arXiv:2105.05233 [Cs, Stat], June.
Dutordoir, Vincent, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. 2022. “Neural Diffusion Processes.” arXiv.
Han, Xizewen, Huangjie Zheng, and Mingyuan Zhou. 2022. “CARD: Classification and Regression Diffusion Models.” arXiv.
Ho, Jonathan, Ajay Jain, and Pieter Abbeel. 2020. “Denoising Diffusion Probabilistic Models.” arXiv:2006.11239 [Cs, Stat], December.
Hoogeboom, Emiel, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. 2021. “Autoregressive Diffusion Models.” arXiv:2110.02037 [Cs, Stat], October.
Hyvärinen, Aapo. 2005. “Estimation of Non-Normalized Statistical Models by Score Matching.” The Journal of Machine Learning Research 6 (December): 695–709.
Jalal, Ajil, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. 2021. “Robust Compressed Sensing MRI with Deep Generative Priors.” In Advances in Neural Information Processing Systems, 34:14938–54. Curran Associates, Inc.
Jolicoeur-Martineau, Alexia, Rémi Piché-Taillefer, Ioannis Mitliagkas, and Remi Tachet des Combes. 2022. “Adversarial Score Matching and Improved Sampling for Image Generation.” In.
Nichol, Alex, and Prafulla Dhariwal. 2021. “Improved Denoising Diffusion Probabilistic Models.” arXiv:2102.09672 [Cs, Stat], February.
Sohl-Dickstein, Jascha, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. “Deep Unsupervised Learning Using Nonequilibrium Thermodynamics.” arXiv:1503.03585 [Cond-Mat, q-Bio, Stat], November.
Song, Jiaming, Chenlin Meng, and Stefano Ermon. 2021. “Denoising Diffusion Implicit Models.” arXiv:2010.02502 [Cs], November.
Song, Yang, Conor Durkan, Iain Murray, and Stefano Ermon. 2021. “Maximum Likelihood Training of Score-Based Diffusion Models.” In Advances in Neural Information Processing Systems.
Song, Yang, and Stefano Ermon. 2020a. “Generative Modeling by Estimating Gradients of the Data Distribution.” In Advances In Neural Information Processing Systems. arXiv.
———. 2020b. “Improved Techniques for Training Score-Based Generative Models.” In Advances In Neural Information Processing Systems. arXiv.
Song, Yang, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. 2019. “Sliced Score Matching: A Scalable Approach to Density and Score Estimation.” arXiv.
Song, Yang, Liyue Shen, Lei Xing, and Stefano Ermon. 2022. “Solving Inverse Problems in Medical Imaging with Score-Based Generative Models.” In. arXiv.
Song, Yang, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2022. “Score-Based Generative Modeling Through Stochastic Differential Equations.” In.
Swersky, Kevin, Marc’Aurelio Ranzato, David Buchman, Nando D. Freitas, and Benjamin M. Marlin. 2011. “On Autoencoders and Score Matching for Energy Based Models.” In Proceedings of the 28th International Conference on Machine Learning (ICML-11), 1201–8.
Vincent, Pascal. 2011. “A connection between score matching and denoising autoencoders.” Neural Computation 23 (7): 1661–74.
Yang, Ling, Zhilong Zhang, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Ming-Hsuan Yang, and Bin Cui. 2022. “Diffusion Models: A Comprehensive Survey of Methods and Applications.” arXiv.
No comments yet. Why not leave one?