Placeholder.
AFAICS, generative models using score-matching to learn and Langevin MCMC to sample. There are various tricks needed to to do it with successive denoising steps and interpretation in terms of SDEs. I am vaguely aware that this oversimplifies a rich and interesting history of convergence of many useful techniques, but not invested enough to perform a reconstruction upon the details.
Training: score matching
Denoising score matching Hyvärinen (2005). See score matching or McAllester (2023) for an introduction.
Sampling: Langevin dynamics
See Langevin samplers.
Diffusion on weird spaces
Incoming
- Lilian Weng, What are Diffusion Models?
- Yang Song, Generative Modeling by Estimating Gradients of the Data Distribution
- Sander Dieleman, Diffusion models are autoencoders
- Denoising Diffusion-based Generative Modeling: Foundations and Applications
- Tutorial on Denoising Diffusion-based Generative Modeling: Foundations and Applications
- What’s the score? (Review of latest Score Based Generative Modeling papers.)
- Anil Ananthaswamy, The Physics Principle That Inspired Modern AI Art
Suggestive connection to thermodynamics (Sohl-Dickstein et al. 2015).
References
Anderson, Brian D. O. 1982. “Reverse-Time Diffusion Equation Models.” Stochastic Processes and Their Applications 12 (3): 313–26.
Dhariwal, Prafulla, and Alex Nichol. 2021. “Diffusion Models Beat GANs on Image Synthesis.” arXiv:2105.05233 [Cs, Stat], June.
Dutordoir, Vincent, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. 2022. “Neural Diffusion Processes.” arXiv.
Han, Xizewen, Huangjie Zheng, and Mingyuan Zhou. 2022. “CARD: Classification and Regression Diffusion Models.” arXiv.
Ho, Jonathan, Ajay Jain, and Pieter Abbeel. 2020. “Denoising Diffusion Probabilistic Models.” arXiv:2006.11239 [Cs, Stat], December.
Hoogeboom, Emiel, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. 2021. “Autoregressive Diffusion Models.” arXiv:2110.02037 [Cs, Stat], October.
Hyvärinen, Aapo. 2005. “Estimation of Non-Normalized Statistical Models by Score Matching.” The Journal of Machine Learning Research 6 (December): 695–709.
Jalal, Ajil, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. 2021. “Robust Compressed Sensing MRI with Deep Generative Priors.” In Advances in Neural Information Processing Systems, 34:14938–54. Curran Associates, Inc.
Jolicoeur-Martineau, Alexia, Rémi Piché-Taillefer, Ioannis Mitliagkas, and Remi Tachet des Combes. 2022. “Adversarial Score Matching and Improved Sampling for Image Generation.” In.
McAllester, David. 2023. “On the Mathematics of Diffusion Models.” arXiv.
Nichol, Alex, and Prafulla Dhariwal. 2021. “Improved Denoising Diffusion Probabilistic Models.” arXiv:2102.09672 [Cs, Stat], February.
Pascual, Santiago, Gautam Bhattacharya, Chunghsin Yeh, Jordi Pons, and Joan Serrà. 2022. “Full-Band General Audio Synthesis with Score-Based Diffusion.” arXiv.
Sharrock, Louis, Jack Simons, Song Liu, and Mark Beaumont. 2022. “Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Models.” arXiv.
Sohl-Dickstein, Jascha, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. “Deep Unsupervised Learning Using Nonequilibrium Thermodynamics.” arXiv:1503.03585 [Cond-Mat, q-Bio, Stat], November.
Song, Jiaming, Chenlin Meng, and Stefano Ermon. 2021. “Denoising Diffusion Implicit Models.” arXiv:2010.02502 [Cs], November.
Song, Yang, Conor Durkan, Iain Murray, and Stefano Ermon. 2021. “Maximum Likelihood Training of Score-Based Diffusion Models.” In Advances in Neural Information Processing Systems.
Song, Yang, and Stefano Ermon. 2020a. “Generative Modeling by Estimating Gradients of the Data Distribution.” In Advances In Neural Information Processing Systems. arXiv.
———. 2020b. “Improved Techniques for Training Score-Based Generative Models.” In Advances In Neural Information Processing Systems. arXiv.
Song, Yang, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. 2019. “Sliced Score Matching: A Scalable Approach to Density and Score Estimation.” arXiv.
Song, Yang, Liyue Shen, Lei Xing, and Stefano Ermon. 2022. “Solving Inverse Problems in Medical Imaging with Score-Based Generative Models.” In. arXiv.
Song, Yang, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2022. “Score-Based Generative Modeling Through Stochastic Differential Equations.” In.
Swersky, Kevin, Marc’Aurelio Ranzato, David Buchman, Nando D. Freitas, and Benjamin M. Marlin. 2011. “On Autoencoders and Score Matching for Energy Based Models.” In Proceedings of the 28th International Conference on Machine Learning (ICML-11), 1201–8.
Torres, Susana Vázquez, Philip J. Y. Leung, Isaac D. Lutz, Preetham Venkatesh, Joseph L. Watson, Fabian Hink, Huu-Hien Huynh, et al. 2022. “De Novo Design of High-Affinity Protein Binders to Bioactive Helical Peptides.” bioRxiv.
Vincent, Pascal. 2011. “A connection between score matching and denoising autoencoders.” Neural Computation 23 (7): 1661–74.
Watson, Joseph L., David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, et al. 2022. “Broadly Applicable and Accurate Protein Design by Integrating Structure Prediction Networks and Diffusion Generative Models.” bioRxiv.
Yang, Ling, Zhilong Zhang, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Ming-Hsuan Yang, and Bin Cui. 2022. “Diffusion Models: A Comprehensive Survey of Methods and Applications.” arXiv.
No comments yet. Why not leave one?