Placeholder.
AFAICS, generative models using score-matching to learn and Langevin MCMC to sample. There are various tricks needed to to do it with successive denoising steps and interpretation in terms of diffusion SDEs. I am vaguely aware that this oversimplifies a rich and interesting history of convergence of many useful techniques, but have not invested enough time to claim actual expertise.
Training: score matching
Denoising score matching Hyvärinen (2005). See score matching or McAllester (2023) for an introduction to the general idea.
Sampling: Langevin dynamics
See Langevin samplers.
Diffusion on weird spaces
Incoming
- Lilian Weng, What are Diffusion Models?
- Yang Song, Generative Modeling by Estimating Gradients of the Data Distribution
- Sander Dieleman, Diffusion models are autoencoders
- CVPR tutorial, Denoising Diffusion-based Generative Modeling: Foundations and Applications Accompanying video
- What’s the score? (Review of latest Score Based Generative Modeling papers.)
- Anil Ananthaswamy, The Physics Principle That Inspired Modern AI Art
Suggestive connection to thermodynamics (Sohl-Dickstein et al. 2015).
References
Ajay, Anurag, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. 2023. “Is Conditional Generative Modeling All You Need for Decision-Making?” In. arXiv.
Anderson, Brian D. O. 1982. “Reverse-Time Diffusion Equation Models.” Stochastic Processes and Their Applications 12 (3): 313–26.
Dhariwal, Prafulla, and Alex Nichol. 2021. “Diffusion Models Beat GANs on Image Synthesis.” arXiv:2105.05233 [Cs, Stat], June.
Dockhorn, Tim, Arash Vahdat, and Karsten Kreis. 2022. “GENIE: Higher-Order Denoising Diffusion Solvers.” In.
Dutordoir, Vincent, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. 2022. “Neural Diffusion Processes.” arXiv.
Han, Xizewen, Huangjie Zheng, and Mingyuan Zhou. 2022. “CARD: Classification and Regression Diffusion Models.” arXiv.
Ho, Jonathan, Ajay Jain, and Pieter Abbeel. 2020. “Denoising Diffusion Probabilistic Models.” arXiv:2006.11239 [Cs, Stat], December.
Hoogeboom, Emiel, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. 2021. “Autoregressive Diffusion Models.” arXiv:2110.02037 [Cs, Stat], October.
Hyvärinen, Aapo. 2005. “Estimation of Non-Normalized Statistical Models by Score Matching.” The Journal of Machine Learning Research 6 (December): 695–709.
Jalal, Ajil, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. 2021. “Robust Compressed Sensing MRI with Deep Generative Priors.” In Advances in Neural Information Processing Systems, 34:14938–54. Curran Associates, Inc.
Jolicoeur-Martineau, Alexia, Rémi Piché-Taillefer, Ioannis Mitliagkas, and Remi Tachet des Combes. 2022. “Adversarial Score Matching and Improved Sampling for Image Generation.” In.
Liu, Ziming, Di Luo, Yilun Xu, Tommi Jaakkola, and Max Tegmark. 2023. “GenPhys: From Physical Processes to Generative Models.” arXiv.
McAllester, David. 2023. “On the Mathematics of Diffusion Models.” arXiv.
Nichol, Alex, and Prafulla Dhariwal. 2021. “Improved Denoising Diffusion Probabilistic Models.” arXiv:2102.09672 [Cs, Stat], February.
Pascual, Santiago, Gautam Bhattacharya, Chunghsin Yeh, Jordi Pons, and Joan Serrà. 2022. “Full-Band General Audio Synthesis with Score-Based Diffusion.” arXiv.
Sharrock, Louis, Jack Simons, Song Liu, and Mark Beaumont. 2022. “Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Models.” arXiv.
Sohl-Dickstein, Jascha, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. “Deep Unsupervised Learning Using Nonequilibrium Thermodynamics.” arXiv:1503.03585 [Cond-Mat, q-Bio, Stat], November.
Song, Jiaming, Chenlin Meng, and Stefano Ermon. 2021. “Denoising Diffusion Implicit Models.” arXiv:2010.02502 [Cs], November.
Song, Yang, Conor Durkan, Iain Murray, and Stefano Ermon. 2021. “Maximum Likelihood Training of Score-Based Diffusion Models.” In Advances in Neural Information Processing Systems.
Song, Yang, and Stefano Ermon. 2020a. “Generative Modeling by Estimating Gradients of the Data Distribution.” In Advances In Neural Information Processing Systems. arXiv.
———. 2020b. “Improved Techniques for Training Score-Based Generative Models.” In Advances In Neural Information Processing Systems. arXiv.
Song, Yang, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. 2019. “Sliced Score Matching: A Scalable Approach to Density and Score Estimation.” arXiv.
Song, Yang, Liyue Shen, Lei Xing, and Stefano Ermon. 2022. “Solving Inverse Problems in Medical Imaging with Score-Based Generative Models.” In. arXiv.
Song, Yang, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2020. “Score-Based Generative Modeling Through Stochastic Differential Equations.” In.
Swersky, Kevin, Marc’Aurelio Ranzato, David Buchman, Nando D. Freitas, and Benjamin M. Marlin. 2011. “On Autoencoders and Score Matching for Energy Based Models.” In Proceedings of the 28th International Conference on Machine Learning (ICML-11), 1201–8.
Torres, Susana Vázquez, Philip J. Y. Leung, Isaac D. Lutz, Preetham Venkatesh, Joseph L. Watson, Fabian Hink, Huu-Hien Huynh, et al. 2022. “De Novo Design of High-Affinity Protein Binders to Bioactive Helical Peptides.” bioRxiv.
Vincent, Pascal. 2011. “A connection between score matching and denoising autoencoders.” Neural Computation 23 (7): 1661–74.
Watson, Joseph L., David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, et al. 2022. “Broadly Applicable and Accurate Protein Design by Integrating Structure Prediction Networks and Diffusion Generative Models.” bioRxiv.
Xu, Yilun, Ziming Liu, Max Tegmark, and Tommi S. Jaakkola. 2022. “Poisson Flow Generative Models.” In Advances in Neural Information Processing Systems.
Xu, Yilun, Ziming Liu, Yonglong Tian, Shangyuan Tong, Max Tegmark, and Tommi S. Jaakkola. 2023. “PFGM++: Unlocking the Potential of Physics-Inspired Generative Models.” In.
Yang, Ling, Zhilong Zhang, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Ming-Hsuan Yang, and Bin Cui. 2022. “Diffusion Models: A Comprehensive Survey of Methods and Applications.” arXiv.
Zamir, Syed Waqas, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2021. “Multi-Stage Progressive Image Restoration.” arXiv.
Zhuang, Peiye, Samira Abnar, Jiatao Gu, Alex Schwing, Joshua M. Susskind, and Miguel Ángel Bautista. 2022. “Diffusion Probabilistic Fields.” In.
No comments yet. Why not leave one?