Placeholder.
AFAICS, generative models using score-matching to learn and Langevin MCMC to sample. There are various tricks needed to to do it with successive denoising steps and interpretation in terms of diffusion SDEs. I am vaguely aware that this oversimplifies a rich and interesting history of convergence of many useful techniques, but have not invested enough time to claim actual expertise.
Training: score matching
Denoising score matching HyvΓ€rinen (2005). See score matching or McAllester (2023) for an introduction to the general idea.
Sampling: Langevin dynamics
See Langevin samplers.
Diffusion on weird spaces
Incoming
- Lilian Weng, What are Diffusion Models?
- Yang Song, Generative Modeling by Estimating Gradients of the Data Distribution
- Sander Dieleman, Diffusion models are autoencoders
- CVPR tutorial, Denoising Diffusion-based Generative Modeling: Foundations and Applications Accompanying video
- Whatβs the score? (Review of latest Score Based Generative Modeling papers.)
- Anil Ananthaswamy, The Physics Principle That Inspired Modern AI Art
Suggestive connection to thermodynamics (Sohl-Dickstein et al. 2015).
References
Ajay, Anurag, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. 2023. βIs Conditional Generative Modeling All You Need for Decision-Making?β In. arXiv.
Anderson, Brian D. O. 1982. βReverse-Time Diffusion Equation Models.β Stochastic Processes and Their Applications 12 (3): 313β26.
Dhariwal, Prafulla, and Alex Nichol. 2021. βDiffusion Models Beat GANs on Image Synthesis.β arXiv:2105.05233 [Cs, Stat], June.
Dockhorn, Tim, Arash Vahdat, and Karsten Kreis. 2022. βGENIE: Higher-Order Denoising Diffusion Solvers.β In.
Dutordoir, Vincent, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. 2022. βNeural Diffusion Processes.β arXiv.
Han, Xizewen, Huangjie Zheng, and Mingyuan Zhou. 2022. βCARD: Classification and Regression Diffusion Models.β arXiv.
Ho, Jonathan, Ajay Jain, and Pieter Abbeel. 2020. βDenoising Diffusion Probabilistic Models.β arXiv:2006.11239 [Cs, Stat], December.
Hoogeboom, Emiel, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. 2021. βAutoregressive Diffusion Models.β arXiv:2110.02037 [Cs, Stat], October.
HyvΓ€rinen, Aapo. 2005. βEstimation of Non-Normalized Statistical Models by Score Matching.β The Journal of Machine Learning Research 6 (December): 695β709.
Jalal, Ajil, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. 2021. βRobust Compressed Sensing MRI with Deep Generative Priors.β In Advances in Neural Information Processing Systems, 34:14938β54. Curran Associates, Inc.
Jolicoeur-Martineau, Alexia, RΓ©mi PichΓ©-Taillefer, Ioannis Mitliagkas, and Remi Tachet des Combes. 2022. βAdversarial Score Matching and Improved Sampling for Image Generation.β In.
Liu, Ziming, Di Luo, Yilun Xu, Tommi Jaakkola, and Max Tegmark. 2023. βGenPhys: From Physical Processes to Generative Models.β arXiv.
McAllester, David. 2023. βOn the Mathematics of Diffusion Models.β arXiv.
Nichol, Alex, and Prafulla Dhariwal. 2021. βImproved Denoising Diffusion Probabilistic Models.β arXiv:2102.09672 [Cs, Stat], February.
Pascual, Santiago, Gautam Bhattacharya, Chunghsin Yeh, Jordi Pons, and Joan SerrΓ . 2022. βFull-Band General Audio Synthesis with Score-Based Diffusion.β arXiv.
Sharrock, Louis, Jack Simons, Song Liu, and Mark Beaumont. 2022. βSequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Models.β arXiv.
Sohl-Dickstein, Jascha, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. βDeep Unsupervised Learning Using Nonequilibrium Thermodynamics.β arXiv:1503.03585 [Cond-Mat, q-Bio, Stat], November.
Song, Jiaming, Chenlin Meng, and Stefano Ermon. 2021. βDenoising Diffusion Implicit Models.β arXiv:2010.02502 [Cs], November.
Song, Yang, Conor Durkan, Iain Murray, and Stefano Ermon. 2021. βMaximum Likelihood Training of Score-Based Diffusion Models.β In Advances in Neural Information Processing Systems.
Song, Yang, and Stefano Ermon. 2020a. βGenerative Modeling by Estimating Gradients of the Data Distribution.β In Advances In Neural Information Processing Systems. arXiv.
βββ. 2020b. βImproved Techniques for Training Score-Based Generative Models.β In Advances In Neural Information Processing Systems. arXiv.
Song, Yang, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. 2019. βSliced Score Matching: A Scalable Approach to Density and Score Estimation.β arXiv.
Song, Yang, Liyue Shen, Lei Xing, and Stefano Ermon. 2022. βSolving Inverse Problems in Medical Imaging with Score-Based Generative Models.β In. arXiv.
Song, Yang, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2020. βScore-Based Generative Modeling Through Stochastic Differential Equations.β In.
βββ. 2022. βScore-Based Generative Modeling Through Stochastic Differential Equations.β In.
Swersky, Kevin, MarcβAurelio Ranzato, David Buchman, Nando D. Freitas, and Benjamin M. Marlin. 2011. βOn Autoencoders and Score Matching for Energy Based Models.β In Proceedings of the 28th International Conference on Machine Learning (ICML-11), 1201β8.
Torres, Susana VΓ‘zquez, Philip J. Y. Leung, Isaac D. Lutz, Preetham Venkatesh, Joseph L. Watson, Fabian Hink, Huu-Hien Huynh, et al. 2022. βDe Novo Design of High-Affinity Protein Binders to Bioactive Helical Peptides.β bioRxiv.
Vincent, Pascal. 2011. βA connection between score matching and denoising autoencoders.β Neural Computation 23 (7): 1661β74.
Watson, Joseph L., David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, et al. 2022. βBroadly Applicable and Accurate Protein Design by Integrating Structure Prediction Networks and Diffusion Generative Models.β bioRxiv.
Xu, Yilun, Ziming Liu, Max Tegmark, and Tommi S. Jaakkola. 2022. βPoisson Flow Generative Models.β In Advances in Neural Information Processing Systems.
Xu, Yilun, Ziming Liu, Yonglong Tian, Shangyuan Tong, Max Tegmark, and Tommi S. Jaakkola. 2023. βPFGM++: Unlocking the Potential of Physics-Inspired Generative Models.β In.
Yang, Ling, Zhilong Zhang, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Ming-Hsuan Yang, and Bin Cui. 2022. βDiffusion Models: A Comprehensive Survey of Methods and Applications.β arXiv.
Zamir, Syed Waqas, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2021. βMulti-Stage Progressive Image Restoration.β arXiv.
Zhuang, Peiye, Samira Abnar, Jiatao Gu, Alex Schwing, Joshua M. Susskind, and Miguel Γngel Bautista. 2022. βDiffusion Probabilistic Fields.β In.
No comments yet. Why not leave one?