Neural denoising diffusion models

Denoising diffusion probabilistic models (DDPMs), score-based generative models, generative diffusion processes, neural energy models…


AFAICS, generative models using score-matching to learn and Langevin MCMC to sample. There are various tricks needed to to do it with successive denoising steps and interpretation in terms of diffusion SDEs. I am vaguely aware that this oversimplifies a rich and interesting history of convergence of many useful techniques, but have not invested enough time to claim actual expertise.

Training: score matching

Denoising score matching HyvΓ€rinen (2005). See score matching or McAllester (2023) for an introduction to the general idea.

Sampling: Langevin dynamics

See Langevin samplers.

Image generation in particular

See image generation with diffusion.


Ajay, Anurag, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. 2023. β€œIs Conditional Generative Modeling All You Need for Decision-Making?” In. arXiv.
Anderson, Brian D. O. 1982. β€œReverse-Time Diffusion Equation Models.” Stochastic Processes and Their Applications 12 (3): 313–26.
Dhariwal, Prafulla, and Alex Nichol. 2021. β€œDiffusion Models Beat GANs on Image Synthesis.” arXiv:2105.05233 [Cs, Stat], June.
Dockhorn, Tim, Arash Vahdat, and Karsten Kreis. 2022. β€œGENIE: Higher-Order Denoising Diffusion Solvers.” In.
Dutordoir, Vincent, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. 2022. β€œNeural Diffusion Processes.” arXiv.
Han, Xizewen, Huangjie Zheng, and Mingyuan Zhou. 2022. β€œCARD: Classification and Regression Diffusion Models.” arXiv.
Ho, Jonathan, Ajay Jain, and Pieter Abbeel. 2020. β€œDenoising Diffusion Probabilistic Models.” arXiv:2006.11239 [Cs, Stat], December.
Hoogeboom, Emiel, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. 2021. β€œAutoregressive Diffusion Models.” arXiv:2110.02037 [Cs, Stat], October.
HyvΓ€rinen, Aapo. 2005. β€œEstimation of Non-Normalized Statistical Models by Score Matching.” The Journal of Machine Learning Research 6 (December): 695–709.
Jalal, Ajil, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. 2021. β€œRobust Compressed Sensing MRI with Deep Generative Priors.” In Advances in Neural Information Processing Systems, 34:14938–54. Curran Associates, Inc.
Jolicoeur-Martineau, Alexia, RΓ©mi PichΓ©-Taillefer, Ioannis Mitliagkas, and Remi Tachet des Combes. 2022. β€œAdversarial Score Matching and Improved Sampling for Image Generation.” In.
Liu, Ziming, Di Luo, Yilun Xu, Tommi Jaakkola, and Max Tegmark. 2023. β€œGenPhys: From Physical Processes to Generative Models.” arXiv.
McAllester, David. 2023. β€œOn the Mathematics of Diffusion Models.” arXiv.
Nichol, Alex, and Prafulla Dhariwal. 2021. β€œImproved Denoising Diffusion Probabilistic Models.” arXiv:2102.09672 [Cs, Stat], February.
Pascual, Santiago, Gautam Bhattacharya, Chunghsin Yeh, Jordi Pons, and Joan SerrΓ . 2022. β€œFull-Band General Audio Synthesis with Score-Based Diffusion.” arXiv.
Sharrock, Louis, Jack Simons, Song Liu, and Mark Beaumont. 2022. β€œSequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Models.” arXiv.
Sohl-Dickstein, Jascha, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. β€œDeep Unsupervised Learning Using Nonequilibrium Thermodynamics.” arXiv:1503.03585 [Cond-Mat, q-Bio, Stat], November.
Song, Jiaming, Chenlin Meng, and Stefano Ermon. 2021. β€œDenoising Diffusion Implicit Models.” arXiv:2010.02502 [Cs], November.
Song, Yang, Conor Durkan, Iain Murray, and Stefano Ermon. 2021. β€œMaximum Likelihood Training of Score-Based Diffusion Models.” In Advances in Neural Information Processing Systems.
Song, Yang, and Stefano Ermon. 2020a. β€œGenerative Modeling by Estimating Gradients of the Data Distribution.” In Advances In Neural Information Processing Systems. arXiv.
β€”β€”β€”. 2020b. β€œImproved Techniques for Training Score-Based Generative Models.” In Advances In Neural Information Processing Systems. arXiv.
Song, Yang, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. 2019. β€œSliced Score Matching: A Scalable Approach to Density and Score Estimation.” arXiv.
Song, Yang, Liyue Shen, Lei Xing, and Stefano Ermon. 2022. β€œSolving Inverse Problems in Medical Imaging with Score-Based Generative Models.” In. arXiv.
Song, Yang, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2020. β€œScore-Based Generative Modeling Through Stochastic Differential Equations.” In.
β€”β€”β€”. 2022. β€œScore-Based Generative Modeling Through Stochastic Differential Equations.” In.
Swersky, Kevin, Marc’Aurelio Ranzato, David Buchman, Nando D. Freitas, and Benjamin M. Marlin. 2011. β€œOn Autoencoders and Score Matching for Energy Based Models.” In Proceedings of the 28th International Conference on Machine Learning (ICML-11), 1201–8.
Torres, Susana VΓ‘zquez, Philip J. Y. Leung, Isaac D. Lutz, Preetham Venkatesh, Joseph L. Watson, Fabian Hink, Huu-Hien Huynh, et al. 2022. β€œDe Novo Design of High-Affinity Protein Binders to Bioactive Helical Peptides.” bioRxiv.
Vincent, Pascal. 2011. β€œA connection between score matching and denoising autoencoders.” Neural Computation 23 (7): 1661–74.
Watson, Joseph L., David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, et al. 2022. β€œBroadly Applicable and Accurate Protein Design by Integrating Structure Prediction Networks and Diffusion Generative Models.” bioRxiv.
Xu, Yilun, Ziming Liu, Max Tegmark, and Tommi S. Jaakkola. 2022. β€œPoisson Flow Generative Models.” In Advances in Neural Information Processing Systems.
Xu, Yilun, Ziming Liu, Yonglong Tian, Shangyuan Tong, Max Tegmark, and Tommi S. Jaakkola. 2023. β€œPFGM++: Unlocking the Potential of Physics-Inspired Generative Models.” In.
Yang, Ling, Zhilong Zhang, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Ming-Hsuan Yang, and Bin Cui. 2022. β€œDiffusion Models: A Comprehensive Survey of Methods and Applications.” arXiv.
Zamir, Syed Waqas, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2021. β€œMulti-Stage Progressive Image Restoration.” arXiv.
Zhuang, Peiye, Samira Abnar, Jiatao Gu, Alex Schwing, Joshua M. Susskind, and Miguel Ángel Bautista. 2022. β€œDiffusion Probabilistic Fields.” In.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.