Neural denoising diffusion models

Denoising diffusion probabilistic models (DDPMs), score-based generative models, generative diffusion processes, neural energy models…

November 11, 2021 — December 6, 2023

approximation
Bayes
generative
Monte Carlo
neural nets
optimization
probabilistic algorithms
probability
statistics
Figure 1

Placeholder.

AFAICS, generative models using score-matching to learn and Langevin MCMC to sample. There are various tricks needed to to do it with successive denoising steps and interpretation in terms of diffusion SDEs. I am vaguely aware that this oversimplifies a rich and interesting history of convergence of many useful techniques, but have not invested enough time to claim actual expertise.

1 Training: score matching

Denoising score matching Hyvärinen (2005). See score matching or McAllester (2023) for an introduction to the general idea.

2 Sampling: Langevin dynamics

See Langevin samplers.

3 Image generation in particular

See image generation with diffusion.

Figure 2

4 Latent

4.1 Generic

4.2 CLIP

Radford et al. (2021)

5 Diffusion on weird spaces

5.1 Proteins

Baker Lab (Torres et al. 2022; Watson et al. 2022)

6 Incoming

Suggestive connection to thermodynamics (Sohl-Dickstein et al. 2015).

Figure 3

7 References

Ajay, Du, Gupta, et al. 2023. “Is Conditional Generative Modeling All You Need for Decision-Making?” In.
Albergo, Boffi, and Vanden-Eijnden. 2023. “Stochastic Interpolants: A Unifying Framework for Flows and Diffusions.”
Albergo, Goldstein, Boffi, et al. 2023. “Stochastic Interpolants with Data-Dependent Couplings.”
Albergo, and Vanden-Eijnden. 2023. “Building Normalizing Flows with Stochastic Interpolants.” In.
Anderson. 1982. “Reverse-Time Diffusion Equation Models.” Stochastic Processes and Their Applications.
Chung, Kim, Mccann, et al. 2023. “Diffusion Posterior Sampling for General Noisy Inverse Problems.” In.
Dhariwal, and Nichol. 2021. “Diffusion Models Beat GANs on Image Synthesis.” arXiv:2105.05233 [Cs, Stat].
Dockhorn, Vahdat, and Kreis. 2022. “GENIE: Higher-Order Denoising Diffusion Solvers.” In.
Dutordoir, Saul, Ghahramani, et al. 2022. “Neural Diffusion Processes.”
Han, Zheng, and Zhou. 2022. “CARD: Classification and Regression Diffusion Models.”
Ho, Jain, and Abbeel. 2020. “Denoising Diffusion Probabilistic Models.” arXiv:2006.11239 [Cs, Stat].
Hoogeboom, Gritsenko, Bastings, et al. 2021. “Autoregressive Diffusion Models.” arXiv:2110.02037 [Cs, Stat].
Hyvärinen. 2005. “Estimation of Non-Normalized Statistical Models by Score Matching.” The Journal of Machine Learning Research.
Jalal, Arvinte, Daras, et al. 2021. “Robust Compressed Sensing MRI with Deep Generative Priors.” In Advances in Neural Information Processing Systems.
Jolicoeur-Martineau, Piché-Taillefer, Mitliagkas, et al. 2022. “Adversarial Score Matching and Improved Sampling for Image Generation.” In.
Lipman, Chen, Ben-Hamu, et al. 2023. “Flow Matching for Generative Modeling.”
Liu, Luo, Xu, et al. 2023. “GenPhys: From Physical Processes to Generative Models.”
McAllester. 2023. “On the Mathematics of Diffusion Models.”
Nichol, and Dhariwal. 2021. “Improved Denoising Diffusion Probabilistic Models.” arXiv:2102.09672 [Cs, Stat].
Pascual, Bhattacharya, Yeh, et al. 2022. “Full-Band General Audio Synthesis with Score-Based Diffusion.”
Preechakul, Chatthee, Wizadwongsa, et al. 2022. “Diffusion Autoencoders: Toward a Meaningful and Decodable Representation.” In.
Radford, Kim, Hallacy, et al. 2021. “Learning Transferable Visual Models From Natural Language Supervision.”
Sharrock, Simons, Liu, et al. 2022. “Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Models.”
Sohl-Dickstein, Weiss, Maheswaranathan, et al. 2015. “Deep Unsupervised Learning Using Nonequilibrium Thermodynamics.” arXiv:1503.03585 [Cond-Mat, q-Bio, Stat].
Song, Yang, Durkan, Murray, et al. 2021. “Maximum Likelihood Training of Score-Based Diffusion Models.” In Advances in Neural Information Processing Systems.
Song, Yang, and Ermon. 2020a. “Generative Modeling by Estimating Gradients of the Data Distribution.” In Advances In Neural Information Processing Systems.
———. 2020b. “Improved Techniques for Training Score-Based Generative Models.” In Advances In Neural Information Processing Systems.
Song, Yang, Garg, Shi, et al. 2019. “Sliced Score Matching: A Scalable Approach to Density and Score Estimation.”
Song, Jiaming, Meng, and Ermon. 2021. “Denoising Diffusion Implicit Models.” arXiv:2010.02502 [Cs].
Song, Yang, Shen, Xing, et al. 2022. “Solving Inverse Problems in Medical Imaging with Score-Based Generative Models.” In.
Song, Yang, Sohl-Dickstein, Kingma, et al. 2020. “Score-Based Generative Modeling Through Stochastic Differential Equations.” In.
———, et al. 2022. “Score-Based Generative Modeling Through Stochastic Differential Equations.” In.
Swersky, Ranzato, Buchman, et al. 2011. “On Autoencoders and Score Matching for Energy Based Models.” In Proceedings of the 28th International Conference on Machine Learning (ICML-11).
Torres, Leung, Lutz, et al. 2022. “De Novo Design of High-Affinity Protein Binders to Bioactive Helical Peptides.”
Tzen, and Raginsky. 2019a. “Theoretical Guarantees for Sampling and Inference in Generative Models with Latent Diffusions.” In Proceedings of the Thirty-Second Conference on Learning Theory.
———. 2019b. “Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit.”
Vincent. 2011. “A connection between score matching and denoising autoencoders.” Neural Computation.
Watson, Juergens, Bennett, et al. 2022. “Broadly Applicable and Accurate Protein Design by Integrating Structure Prediction Networks and Diffusion Generative Models.”
Xu, Liu, Tegmark, et al. 2022. “Poisson Flow Generative Models.” In Advances in Neural Information Processing Systems.
Xu, Liu, Tian, et al. 2023. “PFGM++: Unlocking the Potential of Physics-Inspired Generative Models.” In.
Yang, Zhang, Hong, et al. 2022. “Diffusion Models: A Comprehensive Survey of Methods and Applications.”
Zamir, Arora, Khan, et al. 2021. “Multi-Stage Progressive Image Restoration.”
Zhuang, Abnar, Gu, et al. 2022. “Diffusion Probabilistic Fields.” In.