Neural denoising diffusion models
Denoising diffusion probabilistic models (DDPMs), score-based generative models, generative diffusion processes, neural energy models…
2021-11-10 — 2025-05-28
Placeholder.
The archetypal neural denoising diffusions use score-matching to learn and score diffusion to sample. I won’t explain more than that because this is a super hot area and there are tutorials of sparkling magnificence available, true works of androgogical art. Anything I had time to write would do those works a disservice.
Related names: Denoising Diffusion Probabilistic Models (DDPMs), score-based generative models, generative diffusion processes, neural energy models, denoising diffusion models, denoising diffusion probabilistic models, diffusion probabilistic models, diffusion models, denoising score matching, denoising score matching with Langevin dynamics, denoising score matching with Langevin sampling.
Suggestive connection to thermodynamics (Sohl-Dickstein et al. 2015) and indeed, statistical mechanics of learning.
1 Tutorials
- Das, Building Diffusion Model’s theory from ground up for ICLR Blogposts 2024
- Lilian Weng, What are Diffusion Models?
- Yang Song, Generative Modelling by Estimating Gradients of the Data Distribution
- Sander Dieleman, Diffusion models are autoencoders and Perspectives on diffusion
- CVPR tutorial, Denoising Diffusion-based Generative Modelling: Foundations and Applications Accompanying video
- What’s the score? (Review of latest Score Based Generative Modelling papers.)
- Anil Ananthaswamy, The Physics Principle That Inspired Modern AI Art
- The geometry of data: the missing metric tensor and the Stein score [Part II] | Terra Incognita
- Thoughts on Riemannian metrics and its connection with diffusion/score matching [Part I] | Terra Incognita
2 Sampling
Diffusion models use a score function to sample from a distribution. Unlike classic Langevin samplers, they don’t sample in the data space but construct an artificial diffusion process in a latent space, which produces samples in the data space at the end of the process. I refer to those by the inadequate shorthand of “Score diffusions” purely to disambiguate them from the many other uses of the term “diffusion” in my life. In particular, they are diffusion SDEs (although that was not obvious at the birth of this field) but of a very particular type (reversed!) and with a very particular purpose (sampling).
3 Training: score matching
Modern score matching seems to originate in Hyvärinen (2005), although the original approach was not performant. See score matching or McAllester (2023) for an intro to the general idea of learning the score of a distribution even when you cannot evaluate it. It is wild that this works.
4 Image generation in particular
5 Conditioning
There are lots of ways we might try to condition diffusions, differing sometimes only in emphasis. See neural denoising diffusion models with conditioning for a detailed discussion of the various approaches.
6 Latent
For some ideas about latent representations and their coupling with diffusion models see Multimodal AI.
6.1 Generic
7 Diffusion on weird spaces
See non-Gaussian diffusion for using diffusion models on non-Euclidean spaces.
7.1 Language
See language models for using diffusion models in NLP.
7.2 Solutions satisfying physical constraints
See PDE diffusion models for using diffusion models to generate PDE solutions.
7.3 On PD manifolds
7.4 Proteins
Baker Lab (Torres et al. 2022; Watson et al. 2022)
7.5 Shapes
8 Heavy-tailed
9 Flow matching
See neural flow matching models for a discussion of flow matching, which is a closely related approach to diffusion models.
10 References
Adam, Coogan, Malkin, et al. 2022. “Posterior Samples of Source Galaxies in Strong Gravitational Lenses with Score-Based Priors.”
Aiello, Valsesia, and Magli. 2024. “Fast Inference in Denoising Diffusion Models via MMD Finetuning.” IEEE Access.
Ajay, Du, Gupta, et al. 2023. “Is Conditional Generative Modeling All You Need for Decision-Making?” In.
Albergo, Boffi, and Vanden-Eijnden. 2023. “Stochastic Interpolants: A Unifying Framework for Flows and Diffusions.”
Albergo, Goldstein, Boffi, et al. 2023. “Stochastic Interpolants with Data-Dependent Couplings.”
Albergo, and Vanden-Eijnden. 2023. “Building Normalizing Flows with Stochastic Interpolants.” In.
Anderson. 1982. “Reverse-Time Diffusion Equation Models.” Stochastic Processes and Their Applications.
Bao, Cao, Meir, et al. 2016. “A First Order Scheme for Backward Doubly Stochastic Differential Equations.” SIAM/ASA Journal on Uncertainty Quantification.
Bao, Chipilski, Liang, et al. 2024. “Nonlinear Ensemble Filtering with Diffusion Models: Application to the Surface Quasi-Geostrophic Dynamics.”
Bao, Zhang, and Zhang. 2024. “An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems.”
Bastek, Sun, and Kochmann. 2024. “Physics-Informed Diffusion Models.” In.
Briol, Barp, Duncan, et al. 2019. “Statistical Inference for Generative Models with Maximum Mean Discrepancy.”
Choi, Kim, Jeong, et al. 2021. “ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models.” In.
Chou, Bahat, and Heide. 2023. “Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions.”
Chung, Kim, Mccann, et al. 2023. “Diffusion Posterior Sampling for General Noisy Inverse Problems.” In.
Dhariwal, and Nichol. 2021. “Diffusion Models Beat GANs on Image Synthesis.” arXiv:2105.05233 [Cs, Stat].
Dockhorn, Vahdat, and Kreis. 2022. “GENIE: Higher-Order Denoising Diffusion Solvers.” In.
Dutordoir, Saul, Ghahramani, et al. 2022. “Neural Diffusion Processes.”
Efron. 2011. “Tweedie’s Formula and Selection Bias.” Journal of the American Statistical Association.
Ezawa, Klauder, and Shepp. 1974. “A Path Space Picture for Feynman-Kac Averages.” Annals of Physics.
Ghane, Bao, Akhtiamov, et al. 2025. “Concentration of Measure for Distributions Generated via Diffusion Models.”
Gottwald, Liu, Marzouk, et al. 2025. “Localized Diffusion Models for High Dimensional Distributions Generation.”
Graikos, Malkin, Jojic, et al. 2022. “Diffusion Models as Plug-and-Play Priors.” Advances in Neural Information Processing Systems.
Grechka, Couairon, and Cord. 2024. “GradPaint: Gradient-Guided Inpainting with Diffusion Models.” Computer Vision and Image Understanding.
Guo, Liu, Wang, et al. 2024. “Diffusion Models in Bioinformatics and Computational Biology.” Nature Reviews Bioengineering.
Haitsiukevich, Poyraz, Marttinen, et al. 2024. “Diffusion Models as Probabilistic Neural Operators for Recovering Unobserved States of Dynamical Systems.”
Han, Zheng, and Zhou. 2022. “CARD: Classification and Regression Diffusion Models.”
Heng, De Bortoli, Doucet, et al. 2022. “Simulating Diffusion Bridges with Score Matching.”
Ho, Jain, and Abbeel. 2020. “Denoising Diffusion Probabilistic Models.” In Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS ’20.
Hoogeboom, Gritsenko, Bastings, et al. 2021. “Autoregressive Diffusion Models.” arXiv:2110.02037 [Cs, Stat].
Hyvärinen. 2005. “Estimation of Non-Normalized Statistical Models by Score Matching.” The Journal of Machine Learning Research.
Jalal, Arvinte, Daras, et al. 2021. “Robust Compressed Sensing MRI with Deep Generative Priors.” In Advances in Neural Information Processing Systems.
Jo, Lee, and Hwang. 2022. “Score-Based Generative Modeling of Graphs via the System of Stochastic Differential Equations.” In Proceedings of the 39th International Conference on Machine Learning.
Jolicoeur-Martineau, Piché-Taillefer, Mitliagkas, et al. 2022. “Adversarial Score Matching and Improved Sampling for Image Generation.” In.
Kawar, Elad, Ermon, et al. 2022. “Denoising Diffusion Restoration Models.” Advances in Neural Information Processing Systems.
Kawar, Vaksman, and Elad. 2021. “SNIPS: Solving Noisy Inverse Problems Stochastically.” In.
Kim, and Ye. 2021. “Noise2Score: Tweedie’s Approach to Self-Supervised Image Denoising Without Clean Images.” In.
Kita, Dubiński, Rokita, et al. 2024. “Generative Diffusion Models for Fast Simulations of Particle Collisions at CERN.”
Lipman, Chen, Ben-Hamu, et al. 2023. “Flow Matching for Generative Modeling.” In.
Liu, Ziming, Luo, Xu, et al. 2023. “GenPhys: From Physical Processes to Generative Models.”
Liu, Anji, Niepert, and Broeck. 2023. “Image Inpainting via Tractable Steering of Diffusion Models.”
Liu, Chang, Zhuo, Cheng, et al. 2019. “Understanding and Accelerating Particle-Based Variational Inference.” In Proceedings of the 36th International Conference on Machine Learning.
Li, Yu, He, et al. 2024. “SPD-DDPM: Denoising Diffusion Probabilistic Models in the Symmetric Positive Definite Space.” Proceedings of the AAAI Conference on Artificial Intelligence.
Lugmayr, Danelljan, Romero, et al. 2022. “RePaint: Inpainting Using Denoising Diffusion Probabilistic Models.” In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Mathieu, Dutordoir, Hutchinson, et al. 2023. “Geometric Neural Diffusion Processes.”
McAllester. 2023. “On the Mathematics of Diffusion Models.”
Nair, Mei, and Patel. 2023. “AT-DDPM: Restoring Faces Degraded by Atmospheric Turbulence Using Denoising Diffusion Probabilistic Models.” In.
Nichol, and Dhariwal. 2021. “Improved Denoising Diffusion Probabilistic Models.” In Proceedings of the 38th International Conference on Machine Learning.
Okhotin, Molchanov, Vladimir, et al. 2023. “Star-Shaped Denoising Diffusion Probabilistic Models.” Advances in Neural Information Processing Systems.
Pandey, Pathak, Xu, et al. 2024. “Heavy-Tailed Diffusion Models.”
Pang, Mao, He, et al. 2024. “An Improved Face Image Restoration Method Based on Denoising Diffusion Probabilistic Models.” IEEE Access.
Pascual, Bhattacharya, Yeh, et al. 2022. “Full-Band General Audio Synthesis with Score-Based Diffusion.”
Peebles, and Xie. 2023. “Scalable Diffusion Models with Transformers.” In.
Peng, Qiu, Wynne, et al. 2024. “CBCT-Based Synthetic CT Image Generation Using Conditional Denoising Diffusion Probabilistic Model.” Medical Physics.
Preechakul, Chatthee, Wizadwongsa, et al. 2022. “Diffusion Autoencoders: Toward a Meaningful and Decodable Representation.” In.
Radford, Kim, Hallacy, et al. 2021. “Learning Transferable Visual Models From Natural Language Supervision.”
Rombach, Blattmann, Lorenz, et al. 2022. “High-Resolution Image Synthesis with Latent Diffusion Models.” In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Rozet, and Louppe. 2023a. “Score-Based Data Assimilation.”
Rütte, Fluri, Ding, et al. 2025. “Generalized Interpolating Discrete Diffusion.”
Sharrock, Simons, Liu, et al. 2022. “Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Models.”
Shim, Kang, and Joo. 2023. “Diffusion-Based Signed Distance Fields for 3D Shape Generation.” In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Sohl-Dickstein, Weiss, Maheswaranathan, et al. 2015. “Deep Unsupervised Learning Using Nonequilibrium Thermodynamics.”
Song, Yang, Durkan, Murray, et al. 2021. “Maximum Likelihood Training of Score-Based Diffusion Models.” In Advances in Neural Information Processing Systems.
Song, Yang, and Ermon. 2020a. “Generative Modeling by Estimating Gradients of the Data Distribution.” In Advances In Neural Information Processing Systems.
———. 2020b. “Improved Techniques for Training Score-Based Generative Models.” In Advances In Neural Information Processing Systems.
Song, Yang, Garg, Shi, et al. 2019. “Sliced Score Matching: A Scalable Approach to Density and Score Estimation.”
Song, Jiaming, Meng, and Ermon. 2021. “Denoising Diffusion Implicit Models.” arXiv:2010.02502 [Cs].
Song, Yang, Shen, Xing, et al. 2022. “Solving Inverse Problems in Medical Imaging with Score-Based Generative Models.” In.
Song, Yang, Sohl-Dickstein, Kingma, et al. 2022. “Score-Based Generative Modeling Through Stochastic Differential Equations.” In.
Sui, Ma, Zhang, et al. 2024. “Adaptive Semantic-Enhanced Denoising Diffusion Probabilistic Model for Remote Sensing Image Super-Resolution.”
Swersky, Ranzato, Buchman, et al. 2011. “On Autoencoders and Score Matching for Energy Based Models.” In Proceedings of the 28th International Conference on Machine Learning (ICML-11).
Torres, Leung, Lutz, et al. 2022. “De Novo Design of High-Affinity Protein Binders to Bioactive Helical Peptides.”
Tzen, and Raginsky. 2019a. “Theoretical Guarantees for Sampling and Inference in Generative Models with Latent Diffusions.” In Proceedings of the Thirty-Second Conference on Learning Theory.
———. 2019b. “Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit.”
Vahdat, Kreis, and Kautz. 2021. “Score-Based Generative Modeling in Latent Space.” In.
Vargas, Grathwohl, and Doucet. 2023. “Denoising Diffusion Samplers.”
Vincent. 2011. “A connection between score matching and denoising autoencoders.” Neural Computation.
Watson, Juergens, Bennett, et al. 2022. “Broadly Applicable and Accurate Protein Design by Integrating Structure Prediction Networks and Diffusion Generative Models.”
Wu, Trippe, Naesseth, et al. 2024. “Practical and Asymptotically Exact Conditional Sampling in Diffusion Models.” In.
Xie, and Li. 2022. “Measurement-Conditioned Denoising Diffusion Probabilistic Model for Under-Sampled Medical Image Reconstruction.” In Medical Image Computing and Computer Assisted Intervention – MICCAI 2022.
Xu, Yilun, Liu, Tegmark, et al. 2022. “Poisson Flow Generative Models.” In Proceedings of the 36th International Conference on Neural Information Processing Systems. NIPS ’22.
Xu, Yilun, Liu, Tian, et al. 2023. “PFGM++: Unlocking the Potential of Physics-Inspired Generative Models.” In Proceedings of the 40th International Conference on Machine Learning. ICML’23.
Xu, Mengze, Ma, and Zhu. 2023. “Dual-Diffusion: Dual Conditional Denoising Diffusion Probabilistic Models for Blind Super-Resolution Reconstruction in RSIs.” IEEE Geoscience and Remote Sensing Letters.
Yang, Zhang, Song, et al. 2023. “Diffusion Models: A Comprehensive Survey of Methods and Applications.” ACM Computing Surveys.
Zamir, Arora, Khan, et al. 2021. “Multi-Stage Progressive Image Restoration.”
Zhang, Ji, Zhang, et al. 2023. “Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models.” In Proceedings of the 40th International Conference on Machine Learning. ICML’23.
Zhao, Bai, Zhu, et al. 2023. “DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion.” In.
Zhuang, Abnar, Gu, et al. 2022. “Diffusion Probabilistic Fields.” In.