Inverse problems

X-ray crystallography is a classic artisanal inverse problem, so having a picture related to that makes me look clever.

Robert Ackroyd introduces some handy phrasing for the connections between statistical estimation theory and inverse problem solving.

Photogrammetry, MRIs, SLAM, volumetric reconstruction and X-ray crystallography are all examples of inverse problems. any of these can be constructed as classical belief propagaation, especially Gaussian BP, or in a basic case least squares.

I happen to think that this is a case where it is much easier to explain in terms of Bayesian inference, so my attempt at an actual explanation is under Bayesian inverse problems.

We can do it in terms of frequentist methods, but it does not add much in the way of explanatory value; we end up considering regularizers instead of priors, but the working in between is pretty much the same. (IMO) However, in doing so we focus on point estimates rather than entire densities, which encourages us to solve the problem by optimisation rather than integration, which is a useful insight, for example, when we consider Laplace approximations.

Domain-specific model inversion

PEST, PEST++, and pyemu are some integrated systems for uncertainty quantification that use some weird terminology, such a FOSM (First-order-second-moment) models. They use various linear-algebra tricks to find plausible subspaces and samples.

Interesting specific techniques

Leaning to reconstruct introduces partly-learned, partly designed reconstruction operator trick. πŸ—οΈ

Radiance fields

A fun way of reconstructing objects from photos; differentiable photogrammetry.


Adler, Jonas, and Ozan Γ–ktem. 2018. β€œLearned Primal-Dual Reconstruction.” IEEE Transactions on Medical Imaging 37 (6): 1322–32.
Alberti, Giovanni S., Ernesto De Vito, Matti Lassas, Luca Ratti, and Matteo Santacesaria. 2021. β€œLearning the Optimal Regularizer for Inverse Problems.” arXiv:2106.06513 [Cs, Math, Stat], June.
Aster, Richard C., Brian Borchers, and Clifford H. Thurber. 2019. Parameter Estimation and Inverse Problems. Third. Elsevier.
Basir, Shamsulhaq, and Inanc Senocak. 2022. β€œPhysics and Equality Constrained Artificial Neural Networks: Application to Forward and Inverse Problems with Multi-Fidelity Data Fusion.” Journal of Computational Physics 463 (August): 111301.
Bissantz, Nicolai, Thorsten Hohage, and Axel Munk. 2004. β€œConsistency and Rates of Convergence of Nonlinear Tikhonov Regularization with Random Noise.” Inverse Problems 20 (6): 1773–89.
Borcea, Liliana, Vladimir Druskin, and Leonid Knizhnerman. 2005. β€œOn the Continuum Limit of a Discrete Inverse Spectral Problem on Optimal Finite Difference Grids.” Communications on Pure and Applied Mathematics 58 (9): 1231–79.
Borgerding, Mark, and Philip Schniter. 2016. β€œOnsager-Corrected Deep Networks for Sparse Linear Inverse Problems.” arXiv:1612.01183 [Cs, Math], December.
Brehmer, Johann, Gilles Louppe, Juan Pavez, and Kyle Cranmer. 2020. β€œMining Gold from Implicit Models to Improve Likelihood-Free Inference.” Proceedings of the National Academy of Sciences 117 (10): 5242–49.
Bui-Thanh, Tan. 2012. β€œA Gentle Tutorial on Statistical Inversion Using the Bayesian Paradigm.”
Chen, Yan, and Dean S. Oliver. 2013. β€œLevenberg–Marquardt Forms of the Iterative Ensemble Smoother for Efficient History Matching and Uncertainty Quantification.” Computational Geosciences 17 (4): 689–703.
Cranmer, Kyle, Johann Brehmer, and Gilles Louppe. 2020. β€œThe Frontier of Simulation-Based Inference.” Proceedings of the National Academy of Sciences, May.
Daubechies, I., M. Defrise, and C. De Mol. 2004. β€œAn Iterative Thresholding Algorithm for Linear Inverse Problems with a Sparsity Constraint.” Communications on Pure and Applied Mathematics 57 (11): 1413–57.
Engl, Heinz W., Andreas Hofinger, and Stefan Kindermann. 2005. β€œConvergence Rates in the Prokhorov Metric for Assessing Uncertainty in Ill-Posed Problems.” Inverse Problems 21 (1): 399–412.
Engl, Heinz W., and M. Zuhair Nashed. 1981. β€œGeneralized Inverses of Random Linear Operators in Banach Spaces.” Journal of Mathematical Analysis and Applications 83 (2): 582–610.
FernΓ‘ndez-MartΓ­nez, J. L., Z. FernΓ‘ndez-MuΓ±iz, J. L. G. Pallero, and L. M. Pedruelo-GonzΓ‘lez. 2013. β€œFrom Bayes to Tarantola: New Insights to Understand Uncertainty in Inverse Problems.” Journal of Applied Geophysics 98 (November): 62–72.
Grigorievskiy, Alexander, Neil Lawrence, and Simo SΓ€rkkΓ€. 2017. β€œParallelizable Sparse Inverse Formulation Gaussian Processes (SpInGP).” In arXiv:1610.08035 [Stat].
Holl, Philipp, Vladlen Koltun, and Nils Thuerey. 2022. β€œScale-Invariant Learning by Physics Inversion.” In.
Kaipio, Jari, and E. Somersalo. 2005. Statistical and Computational Inverse Problems. Applied Mathematical Sciences. New York: Springer-Verlag.
Kaipio, Jari, and Erkki Somersalo. 2007. β€œStatistical Inverse Problems: Discretization, Model Reduction and Inverse Crimes.” Journal of Computational and Applied Mathematics 198 (2): 493–504.
Lehtinen, M. S., L. Paivarinta, and E. Somersalo. 1989. β€œLinear Inverse Problems for Generalised Random Variables.” Inverse Problems 5 (4): 599–612.
Mandelbaum, Avi. 1984. β€œLinear Estimators and Measurable Linear Transformations on a Hilbert Space.” Zeitschrift FΓΌr Wahrscheinlichkeitstheorie Und Verwandte Gebiete 65 (3): 385–97.
Mosegaard, Klaus, and Albert Tarantola. 1995. β€œMonte Carlo Sampling of Solutions to Inverse Problems.” Journal of Geophysical Research: Solid Earth 100 (B7): 12431–47.
β€”β€”β€”. 2002. β€œProbabilistic Approach to Inverse Problems.” In International Geophysics, 81:237–65. Elsevier.
Murray-Smith, Roderick, and Barak A. Pearlmutter. 2005. β€œTransformations of Gaussian Process Priors.” In Deterministic and Statistical Methods in Machine Learning, edited by Joab Winkler, Mahesan Niranjan, and Neil Lawrence, 110–23. Lecture Notes in Computer Science. Springer Berlin Heidelberg.
O’Callaghan, Simon Timothy, and Fabio T. Ramos. 2011. β€œContinuous Occupancy Mapping with Integral Kernels.” In Twenty-Fifth AAAI Conference on Artificial Intelligence.
O’Sullivan, Finbarr. 1986. β€œA Statistical Perspective on Ill-Posed Inverse Problems.” Statistical Science 1 (4): 502–18.
Oliver, Dean S. 2022. β€œHybrid Iterative Ensemble Smoother for History Matching of Hierarchical Models.” Mathematical Geosciences 54 (8): 1289–1313.
Pikkarainen, Hanna Katriina. 2006. β€œState Estimation Approach to Nonstationary Inverse Problems: Discretization Error and Filtering Problem.” Inverse Problems 22 (1): 365–79.
Plumlee, Matthew. 2017. β€œBayesian Calibration of Inexact Computer Models.” Journal of the American Statistical Association 112 (519): 1274–85.
Putzky, Patrick, and Max Welling. 2017. β€œRecurrent Inference Machines for Solving Inverse Problems.” arXiv:1706.04008 [Cs], June.
Schnell, Patrick, Philipp Holl, and Nils Thuerey. 2022. β€œHalf-Inverse Gradients for Physical Deep Learning.” arXiv:2203.10131 [Physics], March.
Schwab, C., and A. M. Stuart. 2012. β€œSparse Deterministic Approximation of Bayesian Inverse Problems.” Inverse Problems 28 (4): 045003.
Stuart, Andrew M. 2010. β€œInverse Problems: A Bayesian Perspective.” Acta Numerica 19: 451–559.
Sun, Alexander Y., Bridget R. Scanlon, Himanshu Save, and Ashraf Rateb. 2021. β€œReconstruction of GRACE Total Water Storage Through Automated Machine Learning.” Water Resources Research 57 (2): e2020WR028666.
Tait, Daniel J., and Theodoros Damoulas. 2020. β€œVariational Autoencoding of PDE Inverse Problems.” arXiv:2006.15641 [Cs, Stat], June.
Tarantola, Albert. 2005. Inverse Problem Theory and Methods for Model Parameter Estimation. SIAM.
Tonolini, Francesco, Jack Radford, Alex Turpin, Daniele Faccio, and Roderick Murray-Smith. 2020. β€œVariational Inference for Computational Imaging Inverse Problems.” Journal of Machine Learning Research 21 (179): 1–46.
Tropp, J. A., and S. J. Wright. 2010. β€œComputational Methods for Sparse Solution of Linear Inverse Problems.” Proceedings of the IEEE 98 (6): 948–58.
Wei, Qi, Kai Fan, Lawrence Carin, and Katherine A. Heller. 2017. β€œAn Inner-Loop Free Solution to Inverse Problems Using Deep Neural Networks.” arXiv:1709.01841 [Cs], September.
Welter, David E., John E. Doherty, Randall J. Hunt, Christopher T. Muffels, Matthew J. Tonkin, and Willem A. SchreΓΌder. 2012. β€œApproaches in Highly Parameterized Inversion: PEST++, a Parameter Estimation Code Optimized for Large Environmental Models.”
Welter, David E., Jeremy T. White, Randall J. Hunt, and John E. Doherty. 2015. β€œApproaches in Highly Parameterized Inversionβ€”PEST++ Version 3, a Parameter ESTimation and Uncertainty Analysis Software Suite Optimized for Large Environmental Models.” USGS Numbered Series 7-C12. Techniques and Methods. Reston, VA: U.S. Geological Survey.
White, Jeremy T. 2018. β€œA Model-Independent Iterative Ensemble Smoother for Efficient History-Matching and Uncertainty Quantification in Very High Dimensions.” Environmental Modelling & Software 109 (November): 191–201.
White, Jeremy T., Michael N. Fienen, Paul M. Barlow, and Dave E. Welter. 2018. β€œA Tool for Efficient, Model-Independent Management Optimization Under Uncertainty.” Environmental Modelling & Software 100 (February): 213–21.
White, Jeremy T., Michael N. Fienen, and John E. Doherty. 2016a. β€œpyEMU: A Python Framework for Environmental Model Uncertainty Analysis Version .01.” U.S. Geological Survey.
β€”β€”β€”. 2016b. β€œA Python Framework for Environmental Model Uncertainty Analysis.” Environmental Modelling & Software 85 (November): 217–28.
White, Jeremy T., Randall J. Hunt, Michael N. Fienen, and John E. Doherty. 2020. β€œApproaches to Highly Parameterized Inversion: PEST++ Version 5, a Software Suite for Parameter Estimation, Uncertainty Analysis, Management Optimization and Sensitivity Analysis.” USGS Numbered Series 7-C26. Techniques and Methods. Reston, VA: U.S. Geological Survey.
Zammit-Mangion, Andrew, Michael Bertolacci, Jenny Fisher, Ann Stavert, Matthew L. Rigby, Yi Cao, and Noel Cressie. 2021. β€œWOMBAT v1.0: A fully Bayesian global flux-inversion framework.” Geoscientific Model Development Discussions, July, 1–51.
Zhang, Dongkun, Lu Lu, Ling Guo, and George Em Karniadakis. 2019. β€œQuantifying Total Uncertainty in Physics-Informed Neural Networks for Solving Forward and Inverse Stochastic Problems.” Journal of Computational Physics 397 (November): 108850.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.