Inverse problems in function space

a.k.a. Bayesian calibration, model uncertainty for PDEs and other wibbly surfaces

Inverse problems where the model parameter is in some function space. For me this usually implies a spatiotemporal model, usually in the context of PDE solvers, particularly approximate ones.

Inverse problems arise naturally in tomography, compressed sensing, deconvolution, inverting PDEs and many other areas.

Suppose I have a PDE, possibly with some unknown parameters in the driving equation. All being equal I can do not too badly at approximating that with tools already mentioned. What if I wish to simultaneously infer some unknown inputs? Then we consider it as an inverse problem. This is not quite the same as the predictive problem that many of the methods consider. However, we are free to use simulation-based inference to solve, or MCMC methods to do so for any of the forward-operator-learning approaches. To train the model to solve the inverse problem directly, we might consider GANs or variational inference. At this point we are more or less required to start using a probabilistic network or we will miss essential uncertainty quantification.

We might also be inclined to use approximate methods. We are surprised to find anything like a clean closed-form solution for the posterior distribution of some parameter in a PDE. Why would it?

As for how we might proceed, Liu, Yeo, and Lu (2020) is one approach, generalizing the approach of F. Sigrist, KΓΌnsch, and Stahel (2015b), but for advection/diffusion equations specifically. Generic methods include Bao et al. (2020); Jo et al. (2020); Lu, Mao, and Meng (2019); Raissi, Perdikaris, and Karniadakis (2019); Tait and Damoulas (2020); Xu and Darve (2020); Yang, Zhang, and Karniadakis (2020); Zhang, Guo, and Karniadakis (2020); Zhang et al. (2019).

Bayesian nonparametrics

Since this kind of problem naturally invites functional parameters, we are in the world of Bayesian nonparametrics, which has a slightly different notation than you usually see in Bayes textbooks. I suspect that there is a useful role for various Bayesian nonparametrics here, but the easiest of all is Gaussian process, which I handle next:

Gaussian process parameters

Alexanderian (2021) states a β€˜well-known’ result, that the solution of a Bayesian linear inverse problem with Gaussian prior and noise models is a Gaussian posterior \(\mu_{\text {post }}^{y}=\mathcal{N}\left(m_{\text {MAP }}, \mathcal{C}_{\text {post }}\right)\), where \[ \mathcal{C}_{\text {post }}=\left(\mathcal{F}^{*} \boldsymbol{\Gamma}_{\text {noise }}^{-1} \mathcal{F}+\mathcal{C}_{\text {pr }}^{-1}\right)^{-1} \quad \text { and } \quad m_{\text {MAP }}=\mathcal{C}_{\text {post }}\left(\mathcal{F}^{*} \boldsymbol{\Gamma}_{\text {noise }}^{-1} \boldsymbol{y}+\mathcal{C}_{\text {pr }}^{-1} m_{\text {MAP }}\right). \]

Finite Element Models

Finite Element Models of PDEs of PDEs (annd possibly other representation?) of PDES can be expressed as locally-linear constraints and thus expressed using Gaussian Belief Propagation (Y. El-Kurdi et al. 2016; Y. M. El-Kurdi 2014; Y. El-Kurdi et al. 2015). Note that in this setting, there is nothing special about the inversion process. Inference proceeds the same either way, as a variational message passing algorithm.


Alexanderian, Alen. 2021. β€œOptimal Experimental Design for Infinite-Dimensional Bayesian Inverse Problems Governed by PDEs: A Review.” arXiv:2005.12998 [Math], January.
Bao, Gang, Xiaojing Ye, Yaohua Zang, and Haomin Zhou. 2020. β€œNumerical Solution of Inverse Problems by Weak Adversarial Networks.” Inverse Problems 36 (11): 115003.
Brehmer, Johann, Gilles Louppe, Juan Pavez, and Kyle Cranmer. 2020. β€œMining Gold from Implicit Models to Improve Likelihood-Free Inference.” Proceedings of the National Academy of Sciences 117 (10): 5242–49.
Bui-Thanh, Tan, Omar Ghattas, James Martin, and Georg Stadler. 2013. β€œA Computational Framework for Infinite-Dimensional Bayesian Inverse Problems Part I: The Linearized Case, with Application to Global Seismic Inversion.” SIAM Journal on Scientific Computing 35 (6): A2494–2523.
Bui-Thanh, Tan, and Quoc P. Nguyen. 2016. β€œFEM-Based Discretization-Invariant MCMC Methods for PDE-Constrained Bayesian Inverse Problems.” Inverse Problems & Imaging 10 (4): 943.
Cotter, S. L., M. Dashti, and A. M. Stuart. 2010. β€œApproximation of Bayesian Inverse Problems for PDEs.” SIAM Journal on Numerical Analysis 48 (1): 322–45.
Cox, Dennis D. 1993. β€œAn Analysis of Bayesian Inference for Nonparametric Regression.” The Annals of Statistics 21 (2): 903–23.
Cranmer, Kyle, Johann Brehmer, and Gilles Louppe. 2020. β€œThe Frontier of Simulation-Based Inference.” Proceedings of the National Academy of Sciences, May.
Dashti, Masoumeh, and Andrew M. Stuart. 2015. β€œThe Bayesian Approach To Inverse Problems.” arXiv:1302.6989 [Math], July.
El-Kurdi, Yousef Malek. 2014. β€œParallel Finite Element Processing Using Gaussian Belief Propagation Inference on Probabilistic Graphical Models.” PhD Thesis, McGill University.
El-Kurdi, Yousef, Maryam Mehri Dehnavi, Warren J. Gross, and Dennis Giannacopoulos. 2015. β€œParallel Finite Element Technique Using Gaussian Belief Propagation.” Computer Physics Communications 193 (August): 38–48.
El-Kurdi, Yousef, David Fernandez, Warren J. Gross, and Dennis D. Giannacopoulos. 2016. β€œAcceleration of the Finite-Element Gaussian Belief Propagation Solver Using Minimum Residual Techniques.” IEEE Transactions on Magnetics 52 (3): 1–4.
Florens, Jean-Pierre, and Anna Simoni. 2016. β€œRegularizing Priors for Linear Inverse Problems.” Econometric Theory 32 (1): 71–121.
Franklin, Joel N. 1970. β€œWell-Posed Stochastic Extensions of Ill-Posed Linear Problems.” Journal of Mathematical Analysis and Applications 31 (3): 682–716.
Grigorievskiy, Alexander, Neil Lawrence, and Simo SΓ€rkkΓ€. 2017. β€œParallelizable Sparse Inverse Formulation Gaussian Processes (SpInGP).” In arXiv:1610.08035 [Stat].
Jo, Hyeontae, Hwijae Son, Hyung Ju Hwang, and Eun Heui Kim. 2020. β€œDeep Neural Network Approach to Forward-Inverse Problems.” Networks & Heterogeneous Media 15 (2): 247.
Kaipio, Jari, and E. Somersalo. 2005. Statistical and Computational Inverse Problems. Applied Mathematical Sciences. New York: Springer-Verlag.
Kennedy, Marc C., and Anthony O’Hagan. 2001. β€œBayesian Calibration of Computer Models.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63 (3): 425–64.
Knapik, B. T., A. W. van der Vaart, and J. H. van Zanten. 2011. β€œBayesian Inverse Problems with Gaussian Priors.” The Annals of Statistics 39 (5).
Lasanen, Sari. 2012. β€œNon-Gaussian Statistical Inverse Problems. Part II: Posterior Convergence for Approximated Unknowns.” Inverse Problems & Imaging 6 (2): 267.
Lassas, Matti, Eero Saksman, and Samuli Siltanen. 2009. β€œDiscretization-Invariant Bayesian Inversion and Besov Space Priors.” Inverse Problems and Imaging 3 (1): 87–122.
Liu, Xiao, Kyongmin Yeo, and Siyuan Lu. 2020. β€œStatistical Modeling for Spatio-Temporal Data From Stochastic Convection-Diffusion Processes.” Journal of the American Statistical Association 0 (0): 1–18.
Lu, Lu, Pengzhan Jin, and George Em Karniadakis. 2020. β€œDeepONet: Learning Nonlinear Operators for Identifying Differential Equations Based on the Universal Approximation Theorem of Operators.” arXiv:1910.03193 [Cs, Stat], April.
Lu, Lu, Zhiping Mao, and Xuhui Meng. 2019. β€œDeepXDE: A Deep Learning Library for Solving Differential Equations.” In, 6.
Lu, Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. 2021. β€œDeepXDE: A Deep Learning Library for Solving Differential Equations.” SIAM Review 63 (1): 208–28.
Margossian, Charles C., Aki Vehtari, Daniel Simpson, and Raj Agrawal. 2020. β€œHamiltonian Monte Carlo Using an Adjoint-Differentiated Laplace Approximation: Bayesian Inference for Latent Gaussian Models and Beyond.” arXiv:2004.12550 [Stat], October.
Mosegaard, Klaus, and Albert Tarantola. 2002. β€œProbabilistic Approach to Inverse Problems.” In International Geophysics, 81:237–65. Elsevier.
O’Hagan, A. 2006. β€œBayesian Analysis of Computer Code Outputs: A Tutorial.” Reliability Engineering & System Safety, The Fourth International Conference on Sensitivity Analysis of Model Output (SAMO 2004), 91 (10): 1290–300.
Perdikaris, Paris, and George Em Karniadakis. 2016. β€œModel inversion via multi-fidelity Bayesian optimization: a new paradigm for parameter estimation in haemodynamics, and beyond.” Journal of the Royal Society, Interface 13 (118): 20151107.
Petra, Noemi, James Martin, Georg Stadler, and Omar Ghattas. 2014. β€œA Computational Framework for Infinite-Dimensional Bayesian Inverse Problems, Part II: Stochastic Newton MCMC with Application to Ice Sheet Flow Inverse Problems.” SIAM Journal on Scientific Computing 36 (4): A1525–55.
Plumlee, Matthew. 2017. β€œBayesian Calibration of Inexact Computer Models.” Journal of the American Statistical Association 112 (519): 1274–85.
Raissi, Maziar, Paris Perdikaris, and George Em Karniadakis. 2017a. β€œPhysics Informed Deep Learning (Part I): Data-Driven Solutions of Nonlinear Partial Differential Equations,” November.
β€”β€”β€”. 2017b. β€œPhysics Informed Deep Learning (Part II): Data-Driven Discovery of Nonlinear Partial Differential Equations,” November.
Raissi, Maziar, P. Perdikaris, and George Em Karniadakis. 2019. β€œPhysics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations.” Journal of Computational Physics 378 (February): 686–707.
Roininen, Lassi, Janne M. J. Huttunen, and Sari Lasanen. 2014. β€œWhittle-MatΓ©rn Priors for Bayesian Statistical Inversion with Applications in Electrical Impedance Tomography.” Inverse Problems & Imaging 8 (2): 561.
SΓ€rkkΓ€, Simo, A. Solin, and J. Hartikainen. 2013. β€œSpatiotemporal Learning via Infinite-Dimensional Bayesian Filtering and Smoothing: A Look at Gaussian Process Regression Through Kalman Filtering.” IEEE Signal Processing Magazine 30 (4): 51–61.
Sigrist, Fabio Roman Albert. 2013. β€œPhysics Based Dynamic Modeling of Space-Time Data.” Application/pdf. ETH Zurich.
Sigrist, Fabio, Hans R. KΓΌnsch, and Werner A. Stahel. 2015a. β€œSpate : An R Package for Spatio-Temporal Modeling with a Stochastic Advection-Diffusion Process.” Application/pdf. Journal of Statistical Software 63 (14).
β€”β€”β€”. 2015b. β€œStochastic Partial Differential Equation Based Modelling of Large Space-Time Data Sets.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 77 (1): 3–33.
Stuart, A. M. 2010. β€œInverse Problems: A Bayesian Perspective.” Acta Numerica 19: 451–559.
Stuart, Andrew M., and Aretha L. Teckentrup. 2016. β€œPosterior Consistency for Gaussian Process Approximations of Bayesian Posterior Distributions.” arXiv:1603.02004 [Math], December.
Tait, Daniel J., and Theodoros Damoulas. 2020. β€œVariational Autoencoding of PDE Inverse Problems.” arXiv:2006.15641 [Cs, Stat], June.
Tarantola, Albert. 2005. Inverse Problem Theory and Methods for Model Parameter Estimation. SIAM.
β€”β€”β€”. n.d. Mapping Of Probabilities.
Teckentrup, Aretha L. 2020. β€œConvergence of Gaussian Process Regression with Estimated Hyper-Parameters and Applications in Bayesian Inverse Problems.” arXiv:1909.00232 [Cs, Math, Stat], July.
Welter, David E., Jeremy T. White, Randall J. Hunt, and John E. Doherty. 2015. β€œApproaches in Highly Parameterized Inversionβ€”PEST++ Version 3, a Parameter ESTimation and Uncertainty Analysis Software Suite Optimized for Large Environmental Models.” USGS Numbered Series 7-C12. Techniques and Methods. Reston, VA: U.S. Geological Survey.
White, Jeremy T., Michael N. Fienen, and John E. Doherty. 2016a. pyEMU: A Python Framework for Environmental Model Uncertainty Analysis Version .01. U.S. Geological Survey.
β€”β€”β€”. 2016b. β€œA Python Framework for Environmental Model Uncertainty Analysis.” Environmental Modelling & Software 85 (November): 217–28.
Xu, Kailai, and Eric Darve. 2020. β€œADCME: Learning Spatially-Varying Physical Fields Using Deep Neural Networks.” In arXiv:2011.11955 [Cs, Math].
Yang, Liu, Dongkun Zhang, and George Em Karniadakis. 2020. β€œPhysics-Informed Generative Adversarial Networks for Stochastic Differential Equations.” SIAM Journal on Scientific Computing 42 (1): A292–317.
Zammit-Mangion, Andrew, Michael Bertolacci, Jenny Fisher, Ann Stavert, Matthew L. Rigby, Yi Cao, and Noel Cressie. 2021. β€œWOMBAT v1.0: A fully Bayesian global flux-inversion framework.” Geoscientific Model Development Discussions, July, 1–51.
Zhang, Dongkun, Ling Guo, and George Em Karniadakis. 2020. β€œLearning in Modal Space: Solving Time-Dependent Stochastic PDEs Using Physics-Informed Neural Networks.” SIAM Journal on Scientific Computing 42 (2): A639–65.
Zhang, Dongkun, Lu Lu, Ling Guo, and George Em Karniadakis. 2019. β€œQuantifying Total Uncertainty in Physics-Informed Neural Networks for Solving Forward and Inverse Stochastic Problems.” Journal of Computational Physics 397 (November): 108850.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.