# Inverse problems in function space

## a.k.a. Bayesian calibration, model uncertainty for PDEs and other wibbly surfaces

Inverse problems where the model parameter is in some function space. For me this usually implies a spatiotemporal model, usually in the context of PDE solvers, particularly approximate ones.

Inverse problems arise naturally in tomography, compressed sensing, deconvolution, inverting PDEs and many other areas.

Suppose I have a PDE, possibly with some unknown parameters in the driving equation. All being equal I can do not too badly at approximating that with tools already mentioned. What if I wish to simultaneously infer some unknown inputs? Then we consider it as an inverse problem. This is not quite the same as the predictive problem that many of the methods consider. However, we are free to use simulation-based inference to solve, or MCMC methods to do so for any of the forward-operator-learning approaches. To train the model to solve the inverse problem directly, we might consider GANs or variational inference. At this point we are more or less required to start using a probabilistic network or we will miss essential uncertainty quantification.

We might also be inclined to use approximate methods. We are surprised to find anything like a clean closed-form solution for the posterior distribution of some parameter in a PDE. Why would it?

As for how we might proceed, Liu, Yeo, and Lu (2020) is one approach, generalizing the approach of F. Sigrist, Künsch, and Stahel (2015b), but for advection/diffusion equations specifically. Generic methods include Bao et al. (2020); Jo et al. (2020); Lu, Mao, and Meng (2019); Raissi, Perdikaris, and Karniadakis (2019); Tait and Damoulas (2020); Xu and Darve (2020); Yang, Zhang, and Karniadakis (2020); Zhang, Guo, and Karniadakis (2020); Zhang et al. (2019).

## Bayesian nonparametrics

Since this kind of problem naturally invites functional parameters, we are in the world of Bayesian nonparametrics, which has a slightly different notation than you usually see in Bayes textbooks. I suspect that there is a useful role for various Bayesian nonparametrics here, but the easiest of all is Gaussian process, which I handle next:

## Gaussian process parameters

Alexanderian (2021) states a ‘well-known’ result, that the solution of a Bayesian linear inverse problem with Gaussian prior and noise models is a Gaussian posterior $$\mu_{\text {post }}^{y}=\mathcal{N}\left(m_{\text {MAP }}, \mathcal{C}_{\text {post }}\right)$$, where $\mathcal{C}_{\text {post }}=\left(\mathcal{F}^{*} \boldsymbol{\Gamma}_{\text {noise }}^{-1} \mathcal{F}+\mathcal{C}_{\text {pr }}^{-1}\right)^{-1} \quad \text { and } \quad m_{\text {MAP }}=\mathcal{C}_{\text {post }}\left(\mathcal{F}^{*} \boldsymbol{\Gamma}_{\text {noise }}^{-1} \boldsymbol{y}+\mathcal{C}_{\text {pr }}^{-1} m_{\text {MAP }}\right).$

## Finite Element Models

Finite Element Models of PDEs of PDEs (annd possibly other representation?) of PDES can be expressed as locally-linear constraints and thus expressed using Gaussian Belief Propagation . Note that in this setting, there is nothing special about the inversion process. Inference proceeds the same either way, as a variational message passing algorithm.

## References

Alexanderian, Alen. 2021. arXiv:2005.12998 [Math], January.
Bao, Gang, Xiaojing Ye, Yaohua Zang, and Haomin Zhou. 2020. Inverse Problems 36 (11): 115003.
Brehmer, Johann, Gilles Louppe, Juan Pavez, and Kyle Cranmer. 2020. Proceedings of the National Academy of Sciences 117 (10): 5242–49.
Bui-Thanh, Tan, Omar Ghattas, James Martin, and Georg Stadler. 2013. SIAM Journal on Scientific Computing 35 (6): A2494–2523.
Bui-Thanh, Tan, and Quoc P. Nguyen. 2016. Inverse Problems & Imaging 10 (4): 943.
Cotter, S. L., M. Dashti, and A. M. Stuart. 2010. SIAM Journal on Numerical Analysis 48 (1): 322–45.
Cox, Dennis D. 1993. The Annals of Statistics 21 (2): 903–23.
Cranmer, Kyle, Johann Brehmer, and Gilles Louppe. 2020. Proceedings of the National Academy of Sciences, May.
Dashti, Masoumeh, and Andrew M. Stuart. 2015. arXiv:1302.6989 [Math], July.
El-Kurdi, Yousef Malek. 2014. “Parallel Finite Element Processing Using Gaussian Belief Propagation Inference on Probabilistic Graphical Models.” PhD Thesis, McGill University.
El-Kurdi, Yousef, Maryam Mehri Dehnavi, Warren J. Gross, and Dennis Giannacopoulos. 2015. Computer Physics Communications 193 (August): 38–48.
El-Kurdi, Yousef, David Fernandez, Warren J. Gross, and Dennis D. Giannacopoulos. 2016. IEEE Transactions on Magnetics 52 (3): 1–4.
Florens, Jean-Pierre, and Anna Simoni. 2016. Econometric Theory 32 (1): 71–121.
Franklin, Joel N. 1970. Journal of Mathematical Analysis and Applications 31 (3): 682–716.
Grigorievskiy, Alexander, Neil Lawrence, and Simo Särkkä. 2017. In arXiv:1610.08035 [Stat].
Jo, Hyeontae, Hwijae Son, Hyung Ju Hwang, and Eun Heui Kim. 2020. Networks & Heterogeneous Media 15 (2): 247.
Kaipio, Jari, and E. Somersalo. 2005. Statistical and Computational Inverse Problems. Applied Mathematical Sciences. New York: Springer-Verlag.
Kennedy, Marc C., and Anthony O’Hagan. 2001. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63 (3): 425–64.
Knapik, B. T., A. W. van der Vaart, and J. H. van Zanten. 2011. The Annals of Statistics 39 (5).
Lasanen, Sari. 2012. Inverse Problems & Imaging 6 (2): 267.
Lassas, Matti, Eero Saksman, and Samuli Siltanen. 2009. Inverse Problems and Imaging 3 (1): 87–122.
Liu, Xiao, Kyongmin Yeo, and Siyuan Lu. 2020. Journal of the American Statistical Association 0 (0): 1–18.
Lu, Lu, Pengzhan Jin, and George Em Karniadakis. 2020. arXiv:1910.03193 [Cs, Stat], April.
Lu, Lu, Zhiping Mao, and Xuhui Meng. 2019. In, 6.
Lu, Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. 2021. SIAM Review 63 (1): 208–28.
Margossian, Charles C., Aki Vehtari, Daniel Simpson, and Raj Agrawal. 2020. arXiv:2004.12550 [Stat], October.
Mosegaard, Klaus, and Albert Tarantola. 2002. In International Geophysics, 81:237–65. Elsevier.
O’Hagan, A. 2006. Reliability Engineering & System Safety, The Fourth International Conference on Sensitivity Analysis of Model Output (SAMO 2004), 91 (10): 1290–300.
Perdikaris, Paris, and George Em Karniadakis. 2016. Journal of the Royal Society, Interface 13 (118): 20151107.
Petra, Noemi, James Martin, Georg Stadler, and Omar Ghattas. 2014. SIAM Journal on Scientific Computing 36 (4): A1525–55.
Plumlee, Matthew. 2017. Journal of the American Statistical Association 112 (519): 1274–85.
Raissi, Maziar, Paris Perdikaris, and George Em Karniadakis. 2017a. November.
Raissi, Maziar, P. Perdikaris, and George Em Karniadakis. 2019. Journal of Computational Physics 378 (February): 686–707.
Roininen, Lassi, Janne M. J. Huttunen, and Sari Lasanen. 2014. Inverse Problems & Imaging 8 (2): 561.
Särkkä, Simo, A. Solin, and J. Hartikainen. 2013. IEEE Signal Processing Magazine 30 (4): 51–61.
Sigrist, Fabio Roman Albert. 2013. Application/pdf. ETH Zurich.
Sigrist, Fabio, Hans R. Künsch, and Werner A. Stahel. 2015a. Application/pdf. Journal of Statistical Software 63 (14).
———. 2015b. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 77 (1): 3–33.
Stuart, A. M. 2010. Acta Numerica 19: 451–559.
Stuart, Andrew M., and Aretha L. Teckentrup. 2016. arXiv:1603.02004 [Math], December.
Tait, Daniel J., and Theodoros Damoulas. 2020. arXiv:2006.15641 [Cs, Stat], June.
Tarantola, Albert. 2005. Inverse Problem Theory and Methods for Model Parameter Estimation. SIAM.
———. n.d. Mapping Of Probabilities.
Teckentrup, Aretha L. 2020. arXiv:1909.00232 [Cs, Math, Stat], July.
Welter, David E., Jeremy T. White, Randall J. Hunt, and John E. Doherty. 2015. USGS Numbered Series 7-C12. Techniques and Methods. Reston, VA: U.S. Geological Survey.
White, Jeremy T., Michael N. Fienen, and John E. Doherty. 2016a. pyEMU: A Python Framework for Environmental Model Uncertainty Analysis Version .01. U.S. Geological Survey.
———. 2016b. Environmental Modelling & Software 85 (November): 217–28.
Xu, Kailai, and Eric Darve. 2020. In arXiv:2011.11955 [Cs, Math].
Yang, Liu, Dongkun Zhang, and George Em Karniadakis. 2020. SIAM Journal on Scientific Computing 42 (1): A292–317.
Zammit-Mangion, Andrew, Michael Bertolacci, Jenny Fisher, Ann Stavert, Matthew L. Rigby, Yi Cao, and Noel Cressie. 2021. Geoscientific Model Development Discussions, July, 1–51.
Zhang, Dongkun, Ling Guo, and George Em Karniadakis. 2020. SIAM Journal on Scientific Computing 42 (2): A639–65.
Zhang, Dongkun, Lu Lu, Ling Guo, and George Em Karniadakis. 2019. Journal of Computational Physics 397 (November): 108850.

### No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.