Running neural nets backwards

January 29, 2022 — January 29, 2022

dynamical systems

linear algebra

machine learning

neural nets

optimization

regression

signal processing

sparser than thou

statmech

stochastic processes

TODO. Various techniques in “running networks backwards”, definition later.

Prior regularisations for ill-posed inverse problems.

Deep dreams probably fit here (Mahendran and Vedaldi 2015; Simonyan, Vedaldi, and Zisserman 2014; Yosinski et al. 2015). For more on that see Inceptionism: Going Deeper into Neural Networks.

I wonder if the reversible architectures (Chang et al. 2018) method gets us anything here?

My focus is more on inverse problems.

Video

Figure 2: Let us run it backwards.

1 References

Adler, and Öktem. 2018. “Learned Primal-Dual Reconstruction.” IEEE Transactions on Medical Imaging.

Anantha Padmanabha, and Zabaras. 2021. “Solving Inverse Problems Using Conditional Invertible Neural Networks.” Journal of Computational Physics.

Ardizzone, Kruse, Wirkert, et al. 2019. “Analyzing Inverse Problems with Invertible Neural Networks.” arXiv:1808.04730 [Cs, Stat].

Bao, Ye, Zang, et al. 2020. “Numerical Solution of Inverse Problems by Weak Adversarial Networks.” Inverse Problems.

Borgerding, and Schniter. 2016. “Onsager-Corrected Deep Networks for Sparse Linear Inverse Problems.” arXiv:1612.01183 [Cs, Math].

Chang, Meng, Haber, et al. 2018. “Reversible Architectures for Arbitrarily Deep Residual Neural Networks.” In arXiv:1709.03698 [Cs, Stat].

Depina, Jain, Mar Valsson, et al. 2021. “Application of Physics-Informed Neural Networks to Inverse Problems in Unsaturated Groundwater Flow.” Georisk: Assessment and Management of Risk for Engineered Systems and Geohazards.

Franklin. 1970. “Well-Posed Stochastic Extensions of Ill-Posed Linear Problems.” Journal of Mathematical Analysis and Applications.

Genzel, Macdonald, and März. 2020. “Solving Inverse Problems With Deep Neural Networks — Robustness Included?” arXiv:2011.04268 [Cs, Math].

Haber, and Ruthotto. 2018. “Stable Architectures for Deep Neural Networks.” Inverse Problems.

Holl, Koltun, Um, et al. 2020. “Phiflow: A Differentiable PDE Solving Framework for Deep Learning via Physical Simulations.” In NeurIPS Workshop.

Holl, Thuerey, and Koltun. 2020. “Learning to Control PDEs with Differentiable Physics.” In ICLR.

Jo, Son, Hwang, et al. 2020. “Deep Neural Network Approach to Forward-Inverse Problems.” Networks & Heterogeneous Media.

Kawar, Vaksman, and Elad. 2021. “SNIPS: Solving Noisy Inverse Problems Stochastically.” In.

Khodayi-Mehr, and Zavlanos. 2019. “VarNet: Variational Neural Networks for the Solution of Partial Differential Equations.” arXiv:1912.07443 [Physics, Stat].

Li, Schwab, Antholzer, et al. 2020. “NETT: Solving Inverse Problems with Deep Neural Networks.” Inverse Problems.

Lucas, Iliadis, Molina, et al. 2018. “Using Deep Neural Networks for Inverse Problems in Imaging: Beyond Analytical Methods.” IEEE Signal Processing Magazine.

Lu, Meng, Mao, et al. 2021. “DeepXDE: A Deep Learning Library for Solving Differential Equations.” SIAM Review.

Lunz, Schoenlieb, and Öktem. 2018. “Adversarial Regularizers in Inverse Problems.” In.

Mahendran, and Vedaldi. 2015. “Understanding Deep Image Representations by Inverting Them.” In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Mowlavi, and Nabi. 2021. “Optimal Control of PDEs Using Physics-Informed Neural Networks.” arXiv:2111.09880 [Physics].

Ogawa, Kosugi, and Kanada. 1998. “Neural Network Based Solution to Inverse Problems.” In 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

Rackauckas, Ma, Dixit, et al. 2018. “A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions.” arXiv:1812.01892 [Cs].

Simonyan, Vedaldi, and Zisserman. 2014. “Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps.” arXiv:1312.6034 [Cs].

Spantini, Solonen, Cui, et al. 2015. “Optimal Low-Rank Approximations of Bayesian Linear Inverse Problems.” SIAM Journal on Scientific Computing.

Tait, and Damoulas. 2020. “Variational Autoencoding of PDE Inverse Problems.” arXiv:2006.15641 [Cs, Stat].

Wei, Fan, Carin, et al. 2017. “An Inner-Loop Free Solution to Inverse Problems Using Deep Neural Networks.” arXiv:1709.01841 [Cs].

Williams, Klanke, Vijayakumar, et al. 2009. “Multi-Task Gaussian Process Learning of Robot Inverse Dynamics.” In Advances in Neural Information Processing Systems 21.

Yosinski, Clune, Nguyen, et al. 2015. “Understanding Neural Networks Through Deep Visualization.” arXiv:1506.06579 [Cs].

Zhang, Guo, and Karniadakis. 2020. “Learning in Modal Space: Solving Time-Dependent Stochastic PDEs Using Physics-Informed Neural Networks.” SIAM Journal on Scientific Computing.