Neural rendering

2021-01-21 — 2023-04-26

Wherein Gaussian splatting and neural radiance fields are surveyed, and implicit neural functions are employed to represent 3D scenes as coordinate-to-color-and-density maps, and practical reconstruction from images is discussed.

dynamical systems
functional analysis
machine learning
neural nets

Using NNs (usually implicit style it seems?) to make graphics.

1 Gaussian splatting

2 Neural Radiance fields

What is a radiance field? Try Nathan Reed’s introduction, The Radiance Field. Cool, it looks like it is a way of representing a 3D scene as a function of 3D coordinates that can be conveniently represented for rendering, so a neural SDF+rendering thing somehow? I’ve also heard it claimed that solving the alignment and registration proble is part of the same popline. Example: Sutro Tower in 3D

OK, how does THAT get neural?

See Dupont et al. (2022) and Mildenhall et al. (2020).

Aras: Gaussian Splatting is pretty cool!

3 References

Bautista, Guo, Abnar, et al. 2022. “GAUDI: A Neural Architect for Immersive 3D Scene Generation.” In.
Dupont, Kim, Eslami, et al. 2022. “From Data to Functa: Your Data Point Is a Function and You Can Treat It Like One.” In Proceedings of the 39th International Conference on Machine Learning.
Lange, and Kutz. 2021. “FC2T2: The Fast Continuous Convolutional Taylor Transform with Applications in Vision and Graphics.” arXiv:2111.00110 [Cs].
Mildenhall, Srinivasan, Tancik, et al. 2020. “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.” arXiv:2003.08934 [Cs].
Takikawa, Litalien, Yin, et al. 2021. “Neural Geometric Level of Detail: Real-Time Rendering with Implicit 3D Shapes.”
Yariv, Gu, Kasten, et al. 2021. “Volume Rendering of Neural Implicit Surfaces.” In.
Zammit-Mangion, Ng, Vu, et al. 2021. “Deep Compositional Spatial Models.” Journal of the American Statistical Association.
Zhu, Peng, Larsson, et al. 2022. “NICE-SLAM: Neural Implicit Scalable Encoding for SLAM.”