Neural nets for “implicit representations”



A cute hack for generative neural nets. Unlike other structures, here we allow the output to depend upon image coordinates, rather than some presumed-invariant latent factors. I am not quite sure what the rational is for implicit being used as a term here. It is a terrible choice. What representations are implicit or explicit are particularly viewpoint-dependent. About the only way to make it worse would be to call them unnamed networks.

To make it worse, implicit representation networks are not the only ones using implicit in a confusing away. See also “implicit layers”, which allows an optimisation problem to be implicitly solved in a neural net.

Awesome implicit representations:

Implicit Neural Representations (sometimes also referred to coordinate-based representations) are a novel way to parameterize signals of all kinds. Conventional signal representations are usually discrete — for instance, images are discrete grids of pixels, audio signals are discrete samples of amplitudes, and 3D shapes are usually parameterized as grids of voxels, point clouds, or meshes. In contrast, Implicit Neural Representations parameterize a signal as a continuous function that maps the domain of the signal (i.e., a coordinate, such as a pixel coordinate for an image) to whatever is at that coordinate (for an image, an R,G,B color). Of course, these functions are usually not analytically tractable — it is impossible to “write down” the function that parameterizes a natural image as a mathematical formula. Implicit Neural Representations thus approximate that function via a neural network.

Implicit Neural Representations have several benefits: First, they are not coupled to spatial resolution any more, the way, for instance, an image is coupled to the number of pixels. This is because they are continuous functions! Thus, the memory required to parameterize the signal is independent of spatial resolution, and only scales with the complexity of the underyling signal. Another corollary of this is that implicit representations have “infinite resolution” — they can be sampled at arbitrary spatial resolutions.

This is immediately useful for a number of applications, such as super-resolution, or in parameterizing signals in 3D and higher dimensions, where memory requirements grow intractably fast with spatial resolution.

However, in the future, the key promise of implicit neural representations lie in algorithms that directly operate in the space of these representations.

At first glance this is a cute hack (does reality plausibly ever know its own coordinates with reference to some grid system?), but maybe it is deeper than that; As an audio guy I cannot help but notice this is very similar to what synthesizer designers do all the time when designing new instruments, which are terrible representations of “real” instruments but are deft at making novel sounds. There is a short speculative essay to write, with a side journey into signal sampling.

In generative art

hardmaru’s Compositional pattern-producing network uses apparently a classic type of implicit rep NNs. See generative art with NNs for more of that.

in PDEs

See ML for PDEs

Position encoding

See position_encoding.

References

Chen, Zhiqin, and Hao Zhang. 2018. Learning Implicit Fields for Generative Shape Modeling,” December.
Dupont, Emilien, Hyunjik Kim, S. M. Ali Eslami, Danilo Rezende, and Dan Rosenbaum. 2022. From Data to Functa: Your Data Point Is a Function and You Can Treat It Like One.” arXiv.
Huang, Zhichun, Shaojie Bai, and J. Zico Kolter. 2021. (Implicit)\(^2\): Implicit Layers for Implicit Representations.” In.
Mescheder, Lars, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2018. Occupancy Networks: Learning 3d Reconstruction in Function Space,” December.
Mildenhall, Ben, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.” arXiv:2003.08934 [Cs], August.
Park, Jeong Joon, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation,” January.
Sitzmann, Vincent, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. 2020. Implicit Neural Representations with Periodic Activation Functions.” arXiv:2006.09661 [Cs, Eess], June.
Sitzmann, Vincent, Michael Zollhoefer, and Gordon Wetzstein. 2019. Scene Representation Networks: Continuous 3d-Structure-Aware Neural Scene Representations.” Advances in Neural Information Processing Systems 32: 1121–32.
Stanley, Kenneth O. 2007. Compositional Pattern Producing Networks: A Novel Abstraction of Development.” Genetic Programming and Evolvable Machines 8 (2): 131–62.
Tancik, Matthew, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. 2020. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains.” arXiv:2006.10739 [Cs], June.
Zhu, Zihan, Songyou Peng, Viktor Larsson, and Weiwei Xu. 2022. “NICE-SLAM: Neural Implicit Scalable Encoding for SLAM,” 17.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.