Position encoding

In transformers
In implicit representation networks
Fourier features
In basis decomposition networks
In spatiotemporal networks
As a means of globally locating a local algorithm
Tooling
References

On passing relative location (or features derived from relative locations) into neural networks. Pops up often. That it pops up often is interesting but I am not sure if there is something general to say; I’m not even sure that the position encodings described here are even the saem kind of object. 🏗️🏗️🏗️

In transformers

Position encoding ends up being important in transformers (Dufter, Schmitt, and Schütze 2022).

In implicit representation networks

Implicit representation networks, such as PINNs and Neural radiance fields act mostly (or only) upon position features.

Fourier features

Encoding the position through its the sine and cosine. See Tancik et al. (2020) for some theory.

Connection to Fourier features in Gaussian Processes?

In basis decomposition networks

This idea is, I think, also implicit in any neural network that does basis decomposition, because basis functions encode a “location” in the same way that fourier features do.

In spatiotemporal networks

Not a headline, but spatiotemporal NNs typically use positional predictors, for example Fourier Neural Operators often pack position encoding in.

As a means of globally locating a local algorithm

Convnet-like NNs are local.

Tooling

tatp22/multidim-positional-encoding: An implementation of 1D, 2D, and 3D positional encoding in pytorch

References

Chen, Zhiqin, and Hao Zhang. 2018. “Learning Implicit Fields for Generative Shape Modeling,” December.

Dufter, Philipp, Martin Schmitt, and Hinrich Schütze. 2022. “Position Information in Transformers: An Overview.” Computational Linguistics 48 (3): 733–63.

Mescheder, Lars, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2018. “Occupancy Networks: Learning 3D Reconstruction in Function Space,” December.

Mildenhall, Ben, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.” arXiv:2003.08934 [Cs], August.

Park, Jeong Joon, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. “DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation.” In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 165–74. Long Beach, CA, USA: IEEE.

Press, Ofir, Noah A. Smith, and Mike Lewis. 2021. “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation.” arXiv:2108.12409 [Cs], August.

Sitzmann, Vincent, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. 2020. “Implicit Neural Representations with Periodic Activation Functions.” arXiv:2006.09661 [Cs, Eess], June.

Sitzmann, Vincent, Michael Zollhoefer, and Gordon Wetzstein. 2019. “Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations.” Advances in Neural Information Processing Systems 32: 1121–32.

Stanley, Kenneth O. 2007. “Compositional Pattern Producing Networks: A Novel Abstraction of Development.” Genetic Programming and Evolvable Machines 8 (2): 131–62.

Tancik, Matthew, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. 2020. “Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains.” In Advances In Neural Information Processing Systems. arXiv.

Wang, Sinong, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. “Linformer: Self-Attention with Linear Complexity.” arXiv.

Also Fourier features