Position encoding

Also Fourier features



On passing relative location (or features derived from relative locations) into neural networks. Pops up often. That it pops up often is interesting but I am not sure if there is something general to say; I’m not even sure that the position encodings described here are even the saem kind of object. πŸ—οΈπŸ—οΈπŸ—οΈ

In transformers

Position encoding ends up being important in transformers (Dufter, Schmitt, and SchΓΌtze 2022).

In implicit representation networks

Implicit representation networks, such as PINNs and Neural radiance fields act mostly (or only) upon position features.

Fourier features

Encoding the position through its the sine and cosine. See Tancik et al. (2020) for some theory.

Connection to Fourier features in Gaussian Processes?

See also Fourier Feature Networks.

In basis decomposition networks

This idea is, I think, also implicit in any neural network that does basis decomposition, because basis functions encode a β€œlocation” in the same way that fourier features do.

In spatiotemporal networks

Not a headline, but spatiotemporal NNs typically use positional predictors, for example Fourier Neural Operators often pack position encoding in.

As a means of globally locating a local algorithm

Convnet-like NNs are local.

References

Chen, Zhiqin, and Hao Zhang. 2018. β€œLearning Implicit Fields for Generative Shape Modeling,” December.
Dufter, Philipp, Martin Schmitt, and Hinrich SchΓΌtze. 2022. β€œPosition Information in Transformers: An Overview.” Computational Linguistics 48 (3): 733–63.
Mescheder, Lars, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2018. β€œOccupancy Networks: Learning 3D Reconstruction in Function Space,” December.
Mildenhall, Ben, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. β€œNeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.” arXiv:2003.08934 [Cs], August.
Park, Jeong Joon, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. β€œDeepSDF: Learning Continuous Signed Distance Functions for Shape Representation.” In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 165–74. Long Beach, CA, USA: IEEE.
Press, Ofir, Noah A. Smith, and Mike Lewis. 2021. β€œTrain Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation.” arXiv:2108.12409 [Cs], August.
Sitzmann, Vincent, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. 2020. β€œImplicit Neural Representations with Periodic Activation Functions.” arXiv:2006.09661 [Cs, Eess], June.
Sitzmann, Vincent, Michael Zollhoefer, and Gordon Wetzstein. 2019. β€œScene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations.” Advances in Neural Information Processing Systems 32: 1121–32.
Stanley, Kenneth O. 2007. β€œCompositional Pattern Producing Networks: A Novel Abstraction of Development.” Genetic Programming and Evolvable Machines 8 (2): 131–62.
Tancik, Matthew, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. 2020. β€œFourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains.” In Advances In Neural Information Processing Systems. arXiv.
Wang, Sinong, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. β€œLinformer: Self-Attention with Linear Complexity.” arXiv.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.