Multilinear algebra

Outer products, tensors, einstein summation

October 8, 2021 — October 8, 2021

functional analysis
Hilbert space
linear algebra
Figure 1

Tensors and stuff. Like linear algebra, but linear in multiple arguments.

Keywords: Ricci calculus, Einstein summation notation, index notation, subscript notation.

1 Tensor calculus

When some of the vectors are differentials.

Should this be smushed into differential geometry?

If we crack open a tensor textbook we get a lot of guff about general relativity and tensor fields and such, which is all very nice but not germane to typical machine learning applications. We want to start with the immediately-needed thing, which is some tidy notation conventions for dealing with multilinear operations without too many squiggles in our notation.

Soeren Laue, Matthias Mitterreiter, Joachim Giesen and Jens K. Mueller have been popularising such an approach recently. In their paper (Laue, Mitterreiter, and Giesen 2018), they argue that derivation of matrix differential results can be greatly simplified with Ricci calculus, and P.S. it can induce faster code. They have a website. which showcases this trick to do the useful special case of matrix calculus; Here are tasty readings on relevant bits of tensor machinery.

2 Tensor decomposition

See tensor decomposition.

3 References

Bowen, and Wang. 1976. Introduction to Vectors and Tensors, Vol 1: Linear and Multilinear Algebra.
Bowen, and Wang. 2006. Introduction to Vectors and Tensors, Vol 2: Vector and Tensor Analysis.
Dullemond, and Peeters. 2021. Introduction to Tensor Calculus.”
Flanders. 1989. Differential Forms with Applications to the Physical Sciences.
Hestenes, and Sobczyk. 1984. Clifford Algebra to Geometric Calculus a Unified Language for Mathematics and Physics.
Lasenby, Lasenby, and Doran. 2000. A Unified Mathematical Language for Physics and Engineering in the 21st Century.” Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences.
Laue, Mitterreiter, and Giesen. 2018. Computing Higher Order Derivatives of Matrix and Tensor Expressions.” In Advances in Neural Information Processing Systems 31.
Lim. 2021. Tensors in Computations.” Acta Numerica.