We can generalise the high school calculus, which is about scalar functions of a scalar argument, in various ways, to handle matrix-valued functions or matrix-valued arguments. One could generalise this further, by to full tensor calculus. But it happens that specifically matrix/vector operations are at a useful point of complexity for lots of algorithms, kind of a MVP. (I usually want this for higher order gradient descent.)

I will mention two convenient and popular formalisms for doing that. In practice a mix of each is often useful.

## Matrix differentials

🏗 I need to return to this and tidy it up with some examples.

A special case of tensor calculus that happens to be handy for some common cases; where the rank of the argument and value of the function is not too big. Fun pain point: agreeing upon layout of derivatives, numerator vs denominator.

If our problem is nice, this often gets us a low-fuss, compact, tidy solution even for some surprising cases where it seems that more general tensors would be more natural. (for which, see below)

- The Matrix Calculus You Need For Deep Learning (Parr and Howard 2018)
- Many, many quick recipes: the Matrix Cookbook Petersen and Pedersen (2012)
- More expository but not as broad, Old and new matrix algebra useful for statistics (Minka 2000)
- Mike Brookes’ Matrix Reference Manual
- autodiff-focussed: Collected Matrix Derivative Results for Forward and Reverse Mode Algorithmic Differentiation (Giles 2008)
- The rough-and-ready notation is occasionally confusing, but it has a functional analysis interpretation.

## Indexed tensor calculus

Filed under multilinear algebra.

## References

*Advances in Automatic Differentiation*, edited by Christian H. Bischof, H. Martin Bücker, Paul Hovland, Uwe Naumann, and Jean Utke, 64:35–44. Berlin, Heidelberg: Springer Berlin Heidelberg.

*Matrix Computations*. JHU Press.

*Kronecker Products and Matrix Calculus: With Applications*. Horwood.

*Matrix Variate Distributions*. CRC Press.

*Advances in Neural Information Processing Systems 31*, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, 2750–59. Curran Associates, Inc.

*Matrix Differential Calculus with Applications in Statistics and Econometrics*. Rev. ed. New York: John Wiley.

*Old and new matrix algebra useful for statistics*.

*Wiley StatsRef: Statistics Reference Online*. American Cancer Society.

*Matrix Algebra Useful for Statistics*. John Wiley & Sons.

*A Matrix Handbook for Statisticians*. Wiley.

*Problems and Solutions in Introductory and Advanced Matrix Calculus*. World Scientific.

*Matrix Calculus Zero-One Matrices*. Cambridge University Press.

## No comments yet. Why not leave one?