# Learning covariance functions

Learning a family of covariances at once

September 16, 2019 — March 1, 2021

The generalisation of covariance matrix estimation to the case of continuous index sets. This is often seen in the context of Gaussian processes where everything can work out nicely if we are lucky.

## 1 Selecting parametric kernel by maximising marginal likelihood

The goal for most of these is to maximise the marginal posterior likelihood, a.k.a. model evidence, as is conventional in Bayesian ML. But we could also apply hyperpriors to kernels.

## 2 Learning kernel composition

Automating kernel design by some composition of simpler atomic kernels. AFAICT this started from summaries like (Genton 2001) and went via Duvenaud’s aforementioned notes to became a small industry (Lloyd et al. 2014; D. K. Duvenaud, Nickisch, and Rasmussen 2011; D. Duvenaud et al. 2013; Grosse et al. 2012). A prominent example was the Automated statistician project by David Duvenaud, James Robert Lloyd, Roger Grosse and colleagues, which works by greedy combinatorial search over possible compositions.

More fashionable, presumably, are the differentiable search methods. For example, the AutoGP system (Krauth et al. 2016; Bonilla, Krauth, and Dezfouli 2019) incorporates tricks like these to use gradient descent to design kernels for Gaussian processes. (Sun et al. 2018) construct deep networks of composed kernels. I imagine the Deep Gaussian Process literature is also of this kind, but have not read it.

## 3 Via neural nets

🏗

## 4 Hyperkernels

Kernels on kernels, for kernel learning kernels 🏗 (Ong, Smola, and Williamson 2005, 2002; Ong and Smola 2003; Kondor and Jebara 2006).

## 5 References

*Foundations and Trends® in Machine Learning*.

*Proceedings of the 21st International Conference on Neural Information Processing Systems*. NIPS’08.

*arXiv:1606.05241 [Stat]*.

*arXiv:1709.10441 [Cs, Math]*.

*Journal of Machine Learning Research*.

*Journal of Machine Learning Research*.

*Proceedings of the 30th International Conference on Machine Learning (ICML-13)*.

*Advances in Neural Information Processing Systems*.

*Journal of Machine Learning Research*.

*Proceedings of the 22nd International Conference on Machine Learning - ICML ’05*.

*Proceedings of the Conference on Uncertainty in Artificial Intelligence*.

*2010 IEEE International Workshop on Machine Learning for Signal Processing*.

*The Annals of Statistics*.

*arXiv:1506.02236 [Stat]*.

*Proceedings of the 19th International Conference on Neural Information Processing Systems*. NIPS’06.

*UAI17*.

*Journal of Machine Learning Research*.

*Twenty-Eighth AAAI Conference on Artificial Intelligence*.

*Journal of Machine Learning Research*.

*Neural Computation*.

*Machine learning: a probabilistic perspective*. Adaptive computation and machine learning series.

*Deterministic and Statistical Methods in Machine Learning*. Lecture Notes in Computer Science.

*Twenty-Fifth AAAI Conference on Artificial Intelligence*.

*Twenty-First International Conference on Machine Learning - ICML ’04*.

*Proceedings of the Twentieth International Conference on International Conference on Machine Learning*. ICML’03.

*Proceedings of the 15th International Conference on Neural Information Processing Systems*. NIPS’02.

*Journal of Machine Learning Research*.

*Journal of Machine Learning Research*.

*Gaussian Processes for Machine Learning*. Adaptive Computation and Machine Learning.

*arXiv:1811.10978 [Cs, Stat]*.

*Advances in Neural Information Processing Systems*.

*IEEE Signal Processing Magazine*.

*Computational Learning Theory*. Lecture Notes in Computer Science.

*Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond*.

*Advanced Lectures on Machine Learning*. Lecture Notes in Computer Science 2600.

*Advances in Neural Information Processing Systems 29*.

*arXiv Preprint arXiv:1806.04326*.

*International Conference on Artificial Intelligence and Statistics*.

*Kernel Methods in Computational Biology*.

*Journal of Machine Learning Research*.

*International Conference on Machine Learning*.

*arXiv:1510.07389 [Cs, Stat]*.

*Machine Learning and Knowledge Discovery in Databases*. Lecture Notes in Computer Science.

*Artificial Intelligence and Statistics*.

*Proceedings of the 30th International Conference on Machine Learning (ICML-13)*.

*International Conference on Machine Learning*.