Multi-output Gaussian process regression


In which I discover for myself whether “multi-task” and “co-regionalized” approaches are different. Álvarez, Rosasco, and Lawrence (2012)

Co-regionalization

[the] community has begun to turn its attention to covariance functions for multiple outputs. One of the paradigms that has been considered (Bonilla, Chai, and Williams 2007; Osborne et al. 2008; Seeger, Teh, and Jordan 2005) is known in the geostatistics literature as the linear model of coregionalization (LMC). In the LMC, the covariance function is expressed as the sum of Kronecker products between coregionalization matrices and a set of underlying covariance functions. The correlations across the outputs are expressed in the coregionalization matrices, while the underlying covariance functions express the correlation between different data points.

Multitask learning has also been approached from the perspective of regularization theory (Evgeniou and Pontil, 2004; Evgeniou et al., 2005). These multitask kernels are obtained as generalizations of the regularization theory to vector-valued functions. They can also be seen as examples of LMCs applied to linear transformations of the input space.

Multi Output Spectral Mixture Kernel

This repository provides a toolkit to perform multi-output GP regression with kernels that are designed to utilize correlation information among channels in order to better model signals. The toolkit is mainly targeted to time-series, and includes plotting functions for the case of single input with multiple outputs (time series with several channels).

The main kernel corresponds to Multi Output Spectral Mixture Kernel, which correlates every pair of data points (irrespective of their channel of origin) to model the signals. This kernel is specified in detail in Parra and Tobar (2017)

References

Álvarez, Mauricio A., and Neil D. Lawrence. 2011. “Computationally Efficient Convolved Multiple Output Gaussian Processes.” Journal of Machine Learning Research 12 (41): 1459–1500. http://jmlr.org/papers/v12/alvarez11a.html.
Álvarez, Mauricio A., Lorenzo Rosasco, and Neil D. Lawrence. 2012. “Kernels for Vector-Valued Functions: A Review.” Foundations and Trends® in Machine Learning 4 (3, 3): 195–266. https://doi.org/10.1561/2200000036.
Bonilla, Edwin V., Kian Ming A. Chai, and Christopher K. I. Williams. 2007. “Multi-Task Gaussian Process Prediction.” In Proceedings of the 20th International Conference on Neural Information Processing Systems, 153–60. NIPS’07. USA: Curran Associates Inc. http://dl.acm.org/citation.cfm?id=2981562.2981582.
Dai, Zhenwen, Mauricio Álvarez, and Neil Lawrence. 2017. “Efficient Modeling of Latent Information in Supervised Learning Using Gaussian Processes.” Advances in Neural Information Processing Systems 30: 5131–39. https://proceedings.neurips.cc/paper/2017/hash/1680e9fa7b4dd5d62ece800239bb53bd-Abstract.html.
Evgeniou, Theodoros, Charles A. Micchelli, and Massimiliano Pontil. 2005. “Learning Multiple Tasks with Kernel Methods.” Journal of Machine Learning Research 6: 615–37. http://www.jmlr.org/papers/v6/evgeniou05a.html.
Evgeniou, Theodoros, and Massimiliano Pontil. 2004. “Regularized Multi–Task Learning.” In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 109–17. KDD ’04. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/1014052.1014067.
Micchelli, Charles A., and Massimiliano Pontil. 2005a. “Learning the Kernel Function via Regularization.” Journal of Machine Learning Research 6 (Jul, Jul): 1099–1125. http://www.jmlr.org/papers/v6/micchelli05a.html.
———. 2005b. “On Learning Vector-Valued Functions.” Neural Computation 17 (1, 1): 177–204. https://doi.org/10.1162/0899766052530802.
Moreno-Muñoz, Pablo, Antonio Artés, and Mauricio Álvarez. 2018. “Heterogeneous Multi-Output Gaussian Process Prediction.” In Advances in Neural Information Processing Systems, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, 31:6711–20. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2018/file/165a59f7cf3b5c4396ba65953d679f17-Paper.pdf.
Moreno-Muñoz, Pablo, Antonio Artés-Rodríguez, and Mauricio A. Álvarez. 2019. “Continual Multi-Task Gaussian Processes.” October 31, 2019. http://arxiv.org/abs/1911.00002.
Osborne, M. A., S. J. Roberts, A. Rogers, S. D. Ramchurn, and N. R. Jennings. 2008. “Towards Real-Time Information Processing of Sensor Network Data Using Computationally Efficient Multi-Output Gaussian Processes.” In 2008 International Conference on Information Processing in Sensor Networks (ipsn 2008), 109–20. https://doi.org/10.1109/IPSN.2008.25.
Parra, Gabriel, and Felipe Tobar. 2017. “Spectral Mixture Kernels for Multi-Output Gaussian Processes.” In Advances in Neural Information Processing Systems, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, 30:6681–90. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2017/file/333cb763facc6ce398ff83845f224d62-Paper.pdf.
Seeger, Matthias, Yee-Whye Teh, and Michael I Jordan. 2005. “Semiparametric Latent Factor Models,” 31. http://infoscience.epfl.ch/record/161465.
Stegle, Oliver, Christoph Lippert, Joris Mooij, Neil Lawrence, and Karsten Borgwardt. 2011. “Efficient Inference in Matrix-Variate Gaussian Models with Iid Observation Noise.” In Proceedings of the 24th International Conference on Neural Information Processing Systems, 630–38. NIPS’11. Red Hook, NY, USA: Curran Associates Inc. https://papers.nips.cc/paper/4281-efficient-inference-in-matrix-variate-gaussian-models-with-iid-observation-noise.pdf.
Williams, Christopher, Stefan Klanke, Sethu Vijayakumar, and Kian M. Chai. 2009. “Multi-Task Gaussian Process Learning of Robot Inverse Dynamics.” In Advances in Neural Information Processing Systems 21, edited by D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, 265–72. Curran Associates, Inc. http://papers.nips.cc/paper/3385-multi-task-gaussian-process-learning-of-robot-inverse-dynamics.pdf.