Self-supervised learning

I just wanna be meeeeee / with high probabilityyy ♬♪

March 4, 2022 — March 4, 2022

hidden variables
likelihood free
nonparametric
statistics
unsupervised
Figure 1

Notebook on an area about which I know little. Probably mostly notes on contrastive learning for now?

1 References

Balestriero, Ibrahim, Sobal, et al. 2023. A Cookbook of Self-Supervised Learning.”
Chehab, Gramfort, and Hyvarinen. 2022. The Optimal Noise in Noise-Contrastive Learning Is Not What You Think.” arXiv:2203.01110 [Cs, Stat].
Gutmann, Michael, and Hyvärinen. 2010. Noise-Contrastive Estimation: A New Estimation Principle for Unnormalized Statistical Models.” In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics.
Gutmann, Michael U., and Hyvärinen. 2012. Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics.” Journal of Machine Learning Research.
Le-Khac, Healy, and Smeaton. 2020. Contrastive Representation Learning: A Framework and Review.” IEEE Access.
Ma, and Collins. 2018. Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency.” arXiv:1809.01812 [Cs, Stat].
Saunshi, Ash, Goel, et al. 2022. Understanding Contrastive Learning Requires Incorporating Inductive Biases.” arXiv:2202.14037 [Cs].
Shwartz-Ziv, and LeCun. 2023. To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review.”
Smith, and Eisner. 2005. Contrastive Estimation: Training Log-Linear Models on Unlabeled Data.” In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL ’05.