Self-supervised learning
I just wanna be meeeeee / with high probabilityyy ♬♪
March 4, 2022 — March 4, 2022
Notebook on an area about which I know little. Probably mostly notes on contrastive learning for now?
1 References
Balestriero, Ibrahim, Sobal, et al. 2023. “A Cookbook of Self-Supervised Learning.”
Chehab, Gramfort, and Hyvarinen. 2022. “The Optimal Noise in Noise-Contrastive Learning Is Not What You Think.” arXiv:2203.01110 [Cs, Stat].
Gutmann, Michael, and Hyvärinen. 2010. “Noise-Contrastive Estimation: A New Estimation Principle for Unnormalized Statistical Models.” In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics.
Gutmann, Michael U., and Hyvärinen. 2012. “Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics.” Journal of Machine Learning Research.
Le-Khac, Healy, and Smeaton. 2020. “Contrastive Representation Learning: A Framework and Review.” IEEE Access.
Ma, and Collins. 2018. “Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency.” arXiv:1809.01812 [Cs, Stat].
Saunshi, Ash, Goel, et al. 2022. “Understanding Contrastive Learning Requires Incorporating Inductive Biases.” arXiv:2202.14037 [Cs].
Shwartz-Ziv, and LeCun. 2023. “To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review.”
Smith, and Eisner. 2005. “Contrastive Estimation: Training Log-Linear Models on Unlabeled Data.” In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL ’05.