Inductive biases
Few-shot learning, learning fast weights, learning to learn
2025-06-15 — 2025-06-15
Wherein Neural Network Architectures Are Treated as Inductive Dispositions to Be Quantified by How They Make Learning Target Phenomena Easier, and Parallels With Human Cognitive Architectures Are Outlined.
functional analysis
how do science
meta learning
model selection
optimization
statmech
On this page
I’m not sure what this truly means or whether anyone is, but I think it means something like quantifying architectures that make it “easier” to learn about the phenomena of interest. This is a practical engineering discipline in NNs, but maybe also interesting to think about in humans.
1 References
Dehghani, Gouws, Vinyals, et al. 2019. “Universal Transformers.” In.
Griffiths, Chater, Kemp, et al. 2010. “Probabilistic Models of Cognition: Exploring Representations and Inductive Biases.” Trends in Cognitive Sciences.
Hamilton. 2020. “Graph Representation Learning.” Synthesis Lectures on Artificial Intelligence and Machine Learning.
Hsu, Hamid, Burns, et al. 2024. “Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning.”
Lin, and Tegmark. 2016a. “Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language.” arXiv:1606.06737 [Cond-Mat].
———. 2016b. “Why Does Deep and Cheap Learning Work so Well?” arXiv:1608.08225 [Cond-Mat, Stat].
Meronen, Trapp, and Solin. 2021. “Periodic Activation Functions Induce Stationarity.” In.
Mukhoti, Kirsch, van Amersfoort, et al. 2021. “Deterministic Neural Networks with Inductive Biases Capture Epistemic and Aleatoric Uncertainty.”
Rolnick, and Tegmark. 2017. “The Power of Deeper Networks for Expressing Natural Functions.” arXiv:1705.05502 [Cs, Stat].
Saunshi, Ash, Goel, et al. 2022. “Understanding Contrastive Learning Requires Incorporating Inductive Biases.” arXiv:2202.14037 [Cs].
Teney, Jiang, Gogianu, et al. 2025. “Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild.”
Udrescu, and Tegmark. 2020. “AI Feynman: A Physics-Inspired Method for Symbolic Regression.” Science Advances.
Wilson, Dann, Lucas, et al. 2015. “The Human Kernel.” arXiv:1510.07389 [Cs, Stat].
