Singular Learning Theory

Placeholder.

As far as I can tell, a first-order approximation to (the bits I vaguely understand of) Singular Learning Theory is something like:

Classical Bayesian statistics has a good theory of well-posed models with a small number of interpretable parameters. Singular Learning Theory is a theory of ill-posed models with a large number of uninterpretable parameters, which provides us with a model of Bayesian statistics by using results from algebraic geometry about singularities in the loss surface.

There are obviously a lot of details missing from that. I think there are non-Bayesian versions too, but I haven’t been exposed to them yet.

Jesse Hoogland, Neural networks generalize because of this one weird trick:

Statistical learning theory is lying to you: “overparametrized” models actually aren’t overparametrized, and generalization is not just a question of broad basins.