Brief links on the theme of scaling in the extremely large model/large data limit and what that does to the behaviour of the models. Possibly a fruitful new front in the complexity of statistics.

As to *how* to scale up these models, see distributed gradient descent.

## Big transformers

One fun result comes from Transformer language models. An interesting observation way back in 2020 was that there seemed to be an unexpected trade-off where you can go faster by training a bigger network. Indeed, there is a whole family of observations in this vein.

nostalgebraist summarises Henighan et al. (2020); Kaplan et al. (2020):

## L(D): information

OpenAI derives a scaling law called L(D). This law is the best you could possibly do – even with arbitrarily large compute/models – if you are only allowed to train on D data points.

No matter how good your model is, there is only so much it can learn from a finite sample. L(D) quantifies this intuitive fact (if the model is an autoregressive transformer).

## L(C): budgeting

OpenAI also derives another a scaling law called L(C). This is the best you can do with compute C, if you spend it optimally.

What does optimal spending look like? Remember, you can spend a unit of compute on

- a bigger model (N), or
- training the same model for longer (S)
…In the compute regime we are currently in, making the model bigger is way more effective than taking more steps.

## Bitter lesson

The bitter lesson of history in AI is that “general methods that leverage computation are ultimately the most effective, and by a large margin.”

## Misc

- Zhang et al. (2020) (how do NNs learn from language as
*n*increases?

## References

*arXiv:2010.14701 [cs]*, November. http://arxiv.org/abs/2010.14701.

*arXiv:2001.08361 [cs, Stat]*, January. http://arxiv.org/abs/2001.08361.

*arXiv:2110.04374 [cs]*, October. http://arxiv.org/abs/2110.04374.

*arXiv:2011.03641 [cs]*, November. http://arxiv.org/abs/2011.03641.

*arXiv:2004.10802 [cs, Stat]*, April. http://arxiv.org/abs/2004.10802.

*arXiv:2011.04946 [cs]*, November. http://arxiv.org/abs/2011.04946.

## No comments yet. Why not leave one?