Operationalising the bitter lessons in compute and cleverness

1 Word salad

Overparameterization and inductive biases. Operationalizing the scaling hypothesis. Compute overhangs. Tokenomics. Amortization of inference. Reduced order modelling. Training time versus inference time. The business model of FLOPS.

Concretely, traditional statistics was shaken when it turned out that labouriously proving things wasn’t as valuable as training a neural network to do the same thing. The field was upset again when it turned out that Foundation models and LLMs made it even weirder: it can be worthwhile to spend a lot of money training a gigantic model on all the data we can get, then use that model for very cheap inference. Essentially, we can pay once for a very expensive training run and thereafter everything is cheap.

A lot of ML problems, in hindsight, are implicitly about when to spend our compute budget. I think it’s useful to group the normative question of doing this well under the heading of when to compute.

2 Train time versus inference time

TBD

3 Amortisation

Bayesians call this the trade-off involved in learning inferential shortcuts amortization. To be continued.

4 In reinforcement learning

RL is an interestingly stark example of when to compute. In some cases, we can spend a lot of compute at training time to learn a policy that is very cheap to execute at inference time. In many RL algorithms, notably ones trained on a pure physical model this is a naked attempt to speed up a thing that we could already calculate. We could in princple calculate the optimal action at each time step by solving a dynamic programming problem, but this is often computationally intractable. Instead, we can use RL to learn a policy that approximates the optimal action.

5 Computing and remembering

TBD. For now, see NN memorization.

6 General compute efficiency

We’re getting very good at using hardware efficiently (Grace 2013). The piece AI and efficiency (Hernandez and Brown 2020) makes this clear:

We’re releasing an analysis showing that since 2012 the amount of compute needed to train a neural net to the same performance on ImageNet classification has been decreasing by a factor of 2 every 16 months. Compared to 2012, it now takes 44 times less compute to train a neural network to the level of AlexNet (by contrast, Moore’s Law would yield an 11x cost improvement over this period). Our results suggest that for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than classical hardware efficiency.

7 Metascience

Sutton’s famous bitter lesson set the terminology:

The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.

We make this point a lot — for example, On the futility of trying to be clever (the bitter lesson redux).

Alternative phrasing: Even the best human minds aren’t very good; the quickest path to intelligence prioritizes replacing that human bottleneck.

Things work better when they can scale up. We should deploy the compute we have effectively.

Ermin Orhan argues:

I do think that there are many interesting research directions consistent with the philosophy of the bitter lesson that can have more meaningful, longer-term impact than small algorithmic or architectural tweaks. I just want to wrap up this post by giving a couple of examples below:

1. Probing the limits of model capabilities as a function of training data size: can we get to something close to human-level machine translation by leveraging everything multi-lingual on the web (I think we’ve learned that the answer to this is basically yes)? Can we get to something similar to human-level language understanding by scaling up the GPT-3 approach a couple of orders of magnitude (I think the answer to this is probably we don’t know yet)? Of course, smaller scale, less ambitious versions of these questions are also incredibly interesting and important.

2. Finding out what we can learn from different kinds of data and how what we learn differs as a function of this: e.g. learning from raw video data vs. learning from multi-modal data received by an embodied agent interacting with the world; learning from pure text vs. learning from text + images or text + video.

3. Coming up with new model architectures or training methods that can leverage data and compute more efficiently, e.g., more efficient transformers, residual networks, batch normalization, self-supervised learning algorithms that can scale to large (ideally unlimited) data (e.g. likelihood-based generative pre-training, contrastive learning).

I think Orhan’s statement is easier to engage with than Sutton’s because he gives concrete examples. His invective is tight.

Orhan’s examples suggest that when data are plentiful, we should use compute-heavy infrastructure to extract as much information as possible.

This doesn’t directly apply everywhere. For example, in my last hydrology project, a single data point cost AUD 700,000 because drilling a thousand-metre well cost that much. Telling me to collect a billion more data points instead of being clever isn’t useful; collapsing the global economy to collect them wouldn’t be clever. There might be ways to get more data, such as pretraining a geospatial foundation model on hydrology-adjacent tasks, but that doesn’t look trivial.

What we generally want isn’t a homily saying being clever is a waste of time, but a trade-off curve quantifying how clever to bother being. That’s probably the best lesson.

8 Trading off against humans

This is partially discussed in economics of LLMs. We still need a general theory, though.

9 Bitter lessons in career strategy

I get the most from the limited compute in my skull by figuring out how to use the larger compute on my GPU.

Career version:

…a lot of the good ideas that did not require a massive compute budget have already been published by smart people who did not have GPUs, so we need to leverage our technological advantage relative to those ancestors if we want to get cited.

10 Information transmission and compression

See epistemic bottlenecks.

11 Thermodynamics of computation

Energy, water and minerals are important inputs to computation. This suggests we should think about the connection between thermodynamics and information and the statistical mechanics of statistics.

12 Incoming

• How Well Does RL Scale? — Toby Ord

  Now that RL-training is nearing its effective limit, we may have lost the ability to effectively turn more compute into more intelligence.

• Will the Need to Retrain AI Models from Scratch Block a Software Intelligence Explosion?

• IsoFLOP curves of large language models are extremely flat – Severely Theoretical

• A Better Lesson – Rodney Brooks

• Compute Goes Brrr: Revisiting Sutton’s Bitter Lesson for Artificial Intelligence

• Reflections on ‘The Bitter Lesson’

• Gwern

  • On GPT-3: Meta-Learning, Scaling, Implications, and Deep Theory
  • The Scaling Hypothesis

• Scale, schlep, and systems

13 References