Sequential experiments
Especially multiple sequential experiments
2021-08-04 — 2021-08-04
Wherein the conduct of successive trials is described, and strategies for minimizing time wasted on negative outcomes are pursued, while Bayesian optimization is invoked to identify superior interventions
I am running lots of experiments and want to waste as little time as possible on ones that will give me negative results, while attaining as much certainty as possible about the positive results.
1 One experiment
2 Many experiments
Alex Birkett’s article is OK on explaining this with regard to bandit problems by an example we all know — selling stuff on the internet: When to Run Bandit Tests Instead of A/B/n Tests.
3 But I want to find the best answer, not just low certainty
This is adaptive design of experiments, a.k.a. Bayesian optimization.