Sequential experiments

Especially multiple sequential experiments

I am running lots of experiments and you want to waste a little time as possible on ones that are going to give me negative results.

One experiment

Fairly well understood. See Chernoff (1959);Lakens (2017).

Many experiments

Alex Birkett’s article is OK on explaining this with regard to bandit problems by an example we all know — selling stuff on the internet: When to Run Bandit Tests Instead of A/B/n Tests.


Allen-Zhu, Zeyuan, Yuanzhi Li, Aarti Singh, and Yining Wang. 2017. Near-Optimal Design of Experiments via Regret Minimization.” In PMLR, 126–35.
Chernoff, Herman. 1959. Sequential Design of Experiments.” The Annals of Mathematical Statistics 30 (3): 755–70.
Even-Dar, Eyal, Shie Mannor, and Yishay Mansour. n.d. “Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems,” 27.
Jamieson, Kevin, and Lalit Jain. n.d. “A Bandit Approach to Multiple Testing with False Discovery Control,” 11.
Kuleshov, Volodymyr, and Doina Precup. 2000. “Algorithms for the Multi-Armed Bandit Problem.” Journal of Machine Learning Research, 32.
Lakens, Daniel. 2017. Performing High-Powered Studies Efficiently With Sequential Analyses.” PsyArXiv.
Loecher, Markus. 2021. The Perils of Misspecified Priors and Optional Stopping in Multi-Armed Bandits.” Frontiers in Artificial Intelligence 4 (July): 715690.
Press, William H. 2009. Bandit Solutions Provide Unified Ethical Models for Randomized Clinical Trials and Comparative Effectiveness Research.” Proceedings of the National Academy of Sciences 106 (52): 22387–92.
Villar, Sofía S., Jack Bowden, and James Wason. 2015. Multi-Armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges.” Statistical Science : A Review Journal of the Institute of Mathematical Statistics 30 (2): 199–215.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.