Adaptive design of experiments

Minesweeper++

April 11, 2017 — May 31, 2024

functional analysis
how do science
model selection
optimization
statmech
surrogate
Figure 1

Closely related to AutoML, in that surrogate optimisation is a popular tool for such, and likewise Bayesian model calibration.

tl;dr

As a working ML guy at the moment I simply use Ax and don’t sweat the fine details. It ahs a great API and is powerful. Documentation is thin, but the project is active. I would be happier if it incorporated more prior information about the target, but AFAICT there is not much better.

1 Problem statement

According to Gilles Louppe and Manoj Kumar:

We are interested in solving

\[x^* = \arg \min_x f(x)\]

under the constraints that

  • \(f\) is a black box for which no closed form is known (nor its gradients);
  • \(f\) is expensive to evaluate;
  • evaluations of \(y=f(x)\) may be noisy.

It is possible to imagine we might even have access to gradients sometimes in which case we will additionally say that, rather than observing \(\nabla f, \nabla^2 f\) we observe some random variables \(G(x),H(x)\) with \(\mathbb{E}G=\nabla f\) and \(\mathbb{E}(H)=\nabla^2 f,\) as in stochastic optimisation.

This is similar to the typical framing of reinforcement learning problems where there is a similar explore/exploit trade-off, although I do not know the precise disciplinary boundaries that may transect these areas.

The typical setup here is: We use a surrogate model of the loss surface and optimise that, on the hope that it be computationally cheaper than evaluating the whole loss surface. An artfully chosen surrogate model can choose where to sample next and so on and estimate unseen loss values, and possibly even give uncertainty estimatess.

When the surrogate model is in particular a Bayesian posterior over parameter values that we wish to learn, a common name is the “Bayesian optimisation”. Gaussian process regression is an obvious method to approximate the loss surface in this case, and this seems to be assumed typically.

This is not crazy. Some of the early work on GP regression (Krige 1951) already includes a surrogate modelling application — How much ore remains in my mine, given my observations? However, GP regressions are not the only possible surrogate models, not even the only possible Bayesian one, and there is nothing intrinsically Bayesian about estimating the unknown function.

Setting that quibble aside, see Apoorv Agnihotri, Nipun Batra, Exploring Bayesian Optimization for a well-illustrated journey into this field.

Fashioable use: hyperparameter/ model selection, in e.g. regularising complex models, which is compactly referred to these days as automl.

You could also obviously use adaptive experiments outside of simulations, e.g. in industrial process control, which is where I originally saw this kind of thing, in the form of sequential ANOVA design, which is an incredible idea itself, although that is now years old so is not nearly so hip. This page is full of things that we migh describe as, effectively, nonlinear, heteroskedastic sequential ANOVA,

2 With side information

The hip method I have heard of that aims at this problem is SEBO (Chan, Paulson, and Mesbah 2023; Liu et al. 2023).

3 BORE

Bayesian optimisation by density ratio estimation (Oliveira, Tiao, and Ramos 2022; Louis C. Tiao et al. 2021).

Bayesian optimization (BO) is among the most effective and widely-used blackbox optimization methods. BO proposes solutions according to an explore-exploit trade-off criterion encoded in an acquisition function, many of which are computed from the posterior predictive of a probabilistic surrogate model. Prevalent among these is the expected improvement (EI). The need to ensure analytical tractability of the predictive often poses limitations that can hinder the efficiency and applicability of BO. In this paper, we cast the computation of EI as a binary classification problem, building on the link between class-probability estimation and density-ratio estimation, and the lesser-known link between density-ratios and EI. By circumventing the tractability constraints, this reformulation provides numerous advantages, not least in terms of expressiveness, versatility, and scalability.

4 Lab bandits

Sequential experiment design in the lab.

5 Acquisition functions

Active learning, acquisition functions. TBD.

6 Connection to RL

TBD.

7 Implementation

7.1 BoTorch/Ax

Botorch is the pytorch-based Bayesian optimization toolbox used by Ax which is an experiment designer.

Ax is a platform for optimizing any kind of experiment, including machine learning experiments, A/B tests, and simulations. Ax can optimize discrete configurations (e.g., variants of an A/B test) using multi-armed bandit optimization, and continuous (e.g., integer or floating point)-valued configurations using Bayesian optimization. This makes it suitable for a wide range of applications.

Ax has been successfully applied to a variety of product, infrastructure, ML, and research applications at Facebook.

7.2 skopt

skopt (aka scikit-optimize)

[…] is a simple and efficient library to minimize (very) expensive and noisy black-box functions. It implements several methods for sequential model-based optimization.

This is a member of the sklearn club which is to say it works well, reliably, predictably, universally has amazing tooling, but is not that fast and few modern fancy fripperies.

7.3 Dragonfly

Dragonfly

…is an open source python library for scalable Bayesian optimisation.

Bayesian optimisation is used for optimising black-box functions whose evaluations are usually expensive. Beyond vanilla optimisation techniques, Dragonfly provides an array of tools to scale up Bayesian optimisation to expensive large scale problems. These include features/functionality that are especially suited for high dimensional optimisation (optimising for a large number of variables), parallel evaluations in synchronous or asynchronous settings (conducting multiple evaluations in parallel), multi-fidelity optimisation (using cheap approximations to speed up the optimisation process), and multi-objective optimisation (optimising multiple functions simultaneously).

Python and Fortran, open-source.

7.4 PySOT

PySOT

Surrogate Optimization Toolbox (pySOT) for global deterministic optimization problems. pySOT is hosted on GitHub

The main purpose of the toolbox is for optimization of computationally expensive black-box objective functions with continuous and/or integer variables. All variables are assumed to have bound constraints in some form where none of the bounds are infinity. The tighter the bounds, the more efficient are the algorithms since it reduces the search region and increases the quality of the constructed surrogate. This toolbox may not be very efficient for problems with computationally cheap function evaluations. Surrogate models are intended to be used when function evaluations take from several minutes to several hours or more.

This has a huge variety of different surrogate options, a long history and promises to parallel asynchronous, but is not especially famous for some reason? (quality?) Based on (Krityakierne, Akhtar, and Shoemaker 2016; Regis and Shoemaker 2013, 2009, 2007). It is one of the ones that does not particularly emphasis Bayesian methods.

7.5 GPyOpt

GPyOpt

Gaussian process optimization using GPy. Performs global optimization with different acquisition functions. Among other functionalities, it is possible to use GPyOpt to optimize physical experiments (sequentially or in batches) and tune the parameters of Machine Learning algorithms. It is able to handle large data sets via sparse Gaussian process models.

By the same lab at Sheffield that brough us GPy.

7.6 Sigopt

sigopt is a commercial product that presumably does a good job. The fact their website does not give even a hint of the price leads me to suspect they are extremely expensive.

7.7 spearmint

spearmint/spearmint2:

Spearmint is a package to perform Bayesian optimization according to the algorithms outlined in the paper (Snoek, Larochelle, and Adams 2012)

The code consists of several parts. It is designed to be modular to allow swapping out various ‘driver’ and ‘chooser’ modules. The ‘chooser’ modules are implementations of acquisition functions such as expected improvement, UCB or random. The drivers determine how experiments are distributed and run on the system. As the code is designed to run experiments in parallel (spawning a new experiment as soon a result comes in), this requires some engineering.

Spearmint2 is similar, but more recently updated and fancier; however it has a restrictive license prohibiting wide redistribution without the payment of fees. You may or may not wish to trust the implied level of development and support of 4 Harvard Professors, depending on your application.

Both of the Spearmint options (especially the latter) have opinionated choices of technology stack in order to do their optimizations, which means they can do more work for you, but require more setup, than a simple little thing like skopt. Depending on your computing environment this might be an overall plus or a minus.

7.8 SMAC

SMAC (AGPLv3)/Python SMAC3.

(sequential model-based algorithm configuration) is a versatile tool for optimizing algorithm parameters (or the parameters of some other process we can run automatically, or a function we can evaluate, such as a simulation).

SMAC has helped us speed up both local search and tree search algorithms by orders of magnitude on certain instance distributions. Recently, we have also found it to be very effective for the hyperparameter optimization of machine learning algorithms, scaling better to high dimensions and discrete input dimensions than other algorithms. Finally, the predictive models SMAC is based on can also capture and exploit important information about the model domain, such as which input variables are most important.

We hope you find SMAC similarly useful. Ultimately, we hope that it helps algorithm designers focus on tasks that are more scientifically valuable than parameter tuning.

8 Wacky

Quasi-oppositional Differential Evolution (Rahnamayan, Tizhoosh, and Salama 2007) is old, and comes from a zany field that cites compas points and Yin-Yang as its inspiration (Mahdavi, Rahnamayan, and Deb 2018). However, it is supposedly powerful and robust (“Dagstuhloid Benchmarking” 2023). What is going on there?

9 Incoming

10 References

Allen-Zhu, Li, Singh, et al. 2017. Near-Optimal Design of Experiments via Regret Minimization.” In PMLR.
———, et al. 2021. Near-Optimal Discrete Optimization for Experimental Design: A Regret Minimization Approach.” Mathematical Programming.
Chaloner, and Verdinelli. 1995. Bayesian Experimental Design: A Review.” Statistical Science.
Chan, Paulson, and Mesbah. 2023. Safe Explorative Bayesian Optimization - Towards Personalized Treatments in Plasma Medicine.” In 2023 62nd IEEE Conference on Decision and Control (CDC).
Chowdhury, and Gopalan. 2017. On Kernelized Multi-Armed Bandits.” In Proceedings of the 34th International Conference on Machine Learning.
Dagstuhloid Benchmarking.” 2023.
Dani, Hayes, and Kakade. 2007. The Price of Bandit Information for Online Optimization.” In Proceedings of the 20th International Conference on Neural Information Processing Systems. NIPS’07.
Dani, Hayes, and Kakade. n.d. “Stochastic Linear Optimization Under Bandit Feedback.”
Feurer, Klein, Eggensperger, et al. 2015. Efficient and Robust Automated Machine Learning.” In Advances in Neural Information Processing Systems 28.
Foster, Jankowiak, Bingham, et al. 2020. Variational Bayesian Optimal Experimental Design.” arXiv:1903.05480 [Cs, Stat].
Franceschi, Donini, Frasconi, et al. 2017. On Hyperparameter Optimization in Learning Systems.” In.
Frazier. 2018. A Tutorial on Bayesian Optimization.”
Garnett. 2023. Bayesian Optimization.
Gelbart, Snoek, and Adams. 2014. Bayesian Optimization with Unknown Constraints.” In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence. UAI’14.
Grumitt, Karamanis, and Seljak. 2023. Flow Annealed Kalman Inversion for Gradient-Free Inference in Bayesian Inverse Problems.”
Grünewälder, Audibert, Opper, et al. 2010. Regret Bounds for Gaussian Process Bandit Problems.” In.
Hennig, Osborne, and Kersting. 2022. Probabilistic Numerics: Computation as Machine Learning.
Higdon, Gattiker, Williams, et al. 2008. Computer Model Calibration Using High-Dimensional Output.” Journal of the American Statistical Association.
Hooten, Leeds, Fiechter, et al. 2011. Assessing First-Order Emulator Inference for Physical Parameters in Nonlinear Mechanistic Models.” Journal of Agricultural, Biological, and Environmental Statistics.
Hutter, Hoos, and Leyton-Brown. 2011. Sequential Model-Based Optimization for General Algorithm Configuration.” In Learning and Intelligent Optimization. Lecture Notes in Computer Science.
Hutter, Hoos, and Leyton-Brown. 2013. An Evaluation of Sequential Model-Based Optimization for Expensive Blackbox Functions.” In Proceedings of the 15th Annual Conference Companion on Genetic and Evolutionary Computation. GECCO ’13 Companion.
Jagalur-Mohan, and Marzouk. 2021. “Batch Greedy Maximization of Non-Submodular Functions: Guarantees and Applications to Experimental Design.” Journal of Machine Learning Research.
Krige. 1951. A Statistical Approach to Some Basic Mine Valuation Problems on the Witwatersrand.” Journal of the Southern African Institute of Mining and Metallurgy.
Krityakierne, Akhtar, and Shoemaker. 2016. SOP: Parallel Surrogate Global Optimization with Pareto Center Selection for Computationally Expensive Single Objective Problems.” Journal of Global Optimization.
Li, Jamieson, DeSalvo, et al. 2017. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization.” The Journal of Machine Learning Research.
Liu, Feng, Eriksson, et al. 2023. Sparse Bayesian Optimization.” In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics.
Mahdavi, Rahnamayan, and Deb. 2018. Opposition Based Learning: A Literature Review.” Swarm and Evolutionary Computation.
Matheron. 1963a. Traité de Géostatistique Appliquée. 2. Le Krigeage.
———. 1963b. Principles of Geostatistics.” Economic Geology.
Močkus. 1975. On Bayesian Methods for Seeking the Extremum.” In Optimization Techniques IFIP Technical Conference: Novosibirsk, July 1–7, 1974. Lecture Notes in Computer Science.
O’Hagan. 1978. Curve Fitting and Optimal Design for Prediction.” Journal of the Royal Statistical Society: Series B (Methodological).
Oliveira, Ott, and Ramos. 2021. No-Regret Approximate Inference via Bayesian Optimisation.” In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence.
Oliveira, Tiao, and Ramos. 2022. Batch Bayesian Optimisation via Density-Ratio Estimation with Guarantees.” In Advances in Neural Information Processing Systems.
Rahnamayan, Tizhoosh, and Salama. 2007. Quasi-Oppositional Differential Evolution.” In 2007 IEEE Congress on Evolutionary Computation.
Regis, and Shoemaker. 2007. A Stochastic Radial Basis Function Method for the Global Optimization of Expensive Functions.” INFORMS Journal on Computing.
———. 2009. Parallel Stochastic Global Optimization Using Radial Basis Functions.” INFORMS Journal on Computing.
———. 2013. Combining Radial Basis Function Surrogates and Dynamic Coordinate Search in High-Dimensional Expensive Black-Box Optimization.” Engineering Optimization.
Ryan, Drovandi, McGree, et al. 2016. A Review of Modern Computational Algorithms for Bayesian Optimal Design.” International Statistical Review / Revue Internationale de Statistique.
Sacks, Schiller, and Welch. 1989. Designs for Computer Experiments.” Technometrics.
Sacks, Welch, Mitchell, et al. 1989. Design and Analysis of Computer Experiments.” Statistical Science.
Shahriari, Swersky, Wang, et al. 2016. Taking the Human Out of the Loop: A Review of Bayesian Optimization.” Proceedings of the IEEE.
Snoek, Larochelle, and Adams. 2012. Practical Bayesian Optimization of Machine Learning Algorithms.” In Advances in Neural Information Processing Systems.
Snoek, Swersky, Zemel, et al. 2014. Input Warping for Bayesian Optimization of Non-Stationary Functions.” In Proceedings of the 31st International Conference on Machine Learning (ICML-14).
Srinivas, Krause, Kakade, et al. 2010. Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design.” In Proceedings of the 27th International Conference on International Conference on Machine Learning. ICML’10.
Staines, and Barber. 2012. Variational Optimization.”
Steinberg, Oliveira, Ong, et al. 2024. Variational Search Distributions.”
Swersky, Rubanova, Dohan, et al. 2020. Amortized Bayesian Optimization over Discrete Spaces.” In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI).
Swersky, Snoek, and Adams. 2013. Multi-Task Bayesian Optimization.” In Advances in Neural Information Processing Systems 26.
Tiao, Louis C, Klein, Archambeau, et al. 2020. “Bayesian Optimization by Density Ratio Estimation.” In.
Tiao, Louis C., Klein, Seeger, et al. 2021. BORE: Bayesian Optimization by Density-Ratio Estimation.” In Proceedings of the 38th International Conference on Machine Learning.
Wang, Dahl, Swersky, et al. 2022. Pre-Training Helps Bayesian Optimization Too.”
Wang, Kim, and Kaelbling. 2018. Regret Bounds for Meta Bayesian Optimization with an Unknown Gaussian Process Prior.”