What if I like the flavours of both Bayesian inference and the implicit model selection of sparse inference? Can I cook Bayesian-Frequentist fusion cuisine with this novelty ingredient? It turns out that yes, we can add a variety of imitation sparsity flavours to Bayes model selection. The resulting methods are handy in, for example, symbolic system identification.
Laplace Prior
Laplace priors on linear regression coefficients include normal lasso as a MAP estimate.
Pro: It is easy to derive frequentist LASSO as a MAP estimate from this prior.
Con: Full posterior is not sparse, only the MAP estimate.
I have no need for this right now, but if I did, I might start with Dan Simpson’s critique.
Spike-and-slab prior
Probabilistic equivalent of a LASSO-type prior, via a hierarchical mixture, i.e. an inclusion probability for each coefficient.
Pro: Full posterior can be sparse in some sense.
Con: Mixtures of discrete and continuous variables like this are fiddly to deal with in MCMC, and just in general.
🏗
Thompson sampling for large model spaces
This looks cool: Liu and Ročková (2023).
Global-local shrinkage hierarchy
Not quite sure what this is, but here are some papers: Bhadra et al. (2016);Polson and Scott (2012);Schmidt and Makalic (2020);Xu et al. (2017).
References
Babacan, Luessi, Molina, et al. 2012.
“Sparse Bayesian Methods for Low-Rank Matrix Estimation.” IEEE Transactions on Signal Processing.
Bhadra, Datta, Polson, et al. 2019.
“Lasso Meets Horseshoe : A Survey.”
Brodersen, Gallusser, Koehler, et al. 2015.
“Inferring Causal Impact Using Bayesian Structural Time-Series Models.” The Annals of Applied Statistics.
Carvalho, Polson, and Scott. 2009.
“Handling Sparsity via the Horseshoe.” In
Artificial Intelligence and Statistics.
Castillo, Schmidt-Hieber, and van der Vaart. 2015.
“Bayesian Linear Regression with Sparse Priors.” The Annals of Statistics.
George, and McCulloch. 1997.
“Approaches for bayesian variable selection.” Statistica Sinica.
Herzet, and Drémeau. 2014.
“Bayesian Pursuit Algorithms.”
Liu, and Ročková. 2023.
“Variable Selection Via Thompson Sampling.” Journal of the American Statistical Association.
Mitchell, and Beauchamp. 1988.
“Bayesian Variable Selection in Linear Regression.” Journal of the American Statistical Association.
Polson, and Scott. 2012.
“Local Shrinkage Rules, Lévy Processes and Regularized Regression.” Journal of the Royal Statistical Society: Series B (Statistical Methodology).
Ročková, and George. 2018.
“The Spike-and-Slab LASSO.” Journal of the American Statistical Association.
Schniter, Potter, and Ziniel. 2008.
“Fast Bayesian Matching Pursuit.” In
2008 Information Theory and Applications Workshop.
Scott, and Varian. 2013.
“Predicting the Present with Bayesian Structural Time Series.” SSRN Scholarly Paper ID 2304426.
Seeger, Steinke, and Tsuda. 2007. “Bayesian Inference and Optimal Design in the Sparse Linear Model.” In Artificial Intelligence and Statistics.
Titsias, and Lázaro-Gredilla. 2011.
“Spike and Slab Variational Inference for Multi-Task and Multiple Kernel Learning.” In
Advances in Neural Information Processing Systems 24.
Wang, Sarkar, Carbonetto, et al. 2020.
“A Simple New Approach to Variable Selection in Regression, with Application to Genetic Fine Mapping.” Journal of the Royal Statistical Society Series B: Statistical Methodology.
Zhou, Chen, Paisley, et al. 2009.
“Non-Parametric Bayesian Dictionary Learning for Sparse Image Representations.” In
Proceedings of the 22nd International Conference on Neural Information Processing Systems. NIPS’09.