Bayesian sparsity

January 8, 2019 — October 25, 2022

Bayes
high d
information
model selection
regression
sparser than thou
statistics

What if you like the flavours of both Bayesian inference and the implicit model selection of sparse inference? Can you cook Bayesian-Frequentist fusion cuisine with this novelty ingredient? Yes, Bayes model selection has a sparsity flavour.

1 Laplace Prior

Laplace priors on linear regression coefficients, includes normal lasso as a MAP estimate.

Pro: It is easy to derive frequentist LASSO as a MAP estimate from this prior.

Con: Not actually sparse for non-MAP uses.

I have no need for this right now, but I did I might start with Dan Simpson’s critique.

2 Spike-and-slab prior

🏗

3 Horseshoe prior

Stan guy, Michael Betancourt introduces some issues with LASSO-type inference for Bayesians with a slant towards Horseshoe-type priors in preference spike and slab, possibly because hierarchical mixtures like spike-and-slab are not that easy in Stan, albeit possible.

4 Global-local shrinkage hierarchy

Bhadra et al. (2016);Polson and Scott (2012);Schmidt and Makalic (2020);Xu et al. (2017).

5 References

Babacan, Luessi, Molina, et al. 2012. “Sparse Bayesian Methods for Low-Rank Matrix Estimation.” IEEE Transactions on Signal Processing.
Bhadra, Datta, Polson, et al. 2016. “Default Bayesian Analysis with Global-Local Shrinkage Priors.” Biometrika.
Bondell, and Reich. 2012. “Consistent High-Dimensional Bayesian Variable Selection via Penalized Credible Regions.” Journal of the American Statistical Association.
Brodersen, Gallusser, Koehler, et al. 2015. “Inferring Causal Impact Using Bayesian Structural Time-Series Models.” The Annals of Applied Statistics.
Carvalho, Polson, and Scott. 2009. “Handling Sparsity via the Horseshoe.” In Artificial Intelligence and Statistics.
———. 2010. “The Horseshoe Estimator for Sparse Signals.” Biometrika.
Castillo, Schmidt-Hieber, and van der Vaart. 2015. “Bayesian Linear Regression with Sparse Priors.” The Annals of Statistics.
George, and McCulloch. 1997. “Approaches for bayesian variable selection.” Statistica Sinica.
Ishwaran, and Rao. 2005. “Spike and Slab Variable Selection: Frequentist and Bayesian Strategies.” The Annals of Statistics.
Madigan, and Raftery. 1994. “Model Selection and Accounting for Model Uncertainty in Graphical Models Using Occam’s Window.” Journal of the American Statistical Association.
Mitchell, and Beauchamp. 1988. “Bayesian Variable Selection in Linear Regression.” Journal of the American Statistical Association.
Piironen, and Vehtari. 2017. “Sparsity Information and Regularization in the Horseshoe and Other Shrinkage Priors.” Electronic Journal of Statistics.
Polson, and Scott. 2012. “Local Shrinkage Rules, Lévy Processes and Regularized Regression.” Journal of the Royal Statistical Society: Series B (Statistical Methodology).
Ročková. 2018. “Bayesian Estimation of Sparse Signals with a Continuous Spike-and-Slab Prior.” The Annals of Statistics.
Ročková, and George. 2018. “The Spike-and-Slab LASSO.” Journal of the American Statistical Association.
Schmidt, and Makalic. 2020. “Log-Scale Shrinkage Priors and Adaptive Bayesian Global-Local Shrinkage Estimation.”
Schniter, Potter, and Ziniel. 2008. “Fast Bayesian Matching Pursuit.” In 2008 Information Theory and Applications Workshop.
Scott, and Varian. 2013. “Predicting the Present with Bayesian Structural Time Series.” SSRN Scholarly Paper ID 2304426.
Seeger, Steinke, and Tsuda. 2007. “Bayesian Inference and Optimal Design in the Sparse Linear Model.” In Artificial Intelligence and Statistics.
Smith, and Kohn. 1996. “Nonparametric Regression Using Bayesian Variable Selection.” Journal of Econometrics.
Tang, Xu, Ghosh, et al. 2016. “Bayesian Variable Selection and Estimation Based on Global-Local Shrinkage Priors.”
Titsias, and Lázaro-Gredilla. 2011. “Spike and Slab Variational Inference for Multi-Task and Multiple Kernel Learning.” In Advances in Neural Information Processing Systems 24.
Xu, Schmidt, Makalic, et al. 2017. “Bayesian Sparse Global-Local Shrinkage Regression for Selection of Grouped Variables.”
Zhou, Chen, Paisley, et al. 2009. “Non-Parametric Bayesian Dictionary Learning for Sparse Image Representations.” In Proceedings of the 22nd International Conference on Neural Information Processing Systems. NIPS’09.