What if you like the flavours of both Bayesian inference and the implicit model selection of sparse inference? Can you cook Bayesian-Frequentist fusion cuisine with this novelty ingredient? Yes, Bayes model selection has a sparsity flavour.
Laplace Prior
Laplace priors on linear regression coefficients, includes normal lasso as a MAP estimate.
Pro: It is easy to derive frequentist LASSO as a MAP estimate from this prior.
Con: Not actually sparse for non-MAP uses.
I have no need for this right now, but I did I might start with Dan Simpsonβs critique.
Spike-and-slab prior
π
Horseshoe prior
Stan guy, Michael Betancourt introduces some issues with LASSO-type inference for Bayesians with a slant towards Horseshoe-type priors in preference spike and slab, possibly because hierarchical mixtures like spike-and-slab are not that easy in Stan, albeit possible.
Global-local shrinkage hierarchy
Bhadra et al. (2016);Polson and Scott (2012);Schmidt and Makalic (2020);Xu et al. (2017).
References
Babacan, S. Derin, Martin Luessi, Rafael Molina, and Aggelos K. Katsaggelos. 2012. βSparse Bayesian Methods for Low-Rank Matrix Estimation.β IEEE Transactions on Signal Processing 60 (8): 3964β77.
Bhadra, Anindya, Jyotishka Datta, Nicholas G. Polson, and Brandon Willard. 2016. βDefault Bayesian Analysis with Global-Local Shrinkage Priors.β Biometrika 103 (4): 955β69.
Bondell, Howard D., and Brian J. Reich. 2012. βConsistent High-Dimensional Bayesian Variable Selection via Penalized Credible Regions.β Journal of the American Statistical Association 107 (500): 1610β24.
Brodersen, Kay H., Fabian Gallusser, Jim Koehler, Nicolas Remy, and Steven L. Scott. 2015. βInferring Causal Impact Using Bayesian Structural Time-Series Models.β The Annals of Applied Statistics 9 (1): 247β74.
Carvalho, Carlos M., Nicholas G. Polson, and James G. Scott. 2009. βHandling Sparsity via the Horseshoe.β In Artificial Intelligence and Statistics, 73β80.
βββ. 2010. βThe Horseshoe Estimator for Sparse Signals.β Biometrika 97 (2): 465β80.
Castillo, IsmaΓ«l, Johannes Schmidt-Hieber, and Aad van der Vaart. 2015. βBayesian Linear Regression with Sparse Priors.β The Annals of Statistics 43 (5).
George, Edward I., and Robert McCulloch. 1997. βApproaches for bayesian variable selection.β Statistica Sinica 7 (2): 339β73.
Ishwaran, Hemant, and J. Sunil Rao. 2005. βSpike and Slab Variable Selection: Frequentist and Bayesian Strategies.β The Annals of Statistics 33 (2): 730β73.
Madigan, David, and Adrian E. Raftery. 1994. βModel Selection and Accounting for Model Uncertainty in Graphical Models Using Occamβs Window.β Journal of the American Statistical Association 89 (428): 1535β46.
Mitchell, T. J., and J. J. Beauchamp. 1988. βBayesian Variable Selection in Linear Regression.β Journal of the American Statistical Association 83 (404): 1023β32.
Piironen, Juho, and Aki Vehtari. 2017. βSparsity Information and Regularization in the Horseshoe and Other Shrinkage Priors.β Electronic Journal of Statistics 11 (2): 5018β51.
Polson, Nicholas G., and James G. Scott. 2012. βLocal Shrinkage Rules, LΓ©vy Processes and Regularized Regression.β Journal of the Royal Statistical Society: Series B (Statistical Methodology) 74 (2): 287β311.
RoΔkovΓ‘, Veronika. 2018. βBayesian Estimation of Sparse Signals with a Continuous Spike-and-Slab Prior.β The Annals of Statistics 46 (1): 401β37.
RoΔkovΓ‘, Veronika, and Edward I. George. 2018. βThe Spike-and-Slab LASSO.β Journal of the American Statistical Association 113 (521): 431β44.
Schmidt, Daniel F., and Enes Makalic. 2020. βLog-Scale Shrinkage Priors and Adaptive Bayesian Global-Local Shrinkage Estimation.β arXiv.
Schniter, Philip, Lee C. Potter, and Justin Ziniel. 2008. βFast Bayesian Matching Pursuit.β In 2008 Information Theory and Applications Workshop, 326β33. San Diego, CA, USA: IEEE.
Scott, Steven L., and Hal R. Varian. 2013. βPredicting the Present with Bayesian Structural Time Series.β SSRN Scholarly Paper ID 2304426. Rochester, NY: Social Science Research Network.
Seeger, Matthias, Florian Steinke, and Koji Tsuda. 2007. βBayesian Inference and Optimal Design in the Sparse Linear Model.β In Artificial Intelligence and Statistics, 444β51. PMLR.
Smith, Michael, and Robert Kohn. 1996. βNonparametric Regression Using Bayesian Variable Selection.β Journal of Econometrics 75 (2): 317β43.
Tang, Xueying, Xiaofan Xu, Malay Ghosh, and Prasenjit Ghosh. 2016. βBayesian Variable Selection and Estimation Based on Global-Local Shrinkage Priors.β arXiv.
Titsias, Michalis K., and Miguel LΓ‘zaro-Gredilla. 2011. βSpike and Slab Variational Inference for Multi-Task and Multiple Kernel Learning.β In Advances in Neural Information Processing Systems 24, edited by J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, 2339β47. Curran Associates, Inc.
Xu, Zemei, Daniel F. Schmidt, Enes Makalic, Guoqi Qian, and John L. Hopper. 2017. βBayesian Sparse Global-Local Shrinkage Regression for Selection of Grouped Variables.β arXiv.
Zhou, Mingyuan, Haojun Chen, John Paisley, Lu Ren, Guillermo Sapiro, and Lawrence Carin. 2009. βNon-Parametric Bayesian Dictionary Learning for Sparse Image Representations.β In Proceedings of the 22nd International Conference on Neural Information Processing Systems, 22:2295β2303. NIPSβ09. Red Hook, NY, USA: Curran Associates Inc.
No comments yet. Why not leave one?