Bayesian sparsity

2019-01-08 – 2022-10-25
Maybe related 🀷:

What if you like the flavours of both Bayesian inference and the implicit model selection of sparse inference? Can you cook Bayesian-Frequentist fusion cuisine with this novelty ingredient? Yes, Bayes model selection has a sparsity flavour.

Laplace Prior

Laplace priors on linear regression coefficients, includes normal lasso as a MAP estimate.

Pro: It is easy to derive frequentist LASSO as a MAP estimate from this prior.

Con: Not actually sparse for non-MAP uses.

I have no need for this right now, but I did I might start with Dan Simpson’s critique.

Spike-and-slab prior

πŸ—

Horseshoe prior

Stan guy, Michael Betancourt introduces some issues with LASSO-type inference for Bayesians with a slant towards Horseshoe-type priors in preference spike and slab, possibly because hierarchical mixtures like spike-and-slab are not that easy in Stan, albeit possible.

Global-local shrinkage hierarchy

Bhadra et al. (2016);Polson and Scott (2012);Schmidt and Makalic (2020);Xu et al. (2017).

References

Babacan, S. Derin, Martin Luessi, Rafael Molina, and Aggelos K. Katsaggelos. 2012. β€œSparse Bayesian Methods for Low-Rank Matrix Estimation.” IEEE Transactions on Signal Processing 60 (8): 3964–77.
Bhadra, Anindya, Jyotishka Datta, Nicholas G. Polson, and Brandon Willard. 2016. β€œDefault Bayesian Analysis with Global-Local Shrinkage Priors.” Biometrika 103 (4): 955–69.
Bondell, Howard D., and Brian J. Reich. 2012. β€œConsistent High-Dimensional Bayesian Variable Selection via Penalized Credible Regions.” Journal of the American Statistical Association 107 (500): 1610–24.
Brodersen, Kay H., Fabian Gallusser, Jim Koehler, Nicolas Remy, and Steven L. Scott. 2015. β€œInferring Causal Impact Using Bayesian Structural Time-Series Models.” The Annals of Applied Statistics 9 (1): 247–74.
Carvalho, Carlos M., Nicholas G. Polson, and James G. Scott. 2009. β€œHandling Sparsity via the Horseshoe.” In Artificial Intelligence and Statistics, 73–80.
β€”β€”β€”. 2010. β€œThe Horseshoe Estimator for Sparse Signals.” Biometrika 97 (2): 465–80.
George, Edward I., and Robert McCulloch. 1997. β€œApproaches for bayesian variable selection.” Statistica Sinica 7 (2): 339–73.
Ishwaran, Hemant, and J. Sunil Rao. 2005. β€œSpike and Slab Variable Selection: Frequentist and Bayesian Strategies.” The Annals of Statistics 33 (2): 730–73.
Madigan, David, and Adrian E. Raftery. 1994. β€œModel Selection and Accounting for Model Uncertainty in Graphical Models Using Occam’s Window.” Journal of the American Statistical Association 89 (428): 1535–46.
Mitchell, T. J., and J. J. Beauchamp. 1988. β€œBayesian Variable Selection in Linear Regression.” Journal of the American Statistical Association 83 (404): 1023–32.
Piironen, Juho, and Aki Vehtari. 2017. β€œSparsity Information and Regularization in the Horseshoe and Other Shrinkage Priors.” Electronic Journal of Statistics 11 (2): 5018–51.
Polson, Nicholas G., and James G. Scott. 2012. β€œLocal Shrinkage Rules, LΓ©vy Processes and Regularized Regression.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 74 (2): 287–311.
RočkovΓ‘, Veronika. 2018. β€œBayesian Estimation of Sparse Signals with a Continuous Spike-and-Slab Prior.” The Annals of Statistics 46 (1): 401–37.
RočkovΓ‘, Veronika, and Edward I. George. 2018. β€œThe Spike-and-Slab LASSO.” Journal of the American Statistical Association 113 (521): 431–44.
Schmidt, Daniel F., and Enes Makalic. 2020. β€œLog-Scale Shrinkage Priors and Adaptive Bayesian Global-Local Shrinkage Estimation.” arXiv.
Schniter, Philip, Lee C. Potter, and Justin Ziniel. 2008. β€œFast Bayesian Matching Pursuit.” In 2008 Information Theory and Applications Workshop, 326–33. San Diego, CA, USA: IEEE.
Scott, Steven L., and Hal R. Varian. 2013. β€œPredicting the Present with Bayesian Structural Time Series.” SSRN Scholarly Paper ID 2304426. Rochester, NY: Social Science Research Network.
Seeger, Matthias, Florian Steinke, and Koji Tsuda. 2007. β€œBayesian Inference and Optimal Design in the Sparse Linear Model.” In Artificial Intelligence and Statistics, 444–51. PMLR.
Smith, Michael, and Robert Kohn. 1996. β€œNonparametric Regression Using Bayesian Variable Selection.” Journal of Econometrics 75 (2): 317–43.
Tang, Xueying, Xiaofan Xu, Malay Ghosh, and Prasenjit Ghosh. 2016. β€œBayesian Variable Selection and Estimation Based on Global-Local Shrinkage Priors.” arXiv.
Titsias, Michalis K., and Miguel LΓ‘zaro-Gredilla. 2011. β€œSpike and Slab Variational Inference for Multi-Task and Multiple Kernel Learning.” In Advances in Neural Information Processing Systems 24, edited by J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, 2339–47. Curran Associates, Inc.
Xu, Zemei, Daniel F. Schmidt, Enes Makalic, Guoqi Qian, and John L. Hopper. 2017. β€œBayesian Sparse Global-Local Shrinkage Regression for Selection of Grouped Variables.” arXiv.
Zhou, Mingyuan, Haojun Chen, John Paisley, Lu Ren, Guillermo Sapiro, and Lawrence Carin. 2009. β€œNon-Parametric Bayesian Dictionary Learning for Sparse Image Representations.” In Proceedings of the 22nd International Conference on Neural Information Processing Systems, 22:2295–2303. NIPS’09. Red Hook, NY, USA: Curran Associates Inc.

Download bibliography

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.