Random binary vectors

The class of distributions that cause you to reinvent Shannon information if you stare at them long enough

2017-02-20 — 2022-03-30

Wherein distributions over n-length binary vectors are examined, and the existence of 2^n possible outcomes is noted, with continuous Gumbel-softmax relaxations and piano-roll representations being described.

classification
compsci
information
metrics
statistics
Figure 1: Robert Fludd’s piano rolls

Distributions over random boolean vectors. Useful in computer science and piano rolls. Not quite the same as categorical distributions, although those can be written as distributions over boolean vectors. In a multi-class classification case, each realisation has only one class; in an \(n\)-class rv, there are \(n\) possible outcomes. In a multivariate Bernoulli distribution, there are \(2^n\) possible outcomes.

1 Continuous relaxations

Multivariate Gumbel-softmax tricks.

2 Paintbox models

Not sure how these work but maybe related. See (Broderick, Pitman, and Jordan 2013; Zhang and Paisley 2019).

3 Matrix models

TBC.

See, e.g. Lumbreras, Filstroff, and Févotte (2020)

4 References

Broderick, Pitman, and Jordan. 2013. “Feature Allocations, Probability Functions, and Paintboxes.” Bayesian Analysis.
Dai, Ding, and Wahba. 2013. “Multivariate Bernoulli Distribution.” Bernoulli.
Lumbreras, Filstroff, and Févotte. 2020. “Bayesian Mean-Parameterized Nonnegative Binary Matrix Factorization.” Data Mining and Knowledge Discovery.
Miettinen, and Neumann. 2020. “Recent Developments in Boolean Matrix Factorization.” In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence.
Reuter, Vo, Vo, et al. 2014. “The Labeled Multi-Bernoulli Filter.” IEEE Transactions on Signal Processing.
Rukat, Holmes, Titsias, et al. 2017. “Bayesian Boolean Matrix Factorisation.” In Proceedings of the 34th International Conference on Machine Learning.
Teugels. 1990. “Some Representations of the Multivariate Bernoulli and Binomial Distributions.” Journal of Multivariate Analysis.
Vo, Ba-Tuong, and Vo. 2013. “Labeled Random Finite Sets and Multi-Object Conjugate Priors.” IEEE Transactions on Signal Processing.
Vo, Ba-Tuong, Vo, and Cantoni. 2009. “The Cardinality Balanced Multi-Target Multi-Bernoulli Filter and Its Implementations.” IEEE Transactions on Signal Processing.
Vo, Ba Ngu, Vo, and Hoang. 2017. “An Efficient Implementation of the Generalized Labeled Multi-Bernoulli Filter.” arXiv:1606.08350 [Stat].
Vo, Ba-Ngu, Vo, and Phung. 2014. “Labeled Random Finite Sets and the Bayes Multi-Target Tracking Filter.” IEEE Transactions on Signal Processing.
Wang, and Yin. 2020. “Relaxed Multivariate Bernoulli Distribution and Its Applications to Deep Generative Models.” In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI).
Zhang, and Paisley. 2019. “Random Function Priors for Correlation Modeling.” In International Conference on Machine Learning.
Zhou, Hannah, Dunson, et al. 2012. “Beta-Negative Binomial Process and Poisson Factor Analysis.” In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics.