Implicit variational inference

Variational inference without densities

2024-05-10 — 2024-10-20

Wherein implicit variational inference is presented, models whose likelihoods are intractable are employed and KL-based losses are recovered via adversarial-style density-ratio estimation.

approximation
metrics
optimization
probabilistic algorithms
probability
statistics
Figure 1

Variational inference using generative models whose density cannot be evaluated. See Variational Inference using Implicit Models.

Even though it does not evaluate likelihoods, implicit VI still seems to use KL divergence as a loss function.

There seems to be a connection to adversarial learning too.

Figure 2: From David Burt and Andrew Foong: Introduction to Implicit Variational Inference.

1 Connection to adversarial training

Related concepts, perhaps? Variational interpretation of adversarial losses:

  • Che et al. (2020)
  • Tiao, Bonilla, and Ramos (2018)

2 Semi-implicit variational inference

Not sure yet. I should check out Conor Hassan’s implementation.

See (Cheng et al. 2024; Lim and Johansen 2024; Moens et al. 2021; Molchanov et al. 2019; Yu et al. 2024; Yu and Zhang 2023).

3 References

Cheng, Yu, Xie, et al. 2024. “Kernel Semi-Implicit Variational Inference.”
Che, Zhang, Sohl-Dickstein, et al. 2020. “Your GAN Is Secretly an Energy-Based Model and You Should Use Discriminator Driven Latent Sampling.” arXiv:2003.06060 [Cs, Stat].
Huszár. 2017. “Variational Inference Using Implicit Distributions.”
Karaletsos. 2016. “Adversarial Message Passing For Graphical Models.”
Lim, and Johansen. 2024. “Particle Semi-Implicit Variational Inference.”
Moens, Ren, Maraval, et al. 2021. “Efficient Semi-Implicit Variational Inference.”
Mohamed, and Rezende. 2015. “Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning.” In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2. NIPS’15.
Molchanov, Kharitonov, Sobolev, et al. 2019. “Doubly Semi-Implicit Variational Inference.” In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics.
Smith, and Loaiza-Maya. 2023. “Implicit Copula Variational Inference.” Journal of Computational and Graphical Statistics.
Tiao, Bonilla, and Ramos. 2018. “Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference.”
Tran, Ranganath, and Blei. 2017. “Hierarchical Implicit Models and Likelihood-Free Variational Inference.” In Advances in Neural Information Processing Systems 30.
Uppal, Stensbo-Smidt, Boomsma, et al. 2023. “Implicit Variational Inference for High-Dimensional Posteriors.”
Yu, Xie, Zhu, et al. 2024. “Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration.” In Proceedings of the 37th International Conference on Neural Information Processing Systems. NIPS ’23.
Yu, and Zhang. 2023. “Semi-Implicit Variational Inference via Score Matching.”