Variational inference

Content

  • Kullback–Leibler divergence and evidence lower bound (ELBO)
  • Estimation methods: coordinate ascent, black-box, automatic differentiation variational inference schemes

Learning objectives

At the end of the chapter, students should be able to

  • explain the link between ELBO and reverse Kullback–Leibler divergence and it use for approximations
  • derive the CAVI recursions for a given factorization
  • use variational inference for Bayesian inference

Readings

Complementary readings

  • Ranganath et al. (2014)
  • Blei et al. (2017)
  • Ormerod & Wand (2010)
  • Hoffman et al. (2013)

Slides

 View all slides in new window  Download PDF of all slides

Code

  • R script of CAVI with Gaussian model, probit regression, K-components univariate Gaussian mixture model and linear mixed models from Ormerod & Wand (2010)
  • R script of ADVI for logistic regression model
  • R script of ADVI for stochastic volatility model (with Stan)

References

Bishop, C. M. (2006). Pattern recognition and machine learning (information science and statistics). Springer-Verlag.
Blei, D. M., Kucukelbir, A., & and, J. D. M. (2017). Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518), 859–877. https://doi.org/10.1080/01621459.2017.1285773
Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. (2013). Stochastic variational inference. Journal of Machine Learning Research, 14(40), 1303–1347. http://jmlr.org/papers/v14/hoffman13a.html
Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., & Blei, D. M. (2017). Automatic differentiation variational inference. Journal of Machine Learning Research, 18(14), 1–45. http://jmlr.org/papers/v18/16-107.html
Ormerod, J. T., & Wand, M. P. (2010). Explaining variational approximations. The American Statistician, 64(2), 140–153. https://doi.org/10.1198/tast.2010.09058
Ranganath, R., Gerrish, S., & Blei, D. (2014). Black box variational inference. In S. Kaski & J. Corander (Eds.), Proceedings of the seventeenth international conference on artificial intelligence and statistics (Vol. 33, pp. 814–822). Pmlr. https://proceedings.mlr.press/v33/ranganath14.html