Variational inference
Content
- Kullback–Leibler divergence and evidence lower bound (ELBO)
- Estimation methods: coordinate ascent, black-box, automatic differentiation variational inference schemes
Learning objectives
At the end of the chapter, students should be able to
- explain the link between ELBO and reverse Kullback–Leibler divergence and it use for approximations
- derive the CAVI recursions for a given factorization
- use variational inference for Bayesian inference
Readings
- Chapter 10 of Bishop (2006)
- Kucukelbir et al. (2017)
- Chapter 10 of the course notes
Complementary readings
Slides
View all slides in new window Download PDF of all slides
Code
References
Bishop, C. M. (2006). Pattern recognition and machine learning (information science and statistics). Springer-Verlag.
Blei, D. M., Kucukelbir, A., & and, J. D. M. (2017). Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518), 859–877. https://doi.org/10.1080/01621459.2017.1285773
Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. (2013). Stochastic variational inference. Journal of Machine Learning Research, 14(40), 1303–1347. http://jmlr.org/papers/v14/hoffman13a.html
Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., & Blei, D. M. (2017). Automatic differentiation variational inference. Journal of Machine Learning Research, 18(14), 1–45. http://jmlr.org/papers/v18/16-107.html
Ormerod, J. T., & Wand, M. P. (2010). Explaining variational approximations. The American Statistician, 64(2), 140–153. https://doi.org/10.1198/tast.2010.09058
Ranganath, R., Gerrish, S., & Blei, D. (2014). Black box variational inference. In S. Kaski & J. Corander (Eds.), Proceedings of the seventeenth international conference on artificial intelligence and statistics (Vol. 33, pp. 814–822). Pmlr. https://proceedings.mlr.press/v33/ranganath14.html