Automatic Differentiation Variational Inference

About

Probabilistic modeling is iterative. A scientist posits a simple model, fits it to her data, refines it according to her analysis, and repeats. However, fitting complex models to large data is a bottleneck in this process. Deriving algorithms for new models can be both mathematically and computationally challenging, which makes it difficult to efficiently cycle through the steps. To this end, we develop automatic differentiation variational inference (ADVI). Using our method, the scientist only provides a probabilistic model and a dataset, nothing else. ADVI automatically derives an efficient variational inference algorithm, freeing the scientist to refine and explore many models. ADVI supports a broad class of models-no conjugacy assumptions are required. We study ADVI across ten different models and apply it to a dataset with millions of observations. ADVI is integrated into Stan, a probabilistic programming system; it is available for immediate use.

Alp Kucukelbir, Dustin Tran, Rajesh Ranganath, Andrew Gelman, David M. Blei · 2016

Related benchmarks

Task	Dataset	Result
Time Series Forecasting	ETTh2	MSE0.2415	796
Time Series Forecasting	ECL	MAE0.2376	345
Time Series Forecasting	ILI	MAE0.9299	141
Spatiotemporal Inference	Synthetic 2D Grid Matérn-3/2 kernel	ESS/sec483.24	35
Time Series Forecasting	Weather	MSE0.1273	14
Image Classification	Pets	Error Rate7.91	12
Image Classification	Aircraft	Error Rate45.42	12
Image Classification	DTD	Error Rate32.83	12
Image Classification	Flowers	Error Rate (%)12.656	12
Spatiotemporal Inference	non-separable spatiotemporal kernel	Inference Time (s)105.48	5