Icebreaker: Element-wise Efficient Information Acquisition with a Bayesian Deep Latent Gaussian Model (NeurIPS 2019)

In this paper we address the ice-start problem, i.e., the challenge of deploying machine learning models when only little or no training data is initially available, and acquiring each feature element of data is associated with costs.

This setting is representative of the real-world machine learning applications. For instance, in the health-care domain, obtaining every single measurement comes with a cost. We propose Icebreaker, a principled framework for element-wise training data acquisition. Icebreaker introduces a full Bayesian Deep Latent Gaussian Model (BELGAM) with a novel inference method, {which} combines recent advances in amortized inference and stochastic gradient MCMC to enable fast and accurate posterior inference. By utilizing BELGAM’s ability to fully quantify model uncertainty, we also propose two information acquisition functions for imputation and active prediction problems. We demonstrate that BELGAM performs significantly better than previous variational autoencoder (VAE) based models, when the data set size is small, using both machine learning benchmarks and real-world recommender systems and health-care applications. Moreover, based on BELGAM, Icebreaker further improves the performance and demonstrates the ability to use a minimum amount of the training data to obtain the highest test time performance.