Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models


September 29, 2011


Emtiyaz Khan


University of British Columbia


Bernoulli-logistic Latent Gaussian Models (bLGMs) subsume many popular models for binary data, such as Bayesian logistic regression, Gaussian process classification, probabilistic principal components analysis, and factor analysis. Fitting this models is difficult due to an intractable logistic-Gaussian integral in the marginal likelihood. Even the standard variational framework, which involves application of Jensen’s inequality, does not make the integral tractable.

In this work, we propose the use of fi xed piecewise linear and quadratic upper bounds to the logistic-log-partition (LLP) function as a way of circumventing this intractable integral. We describe a framework for approximately computing minimax optimal piecewise quadratic bounds, as well a generalized expectation maximization algorithm based on using piecewise bounds to estimate bLGMs. We prove a theoretical result relating the maximum error in the LLP bound to the maximum error in the marginal likelihood estimate.

Through application to real-world data, we show that the proposed bounds achieve better estimation accuracy than existing variational bounds with a little increase in computation. We also show that, unlike existing sampling methods, our methods offer guaranteed convergence, easy convergence diagnostics, and scale well to datasets containing thousands of variables and instances. Finally, we illustrate the application of our bounds to model categorical and ordinal data with latent Gaussian models.

This is joint work with Kevin Murphy and Benjamin Marlin.


Emtiyaz Khan

Emti (aka Mohammad Emtiyaz Khan or MT) is a PhD student in the Department of Computer Science at the University of British Columbia, Vancouver, where he works under the supervision of Prof. Kevin Murphy. His research interests are in the area of machine learning and statistics. His PhD work is focused on developing accurate, efficient, and scalable algorithms for fitting hierarchical Bayesian models. He previously received his MSc from Indian Institute of Science, Bangalore, India where he worked on the application of machine learning algorithms to Brain-computer interfaces. He also worked in the control and communication research group at Honeywell Technology Solutions Lab, Bangalore, India. Over the years, Emti has realized that referring to himself in the third person is acceptable at times.