PAC-Bayesian Machine Learning: Learning by Optimizing a Performance Guarantee

September 16, 2011
Mario Marchand | Laval University

The goal of machine learning algorithms is to produce predictors having the smallest possible risk (expected loss). Since the quantity to optimize (the risk) is defined only with respect to the data-generating distribution, and not with respect to the data itself, we still do not know exactly what should be optimized on the training data in order to produce a predictor having the smallest possible risk. But a natural learning strategy is to try to optimize a good guarantee on the risk provided that such a guarantee can be computed efficiently on the available data. PAC-Bayes theory has recently emerged as a good framework for deriving such guarantees in the form of, so-called, risk bounds which can be computed on the training data. In this talk, I will present several successes that we have obtained recently using this approach—which is to first derive a risk bound and then design a learning algorithm that finds a predictor having a minimal risk bound (and, consequently the best performance guarantee).

Speaker Details

Mario Marchand has obtained a PhD in Physics from Sherbrooke University in 1987. After a Postdoc in Germany (at the IFF der KFA, Jülich), he has joined the Physics department of the University of Ottawa in 1988 where he became Associate Professor in 1995. He then joined the Computer Science department (at the University of Ottawa) in 1996 and then moved to the Computer Science and Software Engineering department at Laval University (in Quebec city) in 2003 where he is currently Professor and Chairman since 2008. Mario Marchand has been working in the field of Machine Learning for more than 20 years. His recent contributions include the development of learning algorithms and risk bounds for sample-compressed classifiers (such as the Set Covering Machine) and the design of kernel methods based on PAC-Bayes theory.