mSDA: A fast and easy-to-use way to improve bag-of-words features

June 12, 2012
Kilian Weinberger | Washington University in St. Louis

Machine learning algorithms rely heavily on the representation of the data they are presented with. In particular, text documents (and often images) are traditionally expressed as bag-of-words feature vectors (e.g. as tf-idf).
Recently Glorot et al. showed that stacked denoising autoencoders (SDA), a deep learning algorithm, can learn representations that are far superior over variants of bag-of-words. Unfortunately, training SDAs often requires a prohibitive amount of computation time and is non-trivial for non-experts.
In this work, we show that with a few modifications of the SDA model, we can relax the optimization over the hidden weights into convex optimization problems with closed form solutions. Further, we show that the expected value of the hidden weights after infinitely many training iterations can also be computed in closed form. The resulting transformation (which we call marginalized-SDA) can be computed in no more than 20 lines of straight-forward Matlab code and requires no prior expertise in machine learning.
The representations learned with mSDA behave similar to those obtained with SDA, but the training time is reduced by several orders of magnitudes. For example, mSDA matches the world-record on the Amazon transfer learning benchmark, however the training time shrinks from several days to a few minutes.

Speaker Details

Kilian Q. Weinberger is an Assistant Professor in the Department of Computer Science & Engineering at Washington University in St. Louis. He received his Ph.D. from the University of Pennsylvania in Machine Learning under the supervision of Lawrence Saul. Prior, he obtained his undergraduate degree in Mathematics and Computer Science at the University of Oxford. During his career he won several best paper awards at ICML, CVPR and AISTATS. In 2012 he was awarded the NSF CAREER award.

Kilian Weinberger’s research is in and around Machine Learning. In particular, he focus on high dimensional data analysis, feature- and metric-learning, machine learned web-search ranking, transfer- and multi-task learning, test-time cost sensitive learning and brain decoding.