Stochastic Dual Coordinate Ascent and its Proximal Extension for Regularized Loss Minimization

  • Tong Zhang | Rutgers University

Stochastic Gradient Descent (SGD) has become popular for solving large scale supervised machine learning optimization problems such as SVM, due to their strong theoretical guarantees. While the closely related Dual Coordinate Ascent (DCA) method has been implemented in various software packages, it has so far lacked good convergence analysis. We present a new analysis of Stochastic Dual Coordinate Ascent (SDCA) showing that this class of methods enjoy strong theoretical guarantees that are comparable or better than SGD. This analysis justifies the effectiveness of SDCA for practical applications.

Moreover, we introduce a proximal version of dual coordinate ascent method. We demonstrate how the derived algorithmic framework can be used for numerous regularized loss minimization problems, including L1 regularization and structured output SVM. The convergence rates we obtain match or improve state-of-the-art results.

Joint work with Shai Shalev-Shwartz

Speaker Details

Tong Zhang received a B.A. in mathematics and computer science from Cornell University in 1994 and a Ph.D. in Computer Science from Stanford University in 1998. After graduation, he worked at IBM T.J. Watson Research Center in Yorktown Heights, New York, and Yahoo Research in New York city. He is currently a statistics professor at Rutgers University. His research interests include machine learning, algorithms for statistical computation, their mathematical analysis and applications.

    • Portrait of Jeff Running

      Jeff Running

Series: Microsoft Research Talks