Optimizing Matrix Computations for Learning

Matrix-vector notation is the predominant idiom in which machine learning formulae are expressed; some models, like Gaussian processes (Rasmussen & Williams, 2006), would be extremely difficult to describe without it. Turning a matrix expression into a computer program is not always easy, however. Although good implementations of primitive matrix operations are available (Golub & Van Loan, 1996) as are packages like MATLAB1, which provide a high-level interface to these primitives, two important tasks must still be carried out manually: (i) computing derivatives of matrix functions and (ii) turning a matrix expression into an efficient computer program. Not having tools to do this can and does harm research: even for the relatively simple example of fitting a linear regression model with gradient methods, the number of types and combinations of basis functions a  researcher can experiment with is limited by the need to manually differentiate the objective function and write code for each version. We have addressed these issues by combining a symbolic matrix algebra engine with a  superoptimizing compiler: an interesting learning problem in itself. We call our system Coconut.