21 Station Road
Cambridge
CB1 2FB
View the whole series on talks.cam (opens in new tab)
Frontiers in Artificial Intelligence is a series of public lectures at Microsoft Research Cambridge featuring leading researchers in the field, focusing on the cutting edge topics at the intersection of machine learning, statistics, and artificial intelligence. Students, scientists, and engineers in academia and industry are all welcome to join us for these exciting talks and the opportunity to socialize with the Cambridge AI/ML community.
How good is your classifier? Revisiting the role of evaluation metrics in machine learning
Sanmi Koyejo – University of Illinois
With the increasing integration of machine learning into real systems, it is crucial that trained models are optimized to reflect real-world tradeoffs. Increasing interest in proper evaluation has led to a wide variety of metrics employed in practice, often specially designed by experts. However, modern training strategies have not kept up with the explosion of metrics, leaving practitioners to resort to heuristics. To address this shortcoming, I will present a simple, yet consistent post-processing rule which improves the performance of trained binary, multilabel, and multioutput classifiers. Building on these results, I will propose a framework for metric elicitation, which addresses the broader question of how one might select an evaluation metric for real world problems so that it reflects true preferences.