Belief Propagation Algorithms for Crowdsourcing

  • Qiang Liu | UC Irvine

Crowdsourcing on platforms like Amazon’s Mechanical Turk have become a popular paradigm for labeling large datasets. However, it has given rise to the computational task of properly aggregating the crowdsourced labels provided by a collection of unreliable and diverse annotators. On the other side, graphical models are powerful tools for reasoning on systems with complicated dependency structures. In this talk, we approach the crowdsourcing problem by transforming it into a standard inference problem in graphical models, and apply powerful inference algorithms such as belief propagation (BP). We show both the naive majority voting and a recent algorithm by Karger, Oh, and Shah are special cases of our BP algorithm under particular modeling choices. With more careful choices, we show that our simple BP performs surprisingly well on both simulated and real-world datasets, competitive with state-of-the-art algorithms based on more complicated modeling assumptions. Our work sheds light on the important tradeoff between better modeling choices and better inference algorithms.

Speaker Details

Qiang Liu is a Ph.D. student in the Bren school of information and computer sciences at UC Irvine. His research focuses on machine learning and probabilistic graphical models, with applications to areas such as sensor networks, computational biology and crowdsourcing. He received a Microsoft Research Fellowship in 2011, and was an intern in Microsoft Research Redmond in summer 2012. He received a notable paper award at the 2011 AI and Statistics conference.

    • Portrait of Jeff Running

      Jeff Running

Series: Microsoft Research Talks