Algorithmic Crowdsourcing

Established: February 1, 2012

We study algorithmic issues for combining the intelligence of human and the computing power of machine to solve the problems that are difficult to solve by either human or machine alone.

Contact person: Denny Zhou

People

Denny Zhou (Microsoft Research)
John Platt (Google)
Xi Chen (Carnegie Mellon University)
Nihar Shah (University of California at Berkeley)
Chao Gao (Yale University)
Qiang Liu (University of California at Irvine)
Yuchen Zhang (University of California at Berkeley)
Yuval Peres (Microsoft Research)
Chris Meek (Microsoft Research)
Tengyu Ma (Princeton University)

Publications

N. Shah and  D. Zhou. Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing.  Journal of Machine Learning Research, 17(165):1-52, 2016

N. Shah and D. Zhou. No Oops, You Won’t Do It Again: Mechanisms for Self-correction in Crowdsourcing.   Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016.

C. Gao, Y. Lu and D. Zhou. Exact Exponent in Optimal Rates for Crowdsourcing.  Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016.

Y. Zhang, X. Chen, D. Zhou and M. I. Jordan. Spectral Methods Meet EM: A Provably Optimal Algorithm for Crowdsourcing.  Journal of Machine Learning Research, 17(102):1-44, 2016. [code/data]

N. Shah and D. Zhou. Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing.  Advances in Neural Information Processing Systems (NIPS) 28, 2015.

N. B. Shah,  D. Zhou and Y. Peres. Approval Voting and Incentives in Crowdsourcing. Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015. (long version)

N. B. Shah and  D. Zhou. On the Impossibility of Convex Inference in Human Computation. Proceedings of the 29th AAAI Conference on Artificial Intelligence, 2015.

X. Chen, Q. Lin, and D. Zhou. Statistical Decision Making for Optimal Budget Allocation in Crowd Labeling. Journal of Machine Learning Research, 16 (Jan):1-46, 2015.

Y. Zhang, X. Chen, D. Zhou and M. I. Jordan. Spectral Methods Meet EM: A Provably Optimal Algorithm for Crowdsourcing.  Advances in Neural Information Processing Systems (NIPS) 27, 2014.

D. Zhou, Q. Liu, J. C. Platt, and C. Meek. Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy. Proceedings of the 31st International Conference on Machine Learning (ICML), 2014. [code/data]

X. Chen, Q. Lin, and D. Zhou. Optimistic Knowledge Gradient Policy for Optimal Budget Allocation in Crowdsourcing. Proceedings of the 30th International Conference on Machine Learning (ICML), 2013.

D. Zhou, J. C. Platt, S. Basu, and Y. Mao. Learning from the Wisdom of Crowds by Minimax Entropy. Advances in Neural Information Processing Systems (NIPS) 25, 2204-2212, 2012. [code/data]

Invited talks

Incentive mechanisms for crowdsourcing data labeling. Distinguished Lecture Series at Jump Trading LLC, Chicago. December 14, 2016.

Incentives in Human Computation. Computer Science Department, Yale University. Hosted by Prof. Dan Spielman.  May 7, 2015.

Incentives in Human Computation. Microsoft TechFest, March 26, 2015.

Double or Nothing: Unique Mechanism to Incentivize MTurkers to Skip. Computer Science and Engineering Department, University of Washington. Hosted by Prof. Anna Karlin. March 6, 2015. (slides)

Double or Nothing: Unique Mechanism to Incentivize MTurkers to Skip. Facebook, March 6, 2015.

Double or Nothing: Unique Mechanism to Incentivize MTurkers to Skip. Microsoft Bing, November 5, 2014.

Algorithmic Crowdsourcing. NIPS Workshop on Crowdsourcing: Theory, Algorithms and Applications, December 9, 2013. (slides)

Learning from the Wisdom of Crowds by Minimax Entropy. Amazon, July 25, 2013.

Learning from the Wisdom of Crowds by Minimax Entropy. UC Berkeley, Neyman Seminar, March 15, 2013. Hosted by Prof. Bin Yu and Aditya Guntuboyina. (slides)

Learning from the Wisdom of Crowds by Minimax Entropy. Facebook, March 14, 2013.

Learning from the Wisdom of Crowds by Minimax Entropy. Joint UW-Microsoft Research Machine Learning Workshop. Oct 26, 2012.

Organized workshops

NIPS’14 workshop: Crowdsourcing and Machine Learning, Montreal, Quebec, Canada, 2014.

ICML’14 workshop: Crowdsourcing and Human Computing, Beijing, China, 2014.

NIPS’13 workshop: Crowdsourcing: Theory, Algorithms and Applications, Lake Tahoe, Nevada, United States, 2013.

ICML’13 workshop: Machine Learning Meets Crowdsourcing, Atlanta, United States, 2013.onta