Decision-Theoretic Control for Crowdsourcing

May 2, 2011
Peng Dai | University of Washington

Crowd-sourcing is a recent framework in which human intelligence tasks are outsourced to a crowd of unknown people (”workers”) as an open call (e.g., on Amazon’s Mechanical Turk). It is also known as “artificial artificial intelligence”. Crowd-sourcing has become immensely popular with hoards of employers (”requesters”), who use it to solve a wide variety of jobs, such as dictation transcription, content screening, translation, information extraction, etc. In order to achieve quality results, requesters often subdivide a large task into a chain of bite-sized sub-tasks that are combined into a complex, iterative workflow in which workers check and improve each others’ results.
In this talk, I will introduce a planner, TURKONTROL, which formulates workflow control as a decision-theoretic optimization problem, trading off the implicit quality of a solution artifact against the cost for workers to achieve it. We learn the models from real data, and demonstrate that the dynamic workflow, generated by the decision-theoretic agent based on the model, produces (statistically-significant) higher-quality worker outputs compared to the existing static workflow, under equal monetary assumptions.
I will also elaborate the mathematical model underneath the planner – Markov decision processes (MDPs), and briefly introduce a couple of new approaches that solve MDPs optimally: one that significantly speeds up the convergence of planning by using the problems’ graphical information, the other that exploits the availability of external memory to solve much larger problems than previously attempted.

Speaker Details

Peng Dai is a PhD candidate in the Department of Computer Science and Engineering at the University of Washington. He received a B.S. degree in Computer Science from Nanjing University, and M.S. degrees in Computer Science from National University of Singapore and University of Kentucky.
His research interests in computer science are in the areas of Artificial Intelligence, with emphasis on decision making under uncertainty, scalable, automated planning, and decision-theoretic planning applications for human computation.
Mr. Dai is the author of over 15 papers and technical reports. He is a member of AAAI since 2005. Mr. Dai has served the program committee member for three top AI conferences and has reviewed for over 10 top AI conferences and journals