{"id":186193,"date":"2011-05-02T00:00:00","date_gmt":"2011-05-03T13:59:34","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/msr-research-item\/decision-theoretic-control-for-crowdsourcing\/"},"modified":"2016-08-22T11:30:42","modified_gmt":"2016-08-22T18:30:42","slug":"decision-theoretic-control-for-crowdsourcing","status":"publish","type":"msr-video","link":"https:\/\/www.microsoft.com\/en-us\/research\/video\/decision-theoretic-control-for-crowdsourcing\/","title":{"rendered":"Decision-Theoretic Control for Crowdsourcing"},"content":{"rendered":"<div class=\"asset-content\">\n<p>Crowd-sourcing is a recent framework in which human intelligence tasks are outsourced to a crowd of unknown people (\u201dworkers\u201d) as an open call (e.g., on Amazon\u2019s Mechanical Turk). It is also known as \u201cartificial artificial intelligence\u201d. Crowd-sourcing has become immensely popular with hoards of employers (\u201drequesters\u201d), who use it to solve a wide variety of jobs, such as dictation transcription, content screening, translation, information extraction, etc. In order to achieve quality results, requesters often subdivide a large task into a chain of bite-sized sub-tasks that are combined into a complex, iterative workflow in which workers check and improve each others\u2019 results.<br \/>\nIn this talk, I will introduce a planner, TURKONTROL, which formulates workflow control as a decision-theoretic optimization problem, trading off the implicit quality of a solution artifact against the cost for workers to achieve it. We learn the models from real data, and demonstrate that the dynamic workflow, generated by the decision-theoretic agent based on the model, produces (statistically-significant) higher-quality worker outputs compared to the existing static workflow, under equal monetary assumptions.<br \/>\nI will also elaborate the mathematical model underneath the planner \u2013 Markov decision processes (MDPs), and briefly introduce a couple of new approaches that solve MDPs optimally: one that significantly speeds up the convergence of planning by using the problems\u2019 graphical information, the other that exploits the availability of external memory to solve much larger problems than previously attempted.<\/p>\n<\/div>\n<p><!-- .asset-content --><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Crowd-sourcing is a recent framework in which human intelligence tasks are outsourced to a crowd of unknown people (\u201dworkers\u201d) as an open call (e.g., on Amazon\u2019s Mechanical Turk). It is also known as \u201cartificial artificial intelligence\u201d. Crowd-sourcing has become immensely popular with hoards of employers (\u201drequesters\u201d), who use it to solve a wide variety of [&hellip;]<\/p>\n","protected":false},"featured_media":196114,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr_hide_image_in_river":0,"footnotes":""},"research-area":[],"msr-video-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-session-type":[],"msr-impact-theme":[],"msr-pillar":[],"msr-episode":[],"msr-research-theme":[],"class_list":["post-186193","msr-video","type-msr-video","status-publish","has-post-thumbnail","hentry","msr-locale-en_us"],"msr_download_urls":"","msr_external_url":"https:\/\/youtu.be\/cBmhEpE3uAk","msr_secondary_video_url":"","msr_video_file":"","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video\/186193","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-video"}],"version-history":[{"count":0,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video\/186193\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/196114"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=186193"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=186193"},{"taxonomy":"msr-video-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video-type?post=186193"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=186193"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=186193"},{"taxonomy":"msr-session-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-session-type?post=186193"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=186193"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=186193"},{"taxonomy":"msr-episode","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-episode?post=186193"},{"taxonomy":"msr-research-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-theme?post=186193"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}