{"id":4601,"date":"2015-11-09T09:00:45","date_gmt":"2015-11-09T17:00:45","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/msr_er\/?p=4601"},"modified":"2016-08-17T16:44:29","modified_gmt":"2016-08-17T23:44:29","slug":"extensive-crowdsourcing-study-reveals-surprising-results-in-reversal-of-platform-design-assumptions","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/extensive-crowdsourcing-study-reveals-surprising-results-in-reversal-of-platform-design-assumptions\/","title":{"rendered":"Extensive crowdsourcing study reveals surprising results in reversal of platform design assumptions"},"content":{"rendered":"<p><em>By John Kaiser, Research News<\/em><\/p>\n<p>A 21-month study on the dynamics of crowdsourcing shows workers depend on collaborating with each other, upending platform designers\u2019 long-held assumptions of crowdsourcing tasks performed in isolation.<\/p>\n<p>\u201cPeople spend far more time on a task, trying to get it right, than we might expect, given the pay rates for many tasks,&#8221; said MSR lead researcher <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/mlg\/\" target=\"_blank\">Mary Gray<\/a>, who presented the study during this week\u2019s Conference on Human Computation & Crowdsourcing in San Diego. \u201cFinding a network of support makes all the difference for people learning how to do crowdwork efficiently.&#8221;<\/p>\n<p>In seeking to replicate the social support networks associated with traditional workplaces, crowdworkers motivate each other and create systems of collaboration to make even low-paying crowdwork pay off, the MSR research shows.<\/p>\n<p>In the conference keynote Tuesday, <em>Crowdwork\u2019s Invisible Engine: Valuing the Organic Collaboration in Crowdsourcing Labor Markets, <\/em>Gray explained how crowdworkers collaborate with members of their networks to manage administrative overhead and find lucrative tasks and reputable employers. But it\u2019s the intangible benefits that crowdworkers derive from recreating real-world social connections that compelled researchers to look deeper into the underlying crowdsourcing ecosystem.<\/p>\n<p>The study, co-authored with MSR researcher <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/suri\/\" target=\"_blank\">Siddharth Suri<\/a>, focused on four platforms: Amazon Mechanical Turk, one of the largest micro-task platforms, Amara (a video captioning and translation service), Leadgenius (sales leads), and UHRS (Microsoft\u2019s micro-task platform).<\/p>\n<p>The MSR researchers collected experiences from crowdworkers in the U.S. and India amassing:<\/p>\n<ul>\n<li>2,762 worker surveys.<\/li>\n<li>138 India-based interviews and 52 U.S.-based interviews.<\/li>\n<li>Millions of workflows and bits of metadata.<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2015\/11\/crowdsourcelocations.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-5102 \" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2015\/11\/crowdsourcelocations.jpg\" alt=\"crowdsource_participant_locations\" width=\"739\" height=\"339\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2015\/11\/crowdsourcelocations.jpg 800w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2015\/11\/crowdsourcelocations-300x138.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2015\/11\/crowdsourcelocations-768x352.jpg 768w\" sizes=\"auto, (max-width: 739px) 100vw, 739px\" \/><\/a><\/p>\n<p style=\"text-align: center;\"><em>Self-reported location of study participants<\/em><\/p>\n<p>Crucially, the extent of crowdworker collaboration proved to be such a surprise that measuring it was not even a goal when research started two years ago.<\/p>\n<p>\u201cWe never would have measured task sharing if we didn\u2019t learn about it through interviews,\u201d the study states.<\/p>\n<p>Gray and Suri&#8217;s findings have transformative implications for crowdsourcing APIs originally developed a decade ago with the assumption that crowdworkers mostly work alone without the need or desire to communicate with each other. The MSR researchers argue that platforms are now \u201cdependent on worker collaboration (they just don\u2019t know it).\u201d To grow and otherwise gain market share, they will need to \u201cbuild in ways for workers to collaborate.\u201d<\/p>\n<p><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2015\/11\/crowdsourceslide.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-5112\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2015\/11\/crowdsourceslide-1024x643.jpg\" alt=\"crowdsource_assumptions\" width=\"709\" height=\"445\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2015\/11\/crowdsourceslide-1024x643.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2015\/11\/crowdsourceslide-300x188.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2015\/11\/crowdsourceslide-768x483.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2015\/11\/crowdsourceslide.jpg 1068w\" sizes=\"auto, (max-width: 709px) 100vw, 709px\" \/><\/a><\/p>\n<p style=\"text-align: center;\"><em>Crowdsourcing assumptions<\/em><\/p>\n<p>\u201cThe challenge is how to get engineers to embrace technology that facilitates social interaction,\u201d Gray says. \u201cThe effort so far has been to engineer the collaboration\u201d instead of providing tools to enable social interaction.<\/p>\n<p>Typical Mturk tasks include describing an image or highlighting purchased items from Walmart sales receipts for 3-5 cents on completion. Higher paying tasks include translating TED talks or transcribing recordings, yielding about $18 for what is estimated to take an hour or more.<\/p>\n<p>The economics of crowdsourcing could soon extend into other industries beyond the existing base of service, technology, management, HR, scientific research, and arts\/design, according to the MSR study. These industries include: Real estate, banking, health care, education, leisure, and tourism.<\/p>\n<p>Are these trends merely a precursor to total machine automation? On the contrary, the MSR study echoes the traditional economic theory of innovation as an overall job creator in what it calls the \u201cParadox of tech innovation\u2019s last mile\u201d creating rather than eliminating markets.<\/p>\n<p>For more computer science research news, visit <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" href=\"http:\/\/researchnews.com\/\" target=\"_blank\">ResearchNews.com<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>By John Kaiser, Research News A 21-month study on the dynamics of crowdsourcing shows workers depend on collaborating with each other, upending platform designers\u2019 long-held assumptions of crowdsourcing tasks performed in isolation. \u201cPeople spend far more time on a task, trying to get it right, than we might expect, given the pay rates for many [&hellip;]<\/p>\n","protected":false},"author":32627,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[],"msr_hide_image_in_river":0,"footnotes":""},"categories":[194479,194459],"tags":[186484,195162,186459,195773],"research-area":[13548,13554],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-4601","post","type-post","status-publish","format-standard","hentry","category-economics","category-researchnews","tag-collaboration","tag-conference-on-human-computation-crowdsourcing","tag-crowdsourcing","tag-hcomp-2015","msr-research-area-economics","msr-research-area-human-computer-interaction","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[],"related-researchers":[],"msr_type":"Post","byline":"","formattedDate":"November 9, 2015","formattedExcerpt":"By John Kaiser, Research News A 21-month study on the dynamics of crowdsourcing shows workers depend on collaborating with each other, upending platform designers\u2019 long-held assumptions of crowdsourcing tasks performed in isolation. \u201cPeople spend far more time on a task, trying to get it right,&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/4601","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/32627"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=4601"}],"version-history":[{"count":3,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/4601\/revisions"}],"predecessor-version":[{"id":278589,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/4601\/revisions\/278589"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=4601"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=4601"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=4601"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=4601"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=4601"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=4601"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=4601"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=4601"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=4601"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=4601"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=4601"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}