{"id":171481,"date":"2015-06-23T21:06:39","date_gmt":"2015-06-23T21:06:39","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/project\/multi-world-testing-mwt\/"},"modified":"2020-04-15T15:29:11","modified_gmt":"2020-04-15T22:29:11","slug":"multi-world-testing-mwt","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/multi-world-testing-mwt\/","title":{"rendered":"Multiworld Testing"},"content":{"rendered":"

\"\"Exponentially better than A\/B testing.<\/strong> Multiworld Testing\u00a0(MWT) is the capability to test and optimize over K policies (context-based decision rules) using an amount of data and computation that scales logarithmically in K, without necessarily knowing these policies before or during data collection. MWT can answer exponentially more detailed questions compared to traditional A\/B testing. The underlying machine learning methodology draws on research on “contextual bandits” and “counterfactual evaluation”.<\/p>\n

A system for interactive learning.<\/strong> We implement MWT as a machine learning system for making context-based decisions. The system supports the full cycle from exploration to logging to training policies to deploying them in production. Built as a cloud service, the system is widely applicable, modular, and easy to use. We currently offer two versions of the system: MWT Decision Service (opens in new tab)<\/span><\/a>\u00a0(self-hosted) and Custom Decision Service (opens in new tab)<\/span><\/a>\u00a0(multitenant). This is an ongoing project, released internally in Jun’15 and announced externally (opens in new tab)<\/span><\/a> in Jul’16. Custom Decision Service in public preview since May’17, as a part of Microsoft Azure Cognitive Services (opens in new tab)<\/span><\/a>. A version of the system is already deployed very successfully with MSN (opens in new tab)<\/span><\/a>.\u00a0\u00a0<\/i>
\n\"Multiworld<\/p>\n

A typical example.<\/strong> Suppose one wants to optimize clicks on suggested news stories. To discover what works, one needs to explore over the possible news stories. Further, if the suggested news story can be chosen depending on the visitor’s profile, then one needs to explore over the possible “policies” that map profiles to news stories (and there are exponentially more “policies” than news stories!).\u00a0Traditional machine learning fails at this because it does not explore. Whereas the Decision Service can explore continuously, and optimize decisions using this exploration data.<\/p>\n

Team.<\/strong>\u00a0We are a diverse group of researchers working on all aspects of MWT, spanning algorithms, machine learning, systems, and economics, and covering the entire range from theory to experiments to practical deployments. Most of us are located at Microsoft Research NYC<\/a>. We can be contacted at mwtdev@microsoft.com<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

Exponentially better than A\/B testing. Multiworld Testing\u00a0(MWT) is the capability to test and optimize over K policies (context-based decision rules) using an amount of data and computation that scales logarithmically in K, without necessarily knowing these policies before or during data collection. MWT can answer exponentially more detailed questions compared to traditional A\/B testing. The […]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"research-area":[13561,13556,13547],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-171481","msr-project","type-msr-project","status-publish","hentry","msr-research-area-algorithms","msr-research-area-artificial-intelligence","msr-research-area-systems-and-networking","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"2013-11-01","related-publications":[],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[419964],"related-articles":[],"tab-content":[{"id":0,"name":"Releases","content":"Custom Decision Service<\/a>\r\nMay 2017: public preview release, as a part of Cognitive Services<\/a>.\r\n\r\nMWT Decision Service<\/a>\r\nJul 2016: external announcement.\r\n\r\nMWT Exploration library<\/a>\r\nA library for MWT, structurally compatible with learning algorithms in Vowpal Wabbit<\/a>.\r\nNov 2014: external release.\r\n\r\nMWT white paper<\/a>\u00a0(rev. March 2016)\r\nJul 2016: rev2 released\r\nSep 2015: released externally\r\n\r\nDeployment<\/strong>: personalized news\u00a0on msn.com<\/a>\r\nDeployed on 100%\u00a0of the traffic; 25% lift in clicks<\/em>.\r\nInnovation Award<\/strong><\/em> from Microsoft's Universal Storefronts."},{"id":1,"name":"Timeline","content":"