ALICE

ALICE

Established: December 5, 2016

Publications

Projects

Projects

Overview

Automated Learning and Intelligence for Causation and Economics

Alice is a project to direct Artificial Intelligence towards economic decision making. We are building tools that combine state-of-the-art machine learning with econometrics – the measurement of economic systems — in order to bring automation to economic decision making. The heart of this project is a striving to measure causation: if you want to understand or make policy decisions in a complex economy, you need to know why the system moves the way it does.

Microsoft has a long tradition of work at the intersection of Economics and Computer Science, and this team brings together researchers from across Social Science, AI, and ML. The Alice project allows us to dramatically scale the success that we’ve had in adapting existing ML technology for economic applications and in developing new deep learning architectures for causal inference. All of our research is tied to concrete policy-relevant applications, including work on demand estimation and price optimization, on measuring effectiveness of advertising and sales strategies, and on the design of incentives for desirable healthcare and education outcomes. The goal is to use AI to improve and democratize economic research while taking from economic theory to push the frontier of AI.

People

Core Team

Affiliates

Case Study

A Microsoft & TripAdvisor Case Study

Causal AI for Customer Segmentation

Asian woman looking at her cellphone pulling a rolling suitcase and wearing a backpack in an airport

TripAdvisor is an online travel research company that empowers people around the world to plan and enjoy the ideal trip. TripAdvisor’s mission is to give travelers a platform to share their experiences to promote transparency in the travel industry and enable informed consumer choices.

The ALICE project at Microsoft Research New England1 applies Artificial Intelligence concepts to economic decision making. To make policy decisions in a complex economy, you need to know why the system moves the way it does. The ALICE team’s innovative tools make this kind of cause and effect analysis more reliable, scalable, and accessible for data scientists without extensive economic training. These tools are collected in the open source, production level quality EconML library.2

TripAdvisor’s ability to provide users with accurate information hinges on other travelers sharing their experiences on the platform. TripAdvisor and MSR joined forces to understand whether a membership model would improve user engagement on the website.

The Challenge

On a cold New England day, a chance encounter between two data scientists working on causal problems, one at MSR and one at TripAdvisor, uncovered an exciting opportunity for collaboration: MSR could better understand the causal questions of interest to business decision makers and build intelligent, flexible and reusable solutions, while TripAdvisor would learn about state-of-the-art tools that they could immediately apply to inform business decisions.

The two teams agreed to work together on a problem of mutual interest: leveraging an existing A/B test for user retention policies to understand the effect of a membership model on user engagement. The ALICE team was perfectly positioned to contribute research and build new machine learning tools to tackle this problem.

Strategy

TripAdvisor wanted to know whether promoting membership on their platform could drive engagement and bookings. They couldn’t just look directly at the existing data, comparing members and non-members, because the customers who had chosen to become members were exactly the most engaged ones. Nor could they run a direct A/B test because they could not force users to sign up for membership. Fortunately, TripAdvisor had just run an experiment to explore user retention by offering a random subset of customers an easier sign-up process for membership.

the workflow diagram of TripAdvisor's membership promotion A/B test

arrow pointing down with a circle around it on a lime green rectangular background

”Developing a deep understanding of our travelers so we can create truly relevant experiences is at the core of what we do at TripAdvisor. Our partnership with Microsoft Research has allowed us to unlock critical insights that inform how to improve those experiences in a 1:1 manner.”
Matthew Dacey, Vice President Membership and Growth at TripAdvisor

Results

The ALICE team developed a new statistical method to exploit this indirect nudge towards membership to measure the direct effects of membership on engagement. Their approach expands on the traditional economic technique of instrumental variables, which adjusts for imperfect compliance: some customers who were offered the easier sign-up still declined membership while some customers without the easier process signed up anyway.

TripAdvisor found that membership does have a positive effect on user engagement, but this effect varies widely across users. Some major drivers of this variation are the platform from which the user accesses the TripAdvisor platform and the pages that the user had visited before the experiment.

graph showing the average visits per user during the experiment, whether they were an iPhone user and if they visited the vacation rental or experience pages

The key innovation that revealed this variation is the development of a Machine Learning (ML) based method for estimating heterogeneous causal effects in A/B tests with non-compliance. Traditional instrumental variable methods do not allow for complex individual-level differences in either compliance with the experiment or the effect of the intervention.

Male tourist in patterned shirt holding a cellphone looking up on busy street

MSR’s novelty was to reduce the problem to a sequence of standard Machine Learning steps and take advantage of the many recent algorithmic and statistical advances in machine learning on non-linear and high-dimensional modelling. The ML algorithms allow data scientists to automatically segment the market in a datadriven manner as opposed to predefining a small set of key user features based on prior, perhaps incorrect, beliefs.

The methodology has been implemented in our production level quality EconML library.2 The technical details of this methodology were published at the 33rd Conference on Neural Information Processing Systems.3

Moving Forward

Thanks to their collaboration with Microsoft Research, TripAdvisor can understand and differentiate their customers in new ways. TripAdvisor is now more focused on providing membership opportunities to iPhone users and customers who visit vacation rental pages.

The EconML software package includes a suite of cutting-edge statistical tools to answer causal questions. In addition to this kind of market segmentation, the EconML package is well suited to estimating price sensitivities and designing personalized pricing plans or attributing growth in revenue across multiple drivers, such as customer engagements or advertising campaigns. For more information visit the ALICE team website1 or the EconML website.2

1 https://www.microsoft.com/en-us/research/project/alice/
2 https://econml.azurewebsites.net/
3 “Machine Learning Estimation of Heterogeneous Treatment Effects with Instruments”(NeurIPS’19)
https://arxiv.org/abs/1905.10176