A Microsoft & TripAdvisor Case Study
Causal AI for Customer Segmentation
TripAdvisor is an online travel research company that empowers people around the world to plan and enjoy the ideal trip. TripAdvisor’s mission is to give travelers a platform to share their experiences to promote transparency in the travel industry and enable informed consumer choices.
The ALICE project at Microsoft Research New England1 applies Artificial Intelligence concepts to economic decision making. To make policy decisions in a complex economy, you need to know why the system moves the way it does. The ALICE team’s innovative tools make this kind of cause and effect analysis more reliable, scalable, and accessible for data scientists without extensive economic training. These tools are collected in the open source, production level quality EconML library.2
TripAdvisor’s ability to provide users with accurate information hinges on other travelers sharing their experiences on the platform. TripAdvisor and MSR joined forces to understand whether a membership model would improve user engagement on the website.
The Challenge
On a cold New England day, a chance encounter between two data scientists working on causal problems, one at MSR and one at TripAdvisor, uncovered an exciting opportunity for collaboration: MSR could better understand the causal questions of interest to business decision makers and build intelligent, flexible and reusable solutions, while TripAdvisor would learn about state-of-the-art tools that they could immediately apply to inform business decisions.
The two teams agreed to work together on a problem of mutual interest: leveraging an existing A/B test for user retention policies to understand the effect of a membership model on user engagement. The ALICE team was perfectly positioned to contribute research and build new machine learning tools to tackle this problem.
Strategy
TripAdvisor wanted to know whether promoting membership on their platform could drive engagement and bookings. They couldn’t just look directly at the existing data, comparing members and non-members, because the customers who had chosen to become members were exactly the most engaged ones. Nor could they run a direct A/B test because they could not force users to sign up for membership. Fortunately, TripAdvisor had just run an experiment to explore user retention by offering a random subset of customers an easier sign-up process for membership.
”Developing a deep understanding of our travelers so we can create truly relevant experiences is at the core of what we do at TripAdvisor. Our partnership with Microsoft Research has allowed us to unlock critical insights that inform how to improve those experiences in a 1:1 manner.”—Matthew Dacey, Vice President Membership and Growth at TripAdvisorResults
The ALICE team developed a new statistical method to exploit this indirect nudge towards membership to measure the direct effects of membership on engagement. Their approach expands on the traditional economic technique of instrumental variables, which adjusts for imperfect compliance: some customers who were offered the easier sign-up still declined membership while some customers without the easier process signed up anyway.
TripAdvisor found that membership does have a positive effect on user engagement, but this effect varies widely across users. Some major drivers of this variation are the platform from which the user accesses the TripAdvisor platform and the pages that the user had visited before the experiment.
The key innovation that revealed this variation is the development of a Machine Learning (ML) based method for estimating heterogeneous causal effects in A/B tests with non-compliance. Traditional instrumental variable methods do not allow for complex individual-level differences in either compliance with the experiment or the effect of the intervention.
MSR’s novelty was to reduce the problem to a sequence of standard Machine Learning steps and take advantage of the many recent algorithmic and statistical advances in machine learning on non-linear and high-dimensional modelling. The ML algorithms allow data scientists to automatically segment the market in a datadriven manner as opposed to predefining a small set of key user features based on prior, perhaps incorrect, beliefs.
The methodology has been implemented in our production level quality EconML library.2 The technical details of this methodology were published at the 33rd Conference on Neural Information Processing Systems.3
Moving Forward
Thanks to their collaboration with Microsoft Research, TripAdvisor can understand and differentiate their customers in new ways. TripAdvisor is now more focused on providing membership opportunities to iPhone users and customers who visit vacation rental pages.
The EconML software package includes a suite of cutting-edge statistical tools to answer causal questions. In addition to this kind of market segmentation, the EconML package is well suited to estimating price sensitivities and designing personalized pricing plans or attributing growth in revenue across multiple drivers, such as customer engagements or advertising campaigns. For more information visit the ALICE team website1 or the EconML website.2
1 https://www.microsoft.com/en-us/research/project/alice/ (opens in new tab)
2 https://econml.azurewebsites.net/ (opens in new tab)
3 “Machine Learning Estimation of Heterogeneous Treatment Effects with Instruments”(NeurIPS’19)
https://arxiv.org/abs/1905.10176 (opens in new tab)