Supercharge A/B testing with automated causal inference | Community Workshop on Microsoft’s Causal Tools

An A/B test consists of splitting the customers into a test and a control group, and choosing a large enough sample size to observe the average treatment effect (ATE) we are interested in, in spite of all the other factors driving outcome variance. With causal inference models, we can do better than that, by estimating the effect conditional on customer features (CATE), thus turning customer variability from noise to be averaged over to a valuable source of segmentation, and potentially requiring smaller sample sizes as a result. Unfortunately, there are many different models available for estimating CATE, with many parameters to tune and very different performance. In this talk, we will present our auto-causality library, which combines the three marvelous packages from Microsoft – DoWhy, EconML, and FLAML – to do fully automated selection and tuning of causal models based on out-of-sample performance, just like any other AutoML package does. We will describe the projects inside Wise currently starting to apply it, and present rather striking results on comparative model performance and out-of-sample segmentation on Wise CRM data.

Date:

May 3, 2022

Speakers:

Egor Kraev

Affiliation:

Wise