Assessing and reducing bias for a Bayesian AB test

Abstract

Working with data from Free To Play videogames, we usually have heavy-tailed distributions that can be quite complex to study. When performing AB tests to evaluate certain changes in the games, the different users’ spending behaviour might lead to a pretest bias that can affect the results. Our aim is to find a way to analyze the tests that considers the information from the users before entering the test but also takes into account the pretest bias. We will study the current method used to analyze AB tests at the company and identify some of its problems, such as the use of the probability to beat the control group as a metric. We will also explore some variance reduction techniques, such as CUPED, to get better results focusing on the ARPU and Conversion.