Sequential hypothesis testing in online controlled experiments

Abstract

[en] This thesis explores the sequential analysis paradigm in the field of mathematical statistics, where sample size is not predetermined, allowing for adaptive decisionmaking. The first chapter outlines the theory’s oundations, particularly in sequential hypothesis testing, introducing the properties of two of the most relevant sequen- tial tests: the Sequential Probability Ratio Test (SPRT) and the mixture Sequential Probability Ratio Test (mSPRT). The second chapter focuses on applying sequential hypothesis testing to online controlled experimentation, using Always Valid Inference. This alternative offers a statistically rigorous and efficient solution, potentially outperforming fixed-horizon methods. Empirical evidence of simulated real-world case-scenarios supports the proposed methodology’s advantages in online controlled experimentation.