A/B testing : A systematic literature review

• DOI: 10.48550/arxiv.2308.04929
• Publication Date: 2023-01-01
• Authors: Federico Quin; Danny Weyns; Matthias Galster; Camila Costa Silva
• Keywords: FOS: Computer and information sciences; Technology; Science & Technology; Programvaruteknik; Systematic literature review; MODELS; systematic literature review; Software Engineering; SOFTWARE; Computer Science, Software Engineering; PRODUCT; A/B test engineering; Software Engineering (cs.SE); Computer Science - Software Engineering; Computer Science, Theory & Methods; Computer Science; EXPERIMENTATION; KNOWLEDGE; A/B testing

In A/B testing two variants of a piece of software are compared in the field from an end user's point of view, enabling data-driven decision making. While widely used in practice, no comprehensive study has been conducted on the state-of-the-art in A/B testing. This paper reports the results of a systematic literature review that analyzed 141 primary studies. The results shows that the main targets of A/B testing are algorithms and visual elements. Single classic A/B tests are the dominating type of tests. Stakeholders have three main roles in the design of A/B tests: concept designer, experiment architect, and setup technician. The primary types of data collected during the execution of A/B tests are product/system data and user-centric data. The dominating use of the test results are feature selection, feature rollout, and continued feature development. Stakeholders have two main roles during A/B test execution: experiment coordinator and experiment assessor. The main reported open problems are enhancement of proposed approaches and their usability. Interesting lines for future research include: strengthen the adoption of statistical methods in A/B testing, improving the process of A/B testing, and enhancing the automation of A/B testing.