Where can I get a sample dataset for A/B split testing?

Question

I'm working on A/B split testing now. Where can I get a sample dataset for A/B split testing?

Answer 1

Based on the answer for this question on Cross-validated there are two great packages in R that include useful datasets from completed A/B and multi-variate tests: Agricolae and Agridat

Answer 2

In the years between the question was asked and this answer is written, a number of A/B test datasets has been made freely available.

The following list include datasets that shows results arising from a randomized controlled experiment run online. Thus, it does not include offline experiment such as those in the R package Agridat.

From Research Archives

A number of research teams have either open-sourced research archives containing A/B test results, or made the results available as supplementary materials for their research articles.

• Upworthy Research Archive records 32,487 media copy experiments run on Upworthy (see article on Nature Scientific Data)
• ASSISTments Dataset records 22 experiments on ASSISTments, an online learning platform for middle school students (see paper in L@S 2016)
• Montana State Uni Library A/B Test Dataset records an UI A/B test run on the Library website (see article on Journal of Library User Experience)
• MeuTutor Dataset records 2 experiments run on MeuTutor, another online learning platform for high school students in Brazil (see article on Computers in Human Behavior)
• ASOS Digital Experiments Dataset records 78 experiments on ASOS.com, an online fashion retail website (see paper in NeurIPS 2021)

Example Datasets from MOOCs

Some MOOCs that has a module on A/B testing also include sample datasets for course participants to play with. These include:

• Udacity A/B Testing - Final Project / Free Trial Screener Experiment (Mirror 1, Mirror 2, Mirror 3)
• Udacity Online Data Analyst Course - Project 3 / Analyse A/B Test Results (Mirror 1, Mirror 2, Mirror 3)
• Datacamp Project - Mobile Games A/B Testing with Cookie Cats (Mirror 1, Mirror 2, Mirror 3)
• Datacamp A/B Testing in R - Experiment Dataset (Link to CSV file)
• Datacamp A/B Testing in R - Data Visualization Website - Apr 2018 Dataset (Link to CSV file)
• Customer Analytics and A/B Testing in Python - AB Testing Results (Link to CSV file)

Miscellaneous Datasets on Kaggle

There are also a few A/B tests available on Kaggle. Documentation on these datasets are generally on the light side and we urge caution when using these datasets to build any serious applications.

• Grocery Website Data for AB Test
• Ad A/B Testing a.k.a. SmartAd AB Data
• Synthetical A/B-Tests - A meta dataset exploring effect of having different experiment parameters in A/B tests

Datasets for Uplift Modelling

Unlike an A/B test, which generally looks for the treatment effect on a population level, Uplift Modelling seeks to under the treatment effect on an individual level. Having that said, both tasks start with a controlled experiment and so datasets constructed with Uplift Modelling tasks in mind may also be applicable.

• The MineThatData E-Mail Analytics And Data Mining Challenge - A 2008 classic.
• Criteo Uplift Modeling Dataset (see paper in AdKDD & TargetAd 2018 Workshop)

Answer 3

The R package "datasets" have several examples. Despite some are not properly "AB test" designs, you can use some factorial covariates as levels (A, B, ...).

You can see details of the detasets here: https://stat.ethz.ch/R-manual/R-devel/library/datasets/html/00Index.html