My research is on estimating average treatment effects using different machine learning models. However, I need a big data set with a truly randomised experiment in order to estimate the "true" average treatment effect. Does anybody have knowledge of such a data set that is publicly available? It does not matter what it is about, just that the treatment assignment is truly random.

  • 1
    What do you mean with "truly random". Clinical trials are randomised, there are a couple of data sets available to interested researchers.
    – Grimaldi
    Sep 23 '17 at 9:33

I don't know what you mean exactly, and I assume you don't mean "double-blind" studies.

However, if you already had your own dataset: You can randomize that data yourself, and take averages of permuted values of interesting columns as a baseline.

Just augment + shuffle/permute the data table correctly, and you'll get a randomized dataset just as you need.

This code is from the Datacamp.com class "Foundations of inference" where this technique is demonstrated at length:

Dataset (2 columns, 40 rows):

      sex      promote
1    male     promoted
2    male     not promoted
3    female   promoted 

R Code:

rep_sample_n <- function (tbl, size, replace = FALSE, reps = 1) 
    n <- nrow(tbl)
    i <- unlist(replicate(reps, sample.int(n, size, replace = replace), 
        simplify = FALSE))
    rep_tbl <- cbind(replicate = rep(1:reps, rep(size, reps)), 
        tbl[i, ])
    dplyr::group_by(rep_tbl, replicate)

# Sample the entire data frame 5 times
# Shuffle the "promote" variable within replicate
# Find the proportion of promoted in each replicate and sex 
#    -> get a simple distribution, should be around mean=0   
# Difference in proportion of promoted across sex grouped by gender
disc %>%
  rep_sample_n(size = nrow(disc), reps = 5) %>%
  mutate(prom_perm = sample(promote)) %>%
  group_by(replicate, sex) %>%
  summarize(prop_prom_perm = mean(prom_perm == "promoted"),
            prop_prom = mean(promote == "promoted"))  %>%
  summarize(diff_perm = diff(prop_prom_perm),
            diff_orig = diff(prop_prom))  # male - female


  replicate   diff_perm diff_orig
      <int>       <dbl>     <dbl>
         1  0.04166667 0.2916667
         2 -0.12500000 0.2916667
         3 -0.04166667 0.2916667
         4 -0.04166667 0.2916667
         5  0.12500000 0.2916667

This shows that 5 randomized datasets derived from the original show max +-12.5 % difference in people promoted, regardless of gender, whereas the original data show a mean difference of 29% difference (when grouped by gender).

  • Thank you @knb for an excellent answer!
    – adam
    Aug 24 '21 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.