Section 21.1 Investigation 4.4: Lingering Effects of Sleep Deprivation

 

The following dotplots reveal the distributions of improvements between the two groups:

As with the “dolphin therapy” experiment from Chapter 3, we again need to judge the strength of evidence that the experimental data provide in support of the researchers’ conjecture that sleep deprivation has a harmful effect on learning. We are now working with a quantitative response variable rather than a categorical one, but we will use the same basic logic of statistical significance: We will ask whether the observed experimental results are very unlikely to have occurred by chance variation if the explanatory variable has no effect, that is, if the values in the two groups are interchangeable. To simulate this randomization test: We will take the observed response variable outcomes, randomly distribute these numerical values between the two groups, compute the statistic of interest (e.g., the difference in group means), repeat this process a large number of times, and see how often the random assignment process alone produces a difference in group means as extreme as in the actual research study. This is similar to what you considered in Investigation 4.1. However, this time it is not really feasible to list out all possible \(C(21,10) = 352,716\) random assignments and consider the value of the statistic for each one, so we will simulate a large number of random assignments instead. Note, we need to know the value of \((\bar{x}_1 - \bar{x}_2)\text{,}\) not just the “number of successes” as with the dolphin therapy study.

 

Although not as convenient as before, we can determine the exact p-value for this randomization test by considering all the different possible random assignments of these 21 subjects into groups of 11 and 10, determining the difference in means (or medians) for each, and then counting how many are at least as extreme as the observed difference of 15.92.

The following histogram (produced using the combn function in R, from the combinat package, assuming imprvs contains the response variable values) shows the distribution of all 352,716 possible differences in group means.

The exact randomization distribution consists of every possible random assignment and calculates the statistic of interest (e.g., difference in means) for each one. Then we simply count how many of the configurations result in a value of the statistic at least as extreme (as defined by the alternative hypothesis) as the actual observed result. As you might expect, it can be extremely tedious, even with computers, to list out all of these possible random assignments. And these group sizes are relatively small! One shortcut is to only count how many assignments give results more extreme than the one observed, but as you will see we can often appeal to a mathematical model as well.

Step 1: Randomly reorder the group membership labels (keeping the same number in each group)

Step 2: Calculate the new difference in means (or other statistic) using the reordered group labels

Step 3: Repeat Steps 1 and 2 many times.

Look at the distribution of the statistic, including the mean and standard deviation, and count how many of the simulated statistics are as or more extreme as the observed.

Applet: Comparing Groups (Quantitative)

I = 10000    # use upper case
diff = 0     # initializes the vector

for (i in 1:I){                                           # the loop for I iterations
  rerandom = sample(improvement)                          # mixes up response values
  boxplot(rerandom~condition, horizontal=TRUE)
  diff[i] = mean(rerandom[1:10]) - mean(rerandom[11:21])  # match up the explanatory variable group sizes
}

hist(diff); abline(v=15.92, col=2)   # adds vertical line
sum(diff >= 15.92)/I                 # p-value