(17.33) A randomized comparative experiment examined the effect of a technique for improving one’s ability to focus one’s attent
ion on performance of undergraduate students on the Verbal portion of the Graduate Record Exam (GRE). The researchers found a statistically significant improvement in performance on the Verbal GRE (P = 0.05). When asked to explain the meaning of (P =0.05) a student says, “This means there is only probability equal to 0.05 that the null hypothesis is true.” Explain what P= 0.05 really means in a way that makes it clear that the student's explanation is wrong. P = 0.05 means that the outcome is at least 5% higher among one group than among the other group. P = 0.05 means that there is a probability of 0.05 that the null hypothesis is false. P = 0.05 means that the probability, assuming the null hypothesis is true, that the test statistic will take a value at least as extreme as that actually observed, is 0.05. P = 0.05 means that the probability, assuming the null hypothesis is true, that the test statistic will take a value at least as extreme as that actually observed, is less than 0.05.
What the significance level tells us is that, in this case, there is only a 5% chance that the change in results from the sample happened by random chance. So that means that most likely the change in results did not happen by random change, and something happen between the times the samples were taken. In this case, the experiment.
