Significance tests, type I and type II errors

The null hypothesis (H₀) is a formal statement that there is no effect, no difference, or no association between variables being studied. A significance test uses the sample data to assess how likely the null hypothesis is to be correct.

For example:

• 'there is no difference in the prevalence of colorectal cancer in patients taking low-dose aspirin compared to those who are not'

The alternative hypothesis (H₁) is the opposite of the null hypothesis, i.e. There is a difference between the two treatments

The p value is the probability of obtaining a result by chance at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. It is therefore equal to the chance of making a type I error (see below).

Two types of errors may occur when testing the null hypothesis

• type I: the null hypothesis is rejected when it is true - i.e. Showing a difference between two groups when it doesn't exist, a false positive. This is determined against a preset significance level ,termed alpha (α). As the significance level is determined in advance the chance of making a type I error is not affected by sample size. It is however increased if the number of end-points are increased. For example if a study has 20 end-points it is likely one of these will be reached, just by chance.
• type II: the null hypothesis is not rejected when it is false - i.e. Failing to spot a difference when one really exists, a false negative. The probability of making a type II error is termed beta (β). It is determined by both sample size and α (alpha is the probability of a type I error)

The power of a study is the probability of (correctly) rejecting the null hypothesis when it is false, i.e. the probability of detecting a statistically significant difference

• power = 1 - the probability of a type II error
• power can be increased by increasing the sample size