Taking account of the play of chance

Format: Texts

,

Language/s: English

,

Target Audience: Self-directed learning

|

Postgraduate and professional

|

Short Description:

Differences in outcome events in treatment comparisons may reflect only the play of chance. Increased numbers of events reduces this problem

Key Concepts addressed:

• 2-8 Consider all of the relevant fair comparisons

Details

When treatments are compared, any differences in outcome events may simply reflect the play of chance. Increasing the number of events studied in research reduces the likelihood of being misled by the play of chance.

When two treatments are compared, any differences in outcome may simply be caused by the play of chance. For example, take a comparison of a new treatment with a standard treatment in which 4 people improved with the former and 6 people improved with the latter. It would clearly be wrong to conclude confidently that the new treatment was worse than the standard treatment: these results might simply reflect the play of chance. If the comparison was repeated, the numbers of patients who improved might be reversed (6 against 4), or come out the same (5 against 5), or in some other ratio.

If, however, 40 people improved with the new treatment and 60 with the standard treatment, chance becomes a less likely explanation for the difference. And if 400 people improved with the new treatment and 600 with the standard treatment, it would be clear that the new treatment was indeed very likely to be worse than the standard. The way to reduce the likelihood of being misled by the play of chance in treatment comparisons is thus to ensure that fair tests include sufficiently large numbers of people who experience the outcomes one hopes to influence, such as increased improvement or reduced deterioration.

In some circumstances very large numbers of people – thousands and sometimes tens of thousands – need to participate in fair tests to obtain reliable estimates of treatment effects. Large numbers of participants are necessary, for example, if the treatment outcomes of interest are rare – for example, heart attacks and strokes among apparently healthy middle-aged women using hormone replacement therapy (HRT). Large numbers are also needed if moderate but important effects of treatments are to be detected reliably – for example, a reduction by 20 per cent in the risk of early death among people having heart attacks.

The text in these essays may be copied and used for non-commercial purposes on condition that explicit acknowledgement is made to The James Lind Library (www.jameslindlibrary.org).

Read more about the evolution of fair comparisons in the James Lind Library.