Home > Read or listen to the book > What are fair tests of treatments? > Taking account of the play of chance

Taking account of the play of chance

Download a pdf of The play of chance and the law of large numbers

Key point

  • Account must be taken of ‘the play of chance’ by assessing the confidence that can be placed in the quality and quantity of evidence available

Introduction:  The play of chance and the law of large numbers

Trustworthy evidence about the effects of treatments relies on preventing biases (and of dealing with those that have not been prevented). Unless these characteristics of fair tests have been achieved, no amount of manipulation of the results of research can solve the problems that will remain, and their dangerous – sometimes lethal – consequences. Even when the steps taken to reduce biases have been successful, however, one can still be misled by the play of chance.

Everyone realizes that if you toss a coin repeatedly it is not all that uncommon to see ‘runs’ of five or more heads or tails, one after the other. And everyone realizes that the more times you toss a coin, the more likely it is that you will end up with similar numbers of heads and tails.

When comparing two treatments, any differences in results may simply reflect this play of chance. Say 40% of patients die after Treatment A compared with 60% of similar patients who die after receiving Treatment B. Table 1 shows what you would expect if 10 patients received each of the two treatments. The difference in the number of deaths between the two treatments is expressed as a ‘risk ratio’. The risk ratio in this example is 0.67.

Treatment A Treatment B Risk Ratio
Number who died 4 6 (4:6 =) 0.67
Out of (total) 10 10
Table 1. Does this small study provide a reliable estimate of the difference between Treatment A and Treatment B?

Based on these small numbers, would it be reasonable to conclude that Treatment A was better than Treatment B? Probably not. Chance might be the reason that some people got better in one group rather than the other. If the comparison was repeated in other small groups of patients, the numbers who died in each group might be reversed (6 against 4), or come out the same (5 against 5), or in some other ratio – just by chance.

But what would you expect to see if exactly the same proportion of patients in each treatment group (40% and 60%) died after 100 patients had received each of the treatments (Table 2)? Although the measure of difference (the risk ratio) is exactly the same (0.67) as in the comparison shown in Table 1, 40 deaths compared with 60 deaths is a more impressive difference than 4 compared with 6, and less likely to reflect the play of chance.

Table 2. Does this moderate-sized study provide a reliable estimate of the difference between Treatment A and Treatment B?
Treatment A Treatment B Risk Ratio
Number who died 40 60 (40:60 =) 0.67
Out of (total) 100 100

So, the way to avoid being misled by the play of chance in treatment comparisons is to base conclusions on studying sufficiently large numbers of patients who die, deteriorate or improve, or stay the same. This is sometimes referred to as ‘the law of large numbers’.

  • S4BE

    The following review of this resource is written by Gareth Grant, a 3rd year medical student. You can view the comment on the Students 4 Best Evidence website here: http://www.students4bestevidence.net/taking-account-play-chance-testing-treatments/

    Roughly how long did it take you to read/complete?
    3 minutes

    What did you think of the resource? (e.g. was it clear or difficult to understand?)
    It was clear and easy to use as there is audio but also text and tables to follow along.

    Do you feel it improved your understanding?
    Yes, it’s short and concise but addresses the point and summarises the topic without wordy explanations. It features good examples, without getting too technical with numbers.

    What problems did you find with it (if any) / how do you think it could be improved?
    I did have to refresh the page to access the second recording but that may have just been me – related to cookies I think. I think the source itself was good and doesn’t need improvement.

    Overall, what would you score the resource out of 5? (and why?)
    5 – it clearly summarises the role of chance and was easy to understand. Also, it’s good if (like me) you prefer audio/video sources to reading.