Reducing the play of chance in systematic reviews

Download a pdf of Reducing the play of chance in systematic reviews

Previously, we explained how the play of chance can be reduced by combining data from similar but separate studies – a process known as ‘meta-analysis’. We used the example of five studies in five different countries organized and funded separately to address a 60-year-old quandary about what blood level of oxygen in prematurely born infants is needed to maximize the likelihood that they will survive with no major disabilities.

That example illustrated how this process could be planned before the results of the studies were available, but the same process can be used after a group of similar studies have been completed. For example, in 1974 a Swedish doctor conducted a systematic review of studies comparing the results of surgery for breast cancer with or without radiotherapy. [6] He found that, in all of the studies, women were more likely to die in the groups receiving radiotherapy.

When all of this evidence was synthesized statistically using meta-analysis, it became clear that this excess mortality was unlikely to reflect the play of chance. Subsequent, more detailed analyses, based on evidence from individual patients, confirmed that the radiotherapy being used during that era did indeed increase mortality. [7] Recognizing this led to the development of safer practices.