Home > Read or listen to the book > What are fair tests of treatments? > Fair tests of treatments > Comparing like with like > Treatments with moderate but important effects > Comparing apparently similar groups of patients who happen to have received different treatments in the same time period

Comparing apparently similar groups of patients who happen to have received different treatments in the same time period

Download a pdf of Comparing apparently similar groups of patients who happen to have received different treatments in the same time period

Comparing the experiences and outcomes of apparently similar groups of patients who happen to have received different treatments in the same time period is still used as a way to try to assess the effects of treatments. However, this approach too can be seriously misleading.

The challenge, as with comparisons using ‘historical controls’, is to know whether the groups of people receiving the different treatments were sufficiently alike before they started treatment for a valid comparison to be possible – in other words, whether like was being compared with like.

As with ‘historical controls’, researchers may use statistical adjustments and analyses to try to ensure that like will be compared with like, but only if relevant features of patients in the comparison groups have been recorded and taken into account. So seldom will these conditions have been met that such analyses should always be viewed with great caution. Belief in them can lead to major tragedies.

A telling example concerns hormone replacement therapy (HRT). Women who had used HRT during and after the menopause were compared with apparently similar women who had not used it. These comparisons suggested that HRT reduced the risk of heart attacks and stroke – which would have been very welcome news if it were true. Unfortunately it wasn’t.

Subsequent comparisons, which were designed before treatment started to ensure that the comparison groups would be alike, showed that HRT had exactly the opposite effect – it actually increased heart attacks and strokes. In this case, the apparent difference in the rates of heart attacks and strokes was due to the fact that the women who used HRT were generally healthier than those who did not take HRT – it was not due to the HRT. Research that has not ensured that like really is being compared with like can result in harm being done to tens of thousands of people.

As the HRT experience indicates, the best way to ensure that like will be compared with like is to assemble the comparison groups before starting treatment. The groups need to be composed of patients who are similar not just in terms of known and measured factors, such as age and the severity of their illness, but also in terms of unmeasured factors that may influence recovery from illness, such as diet, occupation and other social factors, or anxiety about illness or proposed treatments.

It is always difficult – indeed often impossible – to be confident that treatment groups are alike if they have been assembled after treatment has started. The critical question then is this: do differences in outcomes reflect differences in the effects of the treatments being compared, or differences in the patients in the comparison groups?