Home > Key Concepts for assessing claims about treatment effects > Which Key Concepts are most relevant and important?

Which Key Concepts are most relevant and important?

The answer to this question depends on who the learners are and the context. Different approaches can be taken to addressing this question. We offer some general suggestions here and give an example of how  this was done in one particular project, Informed Health Choices (IHC), which attempted to teach the Key Concepts to children and adults in Uganda.

How many concepts should be included?

The number of concepts that should be selected depends largely on how much time is available. In our experience it is best not to try to teach too many concepts at a time. For example, in developing learning resources for primary school children in the IHC project, it was found best not to teach more than one or two concepts in a single lesson and that it was best to have two periods (80 minutes) instead of one 40-minute period for each lesson. Similarly, in developing a podcast for the parents of primary school children, it was best if just one concept was taught in each episode and that the episodes be kept short (5 to 9 minutes). In both cases, this was based on extensive user testing and piloting of the resources. For existing resources, there may already be some information available about how much time is needed.

What criteria should be used to select the concepts?

In general, selecting Key Concepts to teach rests on three judgements about each concept:

  1. Are the learners likely to be able to understand and apply the concept?
  2. How important is the concept to the learners?
  3. Time and resources available

Other criteria might also be relevant in some circumstances. For example, if the learners have already been exposed to some of the concepts, it might be relevant to consider whether the learners are likely to have already learned or mastered them (and to exclude or spend less time on those concepts). Another criterion might be the relevance of the concept to the context in which it is being taught. Some concepts might not be relevant in some contexts. For example, concepts about making informed choices might not be relevant in the context of lessons that focus on critical thinking about claims in the media.

Who should decide which concepts to teach and how?

Judgements about the importance of concepts and learners’ abilities to understand and apply the concepts depends on teachers being familiar with the learners and their circumstances. Because these judgements can be difficult, it is helpful if more than one person makes them. There is likely to be some evidence to inform the judgements; e.g. from published studies, from pilot studies or user testing, or from personal teaching experience. Nonetheless, judgement is required.

If more than one person makes these judgements, it is necessary to reach agreement. This can be done either by informal discussion or by using a formal consensus technique, such as the Nominal Group Technique or the Delphi Method. There are advantages and disadvantages to all of these methods. A systematic review of consensus development methods found that: “Formal methods generally perform as well or better than informal methods, but it is difficult to tell which of the formal methods is best” [1].

An example of prioritisation: choosing Key Concepts for the educational resources in the Informed Health Choices (IHC) Project

In developing the IHC primary school resources, we started with a list of 32 Key Concepts divided into six groups [2]. We first consulted with teachers, who found all six groups of concepts to be relevant for 10 to 12-year-old children in the fifth grade [3]. Based on input from the teachers, we judged that 24 of the 32 concepts could be learned by primary school children. These judgements were made by members of the research team in a face-to-face meeting using informal discussion to reach a consensus.

We prototyped, piloted and user tested learning resources for those 24 concepts and found that that they were too numerous to teach in a single school term. We considered the importance of the concepts and the difficulty that the children had learning them when we piloted the resources. The importance of the concepts was based on judgements made by members of the research team by:

  • Each person individually identifying which of the 24 Key Concepts they considered most important
  • Compilation and discussion of those judgements
  • Voting on the concepts
  • Reaching a consensus by informal discussion

We reached agreement that eight of the concepts were “core”, the most important. Three members of the research team then reviewed data from our piloting and user-testing and identified concepts that appeared to be too difficult to teach to 10-12 year old children. In addition, we considered how the concepts were grouped in the lessons and the number of concepts being taught in each lesson. Based on this, three of the team members selected 12 of the 24 concepts, with plans to develop a second learning resource for teaching the other 12 concepts, in another school term. This was then discussed with the rest of the team and agreement reached.

Key Concepts considered relevant for primary school children

Key Concepts taught in The Health Choices Book

CLAIMS: ARE THEY JUSTIFIED?

  • Treatments may be harmful
  • Personal experiences or anecdotes (stories) are an unreliable basis for assessing the effects of most treatments
  • Widely used treatments or treatments that have been used for a long time are not necessarily beneficial or safe
  • New, brand-named, or more expensive treatments may not be better than available alternatives
  • Opinions of experts or authorities do not alone provide a reliable basis for deciding on the benefits and harms of treatments
  • Conflicting interests may result in misleading claims about the effects of treatments.

COMPARISONS: ARE THEY FAIR AND RELIABLE?

  • Evaluating the effects of treatments requires appropriate comparisons
  • Apart from the treatments being compared, the comparison groups need to be similar (i.e. ‘like needs to be compared with like’)
  • If possible, people should not know which of the treatments being compared they are receiving
  • Small studies in which few outcome events occur are usually not informative and the results may be misleading
  • The results of single comparisons of treatments can be misleading.

CHOICES: MAKING INFORMED HEALTH CHOICES

  • Treatments usually have beneficial and harmful effects.

Other Key Concepts considered relevant for primary school children

CLAIMS: ARE THEY JUSTIFIED?

  • An outcome may be associated with a treatment, but not caused by the treatment
  • Increasing the amount of a treatment does not necessarily increase the benefits of a treatment and may cause harm
  • Hope or fear can lead to unrealistic expectations about the effects of treatments
  • Beliefs about how treatments work are not reliable predictors of the actual effects of treatments
  • Large, dramatic effects of treatments are rare.

COMPARISONS: ARE THEY FAIR AND RELIABLE?

  • People in the groups being compared need to be cared for similarly (apart from the treatments being compared)
  • If possible, people should not know which of the treatments being compared they are receiving
  • It is important to measure outcomes in everyone who was included in the treatment comparison groups
  • Results for a selected group of people within fair comparisons can be misleading
  • Reviews of treatment comparisons that do not use systematic methods can be misleading
  • Well done systematic reviews often reveal a lack of relevant evidence, but they provide the best basis for making judgements about the certainty of the evidence.

CHOICES: MAKING INFORMED HEALTH CHOICES

  • Fair comparisons of treatments should measure outcomes that are important

References

  1. Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, Marteau T. Consensus development methods, and their use in clinical guideline development. Health Technology Assessment 1998, 2:i-88.
  2. Austvoll-Dahlgren A, Oxman AD, Chalmers I, Nsangi A, Glenton C, Lewin S, Morelli A, Rosenbaum S, Semakula D, Sewankambo N. Key concepts that people need to understand to assess claims about treatment effects. Journal of Evidence-Based Medicine 2015; 8(3):112-25.
  3. Nsangi A, Semakula D, Oxman AD, Sewankambo N. Teaching children in low-income countries to assess claims about treatment effects; Prioritization of key concepts.  Journal of Evidence-Based Medicine 2015; 8(4):173-80.