About Testing Treatments interactive (TTi)

Why TTi?

Claims about what might improve or harm health are everywhere.

Many treatment claims are untrustworthy, however.  Unwarranted faith in untrustworthy claims causes people to suffer by using ineffective or harmful treatments, or failing to use effective ones.

Unfortunately, it is often hard to tell whether a treatment claim is untrustworthy. Skills are needed to spot claims that are unreliable.

Testing Treatments interactive (TTi) provides open access to learning resources for teachers and others who want to help others to acquire those skills.

Please note that TTi is NOT for specific advice on particular treatments.  This information is available elsewhere.

However, TTi SHOULD help to promote critical thinking about the treatment claims that people encounter.

TTi builds on Testing Treatments, a successful book written for the public, which is available to download for free in more than a dozen languages (see  the top of the page).

Who is TTi for?

Testing Treatments interactive has been established to provide easy access to useful learning resources, particularly for:

• teachers in primary, secondary, and tertiary education;
• communicators, such as journalists and science writers;
• advisers, such as those developing health and other policies;
• researchers who want to assess the effects of learning resources;
• learners, anyone who wishes to teach themselves how to assess treatment claims, for example, interested members of the public, and undergraduate, postgraduate or professional students.

• Learn more about why TTi is targeting “intermediaries” [Mickan and Burls]

How is material in TTi organised?

The material in the website is organised to reflect over 30 Key Concepts relevant to assessing the trustworthiness of claims about the effects of treatments.

Our definition of ‘treatments’ includes any action intended to improve health or relieve suffering. These include changes in behaviour; screening programmes; drugs, surgery, physical and psychological treatments; and public health and healthcare system changes.

The Key Concepts developed by the Informed Health Choices project have been grouped under three headings:

• Claims: Are they justified?
• Comparisons: are they fair and reliable?
• Choices: are the findings relevant?
• Learn more about The Key Concepts [Austvoll-Dahlgren et al.]

What kind of learning resources does TTi contain?

The TTi site has a library of resources at its heart – the Critical thinking and Appraisal Resource Library (CARL). This contains Learning Resources to help people recognise and understand Key Concepts, and how use them to evaluate treatment claims.

Each of the Learning Resources included in CARL is open access and categorised by:

• the Key Concept(s) it addresses
• the target learner group (children and young people, undergraduates, postgraduates, others)
• the resource format (texts, audio, video, cartoons, websites, and lessons)

Resources that have been formally evaluated to assess whether they improve peoples’ understanding of Key Concepts are identified, and there are links to published accounts of how a Learning Resource was evaluated. [Albarquoni et al, in press]

How do we identify resources for CARL?

Populating the CARL database with teaching/learning resources has used candidate items from several sources, including pre-existing compilations and relevant systematic reviews. These include:

1. Interactive teaching resources compiled in preparation for a multidisciplinary meeting in Oxford in 2011 [Krause et al. 2011].
2. Resources compiled as part of the European Communication on Research Awareness Needs (ECRAN) project [Mosconi et al. 2016].
3. Resources identified by the Informed Health Choices project.
4. Resources in Testing Treatments and this website
5. Resources in the James Lind Library [Chalmers I et al. 2008].
6. Resources available through the Educational Endowment Foundation.
7. Resources available through the Times Education Service.
8. Resources identified through systematic reviews of teaching/learning interventions [Nordheim et al.; Austvoll-Dahlgren et al.; Cusack et al.; Albarqouni et al.]
9. Online snowball searches for additional material.

Decisions to include/exclude resources

In all instances a resource’s relevance to the Key Concepts or value in promoting critical thinking skills have been the main determinants of whether a resource would be included. Where doubts arose about inclusion from the main assessor (JCC), decisions were made in collaboration with IC and in some cases with other members of the CARL editorial group. Resource format-specific criteria were also formulated, given the varied nature of resources. The processes for deciding whether a resource should be included are listed separately in Appendix 2.

Teaching/learning resources that have been formally evaluated, for instance in randomised trials, are of particular importance. Separate inclusion criteria for formal evaluations are listed in Appendix 3].

Coding resources

In order to facilitate easy navigation of the resources included in CARL, each resource has been coded using basic categories judged important by members of the CARL editorial group:

1. Unique resource identification code.
2. Name/Title (as stated by the resource host)
3. Format: Text; Video; Audio; Lessons (including presentations, e-Learning modules and specific materials for teaching students such as learning exercises or worksheets); Cartoons and Websites/pages.
4. Reference/URL
5. Language
6. Key concepts to which the resource is relevant
7. Effects of a resource on knowledge/understanding, with links to reports of the evaluations.
8. Target user groups (see Who is TTi for?)

Target user groups were decided by taking into account interests of the editorial group and the most commonly mentioned audiences by resource developers. Those with expertise or interests in specific users will be able to divide them into more specific groups. For example, resources for ‘Schools’ may be divided into Resources for Primary School Children and Resources for Secondary school children.

Does the website contain tests to assess understanding of Key Concepts?

If you want a foretaste of the kind of evaluation questions that have been developed and validated by the Informed Health Choices project, try our illustrative fun quiz.

This quiz will give you an idea of the format of the The Claim Evaluation Tools, which have been formally evaluated to assess people’s understanding of the Key Concepts needed to assess the trustworthiness of treatment claims.

• Learn more about The Claim Evaluation Tools [Austvoll-Dahlgren et al.]

References

• Albarqouni L, Glasziou P, Hoffmann T (in preparation): Completeness of the reporting of the EBP educational interventions: a systematic review.
• Austvoll-Dahlgren A, Oxman AD, Chalmers I, Nsangi A, Glenton C, Lewin S, Morelli A, Rosenbaum S, Smeakula D and Sewankambo N.  Key concepts that people need to understand to assess claims about treatment effects.  Journal of Evidence-Based Medicine 2016; 8:112-25.
• Castle JC, Chalmers I, Atkinson P, Badenoch D, Oxman AD, Austvoll-Dahlgren A, Krause K, Schwartz LM, Woloshin S, Burls A, Mosconi P, Nordheim L, Hoffmann T, Cusack L, Albarqouni L, Glasziou P (in preparation). Establishing a library of resources to help people understand Key Concepts in assessing treatment claims – The Critical thinking and Appraisal Resource Library (CARL). Not yet submitted.
• Chalmers I, Milne I, Trӧhler U, Vandenbrouche J, Morabia A, Tait G, Dukan E, James Lind Library Editorial Team. The James Lind Library: explaining and illustrating the evolution of fair tests of medical treatments. Journal of the Royal College of Physicians of Edinburgh. 2008;38(3):259-64.
• Krause LK, Schwartz L, Woloshin S, Oxman A (2011). Interactive tools to teach the public to be critical consumers of information about health care: What’s out there? Poster presentation at Cochrane Colloquium.
• Mickan S, Burls A (2011). Enhancing Public Understanding of Health Research. Report of a conference held at Kellogg College, Oxford University, 7-8 April.
• Mosconi P, Antes G, Barbareschi G, Burls A, Demotes-Mainard J, Chalmers I, Colombo C,  Garattini S, Gluud C, Gyte G, Mcllwain C, Penfold M, Post N, Satolli R, Valetto MR, West B, Wolff S (2016). A European multi-language initiative to make the general population aware of independent clinical research: the European Communication on Research Awareness Need (ECRAN) project. Trials 17:19. DOI 10.1186/s13063-015-1146-7.
• Nordheim LV, Gundersen MW, Espehaug B, Guttersud Ø, Flottorp S (2016). Effects of School-Based Educational Interventions for Enhancing Adolescents Abilities in Critical Appraisal of Health Claims: A Systematic Review. PLoS ONE. 2016;11(8):e0161485. DOI: 10.1371/journal.pone.0161485.

Image credits

• Teaching Allie to Fish, Ted Kerwin, CC BY 2.0