Home > About Testing Treatments interactive (TTi) > Inclusion criteria for CARL resources

Inclusion criteria for CARL resources

Here’s how we decide what to include in the Critical thinking and Appraisal and Resource Library (CARL).

Selection criteria for learning resources

Key Criteria

i) Must be a learning resource; i.e. text, video, software, or other material that assists learners (anyone wanting to learn) to understand and apply one or more of the Key Concepts

ii) Must be freely available

iii) Must provide an explanation, and preferably an example* for one or more of the Key Concepts

Text (includes books, book chapters and articles†)

Must satisfy i-iii

  • Individual chapters or sections in books are included only if the majority of the content in the rest of the book does not help the target audience to understand and apply one or more of the Key Concepts, but the chapter or section in question does.

Video (also encompasses animations and interactive games)

Must satisfy i-iii

  • Where the main content of the video does not address a Key Concept but part of the video does, the section can only be used in isolation (with permission) if it does not require further explanation, i.e. is not dependent on context provided by the main content.

Audio (includes songs, audiobooks, podcasts, recorded interviews and talks)

Must satisfy i-iii.

  • Where the main content of the audio does not address a Key Concept but part of the audio does, the section can only be used in isolation (with permission) if it does not require further explanation, i.e. is not dependent on context provided by the main content.

Lessons (includes PowerPoint presentations, recorded presentations, worksheets, lesson plans, etc.)

Must satisfy i-iii

  • Lesson plans must provide sufficient detail to be replicable, eg, objectives, sequence, timings, materials, etc)
  • PowerPoint presentations must be interpretable by people who are not methodologists i.e. must include notes, or be sufficiently clear so that notes are not needed.
  • Quizzes (test yourself resources) must provide access to the correct answers and explanations. This does not apply to learning questions that are embedded in a learning resource, such as those in workshop materials.
  • Checklists must include instructions or explanations.
  • Critical Appraisal Tools are only included if they explain the reasoning for steps in the critical appraisal process as relates to the Key Concepts.

Cartoons

Must satisfy i-iii or address a term included in the GET-IT jargon buster.

  • Cartoons that are related to Key Concepts or GET-IT terms, but do not help people to understand the concept or term, are excluded.

Websites

Must satisfy i-iii

  • Specific webpages and parts of websites are included in isolation only if the majority of content on the site does not help the target audience to understand and apply one or more of the Key Concepts, but the parts of the website in question do.

Selection criteria for evaluations of learning resources

Key Criteria

iv) Must evaluate a learning resource that meets the criteria listed in Appendix 1

v) Must include a description of the methods that were used

Evaluations of effects

Must satisfy iv and v

  • Must be a comparison. Comparisons of knowledge after an intervention with knowledge before are eligible, but ’after only’ evaluations (i.e. without a comparison group or pretest measure) will be excluded.
  • Must have a clearly defined outcome measure; e.g. knowledge, skills, attitudes, behaviour.
  • Measurement of knowledge must be objective.
  • Must report the difference between the comparison groups quantitatively.

Evaluations of user experiences or factors that facilitate or hinder use of a learning resource, including pilot studies and user testing (formative evaluations)

Must satisfy iv and v

  • Must have implications for the design of the learning resource.
  • Can use qualitative or quantitative methods.

Process evaluations

Must satisfy iv and v

  • Must be linked to an assessment of effects.
  • Address one or more questions related to obtaining a better understanding of how or why a learning resource works or does not work; e.g. fidelity (whether the resource was used as intended), explaining variation in effects, identifying potential unintended effects, identifying factors that might improve the resources or how they are used, or that might facilitate or hinder scaling up use of the resource.

Reviews (e.g. book reviews)

Are excluded.

Evaluations of instructed educational interventions (syllabuses for educational courses; journal clubs)

Are excluded unless detailed information of the interventions and teaching resources used are provided and explained, such that the intervention can be reliably repeated without requiring additional information.