Subject and Course Guides: Systematic Reviews: Step 4. Appraisal of Studies

For Quality

(CEBM) Centre for Evidence-Based Medicine
The Centre for Evidence-Based Medicine in Oxford (UK) provides tools to develop, teach and promote evidence-based health care. Their site provides useful tools and downloads for the critical appraisal of medical evidence. Example appraisal sheets are provided together with several helpful examples.
(CASP) Critical Appraisal Skills Programme
The Critical Appraisal Skills Programme (CASP) has created a number of appraisal tools for various types of research.
Joanna Briggs Critical Appraisal Tools
The Joanna Briggs Institute has created a number of critical appraisal checklists, including those for case control studies, randomized controlled trials, and cross sectional studies.
AACODS Checklist
You can use this checklist to evaluate grey literature.
(GRADE) Grading of Recommendations Assessment, Development, and Evaluation
GRADE has two levels: strong and weak recommendations.
AHRQ Checklist: Comparative Effectiveness Reviews
This can be used to ensure assumptions and limitations are understood and taken into account when interpreting validity and generalizability.
(CHEC List) Consensus Health Economic Criteria
Checklist evaluating methodological quality in cost effectiveness analyses & comparative effectiveness reviews.
QUADAS-2
A quality assessment tool for diagnostic accuracy studies, developed by a team at the School of Social and Community Medicine, University of Bristol.
AGREE-II
AGREE is a tool for assessing the quality of Clinical Practice Guidelines. It is not to be used to appraise journal articles reporting the results of clinical trials but instead it is a quality assessment instrument for evaluating or deciding which guidelines could be recommended for use in practice or to inform health policy decisions.
(NOS) Newcastle-Ottawa Scale
This scale was developed to assess the quality of non-randomized studies with its design, content and ease of use directed to the task of incorporating the quality assessments in the interpretation of meta-analytic results.

For Content

The next step is for the researchers to read the full text of each article identified for inclusion in the review and extract the pertinent data using a standardized data extraction/coding form. The data extraction form should be as long or as short as necessary and can be coded for computer analysis if desired.

If you are writing a narrative review to summarize information reported in a small number of studies then you probably don't need to go to the trouble of coding the data variables for computer analysis but instead summarize the information from the data extraction sheets for the included studies. If you are conducting an analytical review with a meta-analysis to compare data outcomes from several clinical trials you may wish to computerize the data collection and analysis processes.

Systematic Review Data Repository (SRDR)
From AHRQ, SRDR is a powerful and easy-to-use tool for extraction and management of data for systematic review or meta-analysis. It is also an open and searchable archive of systematic reviews and their data.
DistillerSR
DistillerSR is an online application designed specifically for the screening and data extraction phases of a systematic review. Subscription Fee.
Covidence
This tool is a collaboration between Cochrane and the team of systematic reviewers, designers and software developers that developed the ReGroup and EROS software prototypes with support from The Alfred Hospital, Monash University, National ICT Australia, University College London and Instituto de Effectividad Clinica y Sanitaria.

Evaluation Documentation

Document all the details of your appraisal and data extraction, including the following:

Articles' quality evaluations and questions used
Reasons why they were excluded
Variables of each study (e.g., number of participants, control and intervention groups' characteristics, and length and frequency of interventions)