Stage 5: Assessing Quality: Step 2: How to Assess Quality of Included Studies

Conducting Quality Appraisal

Once you have identified, pre-tested, and potentially revised your checklist from your quality appraisal tool, it is time to evaluate each of the sources collected from data extraction.

If you have multiple reviewers (which is recommended for best practice), remember that appraisal should be conducted independently.

Establish what Quality of Studies will be Included in Data Synthesis

After completing quality appraisal, you will have to decide which studies will contribute to data synthesis.

Hannes (2011) suggests three outcomes following quality appraisal:

Include only high quality studies; for instance, Carlsen et al. (2007) excluded 5 out of 17 studies that were evaluated to be of low quality
Give more weight to studies that were rated to be of high quality; thus, all studies are included
Describe observations without excluding any studies

For more information on each approach, refer to Hannes (2011) in the Additional References at the bottom of this page.

The approach you choose will depend on your specific review, and should ultimately be established amongst your review team.

Potential Issues when Conducting Quality Appraisal

Problems that may arise will largely be specific to your review. However, here are some answers to potential questions that you may encounter (Monash University Library).

What if I want more details about the study?

Understand that you are only appraising the information available in the study. If you are skeptical about any of the methods or quantitative results, contact the primary author of the study for clarification or more information.

What if I have very few studies to work with?

If you are conducting a review on a newer area of research, you may find that there is currently little evidence available. In this case, it may be necessary to consider including sources appraised to be weak or poor quality. After, this limitation must be clearly recorded and discussed in your findings.
Although including few or poor-quality studies may be perceived as a weakness, keep in mind that it makes a good case for more rigorous studies to be conducted in the future.

Document Quality Appraisal

Your review should clearly outline what types of studies were included and excluded, and how those criteria were established. This will ensure that your methods are transparent and replicable.