Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis

Introduction Systematic reviews are a cornerstone of evidence-based
public health. The method of appraising the quality of different
intervention and observational study designs in such reviews
remains an important challenge. This article examines the
applicability of selected quality appraisal tools (QATs) and the
impact of choice of tool on the meta-analysis of a published
systematic review. Methods The authors selected a systematic review
on the effectiveness of hand washing with soap in preventing
diarrhoea, covering a range of epidemiological study designs. 6
QATs were used to assess 13 studies meeting their inclusion
criteria; component sections/questions were coded numerically to
derive a summary score between -1 (low quality) and +1 (high
quality) for each QAT and study. Heterogeneity in study quality was
evaluated graphically using traffic light schemes and spider
charts. Random effects meta-analysis was undertaken for all
studies; sensitivity analyses for each QAT included only those
studies with a score of 0 or above. Results The authors found
substantial heterogeneity in summary scores for a given study.
Their main meta-analysis yielded an OR of 0.60 (95% CI 0.47 to
0.77) with most sensitivity analyses giving similar pooled effect
sizes with wider CIs. Discussion The six QATs differ greatly in
applicability across study designs, approach to quality appraisal
(ie, scale vs checklist, presence/absence of summary score),
coverage of domains and quality of component questions and answers.
Learning from advantages and disadvantages of each QAT, we
recommend research into the development of a reliable QAT with a
broad applicability across study designs.