Use of comparison groups in quality improvement: A review of analyses under the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project
Introduction: While evaluations of quality improvement interventions often show significant improvements in performance, they often lack comparison groups. The USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project was mandated to validate 10% of country-reported indicators, conduct economic analysis in at least one activity in each country and collect data from comparison sites for 10% of country-reported indicators. This report synthesizes findings
Methods: We reviewed ASSIST reports that included data from comparison sites for the following information: methodology, content area, and key findings. Comparison group analyses from eight countries were included in this synthesis report. A total of 63 indicators were compared for anemia, HIV, maternal, newborn, and child health (MNCH) and orphans and vulnerable children (OVC) programs.
Results: Follow-up time varied from five to 14 months across studies. Findings showed that improvements were greater in ASSIST sites compared to non-ASSIST sites for more than half the indicators (38 out of 63 indicators). All country analyses reported improvements in at least one indicator. Overall, indicators with high baseline values showed less improvement over time.
Limitations: The main limitation of these comparison analyses is that country reports did not include specific information to assess the extent to which comparison sites were similar to intervention sites. Reports also failed to describe contextual factors that may have an impact on performance. The heterogeneity in methodological approaches and follow-up time also limits our ability to compare findings across studies.
Conclusions: This synthesis of comparison analyses conducted under ASSIST contribute to our understanding of the overall impact of quality improvement approaches in improving care. Overall, improvements were greater in ASSIST sites compared to non-ASSIST sites.. Findings are, howeverm subject to limitations dues to differences in methodologies and inadequate detail about sites' characteristics. Contamination and high baseline performance may have diluted differences between ASSIST and non-ASSIST sites. While classical experimental designs such as randomized trials may not be feasible in the context of complex interventions, future comparison group studies should use stronger methods to assess effectiveness, select comparison sites that are similar to ASSIST sites, describe key contextual factors, and include longer follow-up time to assess the sustainability of improvement approaches.