Research and Evaluation

The USAID ASSIST Project supports country-led research and synthesizes learning across country programs on topics such as the validity of improvement-related data, sustainability and institutionalization, scale-up, and cost-effectiveness of improvement activities. The ASSIST Research and Evaluation unit provides technical support to these efforts and works to disseminate knowledge from these studies to encourage wider adoption of improvement methods.

USAID ASSIST Project Semi-Annual Performance Monitoring Report FY18

University Research Co., LLC (URC) and its partners have completed 5.5 years of implementation of the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project. This report is the eleventh Semi-Annual Performance Monitoring Report for ASSIST, summarizing the project’s accomplishments and results during the first two quarters (Q1-2) of Fiscal Year 2018 (FY18).  In late September 2017, the project was awarded a two-year costed extension with additional funding for Zika-related activities.  No further funding was provided for non-Zika work.  The project’s FY18 work plan thus focuses mainly on Zika-related support in eight countries and completion of previously funded work in other countries.

Overall goals
The USAID ASSIST Project fosters improvements in a range of health care processes through the application of modern improvement methods by host country providers and managers in USAID-assisted countries. The project’s central purpose is to build the capacity of host country health and social service systems to improve the effectiveness, efficiency, client-centeredness, safety, accessibility, and equity of the services they provide. In addition to supporting the implementation of improvement strategies, the project seeks to generate new knowledge to increase the effectiveness and efficiency of applying improvement methods in low- and middle-income countries.
USAID ASSIST country programs align with the goals of United States Government’s global initiatives and policies, including preventing child and maternal deaths, achieving HIV epidemic control, combating other public health threats, protecting life, and addressing gender inequalities.

Where we work
During the first half of FY18, USAID ASSIST provided technical support in 23 countries through field and core funding. USAID Mission funds supported work in nine countries: Cambodia, Côte d’Ivoire, India, Indonesia, Lesotho, Mali, Swaziland, Tanzania, and Uganda. USAID core funds from the Office of HIV/AIDS supported activities in seven countries: Kenya, Lesotho, Malawi, Mozambique, Namibia, Tanzania, and Zimbabwe.  Funds for Neglected Tropical Diseases supported improvement work to combat the Zika virus in the Dominican Republic, Ecuador, El Salvador, Guatemala, Honduras, Jamaica, Nicaragua, Paraguay, and Peru.  USAID cross-bureau-funded activities, through the Office of Health Systems, supported malaria prevention activities in Malawi and global and regional initiatives that contribute to local and global learning in improvement.

FY18 Q1-2 accomplishments and results:

  • Improvement in key indicators: As discussed in this report, ASSIST-supported programs demonstrated improved care and outcomes for a range of services, including antenatal and postnatal care, essential obstetric and newborn care, screening of pregnant women and newborns for Zika-related signs and symptoms, Zika care and support, PMTCT, and HIV prevention.
  • Research and evaluation studies: At of the end of the reporting period, the project had 29 research studies in planning, underway or completed in 14 countries. Three are multi-country studies.
  • Promoting the use of improvement methods: During the reporting period, project staff published nine peer-reviewed articles; seven case studies; four technical and research reports; and 10 guides, tools, and job aids. ASSIST staff led six sessions and made eight oral presentations at six regional and international conferences. ASSIST’s work continued to be promoted through the project’s website and social media engagement, with some 27,867 visitors viewing 61,572 pages within the ASSIST Knowledge Portal during the past two quarters.

Research and Evaluation Report FY17

The research and evaluation (R&E) unit of the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project provides technical assistance and guidance on country-led research and synthesizes learning across country- and centrally-funded activities. Topics for investigation include the validity of improvement-indicator data, sustainability and institutionalization of improvement activities, and economic analysis of improvement interventions.

The R&E unit works with the knowledge management unit to disseminate information from these studies through web-published reports, peer-reviewed journal articles, webinars, and presentations at international meetings to encourage wider adoption of improvements methods. The unit also provides training to USAID mission officers on cost-effectiveness analysis of health system improvement interventions.

There is a continued demand for evidence of the effectiveness and efficiency of improvement activities within and beyond the countries in which the USAID ASSIST Project is providing support. The rigor and thoroughness with which the evidence is collected, analyzed, and presented is important for promoting continuation of improvement activities in low- and/or middle-income countries.

In fiscal year 2017 (FY17), the R&E unit finalized several key studies. It also began other activities in several countries, including the West Bank, Botswana, and Latin American countries where ASSIST has initiated Zika-related improvement activities.

USAID ASSIST Project Annual Performance Monitoring Report FY17

University Research Co., LLC (URC) and its partners have completed the fifth year of implementation of the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project. This report is the tenth Semi-Annual Performance Monitoring Report for the project and aims to summarize the accomplishments and results toward the program objectives of USAID ASSIST activities during quarter (Q) 1 through 4 of Fiscal Year 2017 (FY17).

Scale of USAID ASSIST work, FY17

FY17 Accomplishments and Results

  • Improvement in key indicators: As discussed in this report, ASSIST-supported programs demonstrated improved care and outcomes for a range of services, including antenatal and postnatal care, essential obstetric and newborn care, family planning, PMTCT, HIV care and treatment, HIV prevention, NACS, TB diagnosis and treatment, malaria diagnosis and case management, and services for vulnerable children and families.
  • Gender integration: Onsite gender training and technical support was provided to country teams in Burundi, Cote d’Ivoire, DRC, Lesotho, Mali, South Africa, and Uganda to integrate gender considerations into their improvement work. Ongoing gender support was provided to all field offices with improvement activities to collect and analyze sex-disaggregated indicators, identify gender-related gaps affecting outcomes, and respond to those gaps.
  • Research and evaluation studies: At of the end of the reporting period, the project had 37 studies underway or completed in 15 countries. Four are multi-country studies.
  • Promoting the use of improvement methods: In FY17, project staff published 14 peer-reviewed articles; 23 case studies; 11 technical and research reports; 42 guides and tools, six short reports; 29 annual reports, and seven multimedia products describing project-supported work and results as well as gender integration learning videos. ASSIST had an active presence at 16 international and national conferences. In all, project staff led 18 sessions and workshops and delivered 14 oral and 14 poster presentations. ASSIST’s work continued to be promoted through strategic engagement in Twitter chats, reaching close to 2,374 followers (26% increase from FY16) and an average of 26,500 unique Twitter users per month (impressions). We continued sharing resources via the ASSIST Facebook page (over 6,200 likes) as well as publishing engaging blog content (28 blogs in FY17).

Why it is Important to Sex-Disaggregate Data in Quality Improvement

The USAID ASSIST Project integrates gender considerations in quality improvement in order to improve outcomes for all — women and men, girls and boys. This video explains a critical part of gender integration in detail: collecting and analyzing sex-disaggregated data.

Learn more about the importance of gender integration in monitoring and evaluation, and sex-disaggregated data in particular.

Evaluating Complex Health Interventions: A Guide to Rigorous Research Designs

“Evaluating Complex Health Interventions: A Guide to Rigorous Research Designs” was co-authored by researchers from ASSIST and IHI, and published by AcademyHealth. This guide provides a framework to guide decision-making around appropriate designs to evaluate public health, and other service interventions, including quality improvement.

It is aimed at those implementing interventions in public health and community settings, including quality improvement, who are involved in evaluation, but may not themselves be evaluators. By arming these stakeholders with the information necessary to understand and assess the strengths, weaknesses, and validity of an evaluation that has already been conducted, the guide aims to help the reader make informed decisions by providing information on the various tradeoffs involved in the selection of an evaluation design.

The evaluation designs presented in this guide represent a mix of experimental, quasi-experimental and observational designs.

Six designs are included in the guide:

  • randomized controlled trial,
  • cluster randomized stepped wedge design,
  • interrupted time series design,
  • controlled before and after design,
  • regression discontinuity design, and
  • natural experiment.

Each design includes:

  • a general description with a diagram to illustrate the design;
  • two examples from the peer-reviewed literature of how the design was used to evaluate a specific health or social service intervention;
  • key strengths and weaknesses of the study design;
  • timeline and budget considerations;
  • policy implications, and
  • considerations for future use.

A flow chart to inform the selection of evaluation designs is featured in the guide. Resources for more detailed information on a broader range of designs are also listed.

Research and Evaluation Report FY16

The USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project, through its research and evaluation (R&E) component, provides technical assistance and guidance on country-led research and synthesizes learning across country- and centrally funded projects on several improvement topics, including the validity of improvement indicator data, sustainability and institutionalization, spread, and economic analysis.

The ASSIST R&E agenda falls under three main categories to address two questions and one broad theme: How can ASSIST improve the way that improvement activities are implemented in the settings in which it works? What is the capacity that countries have to continue to improve health care independent of ASSIST? The major theme is improving the rigor of program evaluation and research conducted in the field of health care improvement. Many R&E activities reported here cover more than one of these broad areas.

In fiscal year (FY) 2016, the R&E unit has continued to finalize several key studies currently underway and launch new studies, especially on the topic of institutionalization and sustainability of quality improvement beyond the period of external technical assistance. There continues to be a major focus on completing ASSIST’s mandates for the use of control groups, economic analysis of improvement, and data validation.

In FY16, the R&E unit supported 39 studies in 15 countries (Burundi, Cote d’Ivoire, Democratic Republic of Congo, Ecuador, Honduras, India, Kenya, Lesotho, Malawi, Mali, South Africa, Swaziland, Tanzania, Uganda, and Ukraine). In addition, the team supported four multi-country studies: 1) experiences from the Partnerships in Community Child Protection in Africa; 2) a multi-country study in Latin America and the Caribbean (LAC) evaluating a web-based improvement collaborative to prevent neonatal infections in hospitals; 3) supporting community providers through a community health system approach in Ethiopia and Tanzania; and 4) a survey of how medical schools in Africa incorporate improvement methods in their curricula. The R&E team also provided technical assistance to Botswana, Burundi, Democratic Republic of Congo (DRC), and Lesotho on validation of their improvement indicators. Data collection from control groups was planned or conducted in Burundi, DRC, Kenya, Tanzania, and Uganda.

The unit is working to disseminate knowledge generated by these studies through web-published reports, peer-reviewed journal articles, and presentations at relevant international meetings to encourage wider adoption of improvements methods.

Cost-effectiveness of an Intervention to Increase Immunization Coverage in Pakistan

Immunization coverage in Pakistan remains low, at 54% nationwide and 39% in Sindh Province in 2012, despite recent efforts by provincial and national Ministries of Health. The Government of Sindh Province in Pakistan sought technical assistance from the USAID-funded Health Systems Strengthening Project to improve immunization uptake in four low-coverage districts of Jacobabad, Kashmore, Tharparkar, and Thatta. This evaluation estimated the effectiveness and efficiency of the intervention to increase immunization uptake and thereby improve population health. This retrospective analysis used program effectiveness data routinely collected by the implementing partner, John Snow International (JSI), and cost data collected retrospectively from accounting records.

Routine immunization program data included the number of children and pregnant women registered and the number immunized. Using data from epidemiological studies, we estimated the number of cases of vaccine-preventable disease expected to be avoided with immunization and determined the disability-adjusted life years (DALYs) attributable to cases of each vaccine-preventable disease. Cost-effectiveness was calculated using decision-tree analysis.

The analysis found that the total cost to USAID of the immunization promotion program implemented in these four districts was US$1.56 million. About 440,000 children and 120,000 women were immunized through the program at an overall cost of $2.80 per vaccination completed. The overall incremental cost-effectiveness ratio comparing it to business-as-usual, from the USAID perspective, was $1.30 per DALY averted. From the Government of Sindh Department of Health perspective, the program decreased costs while improving health because of the reduced expenditure overall by not treating the number of vaccine-preventable disease cases averted. The study concluded that the program is cost-saving while improving population health.

Evaluation 2016: Evaluation of a Beneficiary Identification Tool

Atlanta, Georgia

Evaluation 2016, the American Evaluation Association's annual conference, will be held in Atlanta on October 24-29. Diana Chamrad of the ASSIST Project will be discussing her work evaluating a beneficiary identification tool in Uganda.

Pages