7. Test changes at the site level

Site-level testing of changes to improve care is the heart of CQI.  A useful framework for thinking about how to approach making changes in service delivery processes is the Model for Improvement, developed by Associates in Process Improvement. The Model for Improvement is a simple series of steps for learning about a health care system or set of processes to identify gaps or weaknesses, develop measurable improvement aims, identify changes that can be introduced to improve the care processes, and then test those changes on a small scale using the Plan-Do-Study-Act (PDSA) cycle.

The first step is to analyze the current performance of the system of VMMC care. The baseline CQI assessment is an important source of information about specific VMMC processes that have performance deficiencies.

Facility leadership and the CQI team should prioritize one problem or gap to begin with. The CQI team next develops its improvement aim for solving the problem, using SMART criteria: Specific, Measurable, Achievable, Relevant, Time-bound.  The improvement aim statement describes the numerical gains for improvement, the time period in which the improvement will be achieved, and gives some idea of how the aim will be achieved, to provide guidance for the development of change ideas to test in order to reach the aim.  The aim statement also indicates how the CQI team will evaluate the changes it tests by defining the numerical measure of improvement.  The CQI team confirms what measure (indicator) will tell them whether they have made improvement. This will usually be a measure used in the assessment process.  

Creating a process map or flowchart of how VMMC services are actually delivered can be another useful starting point to identify weak points or missing steps and places where the coordination or handoffs between health workers falter. Teams should note the clinical content which is provided and compare that to clinical guidelines to identify any missing or incorrect clinical practices.  It is important that the flowchart reflect the reality of how care is delivered based on real circumstances to understand where barriers or problems are, not simply be a reiteration of how it is supposed to be delivered according to guidelines.

Once specific gaps and weak areas in the process are identified, it is often necessary to set priorities among them.  Trying to solve too many problems at once can overwhelm a CQI team.  A new team might try to first solve a simple problem to gain experience and confidence before tackling more complicated problems.  Selecting the initial problem on which to focus improvement efforts can be done by voting, a decision matrix, or by using some objective criteria, such as a low CQI baseline assessment score.

Once the focus problem area is chosen, a variety of cause-and-effect analysis tools, like fishbone diagrams and 5-Whys analysis, are useful to identify the root causes of the problem.  Understanding root causes helps the team identify possible solutions to address the underlying causes of performance gaps and understand what improvements should be possible.

The team then develops ideas to test to improve VMMC performance using the Plan-Do-Study-Act (PDSA) cycle. Brainstorming is frequently used by CQI teams to come up with change ideas. Benchmarking—looking at how others addressed the problem—is another strategy, such as visiting other VMMC sites that perform well in the site’s priority problem area to find out how they do it. The Change Ideas and Common Solutions section of this toolkit provides proven changes which can be benchmarked and adapted to a CQI team’s context. The CQI team should ask themselves, “Does this change address the root causes we uncovered?”

Once change ideas have been selected, the team can organize a test of each change. The PDSA cycle is a simple process to follow to test out and adapt a change to a particular setting. Plan involves deciding what needs to be done, who needs to do it, how and when they will do it, and what they predict will happen. In addition, the team needs to think through what information and data they will need to learn from this test—how they will know if the test worked? For example, will they use quantitative data (e.g., How many patients did this work for? How long did it take?) or qualitative information (e.g., team members’ observations about processes: What was successful? What wasn’t successful?), or both? It is important to note that the measures that reflect whether a small test worked may not be the same as the measure which tells the team whether they are meeting their aim.

To test the proposed solution (Do), a team needs to carry out the planned activities and record what happened. Many CQI teams use a team documentation journal to record its aim and tests of change, plotting results on a time series graph and annotating the line graph to show when each change was introduced. Team members need to document what worked and what did not work during the testing process.

Next, Study the results. During this step, the team will decide whether the solution they tested had the desired results.  Here, the team needs to ask, What did we learn from this test?  Was the change successful?

Finally, the team must Act: Based on what was learned from the test, the team decides what action to take. Did the change solve the problem? Should another change be tested? Not every solution that is tested is then adopted.  Sometimes, a change needs to be modified and then tested again. If the test did not show improvement, the change idea is abandoned and a different change is tested.

Through repeated tests of change, all the while continuing to measure, interpret, and document its results, a CQI team works through the identified problem areas.

Moving on to tackle key VMMC care processes

After a CQI team has made gains in the VMMC standards areas where baseline assessments showed weaknesses, the team may move on to address other key processes, such as HIV testing and counseling, STI assessment, management of adverse events, and post-procedure follow-up.  Testing changes to improve these processes requires tracking patient-level indicators:

  • Proportion of VMMC clients counseled and tested for HIV
  • Proportion of VMMC clients assessed for sexually transmitted infections (STIs)
  • Proportion of VMMC clients with signed consent
  • Proportion of circumcised clients that experience moderate to severe adverse events
  • Proportion of circumcised clients that return within 48 hours post-operation
  • Proportion of circumcised clients that return within 7 days for follow-up
  • Proportion of circumcised clients that return after 6 weeks

In Uganda, CQI teams use a data tool (downloadable below) to collect the information needed to calculate these patient-level indicators from VMMC registers.  Teams use the Ministry of Health documentation journal for QI teams to record changes tested and results for each indicator.

Why site-level testing of changes is important: Making change is the fundamental concept of improvement: in order to improve and get a better result, you must change something. If a system is not changed, it can only be expected to continue to achieve the same results it produces now.

Site-level testing, on a small scale, is important for two reasons: First, site staff are the best positioned to see what can be changed in their care system.  Second, by testing out an idea with a quick test on a small scale (i.e., with one or a few patients or for a short time), a CQI team can determine whether the change made led to improvement and if there are any potential negative effects that need to be mitigated before making the change permanent.  Testing out the change with staff involved in the process can help reduce resistance to new ways of doing things.