In this post, Cheyanne Scharbatke-Church and Sandra Sjogren reflect on their experience using Most Significant Change (MSC) in monitoring an anti-corruption program.  We discuss the fit to our context, question why the stories were all about success and share some of the impacts the process had on the Network itself. These reflections build from our prior post, where we described our monitoring needs, why we picked MSC and the steps we took.

The participatory nature of MSC was a good fit, but it took time.

MSC, with its participatory process and focus on storytelling proved to be a good fit for the group for two reasons.  First there is a cultural familiarity with story-telling which meant participants had an immediate affiliation to the approach.  Second, it is second-nature for educated criminal justice professionals to pay attention to detail and articulate arguments.

Despite or possibly because of this, the process of explaining the method, writing stories, sharing and discussing with the network (app. 12-15 people) took half a day to a full day; somewhat longer than expected.  This was partially due to the detail-oriented nature of the group and their desire to confirm they were doing it ‘right’ at each juncture in the process.

The MSC method impacted the Network

Given the Network size at the time, it was not possible to make the stories completely anonymous or have totally different participants in each MSC session.  As a result we expected friendly competition to arise within the group about who had the ‘best’ story during the selection process.  However, we didn’t expect the story selection process to get so competitive that after the first session the program team was jokingly ‘accused’ of favoritism.

In the next MSC session, in an attempt to avoid the perception of favoritism, we dedicated more time to explaining the selection process and criteria.   Nonetheless the program team was sufficiently concerned about a deleterious consequence on network cohesion that it advocated to not have the same person’s story selected more than once.

Conversely, the process also positively impacted the group.  The experience of discussing each story helped create cohesion in the Network.  It reinforced a sense of solidarity amongst members and gave us insight into what mattered most to the group.  To date the positive effects of the MSC process have outweighed the concerning elements of competition.

Did the method promote an overly positive response?

Almost all of the stories that were gathered throughout the MSC were positive in terms of changes in the direction the program intended. We had no reason to believe that there were important negative stories being intentionally hidden or avoided, but since combatting corruption in the justice sector is not without challenge we did expect to hear at least some stories that would reflect experiences of difficulty.

Over time we started to worry about social desirability bias playing too much of a role in the stories that were told and selected.  Upon reflection, there are several additional plausible explanations for the overwhelmingly positive nature of the stories offered.  One is that the nature of the network implicitly pressured participants to only tell stories of resistance against corruption.   Another is that the way the questions were introduced and/or the description of the purpose of the meeting left or led the participants towards positive stories. Another possible explanation could be the complex political context in DRC. With all members of the Network being justice professionals, they may have self-filtered to avoid potential problems with their institution.

Determining Significance of a Story Was Not Simple

The MSC process produced many interesting stories without difficulty. It was notably harder to gain authentic consensus on which of those stories were the most significant.  One of the challenges was navigating the different ‘status’ levels of group members. We found that the status of who wrote the story did not matter in the technical selection process of the most significant stories – when the committee picked. However, the status of author did become important in the discussion after the committee presented their choices about why the story mattered.  People with status e.g. a magistrate or lawyer informally directed the discussion and made comments on the story itself, e.g. what was “good” about it, but not on why they thought the story was important.

Looking back, there should have been more time left for the whole group to explain why each thought the story was significant.

Documenting the stories allowed for multiple uses

Unexpectedly, the written stories provided the M&E team data that could be used beyond the immediate MSC reviews. For instance, the M&E team analyzed all the stories produced in each session for trends such as common types of change and compared them to the program’s theories of change. These stories were also used in the formative evaluation.

Would we use MSC for anti-corruption programming again?

Our most significant takeaway is that using MSC offered us important insights into the real experiences of those involved in the program. Even though it was difficult to dig into significance of each story, MSC gave us more nuance and flexibility than previous monitoring methodologies we’ve used.  This helped us adapt the program quickly, and identify/articulate our assumptions mid-process.

If we were to use MSC again, we would put greater emphasis on questions that sparked reflection on why change(s) happened. This could be done by leaving more time for discussion after the stories are selected: in our context, information given orally would have been best to complete information from the stories but we chose to spend more time or the writing part.