The Challenge of Going Beyond the Usual Metrics

Over the years, numerous people have told me that the value of their community-oriented work is not represented in the metrics they have to report. As a consequence, projects are often de-funded and closed down – despite the positive impact they’re having. Two researchers trying to overcome the ‘value vs metrics’ paradox are Andrew Harrison and Andrew Darnton. I first spoke to them about their initiative, Revaluation, in mid 2017 as part of my Beyond Systems project and got an update from Andrew Harrison last month. 

Pritpal S Tamber: Hi Andrew. It’s good to talk again. We better start with what Revaluation is. 

Andrew Harrison: Revaluation is a social process, a group activity for understanding value. It is based on collecting stories about what people at different levels of a system value. These are told to one another and documented in whatever way is natural. Story tellers then arrange the value in the stories by sorting them into a six-box diagram (see figure 1). The ‘calculate’ row is about numbers (often money), ‘calibrate’ is about judgements of the relative merits of different outcomes, and ‘capacitate’ is about understanding the capacity of a movement or network, including its potential to grow. 

Pritpal: I see a visible and invisible column but before I ask about that just explain how Revaluation came about.  

Andrew: It grew out of a challenge from NHS Horizons, the specialist team within NHS England that supports leaders of change within the NHS (National Health Service). They’d been supporting leaders to make change through a social movement. It was clear that the people involved were benefitting from being involved but orthodox forms of evaluation were struggling to ‘see’ it. In 2015, NHS Horizons asked us to work with them to approach the challenge differently. 

Pritpal: Ok, so why is some of the value ‘invisible’? 

Andrew: Value is what matters to us. It’s often personal, and in systems, including big systems like the NHS, it’s often what intrinsically motivates us. For that reason, it’s entwined with our identities. What we saw when we worked with NHS Horizons was that many people did not want what they valued to be translated into formal metrics. They felt that something was lost in the process, and that what was lost was often what they themselves valued. So, people often hide the value they’re creating as a way to protect what matters to them. They make it invisible because they don’t want to give up part of their identity. 

Pritpal: Wow, that’s heavy stuff. Why do you think this is happening? 

Andrew: I’m not sure but what I think is happening is that in large systems we tend insist on a cognitive framing of experience – using our brains to make sense of things – rather than a non-cognitive, emotional, or relational framing. As a result, the emotional stuff is made invisible. I also think that so-called systems of monitoring and accountability separate what we do from what we value in what we do. Revaluation offers an opportunity to ‘in-source’ measurement; people get to self-measure or co-measure, rather than have to measure what a third party asks for. 

Pritpal: Fascinating. So, when you apply the Revaluation method are you seeing new and surprising types of value? 

Andrew: First of all, rather than see Revaluation as a method we prefer to consider it an approach to understanding value as fully as possible in a given context. Our stance is based on the proposition that value is socially constructed, and so can only really be revealed through a social process. 

Pritpal: Ah, I see. That’s why you called it a ‘social process’.

Andrew: Right. Second, we have to acknowledge that sharing what we value is a pretty intimate activity, far more intimate than we usually are in the work place. It’s not an easy thing for people to do. In fact, one might conjecture that it is instinctual for people to want to protect what they value from control by others. In terms of what we’re seeing when using Revaluation, the process is surprisingly developmental. In every setting we have used it, it has not only created a shared understanding of value it has even helped make value visible to those who have been involved in making it. 

Pritpal: That sounds almost magical. Have you faced any resistance in using the approach? 

Andrew: All of the resistance to using Revaluation so far relate to reporting, and the role that this plays in hierarchies. In essence, I see reporting as providing an account to a third party who has little direct knowledge or experience of the value creation that has been going on. So, the reporting of value is mediated through metrics – usually things that can be counted. Behind this approach is a necessary fantasy that those at the top of the hierarchy can tell what matters everywhere in the system. They can’t. What Revaluation does is provide system players with an account of their value to themselves. 

Pritpal: So, that account may not be of interest to those at the top of the hierarchy? 

Andrew: An edited version of it might be but not all of it. And what that means is that those at the top of the hierarchy are choosing to ignore some of the value they’re creating. 

Pritpal: When you put it that way it does seem odd. Why do we have leadership that’s uninterested to the wider value their systems are creating? 

Andrew: I think it’s because we organise ourselves – and hence our systems – for all kinds of reasons that are rarely acknowledged. For example, we include ourselves in order to exclude others. Or we organise ourselves in groups as a way to regulate emotions – anxiety, fear, hatred, for example. These are the things that really drive system design but we either don’t talk about them or don’t realise it. Instead, we create rational justifications for the design, the ‘cognitive framing’ I mentioned above. 

Pritpal: Thinking about that makes my head hurt. So, does Revaluation help people see their systems for what they really are? 

Andrew: Potentially. Revaluation enables people to hold a mirror to themselves and their work. Whether they do anything with what they see is their decision. It may sound counter-intuitive but value is easily surfaced through a social process based on story telling. The odd thing is how little of what is learnt makes it into the official information that flows through organisations. Organisations seem to operate without considering what those working in them seem to value. Organisations might be more effective if they could change this. We have good evidence that exchanging stories about value and making more shared value visible to all those who create it, creates the conditions for ‘seeing’ systems. 

Pritpal: Fascinating. So, what’s your plan to increase the uptake of Revaluation? 

Andrew: Andrew and I have been joined by a small group of practitioners and we’ve organised into a collective – a community of practice that’s learning through doing – and a consortium – a group able to work on specific pieces of Revaluation work in different settings. Our aim is to develop partnerships wherever we can to explore the approach, including what it leads to. We have ‘revaluated’ systems in nursing, the natural environment, physical inactivity, public health, and education – and we have not yet found a setting where it does not help. In the next 12 months, we hope there will be opportunities in the arts, community development, and logistics. 

Pritpal: Well, I wish you well in your work. It’s challenging stuff but clearly important. 

Andrew: Thanks a lot, Pritpal.

This interview took some interesting twists and turns. My interest in speaking with Andrew centred on helping community-oriented practitioners to help make the case of the value they’re creating, the kind of value that’s not represented in formal metrics. What I did not expect was the way in which trying to surface that value may be complicated by the fact that people want to hide the value they’re creating. I also did not expect to be talking about how systems are actually driven by things like anxiety, fear and hatred. Complicated, though this all is, I’m interested to see how Revaluation might be used to help health care better understand itself and hence be a more self-aware partner to the people it seeks to serve. 

Banner image by Annie Spratt on Unsplash