WHAT’S NOT WORKING – THE INTERVENTION, OR IMPLEMENTATION OF IT?
I watched a really informative webinar on applying implementation science to evaluation a few years back that really struck a chord with me. The facilitator, Dr Jessica Hateley-Brown walked participants through the foundations and key concepts of implementation science, including the Consolidated Framework for Implementation Research (CFIR), which if you get a chance – definitely dig into a little more… but it was a little snippet on the differentiation between intervention failure and implementation failure that blew my mind. In hindsight, it’s still a little embarrassing that I hadn’t understood it so clearly prior to this, but I guess sometimes we can’t see the forest for the trees.
Having spent many years delivering services as a front-line clinician, and then managing and commissioning services from a bit further afar, explaining the obvious difference between intervention failure and implementation failure was like giving me a language that I could finally use to explain what I hadn’t been able to put words to. I had lived and breathed the difference between intervention failure and implementation failure so many times – but I’d never thought about it so simply.
The concept of implementation outcomes – which are the result of deliberate actions and strategies to implement our interventions – was not new to me, and won’t be new to most of you. We often collect data about the implementation of our services… but we don’t review it and use it as much as we should. Implementation outcomes are the things that give us an indication of the quality of the implementation of our program or intervention. It’s things like reach, acceptability, fidelity, cost and sustainability. These things don’t tell us anything about the service outcomes or client outcomes. They are the outcomes of successful implementation – if we’re lucky – and therefore give us an indication of the quality of the implementation. Hopefully the quality is high, which lays the foundations for achieving our desired service outcomes and ultimately the client outcomes.
Service outcomes give us an indication of the quality of the service. This might include things like safety, person-centredness, timeliness, and satisfaction. These things don’t tell us about the quality of implementation, nor about any outcomes experienced by the client.
And finally, client outcomes are the ultimate outcomes we are hoping clients experience – and might look like changes in knowledge, skillset, confidence, wellbeing, functioning or health status.
The outcomes of implementation are distinct from service outcomes, which are distinct from client outcomes. Obvious yet mind-blowing at the same time!!
As front-line staff working with people and implementing programs every day would well be aware, program implementation is dynamic. Of course there’s a service or operating model guiding practice, but minor adjustments are made often to accommodate people, or meet people where they’re at. We may learn that some staff are better suited to certain tasks, or some clients are more engaged on certain days of the week. Noticing these things, and making adjustments, can have significant effects on the reach or acceptability of our programs. It’s an early step towards the client outcomes we are hoping eventuate.
But sometimes… programs don’t work. The people we are working with don’t experience the outcomes both they and the provider were hoping for. Is it intervention that failed, or did we fail in the implementation of it?
Maybe staff weren’t trained adequately to deliver the program; maybe the program was new and never fully embraced by the organisation; maybe the workplace had poor communication channels; maybe the program was seen as an add-on to existing work, and attitudes towards it were negative. All of these things will likely effect implementation quality. In some situations, it might be that the program or intervention never really got a chance, and it was deemed ineffective and phased out… when in fact it was poor implementation, or implementation failure that caused its demise.
When thinking about the programs you deliver, support or manage – can you articulate the outcomes of successful implementation, as distinct from service outcomes and client outcomes? It might be a useful task to undertake as a team. Of course, some programs or interventions are flawed in their design… but in many cases, failure to achieve client outcomes is not always due to intervention failure… but could be partially, of fully, the result of implementation failure.
