In 1987 Martin Buxton observed that “it’s always too early [to evaluate] until suddenly, unfortunately, it’s too late.” This is particularly true in surgery. In most countries, there is rapid diffusion of new surgical techniques. This usually occurs without evaluation, perhaps reflecting a combination of surgeons’ enthusiasm for innovating and a misunderstanding of evidence based practice.

A lack of evaluation may mean that, at best, innovative (and superior) procedures spread haphazardly. At worst, however, it could mean that innovative (and inferior) procedures are introduced widely before their deficits are noticed, potentially leading to harm.

With this in mind, the IDEAL framework for the evaluation of surgical innovation was established. There are five stages (from idea to long term study), and stage 2 is particularly key because it signifies the time at which a randomised controlled trial (RCT) should be considered. IDEAL suggests that before a randomised evaluation is undertaken (stage 3), interventions should have achieved stability and surgeons should have acquired enough experience with the novel procedure.

However, methods for establishing if these milestones have been achieved are lacking. As a result, trials of innovative procedures are often criticised because of disagreements about the way the interventions were delivered and concerns that the results represent the “learning curve.”

So what is “stability?” Ideally, as in a drug trial, it would represent the point at which surgical interventions are identical to each other and delivered in a uniform way. However, surgical interventions are unlikely to ever be identical despite having similar labels. There are so many different ways to achieve an end product and, even if there is an intervention protocol, delivery may vary between surgeons and even with the same surgeon on different occasions.

How, then, should we establish if a surgical intervention is stable enough to be evaluated in the context of an RCT? This is a key area of methodological work, which should help IDEAL stage 2 to be successfully operationalised by surgeons and trialists. The Bristol MRC Hub for Trials Methodology Research is developing methods to define “intervention stability,” informed by work within two surgical RCTs (the By-Band study and the ROMIO study).

The process involves deconstructing the surgical intervention into its constituent components and steps, then studying how surgeons perform them in the operating theatre (to prospectively document variation and technique evolution). Repeating this process enables identification of “stability”—the point at which a crucial amount of the intervention (and the components that are linked to outcomes) is consistently delivered in a similar way (with “similar” meaning within defined boundaries, which vary in flexibility depending on the planned design for IDEAL stage 3). Findings are then reported back to the study team, and consensus methods used to ensure that the “stabilised intervention” sufficiently represents their practice.

Although promising, these methods are themselves innovative, and require further development and testing within other settings and intervention types. We hope that eventually they will form the basis for an interactive online platform enabling surgeons and methodologists to design surgical interventions in a robust way.

Natalie Blencowe (@NatalieBlencowe) is a National Institute for Health Research clinical lecturer and surgical trainee in the Severn Deanery.

Jane Blazeby (@BlazebyJ) is a professor of surgery and honorary consultant upper GI surgeon in Bristol. She directs the MRC ConDuCT-II Hub in Bristol and is a National Institute for Health Research senior investigator.

The views and opinions expressed in this blog are those of the authors and do not necessarily reflect those of the NIHR, NHS, or the Department of Health.

Competing interests: None declared.