No two operations are the same, not least because the surgeon starts each new one with some additional experience and knowledge gained during the previous one. But what counts as innovation in surgery, and how should it be appraised and regulated?
Nearly 40 academic surgeons, methodologists, research funders, editors, and regulators from the US, UK, and elsewhere in Europe met at Rhodes House, Oxford, on 9 and 10 December to discuss the scientific evaluation of surgical and interventional techniques and devices. Over two days we debated, updated, and pondered how to implement a set of proposals called the IDEAL framework (Idea—Development—Exploration—Assessment—Long-term study). These proposals were first mooted last year in a Lancet series by the Balliol Collaboration, an international group of experts who adopted the name of the Oxford college where they first convened.
Innovation in surgery covers a huge range, from using a slightly different approach with one patient to conducting a multicentre randomised controlled trial of, say, a new hip implant. In each case the intervention will be a complex one, not least because it depends on the anatomy and pathology of the patient and the actions, skill, experience, and position on the learning curve of the operator. These complexities, coupled with things like surgeons’ preferences and the challenges of choosing valid comparators make the interventional gold standard – a randomised controlled trial – very hard to do. As Peter McCulloch (Clinical Reader in Surgery at Oxford’s Nuffield Department of Surgical Sciences and co-founder of the Balliol Collaboration) puts it, Buxton’s Law is a big problem in surgical trials. That law says “it’s always too early (for rigorous evaluation) until it’s suddenly too late”: in other words when a newish but unevaluated technique gets widely adopted by surgeons, equipoise goes out the window. Would you, as a patient, say OK to this invitation: “in this study we’re going to anaesthetise and cut you open, but you may get a dummy procedure or other operation and we don’t know if these options are better or worse than the new technique that we’re testing, even though most surgeons are already using it”?
Despite these challenges, the status quo isn’t tenable. Relying mainly on a case series to assess new surgical techniques and only on CE kite marking for new devices (rather than evidence based appraisal and formal regulation) is, surely, woefully inadequate. As one participant at last week’s workshop put it, CE marking does little more than show that a device won’t break or rust.
The IDEAL framework takes a much more evidence based approach. It’s based on the UK Medical Research Council’s recommendations on complex interventions which give researchers and funders a framework to develop and evaluate innovations through several iterative phases; to use experimental rather than observational research designs when feasible (while recognising that they won’t always be feasible or appropriate); to measure outcomes as well as process; and to report detailed descriptions of interventions to improve reproducibility, evidence synthesis, and wider implementation.
Grafting this into a surgical setting, the IDEAL framework proffers five stages to develop and implement surgical innovation. Stage 1 establishes proof of concept for each innovation, with detailed case reports and series being shared through some kind of reporting or registration. Stage 2a is a development phase where other early adopters take up the innovation to check and improve its technical and ethical aspects, then track and report these experiences in prospective development studies (cohort studies or modified consecutive case series) – and , of course, report any adverse events. Stage 2b is an exploration phase where the innovation is more widely taken up; surgical skills and learning curves are extended; and indications for use, potential benefits and harms, and the range of effectiveness are reported prospectively in uncontrolled trials and/or registries.
Stage 3 assesses how the innovation’s clinical effectiveness and cost effectiveness compare with those of established methods – in head to head RCTs whenever feasible and ethical [here’s where Buxton’s Law may kick in], or various modified RCT designs, or other study designs such as controlled interrupted time series analyses. Stage 4 is for long term surveillance of the procedure’s effectiveness and safety in routine practice, using statistical process measures and perhaps quality improvement techniques.
For each of these stages IDEAL recommends new ways to share experience and data; new research methods; improved reporting and regulation of research; and better teaching (for students and postgraduates) and continuing professional development regarding surgical innovation.
Our task at this IDEAL workshop was to start turning the framework into reality. The surgeons, research funders, methodologists, and regulators have a mountain to climb, but it’s got to be done. Editors from both general and specialist journals get the easy bit: spreading the word about IDEAL, fostering debate within the profession and beyond, helping to develop one or more reporting statements for surgical research (including, perhaps, a CONSORT extension), and encouraging submission of papers reporting new surgical research methods.
Trish Groves is the deputy editor, BMJ and editor-in-chief BMJ Open