Innovation is currently fashionable. But new is not necessarily better [1]. Progress rests in sifting out the effective innovations. Edison clearly understood this process: when he developed the light bulb, he tried and discarded thousands of possible filaments. Without testing and recording each option, he may have gone on a random walk and left us in the dark. But persistence despite failures paid off. When a reporter asked him about his lack of success he replied: “Young man, why would I feel like a failure? And why would I ever give up? I now know definitively over 9,000 ways that an electric light bulb will not work. Success is almost in my grasp!”

Medical innovation appears to be similar—many failed attempts, but progress through testing and recording these. A recent systematic review found that proposed new treatments are about as likely to be worse as to be better than the current standard. Djulbegovic and colleagues [2] examined 743 trials from four cohorts of registered studies where they could get all the results—published or not (avoiding the positive spin of publication bias). For survival the average advantage was a relative improvement of just 5%, whereas for the primary endpoint it was a relative improvement of 9%. However, there was a wide spread around this small advance: sometimes new treatments could be much worse. The key lesson is to have humility about our innovations, and to put them to rigorous testing. We can then progress by finding the occasional incremental advance, and the even rarer breakthrough.

The history of chemotherapy for childhood leukaemia illustrates this process well. In the 1940’s in Boston, the pathologist Sidney Farber thought folate might arrest the development of childhood leukaemia. However, the children he treated appeared to get worse, not better. Failure? Only temporarily. Reversing this idea suggested anti-folate drugs (such as methotrexate) might work, which did lead to some remissions, but which were only short lasting. However, that a drug might treat cancer was a paradigm shift that was then honed by trials of different doses, durations, combinations, etc. Most showed no improvement, but some did, so that over three decades of experimentation survival went from 0% to over 80%.

So innovation is vital, but must go hand-in-hand with careful evaluation. Most of the improvements are sufficiently small that only rigorous testing will reliabily detect them.  Without testing we end up, as with managerial re-disorganisation, in a random walk that may go nowhere.

References:
1. Evans I, Thornton H, Chalmers I, Glasziou P (2011). Chapter 1: New—but is it better? Testing Treatments. London: Pinter and Martin. Freely downloadable pdf at www.testingtreatments.org
2. Djulbegovic B, Kumar A, Glasziou PP, Perera R, Reljic T, Dent L, Raftery J, Johansen M, Di Tanna GL, Miladinovic B, Soares HP, Vist GE, Chalmers I. New treatments compared to established treatments in randomized trials. Cochrane Database Syst Rev. 2012 Oct 17;10:MR000024.
3. Spain P, Kadan-Lottick N (2010). Observations of unprecedented remissions following novel treatment for acute leukemia in children in 1948. JLL Bulletin: Commentaries on the history of treatment evaluation (www.jameslindlibrary.org).

Paul Glasziou is professor of evidence based medicine at Bond University and a part time general practitioner.