Richard Smith: Medical journals: “a colossal problem of quality”

Richard SmithWe knew that we had “a colossal problem of quality” when we began the peer review congresses in 1989, said Drummond Rennie, creator of the congresses, at the seventh congress in Chicago earlier this month. That problem is now better described and defined, in large part because of the congresses, but it’s even bigger than we realised. Some steps have been taken to put it right, but many of the consumers of medical journals remain naively unaware of the severe deficiencies in what they are reading.

It’s become a tradition for Rennie to lambast editors at the end of the congress. He started the congress because of the paradox and scandal that editors who spent their time judging the science of others never bothered to examine scientifically their own processes. Some seemed to imagine that their processes could not, like angels or hobgoblins, be studied by science. A defining moment came in the first conference when Iain Chalmers, founder of the Cochrane Collaboration, stood up and said to Bud Relman, then editor of the New England Journal of Medicine: you have criticised almost every single study presented but have not offered one piece of evidence, not one, to support your own assertions. Now we do have a body of evidence, although it is a body full of defects.

Editors, said Rennie, “oiled and polished their vintage machines” but with no idea of what went on within them or how they could be improved. We now know a lot about the problems of the sausage but still little about the sausage machine. Most editors of medical journals do not feel the need to know about the evidence base to their craft, and perhaps the hit phrase of the congress was “hobby editors.” People laughed when the phrase was first used. We laugh most about subjects that evoke anxiety, and most of the audience would define themselves as hobby editors, although by attending the congress they showed their commitment to the evidence behind editing.

My suggestion that it was time to end hobby editing was not well received, although I was repeating a point made by Stephen Lock, my predecessor as editor of the BMJ, in 1995.

He thought that a report into a major scandal of publication misconduct “should end the amateurism in journals.” Sadly it didn’t.
I upset people by implying as an ex-fulltime editor that I was a superior creature, but there are many full time editors who are ignorant of the evidence base of editing. One speaker asked how many of 500 people in the hall had had any training before starting work as editors: six people put up their hands. Luckily the same would not be true of surgeons, but it’s shameful that a multibillion dollar industry that has some of the highest profit margins of any industry should be staffed by often unpaid amateurs who are given no training and have such a poor evidence base. It’s to be expected that journals would have “a colossal problem of quality.”

Rennie quoted a JAMA study from 1947 that looked at 149 studies and found three quarters to be scientifically invalid. Since then we’ve had dozens of studies finding the same, culminating perhaps in the work of John Ioannidis and colleagues showing that most published research findings are false, rendered so by a potent cocktail of bias and random error. We’ve also come to understand how some 50% of studies are not published, with a severe bias towards publication of the positive, and how many studies are not just wrong but are manipulated to give results intended to deceive. Randomised controlled trials, said Rennie, are like an optical illusion that shimmers between a scientific paper and “ a word from our sponsors.” We must worry about the integrity of the evidence base that underpins medicine.

The congresses have helped us greatly in understanding the extent of our failures and ignorance, and as ignorance is the starting point for knowledge and improvement that must be a good thing. But, regretted Rennie, there have been no studies of the cognitive processes of editors and no proper trial of peer review. Editors are still in a position like that of doctors using a drug for which they have no evidence of benefit but considerable evidence of harm. As Rennie has famously said, “If peer review was a drug it wouldn’t get a licence.” One systematic reviewer at the congress said, give us all the wheat and the chaff and we could sort it out, whereas prepublication peer review will block or delay publication of wheat while letting through tons of chaff.

Rennie also regretted that much of what we have learnt has not been implemented. It’s now nearly 20 years since he suggested that authorship in science be replaced by contributorship, which is clearly a better system, but authorship, with all its corruption and misinformation, is still the norm. Rennie has also advocated open peer review (where the authors and perhaps the readers know the identity of the reviewers), but anonymous peer review still predominates, including at JAMA, where Rennie was deputy editor for many years. “Anonymous peer review,” he bemoaned, “is totally unethical. I’ve spent my life working in an unethical system.”

But there are signs of improvement. Although, as Rennie pointed out, it took them 13 years, the International Committee of Medical Journal Editors does now require registration of trials for the journals to publish the results. The Alltrials campaign is trying to ensure the publication of all trials. A drive is on to make all the data behind studies available. The Equator network is setting reporting standards for many different kinds of studies, and many of the newer journals, including BMJ Open and all the Biomed Central journals have open peer review.

But reform is a slow, slow process, and, warned Rennie, it may not matter what journals do because “the world may overtake us.”

RS was given free admission to the congress and had his expenses paid by JAMA because he is presenting a film on the history of evidence based medicine and was master of ceremonies at a “roast” of Drummond Rennie, the creator of the peer review congresses and a great friend of RS.

Richard Smith was the editor of the BMJ until 2004 and is director of the United Health Group’s chronic disease initiative.

  • davea0511

    These are good points but editing experience is a small factor when insuring the science is good – I suggest the problem is systemic and has far more to do with motivations than qualifications. We talk about peer review, but imho *peer* review is the problem. We need bi-lateral review … there are critics to nearly every treatment … why aren’t the critics in on each process in order to introduce true rigor to the system? Real editor experience is nice but it doesn’t utilize the best resources for any new treatment. You can be a hotshot editor but if a treatment is popular then what’s your motivation to not rubber stamp it, or what’s your motivation to approve a submission when the conclusions of the study are unpopular to both you and your professional constituents? Nobody is above the fray when money and professional regard is at stake. The editing system needs to be design in a bi-lateral way to insure accuracy and good science always prevails despite the fallout.

  • IdPnSD

    Everything in our world is controlled by money. We have to obey the money, money power, originated by the central bank. Central
    Bank (CB) is the new king. From a janitor to a prime minister (like Tsipras of Greece) must obey and follow the objectives of the king. Yet, we know money is not an object of nature. Therefore money must be false, free, and abundant at the source, which is the CB. Do we have any choices? There is no democracy or freedom. As we all know capitalism and democracy do not go together.

    You cannot therefore implement a democratic peer review system. It will disrupt the money power. As an example you cannot implement a triple blind peer review system: (1) No one will know the names of the reviewers (2) No one will know the names of the authors (3) Even the system administrator (like website) will also not ask for the names and affiliations of the authors.

    Some journals have implemented double blind systems, items (1) and (2), but not triple blind such as item (3). The complete democracy is not allowed. The risk is that such a system may eliminate papers from large, highly financed institutions, jeopardizing their status and image in the society.

    There are other fundamental questions also. Is it possible for any reviewer to really understand the research of a person who is
    engaged in a highly specialized subject for more than ten years? For example: impact of infinity in mathematical theories. Infinity is not there in nature. Therefore it must be false. No large number can be used to approximate infinity. Replacing infinity by a finite number can dramatically change the characteristics of entire mathematical foundations. Fourier Transform is such a mathematical theory. Quantum mechanics uses it.

    Consider yet another fundamental philosophy – nonexistent cannot become existent. In physics it is also called law of conservation. Nothing can be created or destroyed. Thus no one can really
    invent anything new. All paper ideas must all be therefore wrong and false. Since money is false, you cannot create any truth by using this false money. Thus it can be and has been shown that mathematics, physics, economics, religions are all false. For more details take a look at the free book on Soul Theory at the blog site
    Unless we eliminate money no truth can become visible on any subject. No process can ever be perfect. We will keep on polluting the environment: physical, mental, and social.