We knew that we had “a colossal problem of quality” when we began the peer review congresses in 1989, said Drummond Rennie, creator of the congresses, at the seventh congress in Chicago earlier this month. That problem is now better described and defined, in large part because of the congresses, but it’s even bigger than we realised. Some steps have been taken to put it right, but many of the consumers of medical journals remain naively unaware of the severe deficiencies in what they are reading.

It’s become a tradition for Rennie to lambast editors at the end of the congress. He started the congress because of the paradox and scandal that editors who spent their time judging the science of others never bothered to examine scientifically their own processes. Some seemed to imagine that their processes could not, like angels or hobgoblins, be studied by science. A defining moment came in the first conference when Iain Chalmers, founder of the Cochrane Collaboration, stood up and said to Bud Relman, then editor of the New England Journal of Medicine: you have criticised almost every single study presented but have not offered one piece of evidence, not one, to support your own assertions. Now we do have a body of evidence, although it is a body full of defects.

Editors, said Rennie, “oiled and polished their vintage machines” but with no idea of what went on within them or how they could be improved. We now know a lot about the problems of the sausage but still little about the sausage machine. Most editors of medical journals do not feel the need to know about the evidence base to their craft, and perhaps the hit phrase of the congress was “hobby editors.” People laughed when the phrase was first used. We laugh most about subjects that evoke anxiety, and most of the audience would define themselves as hobby editors, although by attending the congress they showed their commitment to the evidence behind editing.

My suggestion that it was time to end hobby editing was not well received, although I was repeating a point made by Stephen Lock, my predecessor as editor of the BMJ, in 1995.

He thought that a report into a major scandal of publication misconduct “should end the amateurism in journals.” Sadly it didn’t.
I upset people by implying as an ex-fulltime editor that I was a superior creature, but there are many full time editors who are ignorant of the evidence base of editing. One speaker asked how many of 500 people in the hall had had any training before starting work as editors: six people put up their hands. Luckily the same would not be true of surgeons, but it’s shameful that a multibillion dollar industry that has some of the highest profit margins of any industry should be staffed by often unpaid amateurs who are given no training and have such a poor evidence base. It’s to be expected that journals would have “a colossal problem of quality.”

Rennie quoted a JAMA study from 1947 that looked at 149 studies and found three quarters to be scientifically invalid. Since then we’ve had dozens of studies finding the same, culminating perhaps in the work of John Ioannidis and colleagues showing that most published research findings are false, rendered so by a potent cocktail of bias and random error. We’ve also come to understand how some 50% of studies are not published, with a severe bias towards publication of the positive, and how many studies are not just wrong but are manipulated to give results intended to deceive. Randomised controlled trials, said Rennie, are like an optical illusion that shimmers between a scientific paper and “ a word from our sponsors.” We must worry about the integrity of the evidence base that underpins medicine.

The congresses have helped us greatly in understanding the extent of our failures and ignorance, and as ignorance is the starting point for knowledge and improvement that must be a good thing. But, regretted Rennie, there have been no studies of the cognitive processes of editors and no proper trial of peer review. Editors are still in a position like that of doctors using a drug for which they have no evidence of benefit but considerable evidence of harm. As Rennie has famously said, “If peer review was a drug it wouldn’t get a licence.” One systematic reviewer at the congress said, give us all the wheat and the chaff and we could sort it out, whereas prepublication peer review will block or delay publication of wheat while letting through tons of chaff.

Rennie also regretted that much of what we have learnt has not been implemented. It’s now nearly 20 years since he suggested that authorship in science be replaced by contributorship, which is clearly a better system, but authorship, with all its corruption and misinformation, is still the norm. Rennie has also advocated open peer review (where the authors and perhaps the readers know the identity of the reviewers), but anonymous peer review still predominates, including at JAMA, where Rennie was deputy editor for many years. “Anonymous peer review,” he bemoaned, “is totally unethical. I’ve spent my life working in an unethical system.”

But there are signs of improvement. Although, as Rennie pointed out, it took them 13 years, the International Committee of Medical Journal Editors does now require registration of trials for the journals to publish the results. The Alltrials campaign is trying to ensure the publication of all trials. A drive is on to make all the data behind studies available. The Equator network is setting reporting standards for many different kinds of studies, and many of the newer journals, including BMJ Open and all the Biomed Central journals have open peer review.

But reform is a slow, slow process, and, warned Rennie, it may not matter what journals do because “the world may overtake us.”

RS was given free admission to the congress and had his expenses paid by JAMA because he is presenting a film on the history of evidence based medicine and was master of ceremonies at a “roast” of Drummond Rennie, the creator of the peer review congresses and a great friend of RS.

Richard Smith was the editor of the BMJ until 2004 and is director of the United Health Group’s chronic disease initiative.