Comments on: Richard Smith: A woeful tale of the uselessness of peer review

By: JEROME GELB

JEROME GELB — Mon, 05 Mar 2012 22:25:08 +0000

We are deluding ourselves if we think we can detect scientific fraud & misconduct without access to the raw data produced by a study. Electronic posting of such data allows those so inclined to perform their own analysis. So many academics, institutions & oversight authorities claim to carefully review studies free of bias, when their conflict of interest is of such magnitude as to render their findings meaningless.

By: Kamal Mahawar

Kamal Mahawar — Mon, 05 Mar 2012 04:59:39 +0000

Dear Richard,

I am sure you know of this recent BMJ poll which showed that 26% of respondents (118) agreed that journals should only carry out peer review after publication. Did the number bring you hope or disappointment?

Kamal Mahawar
CEO, WebmedCentral

By: Andrew Farke

Andrew Farke — Wed, 30 Nov 2011 22:34:03 +0000

Would things have been easier if reviews had been transferred between journals (rather than just the manuscript alone)? I suspect the editors of some journals, if given access to the reviews from another journal and the authors' response to these reviews, would have accepted the paper. This may be more of an indictment of the “impact” criterion used for acceptance/rejection by some journals than of the peer review system itself.

By: Yannis Guerra

Yannis Guerra — Wed, 16 Nov 2011 19:52:32 +0000

My main issue about post publication review is that it implicitly demands pureness and virtue from the authors. Because there is pre publication peer review, a lot of people that would publish garbage just to pad their CVs don't do it, as they have better things to do with their time. Make it only post publication review, and you will have the literature filled with hundreds of papers that are garbage, but that will take too long to filter out. In the long run we will have 2 groups of papers, the real good ones, and a much larger group of CV inflating bad papers. Is that a better situation than what it is right now? I am not sure.
One clear example of a similar phenomenon that is happening right now is with the retracted/withdrawn papers. A large amount of them continue to be cited as useful, because people either ignore the retraction, or the papers support the position of the new article so you voluntarily "ignore" the retraction.
Always be careful of the law of unintended consequences.
A free market (even of ideas) will always be littered of cheap imitators with a few original ones.

By: Abhijit Bal

Abhijit Bal — Wed, 16 Nov 2011 14:32:13 +0000

I dont think the story of rejections is as fascinating as one would want it to be. From Marshal's interview, it appears that a local scientific society in Australia rejected the abstract and the Lancet accepted the paper- first in June 1983 followed by a more elaborate paper in June 1984. No journal rejected the paper from what I gather from the interview. Correct me if I am wrong.

By: Abhijit Bal

Abhijit Bal — Tue, 15 Nov 2011 22:20:09 +0000

The NEJM paper (Turner et al): This paper describes how positive studies are published while negative studies are not. The authors conclude that there is bias but the reason for the bias was not determined. As you have rightly pointed out, anything can eventually get published so there is no reason why the negative studies would not get published. I therefore suspect that many negative studies were not submitted. Neither pre-publication review nor post-publication review would have any bearing on this.

I agree with the conclusions of the PLoS paper (Young et al). Papers without flaws should be published when there is no restriction on space. Some pre-publication check would be necessary to prevent at least obvious flaws from getting published. Once again, some kind of stratification of papers would be useful, I think, for end-users like me.

The JAMA paper (Ioannidis) is very interesting but we don’t know which studies were correct- the original highly cited papers or the subsequent studies. Also, both studies could be correct because they take place at a different time with different variables. In fact, randomized control trials often have several exclusion criteria. The patients we see in day to day clinical practice are typically the ones who would be excluded from clinical trials. So in that sense nothing is right. But how all this relates to peer review, I am not so sure.

I could not access the other JAMA paper (Ioannidis and Panagiotou).

By: Richard Smith

Richard Smith — Tue, 15 Nov 2011 17:51:54 +0000

I
suggest that you read these three papers. Three show how "top
journals" provide a biased, over positive view of the world, and
the fourth illustrates how we have been hoodwinked into thinking that
antidepressants are much more efficaious than they actually are by
distorted publishing practices.

http://www.plosmedicine.org/ar...

http://jama.ama-assn.org/conte...

http://jama.ama-assn.org/conte...

http://www.nejm.org/doi/full/1...

And
if you can't be bothered with the full articles you can read my blog
on why we should beware of top journals.

http://blogs.bmj.com/bmj/2011/...

By: Abhijit Bal

Abhijit Bal — Tue, 15 Nov 2011 15:23:55 +0000

The top journal in a given specialty does not necessarily publish the top papers of the specialty but overall, it would be fair to say that good papers do get published in good journals as the history of Wislar paper shows (PLoS also has a decent impact factor, which for all its drawbacks is some measure). When was a “Nature-like” paper published in the “Obscure Journal of Science”? Again, this may even have happened but does it happen all the time?

All of us adjust our scientific radars according to our needs. In absence of the source signal, life would be more complicated. We would simply have to wait for someone to do a thorough post-publication peer review or else go through every paper ourselves, good and bad. Who pays for that time? Why must post-publication reviewers be made to review papers which otherwise would never have been submitted by the authors if a minimal filter was available in the first place? Again, who pays for the time it would take to spell out poor papers as poor papers?

Science may have functioned differently 100 years ago but there was also less access to free speech and so only seriously interested people would embark on the difficult journey propagating their findings. Life is different now and anyone can blog anything without spending too much effort which tilts the balance in the other direction.

But overall I think PLoS One has provided some answers. They publish papers which have evidence based conclusion although the cost is very high. A cheaper PLoS One (unless one can justify publishing papers whose conclusions are not evidence based) followed by F1000 would be perfect.

The current system is far more ideal. The alternative that you propose has too many chaotic elements in it. I might be hopelessly wrong but this is what I think.

Sorry I posted twice earlier because I thought there were some problems posting.

By: Richard Smith

Richard Smith — Mon, 14 Nov 2011 13:59:06 +0000

Your assumptions are wrong. I spotted the paper because the authors told me about it soon after it was first written. Since publication the paper has received only two citations–but deserves much more. I hope that postpublication peer review will now begin to work where prepublication peer review has failed.

By: Richard Smith

Richard Smith — Mon, 14 Nov 2011 13:55:19 +0000

As my predecessor as editor of the BMJ showed, you can get anything published in you persist long enough. Prepublication peer review does not weed out papers; rather it takes a long time wastefully to sort them into different journals.
That sorting might be worthwhile if it meant that “the most important papers appeared in the highest profile journals.” Unfortunately it doesn't. What it does do is to introduce a systematic bias into what's published–with the positive, sexy stuff appearing in the “top journals” and the papers spelling out their errors appearing in low profile journals.
The system is bust.

By: Abhijit Bal

Abhijit Bal — Sat, 12 Nov 2011 22:24:38 +0000

Richard,

It is not as easy as you sound it to be. You could spot the paper only because the current process had filtered a lot of rubbish. True there is still rubbish out there but you could ignore to read those journals. In absence of this stratification of published papers and filtering, you wouldnt see the intended paper. True, somebody would see but there would also be time spent in going through several low quality papers. That time also costs money.

By: Abhijit Bal

Abhijit Bal — Fri, 11 Nov 2011 22:24:56 +0000

Richard,

Had it been published straight-away it is likely that it would have been taken less seriously and the paper would have been lost in the ocean of literature. The process has given the paper a "stamp of approval" and so it would be wrong to suggest that nothing would have been lost had it been published without review. You spotted the paper only because PLoS published it. If everything was published, it would be hard to spot papers. In the current process, you can choose to ignore poor quality journals. In fact, the peer review process itself deters many substandard papers as it is not time effective for authors to pursue a poor work all the way through.

Also, only the authors can comment whether they benefited from the review process which in this case does seem a bit protracted to me.

By: Richard Smith

Richard Smith — Fri, 11 Nov 2011 20:45:35 +0000

David
Horrobin, the entrepreneur,
medical researcher, author and editor, was a great critic of peer
review, and his central criticism was that truly original work, the
work that moves science forward, was very likely to be rejected by
peer reviewers—because peer review is a “lower common denominator
process.”
He collected many examples of important studies, several of which led
to Nobel prizes, being rejected by peer reviewers,
http://jama.ama-assn.org/conte...
but he didn't
include one of the best examples—that of Barry Marshall's studies
showing that Helicobacter
pylori was
the cause of peptic ulcer, work that led to a Nobel prize for
Marshall.

When
I graduated in medicine in 1976 it was not known that peptic ulcer
had an infectious cause—and much of the surgery we learnt about was
surgery for peptic ulcer and H2 antagonists were some of the best
selling drugs. The idea that a bacteria might cause peptic ulcer
seemed ludicrous because everybody knew that the stomach is extremely
acidic and that bacteria couldn't flourish there.

I
was talking about case reports at a meeting this week, and I referred
to Marshall's famous paper in which he described infecting himself
with H pylori and giving himself gastritis—so fulfilling
Koch's postulates. Somebody in the audience told me how they had seen
Marshall talk and show slides of the many rejection letters he'd
received. I've tried to find out more, and clearly Marshall did have
much of his work rejected—for example, in February 1983, according
to Wikpedia, the Gastroenterological
Society of Australia rejects Marshall's abstract to present his
research at their yearly conference and deemed it in the bottom 10%
of papers submitted.

Perhaps
somebody can give us more detail on Marshall's rejections.