Richard Smith: What is post publication peer review?

Richard SmithI’ve been tramping from stage to stage arguing that pre publication peer is  slow, expensive ($1.8 billion a year), ineffective, biased, and anti-innovatory and should be dumped in favour of post-publication peer review. But what do I mean by post publication peer review? Despite my best efforts, which are clearly not good enough, people are often mystified.

Many people think of post publication peer review as the comments and letters that appear after scientific articles are published. They point out that these are usually sparse or non existent and don’t add much value. How could they substitute for traditional peer review?

That’s not what I mean by post-publication review, but it is a disappointment to me that comments are so sparse. My friend who blogs for may get 2000 comments in response to a short blog, and blogs in the Guardian, for example, will often have several hundred comments. In contrast, the majority of scientific articles attract no comments whatsoever.

This is worrisome if like me you believe in the Popperian model of science—that the job of scientists is to pose falsifiable hypotheses and then try their hardest through experimentation to show them to be false. Comments from readers should be an important part of that strenuous effort at falsification.

I don’t know exactly why scientists don’t comment, particularly when you often hear them declare particular studies to be hopeless. The main reason is probably that there is no incentive to comment. Instead there are disincentives—fear of upsetting seniors, giving away good ideas, or being wrong.

More published comments could be a useful addition to post-publication peer review, but they will always be only a small part of it. The true post publication peer review, which is for me the real peer review right now, is the process whereby scientific studies are absorbed into the body of knowledge. The “marketplace of ideas” decides whether they are important and should lead to new practices and further research or whether, like most studies, they don’t matter much.

Some post publication peer review is formalised—like the production of systematic reviews or guidelines or the publishing of things like Evidence Based Medicine, Faculty of a 1000, or Journalwatch, which attempt to pick out what is important.

Systematic reviews give a sense of scale to post-publication peer review. Usually the authors will identify hundreds of articles but after applying quality criteria will dismiss all but a dozen or so. The authors will also have to search hard for unpublished studies (perhaps excluded by the lottery of pre publication peer review) and will observe that the best studies are all over the place and not concentrated in the “best” journals.

But more important than these formal types of peer review is the informal, the thousands of comments, decisions, and actions from the many that lead to a sorting of studies. I may hear a study presented or read a paper and be impressed. Others in the audience or other readers might also be impressed. We talk to friends about it. We email colleagues. We put it on listserves. Some of the recipients are impressed and start their own cascade. Others are less impressed and see problems. Perhaps a statistician attracted by the clamour reads the clinical article and sees important flaws that she shares with colleagues. Somebody might incorporate the study into a lecture, a review, or a grant application. And so a study might attract increasing attention and assume a prominent place, or it might fade as its receives more attention and more problems are noticed.

Many studies, in contrast, attract no attention—usually, but not always, rightly.

This decision of the many rather than the few is what I mean by post publication peer review. And the sooner we abandon pre publication peer review and let the real peer review begin as quickly as possible, the better.

Richard Smith
was the editor of the BMJ until 2004 and is director of the United Health Group’s chronic disease initiative.

  • Marc Theilhaber

    Richard, it would be a shame to have your lines about the scarcity of comments regarding scientific topics proved by complete lack of comments here.

    I agree that post peer review will be the way to go. Web 2.0 gave us the technology – we simply need to make use of it. Of course, scientific publishing giants like Elsevier are not exactly keen on this process of democratizing scientific publishing. As PloS has proven, high quality research publishing does not require forfeiting copyright to a publisher. Which, of course, is something we all know.

    Your efforts are laudable. It will be very interesting to see how this game plays out over the next few years.

  • If I correctly interpret your idea of post publication peer review, what you write is not so much as a vetted peerage but more along the lines of crowd voting of studies and knowledge post publication, and dependent on who and how people access the source. That brings into question visibility of a paper, and who gets to comment, and perhaps ushers an era of life course of a paper/evolution of knowledge process, refined through myriads of comments and remarks. In the end, the research will redefine the author.

    I think for this to happen, it calls for a fundamental re-thinking of what constitutes published/established knowledge. Peer review process may have its own problems as you argue, but it has its merits in refinement of prepublication papers when they are done well. I look forward to read the debates in this page. I'd probably go for a more middle ground, and a more creative way to use peer review process with post publication modification and crowd sourcing of refinement, as it were.

  • Jess

    Perhaps I'm not understanding your idea properly. Why would any rational person think that abandoning pre-publication peer review makes any sense at all? Do you think papers are submitted to Journals in perfect condition? The reviewers and editors work along with the authors to develop their ideas and make each paper as clear and concise as possible. This alone makes peer review essential. Not to mention the fact that some papers are just flat out wrong or inappropriate and peer review keeps those types of papers from joining the literature. If people could publish anything they wanted, and we had to rely on the masses to vote the paper up or down, we would have Wikipedia – a good resource, but you better double check the info with a reputable publication.

  • David Kuria

    Any new idea will always be frowned upon due to the lethargy associated with accepting the status quo. While that may be, I still opine that pre-publication peer review is still important since my experience has been that it has greatly enhanced the quality of my publications. It also allows filtering out some otherwise mediocre papers which would lower the integrity of the target journal.

    If the post publication review was to access the same quality of reviews as the pre publication version, with questionable publications being questioned, then the integrity of the process can be vindicated in my opinion. As we venture into it let's integrate the good points of the conventional pre-publication peer review in the process. I think we should consider a strategy of collaborative editing of the initial submission (say for a period of 2 weeks – 2 months) after which the document then becomes adopted as the final version of the paper.

    … Just my thoughts …

  • As I understand it, the idea is to post submitted drafts on the journal website with a big tag like “Under review.” The editors then send the paper out for review and gather comments. They make their editorial decision, the author may be given the opportunity to revise, and eventually the paper can appear as “published” on the website. Is this the idea?
    If so, the positive side is that the paper is also open for discussion while it is under review, which may give even more feedback to the editors and the authors. And it also in some sense “gets the results out there.” On the minus side, considerable efforts will have to be made to distinguish “under review but on the website” from “published.”
    Some kindred thoughts from a recent talk:
    Publishing in the Adjacent Possible

  • Patrice Forget

    Open Access and post-review will probably coexist conjointly with more classical editing ways. But the question is, as NIH discussed, is who can control the publication of the results? So, why not, and why not the most open fashion with good referencing, for the greatest chance of “thousands referees”?