Richard Smith: Twitter to replace peer review?

Richard SmithAn interesting article in Nature gives what may be a glimpse of the future of scientific discourse by telling stories of how social media have done a much better and faster job than traditional prepublication review. Science recently published a paper in which researchers claimed to be able to predict human longevity with 77% accuracy. The paper gathered huge coverage in the media, but almost immediately bloggers and Tweeters recognised major errors. Researchers who regularly used the techniques of the study saw a common pitfall, which was why they reacted so fast. One week after the paper was published the authors acknowledged that they had made a technical error, and shortly afterwards Science issued an “expression of concern,” meaning ignore this paper.

The speed with which the mistake was rectified is in startling contrast to the glacial pace at which most seriously flawed or fraudulent papers are retracted. Indeed, the number of retractions is way below what it should be when we know that the prevalence of serious fraud runs at about 1%. “Most papers sit in a wasteland of silence, attracting no attention whatsoever,” says Phil Davis, a communications researcher at Cornell University, in the Nature article by Apoorva Mandavilli.

It was, of course, bloggers who exposed Hwang Woo-suk, the Korean pioneer of stem cell research who was at the centre of the most spectacular fraud in recent years—also published in Science. And just a month ago another Science paper, which said that bacteria used arsenic rather than phosphorus in their DNA backbone, was rapidly dismembered by bloggers and Tweeters.

We shouldn’t be surprised by these developments. We know that prepublication peer review is a fatally flawed process (my latest critique is available as an open access) and we know that the wisdom of the many is much better than the wisdom of the few. We know too that postpublication review is the real peer review in that most papers disappear into obscurity and just a few emerge as important—and often not papers published in major journals. Our present system of trying to sort information by having the best papers in the best journals not only doesn’t work but it deceives us, giving too much attention to the sexy but often wrong.

Cameron Neylon from the UK Science and Technology Facilities Council is quoted in the Nature article as saying that it makes much more sense to publish everything and filter after the fact. We are moving from a world of  “filter then publish” to a world of “publish then filter.” Why, I have wondered, are we not going faster, but then I read Maynard Keynes quote that “It is difficult to get a man to understand something when his salary depends on him not understanding it.”

What is perhaps surprising in all this is that researchers are happy to blog and use Twitter but are reluctant to comment on the websites of journals. Blogs in the Guardian or on attract hundreds and even thousands of comments, whereas articles in journals rarely attract any. Why are scientists Tweeting and blogging but not commenting on articles in journals? I can only speculate that it’s something to do with the stuffiness, formality, and pomposity of journals compared with the happy go lucky, party atmosphere of the blogosphere.

  • In your paper “Classical peer review: an empty gun” you say:

    “I recently debated peer review in front of around 80 people from the Association of Learned and Scholarly Publishers. Unsurprisingly, I was arguing against peer review. Nobody agreed with my position before my talk – and nobody agreed with me afterwards.”

    This reminds me of an observation made by John Maynard Keynes that I came across recently:

    “It is difficult to get a man to understand something when his salary depends upon his not understanding it. …”

  • Theophane Bukele

    This would be the best thing to avoid imprisonment of science by some wise. And many do not know how it works in this circle. Knowing that such scientific personality is often cited, one is tempted to quote him as to successfully publish an article. Some time ago, I prepared an innovative article for publication. This article was clearly in contradiction with the current view in the field. My first reader told me: “Do not say it particularly because it will not pass and peers will say: 'But who is he?”. It's very sad. Theophane Bukele.

  • Liz Wager

    Interesting — maybe I'm about to be proved wrong (again!) … see…/ for my 2009 thoughts on this!

  • Sunil Bhopal

    Interesting happy-go-lucky blogpost!

    In a recent BBC Horizon programme, James Delingpole, the Daily Telegraph journalist called for precisely the same thing, he called it peer-to-peer review.

    Whilst it seems to have worked in your Science and Hwang Woo-suk cases, in the case Delingpole described – 'climategate' – it led to a scandal when there really was none.

    Peer-review isn't perfect but sometimes 'peer-to-peer' review may be even less so, particularly when politics and ideology are part of the debate…

  • amcunningham

    Recently I came across your comment on Hilary Bekker's editorial last autumn where you stated that people were using BMJ Rapid Responses! But it's true that commenting on journals is not nearly as common as one would expect.

  • amcunningham

    Last night I noticed a comment of yours on the Hilary Bekker editorial where you were remarking on the flurry of Rapid Responses! The BMJ does seem to be much more successful in garnering comments, but it also makes them more obvious. On other journals they feel hidden away. I've made my first comment on a PLOSone paper last night. But I still enjoy tweeting about it as well!

  • Abrooks

    The fact that there are, until now, only 6 comments to your post (this being the seventh) underscores your point. (I also retweeted this link.)

    I think there is simply too much institutional inertia to effectively leapfrog from the current peer-review by the very few then publish to a faster, cheaper and potentially more efficient approach of publishing a paper under, perhaps, a probationary period before it achieves the status of 'peer-reviewed' by the masses post publication.

    Intriguing, all the same.

  • “Led to a scandal when there really was none”

    Ahem – in what way was demonstrated corruption of the peer-review process by a small group of people with significant financial vested interest in maintaining the 'alarmist' status quo not really a scandal…?

    “Peer-to-peer” review, with ALL data and methods out in the open for all to review and comment on is definitely the direction in which science should be moving – whatever your political views…