Richard Smith: Important study points towards a different future

Richard SmithIn what I think is a very important BMJ paper, John Ioannidis and Fotini Karassa have shown that systematic reviews, the highest level of evidence, may be highly deceptive. We did, I suppose, know this in a way, but they illustrate the case beautifully. What disappoints me is that nobody has responded to their paper.

I’m reminded of the first time that I ever wrote to a medical journal—to the Lancet in 1974, when I was a medical student. I wrote in response to Ivan Illich publishing a paper saying that “Major medicine is the major threat to health in the world today.” I said that I was amazed that nobody responded. What did that mean, I asked. Did everybody agree? Did nobody have arguments to refute what Illich said? Or did people not think the paper worthy of comment, in which case, I wondered, why did the Lancet publish it?

Similar questions occur to me with the lack of response to the paper by Ioannidis and Karassa.

John Ioannidis has already shown how most research findings are false and how doctors may receive a very distorted view of the evidence by reading the top medical journals. It’s no wonder that he has reached the attention of the New York Times, but I fear that his messages may be a little too uncomfortable for the average reader of medical journals.

Since the article by Ioannidis and Karassa was published, Fiona Godlee and Elizabeth Loder have told us how systematic reviews have misled us on the effectiveness and safety of the antidepressant reboxetine and argued that “the medical evidence base is distorted by missing clinical trial data” and that “urgent action is needed to restore trust in existing evidence.” But, as Ioannidis has shown, there are many sources of distortion and manipulation in addition to failing to publish data.

Sadly, I’m sceptical that the answer lies with journals. High profile journals are very attractive to those who want to push their idea or product–and it’s not only drug companies who want to do so. They guarantee attention but may also amplify the distortion, as Ioannidis has argued.

As Ian Roberts and I suggested years ago, we need another way to make research available. The job of journals should not be to publish original research but rather to assess critically the research that is published in full on databases. Find the comparatively few studies that matter, test them to destruction, and place them in their full context.

The snag, of course, is economic. Subscriptions from institutions and payments for reprints from drug companies are major sources of revenue for journals, and it seems that institutions (the profitable subscribers) are paying primarily so that their users can access research. They are less willing to pay for analysis and comment.

So we may need to find not only new ways of disseminating research but also new business models to support not necessarily journals but rather the process of critically appraising, digesting, and implementing research. Both are difficult problems, but most interesting problems are difficult.

Competing interest. RS is a former editor of the BMJ and chief executive of the BMJ Publishing Group. He will have a pension from the BMA, the owners of the BMJ, and the profits from publishing are important to the BMA. RS is also on the board of the Public Library of Science, an open access publisher that is now profitable largely because of income from PloS One, which might be described as more a database than a journal, RS receives no payment for being on the board, but has his expenses paid twice a year to attend a board meeting in San Francisco. But an economy flight to and from San Francisco over three days is more a punishment than a reward.

  • Richard, you are spot on about the need for the world to notice John Ioannidis's work.

    However, you oversimplified things when you said that”The job of journals should not be to publish original research but rather to assess critically the research that is published in full on databases”.

    Unfortunately it is a bit more complicated than that. As Deborah Cohen pointed out in her recent article “Rosiglitazone: what went wrong?”, it is sometimes necessary to trawl through the individual patient records to assess risks of bias and error in research papers. See

  • Julian Sheather

    Dear Richard,

    Thank you for your post.

    Earlier I was reading a piece in The Atlantic in which David Freedman interviews Ioannidis. ( I also reviewed Freedman's latest book Wrong, in which he talks at length both about Ioannidis and the extraordinary unreliability of even the most garlanded of our contemporary experts. I am not a scientist. Nor am I medical doctor. I plough the narrow furrow of medical ethics. But I am someone who depends, as we all do, on the opinions of those experts. My health, my wealth, my very wellbeing leans upon them. The compound effect of reading both 'Wrong' and Ionnadis has been little short of seismic. I feel a little like those early believers reading Copernicus: the nature of the universe and the sense of my place in it has changed for good. Like you I am bewildered by the nonchalance with which these revelations are received. Our emperors are naked. Where do we turn now for good counsel?

  • Richard Smith

    I agree, Michael, that reviewing published research is not enough. As the paper by Ioannidis and Karassa shows, you need to consider the broader context. And, as Richard Peto and others have shown, the best systematic reviews are based on individual data, and then Deborah's excellent work shows the importance of looking at individual patient records. The sad truth is that it is easy to deceive–without needing to be frankly dishonest.

  • Richard Smith

    I've noticed that I've made a mistake in my quote from Illich. It should read “Modern [not major] medicine is the major threat to the health in the world today.”

    My impression is that I'm making this kind of mistake more often–just like I'm finding it harder and harder to remember names. This is cognitive decline, which perhaps means that the world should pay less and less attention to what I say. Or could there be some sort of compensation that comes with age? Might it be “wisdom”? I don't think so.

  • Disgruntled Phd

    Your link to the paper does not appear to work (at least from my computer).

    On the substantive, I agree that we need better means of disseminating research (with less time delays – delays of a year or more are common in my field, psychology).

    Personally, i feel that research databases should be funded by governments and institutions, and any commercial interests in scientific knowledge should be avoided.

  • jhwalker

    I have fixed the link to the paper.

    Best wishes,

    Juliet, BMJ blogs editor

  • Great Idea! Michael. This could actually take Richard's suggested database of research work one step further.

    To the best of my knowledge he has already pioneered one such database in the “Cases” journal although it is currently restrictive such that patients (as independent researchers on their own health) may not be able to post on their own.

    I wonder if you fancy analyzing such patient-health professional 'user' driven 'patient-centered research databases.' One of them is the BMJ Case Reports journal although we haven't moved to the 'patient as independent researcher stage yet.


    COI: I work for the last journal although I have never been punished with economy flight tickets to San Francisco. 🙂 I also work for another journal which aims to create a 'user driven database' (although currently languishes in a paper format (a setback that we hope to correct soon)

  • Hi Richard,

    Well a major part of Illich's 'modern' medicine may have changed by now if we consider current Evidence (fresh off the pan)?

    Your uncorrected version “Major part of (rather than modern…why leave out the traditional?:-) ) medicine is the major threat to health in the world today” perhaps still holds good?

    Does cognitive decline add a few slips of newer wisdom, I wonder? 🙂

    Loved your COI statement punchline and all. 🙂

  • Tom Jefferson

    Dear Richard, yes there are many sources of distortion and bias in addition to medical journals. I also agree that as things currently stand medical journals are part of the problem, not part of the solution. I would also add (probably most) systematic reviews to the list because they are mainly based on what is visible, whether it is published or unpublished (and not on the totality of evidence). We still have no comprehensive approach to the problem. As your remarks point out, the majority of readers or users of journals still do not realize that there is a problem. Those that do are overwhelmed by the enormity of its implications (hence they prefer not to react to anything which may rock the boat such as Ioannidis’ work). Readers of prestigious journals do not realize that they are the preferential target of promotional messages. The editors of such journals may realize that they are being targeted, but in this curious symbiosis of big money with big journals they are in need of the cash that reprints and sponsorship bring to their institution.

    In a complex world we have a complex problem, but few of us seem to even want to discuss it or even admit that it exists. Its solution (if we ever get there) will entail a broad coalition of forces, but we could start with doctors who are responsible for signing prescriptions and giving impartial advice to patients.

  • Birte

    More about Freedman and Ioannidis by Kent Anderson on the Scholarly Kitchen blog: “Is There Really a Systematic Problem in Medical Publishing? Or Just a Reporter With a Narrative?”

  • Some of us can't comment because the article is not available to non-subscribers. Sad was the day when BMJ pulled back from being a full free text on the net journal. I used to read BMJ regularly and mention it in my classes and on my website, but no more.

    Steve Simon,