You don't need to be signed in to read BMJ Group Blogs, but you can register here to receive updates about other BMJ Group products and services via our Group site.

Richard Smith: Medical research—still a scandal

31 Jan, 14 | by BMJ

Richard SmithTwenty years ago this week the statistician Doug Altman published an editorial in the BMJ arguing that much medical research was of poor quality and misleading. In his editorial entitled, “The Scandal of Poor Medical Research,” Altman wrote that much research was “seriously flawed through the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods of analysis, and faulty interpretation.” Twenty years later I fear that things are not better but worse.

Most editorials like most of everything, including people, disappear into obscurity very fast, but Altman’s editorial is one that has lasted. I was the editor of the BMJ when we published the editorial, and I have cited Altman’s editorial many times, including recently. The editorial was published in the dawn of evidence based medicine as an increasing number of people realised how much of medical practice lacked evidence of effectiveness and how much research was poor. Altman’s editorial with its concise argument and blunt, provocative title crystallised the scandal.

Why, asked Altman, is so much research poor? Because “researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them.” In other words, too much medical research was conducted by amateurs who were required to do some research in order to progress in their medical careers.
Ethics committees, who had to approve research, were ill equipped to detect scientific flaws, and the flaws were eventually detected by statisticians, like Altman, working as firefighters. Quality assurance should be built in at the beginning of research not the end, particularly as many journals lacked statistical skills and simply went ahead and published misleading research.

“The poor quality of much medical research is widely acknowledged,”  wrote Altman, “yet disturbingly the leaders of the medical profession seem only minimally concerned about the problem and make no apparent efforts to find a solution.”

Altman’s conclusion was: “We need less research, better research, and research done for the right reasons. Abandoning using the number of publications as a measure of ability would be a start.”

Sadly, the BMJ could publish this editorial almost unchanged again this week. Small changes might be that ethics committees are now better equipped to detect scientific weakness and more journals employ statisticians. These quality assurance methods don’t, however, seem to be working as much of what is published continues to be misleading and of low quality. Indeed, we now understand that the problem doesn’t arise from amateurs dabbling in research but rather from career researchers.

The Lancet has this month published an important collection of articles on waste in medical research. The collection has grown from an article by Iain Chalmers and Paul Glasziou in which they argued that 85% of expenditure on medical research ($240 billion in 2010) is wasted. In a very powerful talk at last year’s peer review congress John Ioannidis showed that almost none of thousands of research reports linking foods to conditions are correct and how around only 1% of thousands of studies linking genes with diseases are reporting linkages that are real. His famous paper “Why most published research findings are false” continues to be the most cited paper of PLoS Medicine.

Ioannidis’s conclusion as to why so much research is poor is similar to that of Altman’s: “Most scientific studies are wrong, and they are wrong because scientists are interested in funding and careers rather than truth.” Researchers are publishing studies that are too small, conducted over too short a time, and too full of bias in order to get promoted and secure future funding. An editorial in the Lancet collection on waste in research quotes 2013 Nobel Laureate Peter Higgs describing how he was an embarrassment to his Edinburgh University department because he published so little. “Today,” he said, “I wouldn’t get an academic job. It’s as simple as that. I don’t think I would be regarded as productive enough.” Producing lots of flawed research trumps a few studies that change our understanding of the world, as Higgs’s paper did.

Chalmers, Glasziou, and others identify five steps that lead to 85% of biomedical research being wasted. Firstly, much research fails to address questions that matter. For example, new drugs are tested against placebo rather than against usual treatments. Or the question may already have been answered, but the researchers haven’t undertaken a systematic review that would have told them the research was not needed. Or the research may use outcomes, perhaps surrogate measures, that are not useful.

Secondly, the methods of the studies may be inadequate. Many studies are too small, and more than half fail to deal adequately with bias. Studies are not replicated, and when people have tried to replicate studies they find that most do not have reproducible results.

Thirdly, research is not efficiently regulated and managed. Quality assurance systems fail to pick up the flaws in the research proposals. Or the bureaucracy involved in having research funded and approved may encourage researchers to conduct studies that are too small or too short term.

Fourthly, the research that is completed is not made fully accessible. Half of studies are never published at all, and there is a bias in what is published, meaning that treatments may seem to be more effective and safer than they actually are. Then not all outcome measures are reported, again with a bias towards those are positive.

Fifthly, published reports of research are often biased and unusable. In trials about a third of interventions are inadequately described meaning they cannot be implemented. Half of study outcomes are not reported.

The articles in the Lancet collection concentrate constructively on how wastage in research might be reduced and the quality and dissemination of research improved. But it wouldn’t be unfair simply to repeat Altman’s statement of 20 years ago that: “The poor quality of much medical research is widely acknowledged, yet disturbingly the leaders of the medical profession seem only minimally concerned about the problem and make no apparent efforts to find a solution.”

I reflect on all this in a very personal way. I wasn’t shocked when we published Altman’s editorial because I’d begun to understand about five years’ before that much research was poor. Like Altman I thought that that was mainly because too much medical research was conducted by amateurs. It took me a while to understand that the reasons were deeper. In January 1994 at age 41, when we published Altman’s editorial, I had confidence that things would improve. In 2002 I spent eight marvellous weeks in a 15th century palazzo in Venice writing a book on medical journals, the major outlets for medical research, and reached the dismal conclusion that things were badly wrong with journals and the research they published. I wondered after the book was published if I’d struck too sour a note, but now I think it could have been sourer. My confidence that “things can only get better” has largely drained away, but I’m not a miserable old man. Rather I’ve come to enjoy observing and cataloguing human imperfections, which is why I read novels and history rather than medical journals.

Competing interest: RS was the editor of the BMJ when it published Altman’s article. Doug Altman and Iain Chalmers he counts as friends (they might even make his funeral), and he admires Paul Glasziou and John Ioannidis (to the extent that he can now spell both of their names without having to look them up.) He’d like to think of them as friends as well but worries he would being to forward as he doesn’t know them so well.

Richard Smith was the editor of the BMJ until 2004 and is director of the United Health Group’s chronic disease initiative.

By submitting your comment you agree to adhere to these terms and conditions
  • Bill Cayley

    Thanks for the post!

    Thankfully, the Economist has helped bring some of these issues into the general public discussion as well, with their article last fall “How Science Goes Wrong” (http://www.economist.com/news/leaders/21588069-scientific-research-has-changed-world-now-it-needs-change-itself-how-science-goes-wrong)

  • Peter Gøtzsche

    Richard Smith writes that he has come to enjoy
    observing and cataloguing human imperfections. These abound in science, and people
    who argue in an evidence-based fashion against treasured beliefs are often
    treated so badly by their colleagues that if we had lived under the
    Inquisition, they would have been sentenced to be burned at the stake.

    I catalogued several condemnations in my book “Mammography
    screening: truth, lies and controversy” that screening advocates often use
    against their opponents and here are two of them:

    1) The ‘you’re not
    one of us’ argument, which has been used at all times as an excuse for
    killing people you didn’t like.

    a) An editor wrote to me that perhaps I needed to add
    a clinical oncology perspective to my ‘undoubted expertise in the finer details
    of trial methodology analysis’. I wonder what difference that should have made
    to our Cochrane review, which plenty of people with all sorts of expertise had
    commented on, both before and after publication.

    b) At a meeting at the Danish National Board of Health
    with its breast cancer advisory group,

    we were urged to include surgical expertise when we
    evaluated the trials for our Cochrane review. I replied that we didn’t need
    surgeons, as we were not going to operate on the breasts ourselves. When the
    issue of including appropriate experts came up again, I responded that we had
    learned how to read and didn’t need help with this. John Ioannidis and I share
    a view that it can be harmful for systematic reviews to include content area
    experts as authors and we published a paper about this
    in the BMJ.

    c) Finnish Professor of radiology Peter Dean has
    argued that those who criticise mammography screening are rarely experts on
    screening or have professional experience with diagnosing and treating breast
    diseases. Dean’s views open some interesting perspectives in societal life.
    Does this mean that you are not allowed to criticise poor car repair work
    unless you are a mechanic? Or poor surgery because you are not a surgeon? Or
    excessive interest rates because you are not a banker? Or politicians because
    you are not a politician?

    2) The ‘you are
    killing my patients’ argument. This is the ultimate trump card, used not
    only by screening advocates but by doctors of all kinds whenever anything in
    healthcare is being criticised. The Catholic Church couldn’t have invented a
    better argument aimed at stopping all rational discussion. Anyone who dares
    criticise current dogma in healthcare is responsible for the death of many patients
    and won’t go to Heaven.

  • Cathryn Walker

    It is not just ‘amateurs’ carrying out medical research that perpetuates this scandal. This is a classic demonstration of bioethics running second to biopolitics. My observation is that the epidemiologists and statisticians are not ignorant of their failure to divulge biases and limitations. Selective reporting is about serving political or funding masters. Without independent verification of findings, quantitative research findings are too often designed conclusions to ensure continuity of funding and career.

  • Cathryn Walker

    It is not just ‘amateurs’ carrying out medical research that perpetuates this scandal. This is a classic demonstration of bioethics running second to biopolitics. My observation is that the epidemiologists and statisticians are not ignorant of their failure to divulge biases and limitations just as much in public health regulatory authorities as in research, maybe more. Selective reporting is about serving political or funding masters. Without independent verification of findings, quantitative research findings are too often designed conclusions to ensure continuity of funding and career.

  • SG

    Very interesting and valid thoughts. I’d suggest that the critical driving variable behind the five causes listed above is cash that drives the sellers of medical technologies and substances – and which realistically drives everything in our world. If consumers will buy it – regardless of the truth behind something – then the cash drive will sway things in that direction, whether consciously or not. Fundamentally, for medical journalism to improve, the driving influences behind why poor quality is the norm would have to change. Consumers – who at the end of the day provide the cash behind the driving influence – need to be given the tools to differentiate between qualities of what they are buying. Further up the purchase chain, the lack of time throughout the medical professions available for critical analysis of medical research literature trickles down to the end-line consumer. The same goes for anyone who relies on medical research in their decision making but doesn’t have the tools/time/unbiased reason for critical analysis. A regulatory body on quality of publication who are equipped (with finances, time and adequate expertise/education) for proper critical analysis of medial literature, without drivers of bias such as a need for independent funding might be a good place to start….. Would the governments of the world provide such a service if they knew the savings it could generate in efficiency terms to the medical world?

  • William Ray

    “Most scientific studies are wrong, and they are wrong because scientists are interested in funding and careers rather than truth.” I disagree most emphatically. Most scientists are overwhelmingly interested in knowledge (and truth). Unfortunately, scientists are also painfully aware of the fact that if they are unemployed, and unfunded, they will have no resources with which to seek knowledge and truth, and no ability to disseminate that any that they might happen to find. The current governmental funding models and mechanisms leave scientists no choice but to prostitute themselves in the hopes that by pursuing the “junk” that is favored by the funding agencies and reviewers, they might also have the occasional opportunity to pursue knowledge and truth.

  • William Ray

    There is also a strong secondary problem, in that every medical or biological textbook contains a chapter on statistics, and while the majority of clinicians and wet biological scientists haven’t read it, and/or didn’t understand it, it’s always there, and in the back their minds, that if they just spent the week to learn that 15 pages, they’d be full-fledged statisticians, or bioinformaticians/quantitative scientists/etc too. Until both the researchers on the bio-life-sciences side of the wall, and the funding reviewers who read their proposals, come to grips with the fact that a statistician or quantitative/computational expert takes just as long to learn his craft, as they took to learn theirs, and that that expertise demands just as large an investment in terms of time and dollars in funding a project, quantitative rigor will remain an afterthought in the life sciences, not because of any particular malfeasance on the part of any of the players, but because of the simple fact that a brain cannot be exquisitely trained in /all/ of the necessary disciplines to be both a top-notch wet/clinical researcher, and a top-notch quantitative/statistical expert.

  • RIchard Feinman

    My conversation with clinicians suggests that these problems exist in many fields of medicine but I do not think that you can find a field comparable to nutrition for bad science and control of research by a biased in-circle with unrestrained hostility to opinion that deviates from the party line. The low fat-cholesterol-CVD paradigm has failed so many times and has a two-foot shelf of books expsosing the failures. As Elizabeth Nabel, then head of the NIH said, when she got on television after the failure of the Women’s Health Initiative: “Nothings changed.”

    The book that I am writing tries to emphasize the biochemistry and its relation to metabolism and nutrition. It turned out that I could not avoid the terrible politics and like Dr. Smith I became increasingly discouraged. My additional mistake, however, was to write the book in Brooklyn. Venetian palazzo. Why didn’t I think of that?

  • aphoenix444

    Well that’s even more terrible then.

  • Huw Llewelyn

    The poor quality of medical research may well be due partly
    to human failings, which history tells us will always be with us. However there may also be inadequate knowledge. This gives cause for greater optimism as history also tells us that humanity’s knowledge increases with time!
    For example, the idea of replication of a single observation
    (or a study result based on many subjects) is very important to doctors and scientists. However, it is not widely understood
    how the probability of replication should be estimated logically by incorporating into it the calculations of statistical significance testing. This makes it very difficut to teach those embarking on research what is expected of them so that they can seek appropriate advice before they start, especially from statisticians.
    In essence we must consider the possible causes of
    non-replication (one of which is insufficient numbers of subjects) and then set out to convince future readers with our research that all these possible causes of non-replication have a low probabiity.
    A similar reasoning process is used during diagnosis and considering different scientific hypotheses that might explain findings that have a high probability of replication [ http://blog.oup.com/2013/09/medical-diagnosis-reasoning-probable-elimination/ ].

  • William Ray

    It’s certainly terrible, but it’s a very different kind of terrible than the easy and incorrect lumping of the problem onto “scientists are only interested in their careers and funding” as an excuse.

    The problem is societal. After spending 10-15 years (minimum) perfecting their craft, society presents scientists with the choice of “dance with the devil of funding agencies and the career/tenure process”, or “go start flipping burgers”.

    Most of us think we can make a greater contribution to knowledge, truth and the world if we sell a bit of our souls, than if we go flip burgers.

    Scientists would /love/ to have a third option, but that option will never exist until society changes the way that it funds, promotes, and disseminates science.

  • Patrick Durusau

    Excellent post but 20 years of “jaw boning” doesn’t appear to had an impact on the problem. What if the leading journals were to collaborate on a joint publication that published research also performed by two labs/research groups chosen by the publication?

    Note I didn’t say “duplicated.” Publishing failures to duplicate should be as important, if not more so, than duplication. With the full data sets and work flow from all three labs/research groups.

    Reasoning the social mechanisms around publication, promotion, status, etc. are going to be very difficult to change. But changing the inputs into those social mechanisms, a higher publishing prize journal with different requirements, might over time erode some current publishing practices.

    Funders would have to be encouraged to fund verification work but with leading journals adding their voices to applications for such funding, that could make a telling difference.

You can follow any responses to this entry through the RSS 2.0 feed.
BMJ blogs homepage

BMJ.com

Helping doctors make better decisions. Visit site



Creative Comms logo

Latest from BMJ.com

Latest from BMJ.com

Latest from BMJ.com podcasts

Latest from BMJ.com podcasts

Blogs linking here

Blogs linking here