Wim Weber on OPEN—the European research project to study publication bias

wim_weberOn 23 and 24 May the final workshop of the OPEN project took place in Freiburg, Germany. OPEN—To Overcome failure to Publish nEgative fiNdings—is an EU funded project to study publication bias in medical research. It brings together research groups from Germany, Italy, France, Croatia, Spain, and the UK, and it aimed to assess the extent of the problem and come up with recommendations for the European Commission.

The quality of clinical trials has improved over the years, but publication bias has emerged as a major obstacle for a fair assessment of drugs and devices in medicine. When only trials with welcome results are published, we are still left with a false impression as to the true effect of an intervention.  The pharmaceutical industry has faced massive criticism here, but it is also present in academia.

The OPEN project is coordinated by the German Cochrane Centre in Freiburg and it hosted this final workshop summarising the findings and we spent one day finalising recommendations. This is a difficult problem to study: how do you analyse something that is not there, i.e. non-publication. Most research by the OPEN group was through surveys, and it is safe to assume that the problem is vastly underreported. But even then, the findings are quite sobering:

One third of surveyed medical researchers have unpublished data—20% cite methodological problems for non-publication.

Less than half of surveyed European ethical committees ask for trial registration or check that approved studies are published.

When using trial registries, ethics approval, and congress abstracts as denominator, less than half of medical research is published.

More than half of European research funding bodies say they encourage publication of funded research, but two thirds do not bother to check this.

Almost half of surveyed pharmaceutical companies reported having work rejected by journals for the sole reason of being industry funded.

More than two thirds of surveyed medical journals does not require trial registration, as they see it as a “competitive disadvantage.”

Regulatory bodies in most European countries do not take publication bias into account when assessing new drugs or devices.

Recommendations will be summarised in a final document, but will certainly include obligatory trial registration before ethics approval and legal requirements to disseminate results from clinical studies.

Wim Weber is European research editor of the BMJ. 

  • Tim Heywood

    There is a world of difference between non-publication for methodological problems and non-publication by author’s (or funding body’s) choice. It is a great shame that so much research effort goes unpublished because of methodological problems, but we must accept that that data is probably not to be trusted. I would propose a model in which journals’ editorial and review teams could have input to a trial in the design stage, stating methodological flaws before the data is collected, so that corrections could be made, and scarce research resources not wasted by flaws only becoming evident after the work is completed. In return, the journal involved would guarantee to publish the results, regardless of positivity, either in print or online, providing that the researchers guarantee to include all the data from the trial in question, as well as any data from other trials which are registered as investigating the same intervention. This rigour would require some considerable degree of input, but would surely be reflected in the journal’s impact factor. They would also have a greater ability to predict the supply of publishable data of high methodological quality, meaning that the temptation to publish lower standard work to fill column inches would be reduced. It is possible that a certain length of time of this system would mean that the competitive advantage would swing entirely the other way, as journals not engaging would become less well regarded.