The way in which Cochrane produces its reviews is a boon to pharma, says Tom Jefferson

We do not know how the pharmaceutical industry viewed the birth of the Cochrane Collaboration but we can be fairly sure that today, if industry is watching, it is surely smiling. This is a consequence of the evolution of Cochrane and its output over the past two decades.

Since the start of Cochrane, the production of its reviews has been focused on randomised controlled trials (RCTs). This is true of most reviews in medicine, which use RCTs as their basic evidence blocks. It’s only been in the past decade (since ClinicalTrials.gov began requiring investigators to report trial results under the Food and Drug Administration Amendments Act of 2007) that the emphasis has shifted, partly, to include registers as a source of data.

During that time, we’ve also seen the publication of a series of reviews that identified the presence of distortions and gaps in the evidence base provided by pharmaceutical trials published in journals. These distortions were later grouped under the name of reporting bias, including its most notorious distortion: publication bias.

Cochrane continues to produce reviews of pharmaceutical interventions based in part on journal articles that we know are likely to be unreliable, but which are still synthesised and re-disseminated under the prestigious Cochrane name.

Although Cochrane reviewers, in good faith, appraise trial publications, it is impossible to identify subtle distortions, discrepancies, and missing information when reading a 10 page journal article that represents the synthesis of tens of thousands of pages of underlying data. The amount of detail on the intervention and its comparators that is available in clinical study reports, and which allows a good understanding of an intervention’s properties, is simply not on offer in these publications. In addition, the selection of what to publish is not made on the basis of public health principles but of what sells drugs and biologics. This is why industry runs clinical trials.

The result is that Cochrane reviews are in effect likely to be publicising and amplifying the effects of bias. This risk could be minimised by Cochrane reviewers accessing alternative sources of evidence, such as regulatory data. Yet even though this realisation is now almost a decade old, Cochrane has never pushed for its reviews to do this. A survey of Cochrane authors published earlier this year found that less than 15% of them had incorporated regulatory data into their reviews

As Cochrane evolved and time passed, one of the consequences of its focus on RCTs has been the creation of a panoply of reporting guidelines, checklists, and methodological appraisal tools for RCTs, later extended to observational and other types of studies. Perhaps the most famous of these are CONSORT and Cochrane’s own risk of bias tool. These are mainly authored by Cochrane researchers acting within Cochrane or inspired by its ethos and are based on journal publications. There were initially very good reasons to try (for example) to standardise the content of journal articles that were reporting trials, as any reviewer of the pre-CONSORT era knows.

However, the other side of the coin has largely avoided scrutiny. By providing a roadmap of what should be included in a journal submission and how it will be appraised we have furnished a great facilitator for writers trying to get their product (the submission) accepted for publication. “Tell me what you are looking for and I will make sure it’s there.”

Given the space constraints of journals, the lack of awareness of the degree of detail necessary to understand and appraise an endeavour as complex as a RCT, and the lack of development of the peer review process, a neatly presented article can omit some of the most important details. It can also under-report, or not report at all, the harms of interventions and still be published by a major journal. The irony here is that those very researchers who are intent on identifying or minimising the effects of bias over the years have provided tailor made solutions to introduce undetected bias.

Another problem is the lag time between a drug coming to market and the publication of a Cochrane review. This usually appears years after the introduction of the drug, making its relevance to decision making debatable. Perhaps it is for this reason that the list of major retroactive policy changes on the basis of the results of a Cochrane review is thin. Even if Cochrane publishes a review that is critical of an intervention, these may come too late to change decision making.

Last but not least, recent events have shown that Cochrane is willing to expel from its ranks a vocal industry critic despite the predictable and sustained backlash. The events have shown how the organisation promotes a false sense of plurality when in fact the information released from Cochrane is now tightly controlled. This last is a constant characteristic of how industry operates.

At present Cochrane is performing useful functions for industry by creating a roadmap for the publication of commercial trials and then magnifying the effects of bias by including those trials too readily in its reviews.

Cochrane’s failure to officially and actively push for open access to regulatory data and their inclusion in its reviews, or to address in a coherent, decisive way the under-reporting of harms, align it with the aims of industry, either by design or accident.

Tom Jefferson, honorary research fellow, Centre for Evidence Based Medicine, Oxford.

Disclosure: TJ was a recipient of a UK National Institute for Health Research grant for a Cochrane review of neuraminidase inhibitors for influenza. In addition, TJ receives royalties from his books published by Il Pensiero Scientifico Editore, Rome and Blackwells. TJ is occasionally interviewed by market research companies about phase I or II pharmaceutical products. In 2011-13, TJ acted as an expert witness in litigation related to the antiviral oseltamivir, in two litigation cases on potential vaccine related damage (including the vaccine Pandemrix (2015-2017) and in a labour case on influenza vaccines in healthcare workers in Canada. He has acted as a consultant for Roche (1997-99), GSK (2001-2), Sanofi-Synthelabo (2003), and IMS Health (2013). In 2014 he was retained as a scientific adviser to a legal team acting on oseltamivir. TJ has a potential financial conflict of interest in the drug oseltamivir. In 2014-16, TJ was a member of three advisory boards for Boerhinger Ingelheim. TJ was holder of a Cochrane Methods Innovations Fund grant to develop guidance on the use of regulatory data in Cochrane reviews. TJ was a member of an independent data monitoring committee for a Sanofi Pasteur clinical trial on an influenza vaccine. Between 1994 and 2013, TJ was the coordinator of the Cochrane Vaccines Field. TJ was a co-signatory of the Nordic Cochrane Centre Complaint to the European Medicines Agency (EMA) over maladministration at the EMA in relation to the investigation of alleged harms of HPV vaccines and consequent complaints to the European Ombudsman. TJ is co-holder of a John and Laura Arnold Foundation grant for development of a RIAT support centre (2017-2020) and Jean Monnet Network Grant, 2017-2020 for The Jean Monnet Health Law and Policy Network. TJ is an unpaid collaborator to the project Beyond Transparency in Pharmaceutical Research and Regulation led by Dalhousie University and funded by the Canadian Institutes of Health Research (2018-2022).