My earlier blog outlined BMJ reader feedback to the two updated Cochrane reviews on the benefits and harms in influenza of the neuraminidase inhibitors oseltamivir (Tamiflu) and zanamivir (Relenza). But the two research articles also attracted a great deal of attention in the blogosphere.
Jon Brassey of clinical search engine Trip is increasingly worried about the accuracy of systematic reviews. He muses on the Cochrane reviewers’ approach – meta-analysing only the data from the trials’ Clinical Study Reports (CSRs), and discarding the published trials as incompletely reported. “…What does the evidence say about relying on published trials?” he asks. “It is [now] evident that systematic reviews [most of which analyse only published studies] cannot be relied upon for an accurate assessment of an average effect size for an intervention…all I can conclude is that a systematic review gives a ball-park figure for the actual effect size…We use systematic reviews as a heuristic for accuracy. I have used myself and heard many others say something along the lines of ‘There’s a recent Cochrane systematic review, so no need to look any further’… [But] can we really say any more than ‘This is likely to effective and the effect size is likely to be in the range of…’? …Systematic reviews producers probably don’t say that they produce accurate results, it’s assumed. But, they probably don’t do enough to highlight the shortcomings either. Accuracy sells, ballpark doesn’t.”
Sociologist Robert Dingwall pulls no punches in his much-tweeted blog entitled Tamiflu and the Ethics of the British Medical Journal. He also discusses his competing interests up front as “a member of ethical advisory panels to both the UK Department of Health (2006 onwards) and Roche Pharmaceuticals (2007-10), the manufacturers of Tamiflu”. Dingwall says “The BMJ is talking nonsense when it suggests that national governments were victims of a confidence trick by a wicked drug company. The policy community fully understood the limitations of the evidence and the likelihood that the benefits would be moderate. No insider ever thought it was a wonder drug.”
But he reckons that governments now being damned for stockpiling Tamiflu could well have been damned for not doing so: their job, after all, is to keep countries functioning. And if the clinical trial data were so weak, why didn’t the Cochrane reviewers simply say that and save themselves the bother of a lot of complicated statistical analysis? Dingwall asserts that “This story, however, tells us less about Tamiflu than about the circulation wars between leading medical and scientific journals, about the credulity of some science and medical journalists, about the exaggerated public expectations of modern pharmaceuticals, and about the Talibanization of the Cochrane Collaboration. If public and professional confidence in a moderately helpful drug has been damaged, it would not be a surprise if there are deaths to be laid at the doors of the BMJ and the Cochrane Collaboration.”
Canadian pharmacist and blogger Scott Gavura, in a long and detailed critique, agrees that, despite the weak trial evidence and the very uncertain cost-benefit equation, stockpiling was probably inevitable because governments, public health agencies, and the public were so fearful of a major pandemic. He’s unimpressed by the updated reviews on oseltamivir and zanamivir and makes it personal, attacking the track record of two of the Cochrane reviewers. “Tom Jefferson is a vocal critic of the influenza vaccine and his strict approach to data been criticized as ‘methodolatry’ by some epidemiologists,” he says, and “Peter Doshi.. is so stridently anti-vaccine he makes Tom Jefferson look pro-vaccine. Doshi believes that influenza itself is an example of “selling sickness,” and questions the morbidity and mortality estimates of the disease.”
Another long blog post that did the rounds via Twitter is by anonymous microbiologist Rectofossa. This covers much of the same ground as the two blogs above, but in a much more gung-ho style with phrases such as “Predicting history is easy,” and “Quite simply, these people [the Cochrane reviewers and The BMJ] all have a dog in the fight.” And the writer doesn’t much like systematic reviews: ” these sorts of analyses can be of use when you understand how they work and what the limitations are.
A big limitation is that unless you also take into account basic plausibility – which the Cochrane Collaboration doesn’t – you get Tooth Fairy Science. And they are notoriously wishy-washy; were the Cochrane Collaboration to investigate a murder scene they would insist equal weight be given to the idea that the victim backed onto the knife a dozen times themselves.”
I’m sure the Cochrane reviewers would be happy to reply to substantive criticisms and comments made through The BMJ’s rapid responses. In the meantime, they have posted this response ”It took us four years to obtain the full set of reports. The story relating to the acquisition has been documented at The BMJ’s open data campaign. If you disagree with our findings, or if you want to carry out your own analysis or just want to see what around 150,000 pages of data look like, they are one click away.”
Competing interests: I chaired the research manuscript committee meeting where we accepted the updated Cochrane reviews on oseltamivir and zanamivir. Last year I observed and blogged about a key meeting of the Multiparty Group for Advice on Science (MUGAS). On behalf of The BMJ I am an active participant in the AllTrials campaign and the Ottawa Group’s IMPACT study, have contributed to several working groups that led to revision of the EU Clinical Trials Directive, and regularly act as an advocate for and give talks on data sharing and particularly on increasing transparency in the reporting of clinical trials.
Trish Groves is deputy editor and head of research, The BMJ