In September 2015, under the banner “Trusted evidence, Informed decisions, Better Health,” a Cochrane Collaboration press release proclaimed to the public that “preoperative statin therapy reduces the odds of postoperative atrial fibrillation and shortens the patient’s stay on the ICU and in hospital.” No adverse effects, no caveats, no calls for bigger and better trials, just a strong recommendation for practice. The relevant systematic review included 17 trials with a total of 2138 participants and reported a halving of the odds of post-operative atrial fibrillation with statin treatment (OR=0.54 95%CI 0.43 to 0.67; p<0.01). [1] A little over six months later, this claim was refuted by a randomized trial (the STICS trial) that included more outcome events (cases of atrial fibrillation) than all the previous trials combined. [2] It found no reduction in atrial fibrillation (OR=1.04; 95%CI 0.84 to 1.30; p=0.72) and no reductions in hospital stay. However, acute kidney injury was more common in statin treated patients. Although the new trial cannot rule out a smaller treatment effect, it does rule out reductions in atrial fibrillation considerably smaller (15%-20%) than those found in the Cochrane review.

In this case the “trusted evidence” was not as trustworthy as advertised. The Cochrane conclusion appears to be mistaken about benefit and possibly mistaken about harm. The discrepancy, which must be considered in the context of the available methodological research on the unreliability of meta-analyses of small trials, calls into question the value for money provided by systematic reviews of small trials. [3,4] Chalmers and Glasziou disagree and argue that “funders and regulators cannot be expected to support and endorse large studies without some reassurance from the results of smaller existing studies that the substantial investment needed is justified.” [5] This argument does not stand scrutiny. Meta-analyses of small trials do not provide such reassurance because their results are often misleading. The STICS trial investigators were appropriately skeptical of the large treatment benefits suggested by the review of small trials and conducted a trial with sufficient power to exclude a more plausible effect. Whilst new trial research should be preceded by a systematic review of the existing evidence, the main role of reviews in setting the research agenda is to establish whether or not any adequately powered high quality clinical trials have been conducted.

For an organization making strong claims about the validity of its conclusions the apparent refutation of the Cochrane conclusions should be considered a “never event” and further investigation into the reliability (or otherwise) of the available trials may be informative. Cochrane might then issue a press release that clarifies for the public the potential inconsistency.

References:

1. Kuhn EW, Slottosch I, Wahlers T, Liakopoulos OJ. Preoperative statin therapy for patients undergoing cardiac surgery. Cochrane Database Syst Rev 2015;8:CD008493-CD008493

2. Zheng Z, Jayaram R, Jiang L, Emberson J, Zhao Y, Li Q, Du J, Guarguagli S, Hill M, Chen Z, Collins R, Casadei B. Perioperative Rosuvastatin in Cardiac Surgery. N Engl J Med 2016; 374:1744-1753

3. Roberts, I and Ker, K. How systematic reviews cause research waste. Lancet. 2015; 386: 1536

4. Dechartres, A, Trinquart, L, Boutron, I, and Ravaud, P. Influence of trial sample size on treatment effect estimates: meta-epidemiological study. BMJ. 2013; 346: f2304

5. Chalmers I, Glasziou P. Systematic reviews and research waste. Lancet 2016;387:122-3

Ian Roberts is Professor of Epidemiology and Public Health at the London School of Hygiene & Tropical Medicine and Director of the Clinical Trials Unit.

Conflicts of interest: Ian Roberts is a coordinating editor of the Cochrane Injuries Group.