In a new research paper published in The BMJ, we examine the evidence base supporting the European Medicines Agency (EMA)’s approval of cancer drugs. Of 32 new cancer drugs approved by the EMA during 2014-2016, only nine had at least one randomised controlled trial that we judged to be at low risk of bias and without major criticism from the EMA’s Committee for Medicinal Products for Human Use (CHMP). Our new study adds to a growing body of research on the regulation of new cancer drugs that has raised questions about their therapeutic benefits and economic value, and the strength and credibility of the evidence supporting their market entry. 

Our paper is concerned with the validity of clinical studies. Since the publication of a landmark paper by Schulz and colleagues in 1995, a large body of research has shown that deficits in the way clinical studies are designed, conducted, analysed, or reported can affect their results. Often, studies with methodological shortcomings produce biased findings, exaggerating the magnitude of benefit associated with the treatment under investigation. 

Such concerns are especially pertinent in cancer. Cancer drugs are now the single largest category of new drug approvals and products in development. This is cause for optimism for patients who may benefit from these developments. However, there is also growing cause for concern. Most cancer drugs enter the market on the basis of their effects on surrogate endpoints, rather than clinical outcomes that matter to patients and their caregivers – overall survival and quality of life. Cancer drugs that appear effective on surrogate endpoints may turn out to have no effect on overall survival (recent examples include bevacizumab for glioblastoma; bevacizumab for metastatic breast cancer; bevacizumab for advanced ovarian cancer; axitinib for advanced renal cell carcinoma; everolimus for metastatic breast cancer; atezolizumab for urothelial cancer). Worse yet, cancer drugs that appear effective on surrogate endpoints may actually have detrimental effects on survival. In the recent BELLINI trial, patients with multiple myeloma who received venetoclax had shorter survival than those who received a control treatment (even though venetoclax appeared more effective than the control on the basis of commonly-used surrogate endpoints like progression-free survival and response rate). 

Complicating this picture further, cancer drug trials are often small and have relatively short follow-up durations. For example, olaratumab did not demonstrate evidence of overall survival benefits among patients with soft tissue sarcoma in the phase-3 ANNOUNCE trial despite indicative data from a smaller phase-2 trial that supported its initial regulatory approval. In a recent paper, Del Paggio and Tannock showed that the results of clinical trials supporting cancer drug approvals are “fragile”: positive results in cancer drug trials can lose significance with a change in designation of very few events. This reality is starkly contrasted with the significant hype associated with new cancer drugs when they are discussed at professional conferences and reported in the media.

Cancer drug trials are complex, and some methodological deficits that may lead to bias may be inevitable. In the recently revised Cochrane risk of bias tool 2.0, the following 5 bias domains are considered: (1) bias arising from the randomisation process; (2) bias due to deviations from intended interventions; (3) bias due to missing outcome data; (4) bias in measurement of the outcome; and (5) bias in selection of the reported result. In our paper, concerns due to missing outcome data and measurement of the outcome were the most common domains leading to high risk of bias judgments.

Even when trials have been labelled as “double-blind,” blinding may be compromised by occurrence of side effect profiles that are specific to one intervention. While compromised blinding does not necessarily lead to bias, it may be particularly problematic if a subjective outcome is assessed by investigators who have deduced the patient’s intervention group assignment. EMA scientists acknowledge that a risk of compromised blinding is inherent to many cancer trials (“real effectiveness of the blinding for cancer drugs can always be questioned”). Yet, regulatory documents and published reports rarely discuss the potential implications of compromised blinding and any measures taken to ensure that patients are treated and evaluated in accordance with the trial protocol. 

Another issue that may be difficult to address in cancer trials is ensuring that outcome data are available for all (or nearly all) patients. We observed that the proportion of patients who withdrew their consent to continue participating in a trial often varied between the experimental and control arms. Whether outcome data were available from patients who withdrew their consent was not consistently discussed. In addition, sensitivity of trial findings to missing outcome data were not routinely tested. Sometimes, trials also censored patients when they changed their assigned treatment. This is inappropriate, and likely to lead to selection bias, when estimating intention-to-treat effects. These issues were not fully disclosed and discussed in regulatory documents and published trial reports. 

Where does this leave patients and clinicians who need to interpret trial findings and make decisions about the relative benefits and harms of alternative treatment strategies? The validity of the available evidence is a pre-requisite for shared decision-making. Currently, it is difficult, if not impossible, for patients and clinicians to gauge the validity of the clinical studies that form the basis of cancer drug approvals. In our experience, key methodological details are scattered across different documents; results of important sensitivity analyses are tucked away in extensive supplementary appendices; and trial protocols and statistical analysis plans are difficult to navigate. 

We recommend devising and testing novel strategies for collating and communicating information about the validity of clinical studies in the future. Several strategies could be considered. First, EMA’s European Public Assessment Reports could incorporate a formal risk of bias assessment for all pivotal studies that support drug approvals. Second, the European Clinical Trials Register could require the submission of a risk of bias assessment alongside completed trial results. Third, trial investigators could complete a risk of bias assessment to accompany the publication of their trial reports in the peer-reviewed literature. Fourth, with financial support from non-conflicted entities, researchers from the academic community could develop a stand-alone website to routinely evaluate and report the risk of bias in clinical studies that support new drug approvals. 

We call for close collaboration among regulators, cancer trialists, academic researchers, and patients. Recently, the EMA has been at the forefront of regulatory transparency and data sharing initiatives. Future efforts aimed at improving the reporting standards of trial validity would greatly benefit from EMA’s input and involvement. Active engagement from patients and clinicians is also essential to ensure that risk of bias information is presented in an intuitive manner that can inform patient choice and contribute to shared decision-making in clinical practice. 

Huseyin Naci — Assistant Professor of Health Policy, Department of Health Policy, London School of Economics and Political Science, London, UK

Courtney Davis— Reader, Department of Global Health and Social Medicine, King’s College London, London, UK

Bishal Gyawali— Assistant Professor of Public Health Sciences, Cancer Research Institute, Queen’s University at Kingston, Kingston, Ontario, Canada

Xotchil Romo-Sandoval— Research Assistant,  Department of Health Policy, London School of Economics and Political Science, London, UK

Christopher Booth— Professor of Oncology,  Cancer Research Institute, Queen’s University at Kingston, Kingston, Ontario, Canada