What level of evidence should we have before costly, toxic cancer drugs are given to patients?

I am a haematologist oncologist who treats patients with cancer, studies the medical evidence, and frequently lectures oncologists in training as assistant programme director of the haematology oncology fellowship programme at Oregon Health and Science University. As part of that last role, I supervise trainees as they care for patients with cancer. Not a week goes by without me hearing some variation of, “The National Comprehensive Cancer Network (NCCN) recommends . . . ” followed by what my fellow wants to do for a patient. The fellows know me well enough to know my response, “The NCCN recommends that, yes, but on what basis? What’s the evidence?”

It isn’t just my fellows who turn to guidance from the NCCN. The NCCN compendia recommendations determine coverage decisions by the US Centers for Medicare and Medicaid Services (CMS), and many commercial insurers. After all, how are they going to decide which drugs to pay for (and for what indications)? You may think payers should limit reimbursement only to drugs that are approved by the US Food and Drug Administration (FDA), but that approach would result in lack of payment for many cancer drugs and combinations of indisputable benefit. Some older cancer drugs are widely used and based on credible studies, but there is little financial incentive to seek formal FDA approval, so they don’t have it. Hence, the need for some other mechanism for coverage; hence, the compendia.

But, in recent years, things have changed. Newer cancer drugs come to market routinely priced in excess of $100 000 per year of treatment. New drugs have industry sponsors with the resources and ability to seek formal FDA approval. At the same time, these sponsors also have the compendia. Instead of going through the lengthy, costly process needed to obtain formal FDA approval they can instead conduct studies and gain coverage of their drugs through recommendations in places like the NCCN guidelines.

In early 2016, my colleagues and I set out to investigate the evidence base behind NCCN recommendations. We did something simple. We picked a set of contemporary cancer drugs, and on a date in March 2016, we “locked” the entire NCCN guidelines and FDA drug labels. Locked is a fancy word for downloaded the PDFs. Then, in the months that followed, we met and painstakingly compared these two documents.

We wanted to know how many FDA approved uses were included in the NCCN guidance. The answer was all of them. All 69 approved indications for 47 drugs. We also wanted to know how many additional recommendations the NCCN made for these drugs that went beyond the guidance given by the FDA in approving the drugs. We found 44 additional recommendations. Of the total number of NCCN recommendations, 39% (44/113) were additional recommendations or extrapolations.

We then searched the NCCN document for the cited level of evidence supporting these additional recommendations. We also classified the type of additional recommendations.

Only 23% were based on randomised data, but a plurality of the additional recommendations was based on no cited data (36%) or lower forms of evidence. Removing requirements (i.e. inclusion criteria) stipulated by the FDA, such as the need for prior therapy, or other inclusion criteria, were common types of additional recommendations. Expanding use to a different cancer, permitting novel combinations, or removing the requirement for concomitant medication were other extrapolations. As a general rule, additional recommendations pushed the use of a drug beyond the studies supporting FDA approval.

During the tremendous peer review process at The BMJ, we went back through these additional recommendations. We asked, 21 months after the original analysis, how many of the additional recommendations received a new FDA approval. How many times did the FDA later support the NCCN—the answer was just six of 44 (14%). Together, these findings made one question the strength of the NCCN additional recommendations.

Before, I hit the takeaway message, I want to thank all of the authors who worked tirelessly on this project: Jeffrey Wagner, John Marquart, Julia Ruby, Austin Lammers, Sham Mailankody, and Victoria Kaestner. Collectively, they invested a thousand hours of their time studying these documents—until our eyes went blurry. I want to thank John and Jeff for working through the revisions with me. Their dedication was inspiring, and had it not been for their help you would not be reading this.

Now, the takeaway: We have a system in the US that needs an audit. Is it really the best idea that a guidelines body with high rates of financial conflict of interest—studies put rates of personal financial ties between NCCN writers and the industry at around 80%—dictates when CMS and commercial insurers should pay for drugs? Is the impetus for the compendia—older drugs with robust evidence that lack FDA approval and are not about to get it any time soon—still relevant in 2018, when new cancer drugs generate large revenue and companies have the resources and experience of seeking FDA approval? Does having a system like compendia coverage encourage manufacturers to conduct unrepresentative trials, knowing that bodies like the NCCN will later expand use of their medications? And, most importantly, what is the level of evidence we should have before costly, toxic cancer drugs are given to patients with cancer?

We don’t have the answers to these questions, but our study suggests that the NCCN and compendia coverage warrant closer examination.

Vinay Prasad is an assistant professor of medicine in the Division of Hematology Oncology in the Knight Cancer Institute, Department of Public Health and Preventive Medicine, Oregon Health and Sciences University, Portland, Oregon. He is also a senior scholar in the university’s Center for Health Care Ethics.

Competing interests: Vinay Prasad wrote Ending Medical Reversal in 2015 and receives royalties from that. One of his research projects is funded by the Laura and John Arnold Foundation.