The FDA should demand adequate, controlled studies with long term follow up, and make data publicly available, before granting full approval to covid-19 vaccines, says Peter Doshi

On 28 July 2021, Pfizer and BioNTech posted updated results for their ongoing phase 3 covid-19 vaccine trial. The preprint came almost a year to the day after the historical trial commenced, and nearly four months since the companies announced vaccine efficacy estimates “up to six months.”

But you won’t find 10 month follow-up data here. While the preprint is new, the results it contains aren’t particularly up to date. In fact, the paper is based on the same data cut-off date (13 March 2021) as the 1 April press release, and its topline efficacy result is identical: 91.3% (95% CI 89.0 to 93.2) vaccine efficacy against symptomatic covid-19 through “up to six months of follow-up.”

The 20 page preprint matters because it represents the most detailed public account of the pivotal trial data Pfizer submitted in pursuit of the world’s first “full approval” of a coronavirus vaccine from the Food and Drug Administration. It deserves careful scrutiny.

The elephant named “waning immunity”

Since late last year, we’ve heard that Pfizer and Moderna’s vaccines are “95% effective” with even greater efficacy against severe disease (“100% effective,” Moderna said).

Whatever one thinks about the “95% effective” claims (my thoughts are here), even the most enthusiastic commentators have acknowledged that measuring vaccine efficacy two months after dosing says little about just how long vaccine-induced immunity will last. “We’re going to be looking very intently at the durability of protection,” Pfizer senior vice president William Gruber, an author on the recent preprint, told the FDA’s advisory committee last December.

The concern, of course, was decreased efficacy over time. “Waning immunity” is a known problem for influenza vaccines, with some studies showing near zero effectiveness after just three months, meaning a vaccine taken early may ultimately provide no protection by the time “flu season” arrives some months later. If vaccine efficacy wanes over time, the crucial question becomes what level of effectiveness will the vaccine provide when a person is actually exposed to the virus? Unlike covid vaccines, influenza vaccine performance has always been judged over a full season, not a couple months.

And so the recent reports from Israel’s Ministry of Health caught my eye. In early July, they reported that efficacy against infection and symptomatic disease “fell to 64%.” By late July it had fallen to 39% where Delta is the dominant strain. This is very low. For context, the FDA’s expectation is of “at least 50%” efficacy for any approvable vaccine.

Now Israel, which almost exclusively used Pfizer vaccine, has begun administering a third “booster” dose to all adults over 40. And starting 20 September 2021, the US plans to follow suit for all “fully vaccinated” adults eight months past their second dose.

Delta may not be responsible

Enter Pfizer’s preprint. As an RCT reporting “up to six months of follow-up,” it is notable that evidence of waning immunity was already visible in the data by the 13 March 2021 data cut-off.  

“From its peak post-dose 2,” the study authors write, “observed VE [vaccine efficacy] declined.” From 96% to 90% (from two months to <4 months), then to 84% (95% CI 75 to 90) “from four months to the data cut-off,” which, by my calculation (see footnote at the end of the piece), was about one month later.

But although this additional information was available to Pfizer in April, it was not published until the end of July.

And it’s hard to imagine how the Delta variant could play a real role here, for 77% of trial participants were from the United States, where Delta was not established until months after data cut-off.

Waning efficacy has the potential to be far more than a minor inconvenience; it can dramatically change the risk-benefit calculus. And whatever its cause—intrinsic properties of the vaccine, the circulation of new variants, some combination of the two, or something else—the bottom line is that vaccines need to be effective.

Until new clinical trials demonstrate that boosters increase efficacy above 50%, without increasing serious adverse events, it is unclear whether the 2-dose series would even meet the FDA’s approval standard at six or nine months.  

The “six month” preprint based on the 7% of trial participants who remained blinded at six months

The final efficacy timepoint reported in Pfizer’s preprint is “from four months to the data cut-off.” The confidence interval here is wider than earlier time points because only half of trial participants (53%) made it to the four month mark, and mean follow-up is around 4.4 months (see footnote).

This all happened because starting last December, Pfizer allowed all trial participants to be formally unblinded, and placebo recipients to get vaccinated. By 13 March 2021 (data cut-off), 93% of trial participants (41,128 of 44,060; Fig 1) were unblinded, officially entering “open-label followup.” (Ditto for Moderna: by mid April, 98% of placebo recipients had been vaccinated.)

Despite the reference to “six month safety and efficacy” in the preprint’s title, the paper only reports on vaccine efficacy “up to six months,” but not from six months. This is not semantics, as it turns out only 7% of trial participants actually reached six months of blinded follow-up (“8% of BNT162b2 recipients and 6% of placebo recipients had ≥6 months follow-up post-dose 2.”) So despite this preprint appearing a year after the trial began, it provides no data on vaccine efficacy past six months, which is the period Israel says vaccine efficacy has dropped to 39%.

It is hard to imagine that the <10% of trial participants who remained blinded at six months (which presumably further dwindled after 13 March 2021) could constitute a reliable or valid sample to produce further findings. And the preprint does not report any demographic comparisons to justify future analyses.

Severe disease

With the US awash in news about rising cases of the Delta variant, including among the “fully vaccinated,” the vaccine’s efficacy profile is in question. But some medical commentators are delivering an upbeat message. Former FDA commissioner Scott Gottlieb, who is on Pfizer’s board, said: “Remember, the original premise behind these vaccines were [sic] that they would substantially reduce the risk of death and severe disease and hospitalization. And that was the data that came out of the initial clinical trials.”

Yet, the trials were not designed to study severe disease. In the data that supported Pfizer’s EUA, the company itself characterized the “severe covid-19” endpoint results as “preliminary evidence.” Hospital admission numbers were not reported, and zero covid-19 deaths occurred.

In the preprint, high efficacy against “severe covid-19” is reported based on all follow-up time (one event in the vaccinated group vs 30 in placebo), but the number of hospital admissions is not reported so we don’t know which, if any, of these patients were ill enough to require hospital treatment. (In Moderna’s trial, data last year showed that 21 of 30 “severe covid-19” cases were not admitted to hospital; Table S14).

And on preventing death from covid-19, there are too few data to draw conclusions—a total of three covid-19 related deaths (one on vaccine, two on placebo). There were 29 total deaths during blinded follow-up (15 in the vaccine arm; 14 in placebo).

The crucial question, however, is whether the waning efficacy seen in the primary endpoint data also applies to the vaccine’s efficacy against severe disease. Unfortunately, Pfizer’s new preprint does not report the results in a way that allows for evaluating this question.

Approval imminent without data transparency, or even an advisory committee meeting?

Last December, with limited data, the FDA granted Pfizer’s vaccine an EUA, enabling access to all Americans who wanted one. It sent a clear message that the FDA could both address the enormous demand for vaccines without compromising on the science. A “full approval” could remain a high bar.

But here we are, with FDA reportedly on the verge of granting a marketing license 13 months into the still ongoing, two year pivotal trial, with no reported data past 13 March 2021, unclear efficacy after six months due to unblinding, evidence of waning protection irrespective of the Delta variant, and limited reporting of safety data. (The preprint reports “decreased appetite, lethargy, asthenia, malaise, night sweats, and hyperhidrosis were new adverse events attributable to BNT162b2 not previously identified in earlier reports,” but provides no data tables showing the frequency of these, or other, adverse events.)

It’s not helping matters that FDA now says it won’t convene its advisory committee to discuss the data ahead of approving Pfizer’s vaccine. (Last August, to address vaccine hesitancy, the agency had “committed to use an advisory committee composed of independent experts to ensure deliberations about authorization or licensure are transparent for the public.”)

Prior to the preprint, my view, along with a group of around 30 clinicians, scientists, and patient advocates, was that there were simply too many open questions about all covid-19 vaccines to support approving any this year. The preprint has, unfortunately, addressed very few of those open questions, and has raised some new ones.

I reiterate our call: “slow down and get the science right—there is no legitimate reason to hurry to grant a license to a coronavirus vaccine.”

FDA should be demanding that the companies complete the two year follow-up, as originally planned (even without a placebo group, much can still be learned about safety). They should demand adequate, controlled studies using patient outcomes in the now substantial population of people who have recovered from covid. And regulators should bolster public trust by helping ensure that everyone can access the underlying data.

Peter Doshi, senior editor, The BMJ.

Competing interests: I helped organize the Coalition Advocating for Adequately Licensed Medicines (CAALM), which has formally petitioned the FDA to refrain from fully approving any covid-19 vaccine this year (docket FDA-2021-P-0786). A full list of competing interests is available here. The views and opinions expressed here are mine and do not necessarily reflect official policy or the position of the University of Maryland.

Provenance: commissioned; externally peer-reviewed.

Footnote: Calculations in this article are as follows. “About 1 month” past month 4 is based on the final row of Fig 2 in the preprint: 1030/12670*12 = 0.98 months (vaccine group) and 895/11802*12 = 0.91 months (placebo group). “53%” is based on Fig 2: (12670+11802)/(23040+23037). “4.4 months” is based on the average of 8412/22505*12 = 4.5 (vaccine) and 8124/22434*12 = 4.3 (placebo) in Fig 2.