Only full transparency and rigorous scrutiny of the data will allow for informed decision making, argues Peter Doshi

In the United States, all eyes are on Pfizer and Moderna. The topline efficacy results from their experimental covid-19 vaccine trials are astounding at first glance. Pfizer says it recorded 170 covid-19 cases (in 44,000 volunteers), with a remarkable split: 162 in the placebo group versus 8 in the vaccine group. Meanwhile Moderna says 95 of 30,000 volunteers in its ongoing trial got covid-19: 90 on placebo versus 5 receiving the vaccine, leading both companies to claim around 95% efficacy.

Let’s put this in perspective. First, a relative risk reduction is being reported, not absolute risk reduction, which appears to be less than 1%. Second, these results refer to the trials’ primary endpoint of covid-19 of essentially any severity, and importantly not the vaccine’s ability to save lives, nor the ability to prevent infection, nor the efficacy in important subgroups (e.g. frail elderly). Those still remain unknown. Third, these results reflect a time point relatively soon after vaccination, and we know nothing about vaccine performance at 3, 6, or 12 months, so cannot compare these efficacy numbers against other vaccines like influenza vaccines (which are judged over a season). Fourth, children, adolescents, and immunocompromised individuals were largely excluded from the trials, so we still lack any data on these important populations.

I previously argued that the trials are studying the wrong endpoint, and for an urgent need to correct course and study more important endpoints like prevention of severe disease and transmission in high risk people. Yet, despite the existence of regulatory mechanisms for ensuring vaccine access while keeping the authorization bar high (which would allow placebo-controlled trials to continue long enough to answer the important question), it’s hard to avoid the impression that sponsors are claiming victory and wrapping up their trials (Pfizer has already sent trial participants a letter discussing “crossing over” from placebo to vaccine), and the FDA will now be under enormous pressure to rapidly authorize the vaccines.

But as conversation shifts to vaccine distribution, let’s not lose sight of the evidence. Independent scrutiny of the underlying trial data will increase trust and credibility of the results. There also might be important limitations to the trial findings we need to be aware of.

Most crucially, we need data-driven assurances that the studies were not inadvertently unblinded, by which I mean investigators or volunteers could make reasonable guesses as to which group they were in. Blinding is most important when measuring subjective endpoints like symptomatic covid-19, and differences in post-injection side-effects between vaccine and placebo might have allowed for educated guessing. Past placebo-controlled trials of influenza vaccine were not able to fully maintain blinding of vaccine status, and the recent “half dose” mishap in the Oxford covid-19 vaccine trial was apparently only noticed because of milder-than-expected side-effects. (And that is just one of many concerns with the Oxford trial.)

In contrast to a normal saline placebo, early phase trials suggested that systemic and local adverse events are common in those receiving vaccine. In one Pfizer trial, for example, more than half of the vaccinated participants experienced headache, muscle pain and chills—but the early phase trials were small, with large margins of error around the data. Few details from the large phase 3 studies have been released thus far. Moderna’s press release states that 9% experienced grade 3 myalgia and 10% grade 3 fatigue; Pfizer’s statement reported 3.8% experienced grade 3 fatigue and 2% grade 3 headache. Grade 3 adverse events are considered severe, defined as preventing daily activity. Mild and moderate severity reactions are bound to be far more common.

One way the trial’s raw data could facilitate an informed judgment as to whether any potential unblinding might have affected the results is by analyzing how often people with symptoms of covid-19 were referred for confirmatory SARS-CoV-2 testing. Without a referral for testing, a suspected covid-19 case could not become a confirmed covid-19 case, and thus is a crucial step in order to be counted as a primary event: lab-confirmed, symptomatic covid-19. Because some of the adverse reactions to the vaccine are themselves also symptoms of covid-19 (e.g. fever, muscle pain), one might expect a far larger proportion of people receiving vaccine to have been swabbed and tested for SARS-CoV-2 than those receiving placebo.

This assumes all people with symptoms would be tested, as one might expect would be the case. However the trial protocols for Moderna and Pfizer’s studies contain explicit language instructing investigators to use their clinical judgment to decide whether to refer people for testing. Moderna puts it this way:

“It is important to note that some of the symptoms of COVID-19 overlap with solicited systemic ARs that are expected after vaccination with mRNA-1273 (eg, myalgia, headache, fever, and chills). During the first 7 days after vaccination, when these solicited ARs are common, Investigators should use their clinical judgement to decide if an NP swab should be collected.”

This amounts to asking investigators to make guesses as to which intervention group patients were in. But when the disease and the vaccine side-effects overlap, how is a clinician to judge the cause without a test? And why were they asked, anyway?

Importantly, the instructions only refer to the first seven days following vaccination, leaving unclear what role clinician judgment could play in the key days afterward, when cases of covid-19 could begin counting towards the primary endpoint. (For Pfizer, 7 days after the 2nd dose. For Moderna, 14 days.)

In a proper trial, all cases of covid-19 should have been recorded, no matter which arm of the trial the case occurred in. (In epidemiology terms, there should be no ascertainment bias, or differential measurement error). It’s even become common sense in the Covid era: “test, test, test.” But if referrals for testing were not provided to all individuals with symptoms of covid-19—for example because an assumption was made that the symptoms were due to side-effects of the vaccine—cases could go uncounted.

Data on pain and fever reducing medicines also deserve scrutiny. Symptoms resulting from a SARS-CoV-2 infection (e.g. fever or body aches) can be suppressed by pain and fever reducing medicines. If people in the vaccine arm took such medicines prophylactically, more often, or for a longer duration of time than those in the placebo arm, this could have led to greater suppression of covid-19 symptoms following SARS-CoV-2 infection in the vaccine arm, translating into a reduced likelihood of being suspected for covid-19, reduced likelihood of testing, and therefore reduced likelihood of meeting the primary endpoint. But in such a scenario, the effect was driven by the medicines, not the vaccine.

Neither Moderna nor Pfizer have released any samples of written materials provided to patients, so it is unclear what, if any, instructions patients were given regarding the use of medicines to treat side effects following vaccination, but the informed consent form for Johnson and Johnson’s vaccine trial provides such a recommendation:

“Following administration of Ad26.COV2.S, fever, muscle aches and headache appear to be more common in younger adults and can be severe. For this reason, we recommend you take a fever reducer or pain reliever if symptoms appear after receiving the vaccination, or upon your study doctor’s recommendation.”

There may be much more complexity to the “95% effective” announcement than meets the eye—or perhaps not. Only full transparency and rigorous scrutiny of the data will allow for informed decision making. The data must be made public.

Spanish translation of this article

German translation of this article

Peter Doshi, associate editor, The BMJ.

Competing interests: I have been pursuing the public release of vaccine trial protocols, and have co-signed open letters calling for independence and transparency in covid-19 vaccine related decision making.