Peter Doshi: Pfizer and Moderna vaccines “95% effective” - we need more details and raw data.
Five weeks ago, when I raised questions about the results of Pfizer and Moderna's Covid-19 vaccine tests, all that was in the public domain was the study protocols and some press releases. Today, two papers and approximately 400 pages of summary data are available in the form of multiple reports submitted by and to the FDA prior to the agency's release of each company's mRNA vaccine for emergency authorization. While some of the additional details are reassuring, others are not. Here I outline new concerns about the reliability and significance of the reported efficacy results.
Attention was totally focused on the excellent efficacy results: Pfizer reported 170 cases of Covid-19, divided into 8 in the group of vaccinated subjects and 162 among those assigned to placebo. But these numbers were reduced by a category called “Covid-19 suspects” - that is, those with Covid-19 symptoms not confirmed by the PCR test. According to Pfizer's FDA vaccine report, there were 3410 total cases of 'suspected but unconfirmed Covid-19' in the overall study population and of these 1594 occurred in the vaccine group versus 1816 in the placebo group.
With 20 times more suspected cases than confirmed cases, this category of sufferers cannot be ignored simply because there was no positive PCR test result.
Indeed, this makes understanding even more urgent. A rough estimate of the vaccine's effectiveness against developing Covid-19 symptoms, with or without a positive PCR test result, would represent a 19% relative risk reduction (see footnote) - far below the 50% effectiveness threshold required by regulatory authorities for authorization.
Even after removing cases that occurred within seven days of vaccination (409 on Pfizer vaccine vs 287 on placebo), which should include most of the symptoms due to short-term vaccine reactogenicity, the efficacy of the vaccine remains low and reaches 29%. (see footnotes).
If many or most of these suspected cases had occurred in people who had a false positive result on the PCR test, this would imply a dramatic reduction in the effectiveness of the vaccine. But considering that flu-like illnesses have always had a variety of causes - rhinoviruses, influenza viruses, other coronaviruses, adenoviruses, respiratory syncytial virus, etc. - some or many of the suspected covid-19 cases could be attributable to different causative agents. .
But why should etiology be considered? If "suspected covid-19" had essentially the same clinical course as confirmed covid-19 cases, then "suspected plus confirmed" could be a more significant clinical endpoint than confirmed Covid-19 cases alone.
However, if the confirmed cases were on average more serious than the suspected cases, we would still have to consider that in the end it is not the average clinical severity that is relevant, but the incidence of hospitalizations.
With 20 times more suspected cases than confirmed, and with trials not designed to assess whether vaccines can stop viral transmission, an analysis of disease severity regardless of causative agent - i.e., hospitalization rates, ICU cases and deaths among study participants - seems the least to be done, and is the only way to assess the true ability of vaccines to contain the pandemic.
Clearly data is needed to answer these questions, but Pfizer's 92-page report did not mention the 3410 cases of suspected Covid-19. Not even its publication on the New England Journal of Medicine. Nor any of the reports on the Moderna vaccine. The only source that appears to have reported this is the FDA review of the Pfizer vaccine.
The 371 individuals excluded from the Pfizer vaccine efficacy analysis.
Another reason we need more data is to analyze an inexplicable detail identified in a table in the FDA review of the Pfizer vaccine: 371 individuals were excluded from the efficacy analysis for "major deviations from the protocol within or before 7 days after the second dose ". What is worrying is the imbalance between the randomized groups in the number of these individuals excluded: 311 from the vaccine group versus 60 from the placebo. (In contrast, in the Moderna trials only 36 participants were excluded from the efficacy analysis for "important protocol deviations" - 12 vaccine group vs 24 placebo group.)
What were these protocol deviations in Pfizer's study and why were five times as many vaccine group participants excluded? The FDA report doesn't say so, and these exclusions are hard to spot even in Pfizer's report and journal publication.
Pain and fever medications, unmasking and primary event assessment committees.
Last month, I expressed my concern about the potential confounding role of pain relievers and antipyretics to treat symptoms. I have speculated that some drugs may mask symptoms, leading to under-detection of Covid-19 cases, probably more among people who have received the vaccine in an attempt to prevent or treat adverse events. However, it seems that this ability to confuse the results is quite limited: even if the results indicate that these medicines were taken between 3 and 4 times more in the vaccine group than in the placebo group (at least as regards Pfizer vaccine, Moderna did not clearly highlight this in the report), their use was presumably concentrated in the first week after vaccine administration, in order to soothe local adverse events at the injection site and systemic ones.
But the cumulative incidence curves indicate a fairly constant rate of confirmed cases of covid-19, with symptom onset dates extending well beyond a week after administration.
That said, the higher rate of drug use in the vaccinated arm provides more than one cause for concern about unofficial exposure.
Given the reactogenicity of vaccines, it is difficult to imagine that the participants and investigators did not imagine what group they were in.
The primary endpoint in trials is quite subjective, making double-blind an important issue. But, neither the FDA nor the companies appear to have formally probed the reliability of the double-blind procedure, and its effect in the reported results.
We also don't know enough about the processes of the primary event assessment boards that counted the covid-19 cases.
Were they in the dark about the information about the patients' antibody titers and symptoms in the first week after vaccination? What criteria did they use and why, with a primary event consisting of a patient-reported outcome (Covid-19 symptoms) and PCR test result, was this kind of committee even necessary? It is also important to understand who was on these committees. Although Moderna has appointed its four-member evaluation committee - all university-affiliated doctors - Pfizer's protocol states that 3 Pfizer employees have worked there. Yes, Pfizer staff members.
Effectiveness of vaccination in people who have already had Covid?
Individuals with known prior SARS-CoV-2 infection or a previous diagnosis of Covid 19 were excluded from the Moderna and Pfizer studies. Yet, 1125 (3,0%) and 675 (2,2%) participants in the Moderna and Pfizer trials, respectively, were considered positive at baseline.
The safety and efficacy of the vaccine in these participants has not received much attention, but as ever larger portions of the population of many countries may be "post-Covid", these data seem important - and all the more so because the US CDC recommends offer the vaccine “regardless of history of previous symptomatic or asymptomatic SARS-CoV-2 infection”.
This ties in with the agency's conclusions regarding the Pfizer vaccine having a ≥92% efficacy and "no particular safety concern" in people with previous cases of SARS-CoV-2 infection.
According to my calculations, Pfizer reported 8 cases of symptomatic confirmed Covid-19 in people positive for SARS-CoV-2 at the start of the trial (1 in the vaccine group, 7 in the placebo group, according to the differences between Tables 9 and 10) and Moderna, 1 case (placebo group, table 12). But with only 4 to 31 documented reinfections globally, how could there be nine confirmed cases of Covid-19 among those infected with SARS-CoV-2 at the start of the trial, in studies of tens of thousands, with a follow-up median up of two months? Is this representative of significant vaccine efficacy, as the CDC appears to have claimed? Or could it be something else, like preventing Covid-19 symptoms, possibly with the vaccine or the use of drugs that suppress symptoms, and nothing to do with re-infection? "
We need raw data.
Addressing the many open questions about these studies requires access to raw data. But no company seems to have shared the data with third parties so far.
Pfizer says it will make the data available "on request and subject to review". This avoids making the data publicly available, but at least leaves the door open. How open is unclear, as the study protocol says Pfizer will begin making data available 24 months after the studies are completed.
Moderna's statement on data sharing states that “data may be available upon request once the study is completed”. This translates into the second half of 2022, as the follow-up is scheduled for two years.
Things may be no different for the Oxford / AstraZeneca vaccine which promised patient data "when the study is complete." And for the Russian Sputnik V vaccine, ClinicalTrials.gov says there is no planned sharing of participant data.
However, the European Medicines Agency and Health Canada may share data for any vaccine authorized much earlier. EMA has already committed to publishing the data submitted by Pfizer on its website "in due course", as has Health Canada.
peter doshi, associate editor, The BMJ
Conflicts of interest: I have dedicated myself to public release of vaccine testing protocols and co-signed open letters calling for independence and transparency in the decision-making process related to the covid-19 vaccine.
The calculations in this article are as follows: 19% = 1 - (8 + 1594) / (162 + 1816); 29% = 1 - (8 + 1594 - 409) / (162 + 1816 - 287) I ignored the denominators as they are similar in groups.