Editors’ Review of BMJ EBM’s Analysis of Cochrane’s HPV Systematic Review

On 27 July 2018, BMJ EBM published an analysis [1] that outlined problems with Cochrane’s systematic review of human papillomavirus (HPV) vaccine, which we shall refer to as the Cochrane HPV Review, published in May 2018; this included missing eligible trials, reporting bias, and biased trial designs and conflicts of interest. [2,3]

In response to this analysis article, on 9 August the Cochrane Editor in Chief launched an urgent investigation [4] which resulted in a report, published 3 September, which defended the Cochrane HPV Review and asked whether the conclusions of the BMJ EBM article were “justified and proportionate.” [5] Subsequently, we wrote to the authors of the BMJ EBM analysis article and received a response in which they assert that ‘[their] analysis was appropriate and that the Cochrane editors substantially ignored several of [their] criticisms’. [6]

Our review of the dispute follows. In it, we set out the contentious questions arising from the Cochrane Editors’ response [5] to the analysis piece and consider whether the BMJ EBM analysis [1] requires further editorial comments or corrections. Given the potential to introduce error and the complexity of the issues at hand, we have sent a copy of this review to all parties concerned, and have additionally posted it on BMJ EBM Spotlight to seek further public comments and consensus commenting further on the BMJ EBM HPV analysis. [1] Following this consultation (tentatively over a 6-week period), we will move to issue a formal correction and/or clarification, if necessary.

Carl Heneghan, Editor in Chief BMJ EBM

Igho Onakpoya, Deputy Editor BMJ EBM

Questions:

Did the Cochrane HPV Review miss “nearly half” of the eligible trials, as reported by the BMJ EBM article?
Was the BMJ EBM article calculation of the number of missing participants in the 20 “missing trials” correct?
Did the Cochrane HPV Review authors mistakenly use the term placebo to describe active comparators, as reported by the BMJ EBM article?
Did the BMJ EBM article [1] substantially overstate its criticisms?
Were the serious adverse events rates accurately reported by the authors of the BMJ EBM article?
Were all the trials that were included in the Cochrane HPV Review funded by the HPV vaccine manufacturers, as reported by the BMJ EBM article?
Did the Cochrane HPV Review comply with Cochrane’s COI policy?

The articles that inform this report are:

1. Did the Cochrane HPV Review miss “nearly half” of the eligible trials, as reported by the BMJ EBM article?

The authors of the BMJ EBM analysis article said: “The Cochrane HPV Review missed nearly half of the eligible trials.” The Cochrane editors’ dispute this, saying: “The Cochrane Review did not miss “nearly half of the eligible trials”. A small number of studies were missed due to the primary focus on peer-reviewed reports in scientific journals.”

Our analysis: (Google sheet here for the analysis of these articles:)

“The Cochrane HPV Review aimed to include RCTs of HPV vaccines in women who reported efficacy and/or safety outcomes/occurrence of adverse events.
The comparators were the administration of a placebo containing no active product or only the adjuvant of the HPV vaccine, or another non‐HPV vaccine.
The Cochrane HPV Review searched clinicaltrials.gov, www.isrctn.com, and www.cancer.gov/clinicaltrials for unpublished or ongoing trials.
Twenty studies were identified by the BMJ EBM article and one by the Cochrane editors (NCT00834106), giving a total of 21 additional eligible.*
In our view, we consider that one of the 21 studies (MENACWY-TT054-NCT01755689)** is unlikely to be eligible for inclusion, bringing the number of additional eligible trials down to 20.
Of the 20 studies in the BMJ EBM article table, four were already included in the Cochrane HPV review (NCT00929526; NCT00518336; NCT00652938; NCT00578227).
The authors of the BMJ EBM article [6] report additional unpublished data on safety outcomes/occurrence of adverse events for these four studies that were not included in the Cochrane HPV Review. [3]
In their response, [6] the authors of the BMJ EBM article point out that the Cochrane editors updated 20 of the Cochrane authors meta-analyses (20/66, 30%) with additional data *** and added seven meta-analyses of the HPV vaccine Gardasil (9). However, as of 14 September 2018, the Cochrane review itself had not yet included the additional data [3].

The analysis of trial eligibility is made difficult by the trial identifiers in the Cochrane HPV Review. Using the trial registry number as the main identifier could improve trial verification across multiple sources. Analysis of trial eligibility and comparisons are further hindered by discrepancies across multiple sources of trial information. For example:

The trial identified as Co‐vaccination_HAB trial (Ph3, 2v) in Table 1 of the Cochrane HPV Review included 494 participants in the safety analysis. The characteristics of the included studies table states that 813 girls were enrolled and 543 were given the vaccine in the Cervarix + Twinrix group compared with the Twinrix Group.
The trial registry has 814 participants enrolled and 543 who started vaccination and were analysed (534 completed). (NCT00578227)
The trial study (Pedersen et al. 2012) [7] reports that 814 girls were enrolled and 813 received at least one vaccine dose; 543 were vaccinated (272 in the Cervarix + Twinrix group and 271 in the Twinrix group); 534 completed (267 in each group) and 520 (257 and 263) were included according to the protocol for immunogenicity.
In the trial study (Pedersen et al. 2012) ten subjects reported 11 SAEs, but two of us (IO and CH) could not find the Pederson et al. study in Figure 10 of the Cochrane HPV Review (Figure 10 shows the data on harms from peer-reviewed publications only). This is potentially because the analysis compares vaccinated against ‘placebo’, but we could not locate the serious adverse events (SAEs) to verify the data across the review.

Our conclusion: The following corrections and clarifications to the BMJ EBM article are required:

The table of 20 trials (See; Cochrane HPV Review – Table 1) should have been included in the article.
16 additional trials were eligible for inclusion in the Cochrane HPV Review (not 20 as stated in the article).
Additional data from four trials already included in the Cochrane HPV Review (NCT00929526; NCT00518336; NCT00652938; NCT00578227) were eligible for inclusion in the Cochrane HPV Review.

*Eligibility criteria for potential studies define the population to which the inclusion and exclusion criteria should be applied, and predefined, unambiguous eligibility criteria are therefore a fundamental prerequisite for a systematic review. (see Methodological Expectations of Cochrane Intervention Reviews).

**MENACWY-TT-054 is a five-arm trial in which, during a one-month window, exposure to Cervarix was directly compared with another vaccine (Nimenrix), but at study completion, all arms may have received Cervarix.

*** The additional data were from five eligible completed studies with available data representing 5267 women, which according to the Cochrane editors’ investigation may have been missed from the Cochrane Review. [5]

2. Was the BMJ EBM article calculation of the number of missing participants in the 20 “missing trials” correct?

The authors of the BMJ EBM article stated: “The number of randomised participants could be assessed for 42 of the 46 trials and was 121 704. With nearly half of the trials and half of the participants missing, the Cochrane authors’ conclusion, ‘that the risk of reporting bias may be small’, was inappropriate.” [1]

The Cochrane editors’ report [5] disputed this figure of missing participants, and report: ‘The Cochrane Review included 26 trials (73,428 participants) that matched the predetermined study criteria. As a result of our investigations, we believe that five eligible completed studies with available data representing 5,267 women may have been missed from the Cochrane Review.’

In their response, the authors of the BMJ EBM article [6] say: “For numbers of participants, we did not subtract the male participants that were included in three of the studies, as we should have done.”

The BMJ EBM article authors [6] further state ‘Our analysis reinforces the view that Cochrane HPV review is incomplete. We found an additional 25,550 females (and possibly up to 30,195 for the Cochrane HPV review’s serious adverse events meta-analyses) that are eligible for the Cochrane HPV review’s meta-analyses. ‘

Our conclusion:

The following correction is required: The authors of the BMJ EBM article [1] should have said: “The number of randomised participants could be assessed for 42 of the 46 trials, and we found an additional 25,550 females (and possibly up to 30,195 for the Cochrane HPV Review’s serious adverse events meta-analyses) who are eligible for the Cochrane HPV Review’s meta-analyses.’ With nearly half of the trials and possibly half of the participants missing, the Cochrane authors’ conclusion, ‘that the risk of reporting bias may be small’, was inappropriate.”

3. Did the Cochrane HPV Review authors mistakenly use the term placebo to describe active comparators, as reported by BMJ EBM article?

The BMJ EBM Analysis article [1] states: “The Cochrane authors mistakenly used the term placebo to describe the active comparators.” The Cochrane editors dispute this, saying that “trials comparators were transparently and accurately described.”

We found that the Cochrane HPV Review [3] did indeed define placebo (e.g. in the “Selection criteria” section of the Abstract), but it also described the comparison intervention using the single unqualified word “placebo”, e.g. in the Summary of Findings table (PDF p.9).

Our conclusion: We consider this to be a matter of opinion, and that, therefore, there is no requirement for a correction or clarification.

4. Did the BMJ EBM article [1] substantially overstate its criticisms?

The BMJ EBM article is titled: “The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias”, and it concludes: “We do not find the Cochrane HPV vaccine review to be ‘Trusted evidence’, as it was influenced by reporting bias and biased trial designs. We believe that the Cochrane review does not meet the standards for Cochrane reviews or the needs of the citizens or healthcare providers that consult Cochrane reviews to make ‘Informed decisions’, which also is part of Cochrane’s motto. We recommend that authors of Cochrane reviews make every effort to identify all trials and their limitations and conduct reviews accordingly.”

The Cochrane editors say that the authors of the BMJ EBM article substantially overstate their case. “In our judgement, the criticisms were overstated. For example, the potentially missing studies do not seem to represent anywhere close to “half of the eligible studies”.”

This is complex. If the BMJ EBM article’s title and conclusion were based only or primarily on the number of missing trials, the Cochrane editors might be considered to have a reasonable case.

However, the BMJ EBM article discusses several other issues that are not substantively addressed in the Cochrane editors’ response and which do not in our view require correction, such as incomplete reporting of serious adverse events in several of the included studies; considerations of the trials’ adjuvant and vaccine comparators; assessment of the Cochrane authors’ conflicts of interest; and the use of surrogate outcomes.

Our conclusion: The BMJ EBM article title and conclusions are justified and do not require correction or clarification.

5. Were the serious adverse events rates accurately reported by the authors of the BMJ EBM article?

The authors of the BMJ EBM article state: [1] ‘As an example, the PATRICIA trial publication only included two thirds (1400/2028) of the serious adverse events listed on ClinicalTrials.gov. The Cochrane authors included 701 vs 699 serious adverse events (1400) from the PATRICIA trial publication (see the Cochrane review’s ‘Figure 10, Analysis 7.6.2’) and 835 vs 829 serious adverse events from its ClinicalTrials.gov entry (see ‘Comparison 7, Analysis 6: 7.6.2’; both analyses were called ‘7.6.2’). We found 1046 vs 982 serious adverse events (2028) when we summarised the data from ClinicalTrials.gov (see ‘Results: Serious Adverse Events’).’

The Cochrane editors report that ‘the figures accurately match the number of women experiencing one or more serious adverse events; and a PLoS blog by Hilda Bastian [10] reports that ‘The Copenhagen critique [Jørgensen et al.] is talking about numbers of SAEs. But the Cochrane review, for both its comparisons, as well as the PATRICIA publication, used the women with reported SAEs as the unit of measure – and one woman could obviously have more than one SAE over a period of years.’

Analysing the ClinicalTrials.gov entry (NCT00122681) we were able to replicate the BMJ EBM article’s calculation (1046 vs 982) by summing all the numerators of all the types of serious adverse events (SAEs) listed in the NCT00122681 Study results under “Serious Adverse Events”.

The total number of SAEs and the individual types of SAEs in PATRICIA’s ClinicalTrials.gov entry both use the total number of participants as numerator and denominator:

“Total, Serious Adverse Events: # participants affected / at risk: 829/9325 (8.89%) 835/9319 (8.96%)”
Individual, e.g. “Blood and lymphatic system disorders, Anaemia * 1, # participants affected / at risk 3/9325 (0.03%) 1/9319 (0.01%)”

In ClinicalTrials.gov (see: NCT00122681 Study results) the results are reported as “# participants affected / at risk”.

The number of women with an SAE is the number reported at the very top of the table, as used by the Cochrane HPV Review (i.e. 835 vs 829). [3] This is much lower than the 1046 vs 982 calculation in the BMJ EBM article, [1] because many women had multiple types of SAEs.

The 2012 abstract of the PATRICIA paper (Lancet Oncol 2012 Jan;13(1): e1) [9] reports 1664 SAEs (“Serious adverse events occurred in 835 (9·0%) and 829 (8·9%) women in the vaccine and control groups women with SAEs.”) The authors of the Cochrane HPV Review report that their data were obtained from the PATRICIA 2009 publication (Lancet 2009; 374(9686): 301-314).

The authors of the BMJ EBM article conclude that the PATRICIA trial publication under-reports the number of SAEs compared with what is reported on ClinicalTrials.Gov. Their Response to Cochrane editors [6] reiterates the point that “the PATRICIA trial publication only included two thirds (1400/2028) of the serious adverse events listed on ClinicalTrials.Gov”.

Our conclusion:

The following clarification to the BMJ EBM article is required:

“We found 1046 vs 982 serious adverse events (2028) when we summarised the data from ClinicalTrials.gov (see ‘Results: Serious Adverse Events’).”

Change to:

“We found 1046 vs 982 serious adverse events (2028) when we summarised the data from ClinicalTrials.gov (see ‘Results: Serious Adverse Events’); the PATRICIA publication reports the total number of serious adverse events whereas the Cochrane HPV Review [3] refers to the number of women with serious adverse events.”

6. Were all trials that were included in the Cochrane HPV Review funded by the HPV vaccine manufacturers, as reported by the BMJ EBM article?

The authors of the BMJ EBM article [1] state: “According to ClinicalTrials.gov, this particular trial (‘CVT’ or ‘Costa Rica trial’1) was sponsored by GlaxoSmithKline. Therefore, all included trials were funded by the HPV vaccine manufacturers.”

The ClinicalTrials.gov entry for this trial (NCT00128661) and the trial publication state that the trial was funded and sponsored by the National Cancer Institute (NCI).

“The Costa Rican Vaccine Trial is a longstanding collaboration between investigators in Costa Rica and NCI. The trial is sponsored and funded by NCI (N01-CP-11005) with support from the NIH Office of Research on Women’s Health and conducted in agreement with the Ministry of Health of Costa Rica.”

The trial publication reports “Vaccine was provided for our trial by GSK Biologicals, under a Clinical Trials Agreement with NCI. GSK also provided support for aspects of the trial associated with regulatory submission needs of the company under FDA BB-IND 7920. D R Lowy and J T Schiller are named inventors on the US government-owned HPV vaccine patents that are licensed to GSK and Merck, and so are entitled to limited royalties as specified by federal law.”

We recognise that the terms “study funder” and the “study sponsor” are not synonymous (see here). ‘The term sponsor conveys a different meaning in the context of FDA-regulated research. The sponsor is the individual or entity that is responsible for the oversight of the clinical trial and for all communications with the FDA’. [11]

In the context of FDA regulations, the trial may have been sponsored by GSK, but it is not clear if the trial received any funding from GSK. We consider it is reasonable to accept that GSK provided funding, at least in kind, since it provided vaccines and provided support related to the regulatory submission.

Our conclusion:

The following correction is required: The authors of the BMJ EBM should have said: “Therefore, all included trials were funded or sponsored by the HPV vaccine manufacturers.”

7. Did the Cochrane HPV Review comply with Cochrane’s COI policy?

The authors of the BMJ EBM article [1] stated that “the review’s first author currently leads EMA’s post-marketing surveillance of HPV vaccination effects in non-Nordic member states of the European Union’, which is funded by Sanofi-Pasteur-MSD, the co-manufacturer of Gardasil.”

The Cochrane editors’ response states that “The review was compliant with Cochrane’s current conflict of interest policy”, and ‘Professor Arbyn is not funded by the European Medicine Agency nor by any vaccine manufacturer.’

The authors of the BMJ EBM article [6] responded with the following:

‘In 2011, “EUROGIN covered his [Marc Arbyn’s] travel and lodging expenses … EUROGIN conferences are financially supported by a range of pharmaceutical companies with an interest in cervical cancer” (26); in 2014, “Marc Arbyn’s research unit at The Scientific Institute of Public Health received research support not exceeding 48,000 Euros from MSD-Sanofi Pasteur [co-manufacturer of Gardasil] for a surveillance study of the effects of HPV vaccination in Belgium (SEHIB study) … His [Marc Arbyn’s] unit has also received research support from BD, Bio-Greiner, Abbot, and Cepheid for validation studies of HPV genotyping tests (through the VALGENT studies, valued at 21,000, 21,000 & 38,000€ respectively)” (27); and in 2018, Marc Arbyn is on the EUROGIN programme committee where Merck is a platinum sponsor (28). The Cochrane review’s last author, Dr Markowitz is sponsored by Merck via Medscape (“sponsored by the manufacturer of the quadrivalent vaccine (“supported by an independent educational grant from Merck”) (29).’

*EUROGIN is a platform on education, training, scientific research, expertise and networking on Human Papillomavirus and associated cancers in both genital and non-genital sites.

References from Jørgensen et al. [6]

26. Information NC for B, Pike USNL of M 8600 R, MD B, Usa 20894. Declarations of interest. World Health Organization; 2013. Available from: https://www.ncbi.nlm.nih.gov/books/NBK195238/
27. Cancer Screening in the European Union (2017) Report on the implementation of the Council Recommendation on cancer screening. Health and Food Safety. Available from: https://docplayer.net/43107833-Cancer-screening-in-the-european-union-2017-report-on-the-implementation-of-the-council-recommendation-on-cancer-screening-health-and-food-safety.htm
28. EUROGIN 2018. Available from: https://www.eurogin.com/2018/341-program-committee.html
29. Conflicts of interests. Available from: http://www.nogracias.eu/wp-content/uploads/2013/05/vacc-HPV-Cochrane_HPV_authorship_201212101.pdf

Our conclusion: No correction or clarification required. However, what constitutes a COI in the context of Cochrane reviews should be open to further debate.

We welcome feedback on the complex issues outlined in this report, particularly where there are areas of disagreement with our analysis, or where there are disagreements with our conclusions, or there are errors that require further changes.

Acknowledgements:

Peter Doshi and Fiona Godlee (BMJ) provided comments and criticisms on this draft and Jeff Aronson, Kamal Mahtani, and Rafael Perera (BMJ EBM) provided advice on the Editors’ Review of BMJ EBM’s Analysis of Cochrane’s HPV Systematic Review.

COIs

CH has co-authored 16 peer-reviewed articles with Tom Jefferson (two of which are Cochrane reviews) and holds grant funding jointly with Tom Jefferson from Cochrane on methods for deciding when to prioritise the use of clinical study reports in systematic reviews. CH has been an author 21 Cochrane reviews including updates. He is a founder of AllTrials and an advisor to the WHO International Clinical Trial Registry Platform. He has received expenses and fees for his media work (including payments from BBC Radio 4 Inside Health). He holds grant funding from the NIHR, the NIHR School of Primary Care Research, NIHR BRC Oxford and Cochrane. He has received financial remuneration from an asbestos case and given free legal advice on mesh cases. He has also received income from the publication of a series of toolkit books published by Blackwells. On occasion, he receives expenses for teaching EBM and is also paid for his NHS GP work in urgent care (contract with Oxford Health NHS Foundation Trust). He is Director of CEBM at the University of Oxford, which jointly runs the EBMLive Conference with the BMJ and the Overdiagnosis Conference with international partners, based on a non-profit making model. He is Editor in Chief of BMJ Evidence-Based Medicine. IO is Research Fellow in Evidence Synthesis at the CEBM, University of Oxford, a clinician who works across several NHS Trusts and is funded by the NIHR School of Primary Care Research. IO has co-authored eight peer-reviewed articles jointly with Tom Jefferson (one of these is a Cochrane review).

References

1 Jørgensen L, Gøtzsche PC, Jefferson T. The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias. BMJ Evid Based Med 2018;23:165–8.

2 Hawkes N. Cochrane editors hit back at criticisms of HPV vaccine review. BMJ 2018;362:k3834.

3 HPV vaccination to prevent cancer and pre-cancerous changes of the cervix. doi:10.1002/14651858.CD009069.pub3

4 Hawkes N. HPV vaccine safety: Cochrane launches urgent investigation into review after criticisms. BMJ 2018;:k3472.

5 Tovey D, Soares-Weiser K. Cochrane’s Editor in Chief responds to a BMJ Evidence-Based Medicine article criticizing the Cochrane Review of HPV vaccines. https://www.cochrane.org/sites/default/files/public/uploads/cochrane_hpv_response_3sep18.pdf

6 Jørgensen L Gøtzsche Pc Jefferson. Response to Cochrane editors: Jørgensen, Gøtzsche and Jefferson | BMJ EBM Spotlight. BMJ EBM Spotlight. 2018. https://blogs.bmj.com/bmjebmspotlight/2018/09/23/response-to-cochrane-editors-jorgensen-gotzsche-and-jefferson/ (accessed 27 Sep 2018).

7 Pedersen C, Breindahl M, Aggarwal N, et al. Randomized Trial: Immunogenicity and Safety of Coadministered Human Papillomavirus-16/18 AS04-Adjuvanted Vaccine and Combined Hepatitis A and B Vaccine in Girls. J Adolesc Health Care 2012;50:38–46.

8 Rodgers MA, Brown JVE, Heirs MK, et al. Reporting of industry funded study outcome data: comparison of confidential and published data on the safety and effectiveness of rhBMP-2 for spinal fusion. BMJ 2013;346:f3981–f3981.

9 Lehtinen M E al. Overall efficacy of HPV-16/18 AS04-adjuvanted vaccine against grade 3 or greater cervical intraepithelial neoplasia: 4-year end-of-study analysis o… – PubMed – NCBI. https://www.ncbi.nlm.nih.gov/pubmed/22075171?dopt=Abstract (accessed 3 Oct 2018).

10 Bastian H. The HPV Vaccine: A Critique of a Critique of a Meta-Analysis | Absolutely Maybe. Absolutely Maybe. 2018.https://blogs.plos.org/absolutely-maybe/2018/08/25/the-hpv-vaccine-a-critique-of-a-critique-of-a-meta-analysis/ (accessed 3 Oct 2018).

11 Study Sponsor vs. Funder | CHOP Institutional Review Board. https://irb.research.chop.edu/sponsorship-and-funding (accessed 3 Oct 2018).