Compared to the SARS outbreak in 2003, the covid-19 pandemic has led to substantially more scientific publications during the first four months. Preprints have become the medium of choice. The rapidly increasing number of publications combined with the urgency to quickly understand the new pathogen presents a significant challenge for maintaining the integrity of the underlying evidence base, and to ensure that research is conducted according to global standards of research integrity [1,2].

Rapid publication can aggravate questionable research practices

Competition and the pressure to publish quickly can lead researchers to (inadvertently) base studies on questionable methods. This has recently been a major issue in discussions of the models used to plan responses and predict the future course of the pandemic. A recent research paper in The BMJ found that: “proposed models are poorly reported, at high risk of bias, and their reported performance is probably optimistic.” [3] Similar concerns are currently being raised about serological tests coming onto the market rapidly as a way to determine the extent of immunity against covid-19. Manufacturers claim their tests have high sensitivity and specificity, but with little or no published data yet to back this, it is hard to assess the basis of these claims. This lack of transparency on the research behind these tests is worrying as more and more countries turn to testing to guide easing of lockdowns [4].

Rapid publication amplifies weaknesses in peer review

Competition and the pressure to publish quickly has produced a flood of un-peer-reviewed papers published as preprints. Publishing and sharing  preprints encourages scientific collaboration, transparency, and fast sharing of data.  But, in a crisis, it could lead to the spread and use of controversial information that needs further peer evaluation and validation. A recent case is the fierce scientific debate [5] on twitter and blog posts around a Stanford preprint [6] that claimed the case fatality rate of covid-19 was similar to seasonal influenza. Researchers were quick to raise questions about sampling, the validity of the antibody tests used and statistical calculations. Much has been written about social media’s ability to spread (mis)information rapidly. Recent reports analysed preprints’ viral potential on social media and news outlets [7]. Examples include a study showing a link between covid-19 and HIV, which was tweeted at least 17,000 times and reached 25 news outlets despite being criticised widely by scientists and rapidly retracted [1]. 

The most troubling consequence of hurried decisions to publish papers based on inadequate or flawed research, is the impact this can have on health policy decisions. 

This is highlighted by the ongoing debate around the use of hydroxychloroquine, an anti-malaria drug, for the treatment of covid-19, despite the fact that rigorous clinical trial evidence of its efficacy in the treatment of covid-19 is absent [8]. A study by Didier Raoult and colleagues claimed that hydroxychloroquine is effective in treating covid-19. The study has since come under scrutiny and Elsevier have announced that they are investigating the paper. However, this hasn’t stopped politicians from promoting its use, even though increasingly new evidence suggests it is not an effective treatment, and may even cause harm.  

Another example is the UK government’s strategy to manage the pandemic. In the UK, a herd immunity response was initially proposed. However, the modelling on which this was based, was not made public and it took a few days before a group at Imperial College, led by Ferguson et al. published their modelling [10]. This gap left people wondering what the decision was based on. When the modelling was published, it showed that having herd immunity as an end goal could mean that 60% of the total population would need to be infected, with nearly half a million deaths [10]. The UK government aborted the strategy, opting for more stringent measures instead. However, during this brief period, the number of confirmed cases increased from 590 on 12 March 2020 to more than 2500 cases by 17 March 2020, increasing deaths and complicating the challenge of preventing further spread [11]. The UK government has come under considerable criticism for their handling of the pandemic so far. This policy change and the ensuing delay has made the UK one of the worst countries worldwide in terms of covid-19 fatalities [11].

Solutions

The speed and volume of data that have been published in the past few months on covid-19, emphasise the long recognised importance of maintaining research integrity. It is well known that the systems and processes for maintaining integrity in research are not without their flaws. The current crisis has made these flaws more apparent due to their potential for immediate impact on policy making and resulting consequences for public health. While there are no quick ways to change behaviour and raise the standards of research integrity, we believe that some immediate steps should be considered.

First, the scientific community, both individuals and professional societies, should actively speak out when they feel the standards for integrity in research [12] have been compromised. Identifying unreliable research is a global responsibility of the scientific community.

Second, government advisory committees should be fully transparent [13], disclosing all real and apparent conflicts of interest and clearly identifying and making open the research that supports their recommendations. They should have balanced memberships that reflect a range of views and disclose these memberships openly.

Third, editors and authors should be proactive in assuring that the review status of their publications is clearly understood. Journals should explicitly state whether a publication is peer reviewed or not and if peer reviewed, was it under a rapid review. All preprints platforms should as standard policy have warning banners to readers that the studies are not peer reviewed and should be read with great caution. Many preprint platforms already do this. 

Fourthly, there are calls for scientists to open up their research to critique early on in the process i.e. before data collection starts so the robustness of the study methods, materials and analytical approach can be first “peer reviewed” either formally or informally. This is exactly what the Registered Report, started by the journal Royal Society Open Science aims to do with its covid-19 studies. Here, a study is peer reviewed prior to data collection, so the methods and analysis plan can first be checked for robustness [14]. A similar call was recently made for preprints [15]. In this way, the quality of the research methods and analysis can be validated before data are collected thereby assuring quality of its findings eventually.

As some countries have begun to ease lockdown measures and others consider what approach to take, the reliability of the research used to make these decisions will be crucial. The use of flawed studies that lacked proper peer review by predisposed advisory committees could re-ignite the pandemic and lead to further economic instability. The millions currently being invested in research will be of little use if it is not conducted, published and used with integrity.

Gowri Gopalakrishna, PhD/Postdoctoral Researcher, Amsterdam University Medical Centers, Amsterdam, The Netherlands.

Lex Bouter, PhD/Professor of Methodology and Integrity, Amsterdam University Medical Centers, Amsterdam, The Netherlands.

Tony Mayer, Former Research Integrity Adviser, Nanyang Technological University, Singapore.

Nicholas Steneck, PhD/Professor Emeritus of History, University of Michigan, Ann Arbor, United States.

Competing interests: NS and TM were and LB is playing a leading, but unpaid role in organizing the World Conferences on Research Integrity. NS is receiving royalties from the sales of the Epigeum research integrity course by Oxford University Press. GG the lead postdoctoral researcher on the Dutch National Survey on Research Integrity.

References:

1. Speed Science: The risks of swiftly spreading coronavirus research. Reuters. Published 19 Feb 2020 https://graphics.reuters.com/CHINA-HEALTH-RESEARCH/0100B5ES3MG/index.html
2. https://wcrif.org/guidance/singapore-statement (accessed 13 May 2020)
3. Wynants L, Van Calster B, Bonten MJ et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ, 2020; 369: m1328; https://doi- org.vu-nl.idm.oclc.org/10.1136/bmj.m1328 (accessed 13 May 2020)
4. https://jamanetwork.com/journals/jama/fullarticle/2764954 (accessed 13 May 2020)
5. https://arstechnica.com/science/2020/04/experts-demolish-studies-suggesting-covid-19-is-no-worse-than-flu/) (accessed 13 May 2020)
6. https://www.medrxiv.org/content/10.1101/2020.04.14.20062463v1.full.pdf (accessed 13 May 2020)
7. https://www.economist.com/science-and-technology/2020/05/07/scientific-research-on-the-coronavirus-is-being-released-in-a-torrent (accessed 13 May 2020)
8. https://www.bmj.com/content/369/bmj.m2018
9. https://retractionwatch.com/2020/04/12/elsevier-investigating-hydroxychloroquine-covid-19-paper/ (accessed 13 May 2020)
10. Impact of non-pharmaceutical interventions (NPIs) to reduce COVID19 mortality and healthcare demand, 16 March 2020, Imperial College COVID-19 Response Team https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida- fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf (accessed 14 April 2020)
11. https://www.statista.com/topics/6112/coronavirus-covid-19-in-the-uk/ (accessed 14 April 2020)
12. https://wcrif.org/foundation/mission
13. https://www.nature.com/articles/d41586-020-00772-4
14. https://www.natureindex.com/news-blog/rapid-registered-report-coronavirus-aims-to-stop-researchers-following-false-research-leads
15. Lakens D. Nature 581, 121 (2020); doi: 10.1038/d41586-020-01392-8