Testing for covid-19 is critically important for the management of disease and prevention of spread. There has been understandable concern about the need to develop good tests and ensure that the tests which are used perform well. The debate has been particularly heated for serology tests (tests for antibodies to the SARS-CoV-2 virus) which are being rolled out as part of the UK government’s testing and contact tracing strategy, but with concerns remaining about the tests and how effective the strategy will be. [1]
But what evidence has been used to inform decisions on which serology tests to use and whether the tests chosen are fit-for-purpose? In my view, much greater care is required in understanding and using the results of the available research.
Official comments on the best serology tests have predominantly relied on the concept of test accuracy, frequently expressed as estimates of sensitivity and specificity.
Concerning antibody tests which had been ordered by the UK in early April, John Newton, national coordinator of the UK Coronavirus Testing Programme, said “All the tests that have been made available have been tested. None of them performed well enough. In particular they didn’t have sufficient sensitivity, which is the ability to correctly identify people who have been infected. We set a clear target for a test to achieve and none of them frankly were close.” [2]
And concerning the announcement of validated antibody tests in May, Newton said an independent evaluation carried out at the PHE Porton Down centre reported that the test was a “highly specific assay with specificity of 100%.” [1]
Test accuracy seeks to measure the ability of a test to identify a target condition by comparing the results obtained by that test relative to another test which “truthfully” captures the target condition, the so called “reference standard.” Tests which correctly identify a high proportion of samples who have the target condition by the reference standard have high sensitivity; tests which correctly identify a high proportion of samples who do not have the target condition by the reference standard have high specificity. The concept is absolutely dependent on having a reference standard which captures the target condition.
But what is the “target condition” for serology? Although not as straight-forward as if we were considering the diagnosis of a clearly characterised disease like lung cancer, there are several options:
- The presence of covid-19 disease, frequently based on the presence of viral RNA in swabs
- The true presence of covid-19 antibodies in a sample
- The presence of a serological response in a patient
- Confirmed protection against the SARS-CoV-2 in a patient
None is correct or incorrect, as each addresses a relevant question, but what is clear is that the accuracy will vary depending on the reference standard used. Thus sensitivity and specificity can only be understood where the reference standard is indicated and this is not apparent in official pronouncements. So is it the users of research who are lax in faithfully transmitting the results of the research, or is it the researchers who are not reporting accuracy correctly?
So far there have been 7 studies published in peer-reviewed journals evaluating serology tests for covid-19. Five refer to sensitivity and/or specificity and three rely on it. Only one of these five provided qualification of what their accuracy estimate referred to (and then only did it in a general way): “Compared to molecular detection, the sensitivity of serum IgM and IgG antibodies to diagnose COVID-19 were 48.1% and 88.9%,….” [3] None discussed the influence of their chosen reference standard on their estimates, particularly differentiating single viral RNA tests, from multiple viral RNA tests, from viral RNA profile in combination with clinical pattern.
So it seems that part of the problem lies with the reporting of research. However, two studies possibly understanding that the notion of a reference standard early in the understanding of a disease is premature, relied on positivity rates to express their results, raising the possibility that these estimates may be reinterpreted as sensitivity or specificity by readers inappropriately. [4,5] These studies also clearly demonstrate time dependence of serology test results, in turn suggesting another feature which must be specified when the accuracy of serology tests is quoted.
In conclusion great care is required in judging the strengths of different serology tests, particularly when relying on accuracy estimations. Researchers and decision-makers should: be precise about what they are talking about when trading estimates of sensitivity and specificity, particularly the nature of the reference standard and the time the tests were done relative to when infection is likely to have occurred; be critical about the limitations of many of the reference standards currently being used in accuracy studies; be open to the possibility that studies other than those measuring accuracy may be more helpful in understanding the value of serology tests.
When the crisis of covid has passed, the lessons we learn about the limitations of accuracy may need to reapplied to tests generally, as should attention to broadening the range of test evaluative studies beyond accuracy and considering standards for such studies
Chris Hyde is a medically qualified public health academic with expertise in health technology assessment, particularly as applied to tests. He is a member of NICE’s Diagnostic Advisory Committee and the National Screening Committee, but is offering this opinion in a personal capacity.
Conflicts of interest: I have advised companies developing tests for COVID but have received no direct or indirect reward for such advice.
Twitter: @ExeterTestGroup
References:
1] Mahase E. Covid-19: “Unacceptable” that antibody test claims cannot be scrutinised, say experts. BMJ 2020; 369 doi: https://doi.org/10.1136/bmj.m2000
2] Iacobucci G. Covid-19: Antibody tests will not be rolled out in UK until at least May, MPs hear. BMJ 2020; 369 doi: https://doi.org/10.1136/bmj.m1449
3] Jin Y, Wang M, Zuo Z, Fan C, Ye F, Cai Z, et al. Diagnostic value and dynamic variance of serum antibody in coronavirus disease 2019. Int J Infect Dis 2020 DOI:10.1016/j.ijid.2020.03.065
4] Gao H X, Li Y N, Xu Z G, Wang Y L, Wang H B, Cao J F, et al. Detection of serum immunoglobulin M and immunoglobulin G antibodies in 2019-novel coronavirus infected cases from different stages. Chinese Medical Journal 2020;26 DOI:10.1097/CM9.0000000000000820
5] Guo L, Ren L, Yang S, Xiao M, Chang, Yang F, et al. Profiling Early Humoral Response to Diagnose Novel Coronavirus Disease (COVID-19). Clinical infectious diseases 2020 DOI:10.1093/cid/ciaa310