The Royal Statistical Society has today published its review of the statistical evidence needed to assure the performance of future diagnostic tests, so we are better prepared for future pandemics.
The Royal Statistical Society (RSS) has today published its review of the statistical evidence needed to assure the performance of future diagnostic tests, so we are better prepared for future pandemics.
The RSS Working Group on Diagnostic Tests, which is co-chaired by University of Birmingham’s Professor Jon Deeks, is calling on the Medicines and Healthcare Products Regulatory Agency (MHRA) to review and revise the national licensing process for in vitro diagnostic tests, to ensure that reliable evidence about the performance of tests is available and public safety is protected.
Present legislation does not require tests to be evaluated in the settings where they will be used, and for the evidence to be independently scrutinised and publicly available. Currently, the performance of diagnostic tests is not held to a common statistical benchmark in either the UK, US, or EU, whereas the regulatory regime for drugs and vaccines is more stringent. Covid-19 has brought this longstanding problem to the fore, as tests have come to market without evidence of their accuracy for many of the uses to which they are being applied and have been marketed with claims not supported by strong studies.
The RSS experts stress that the assessment of a test’s safety must go beyond just the safety of the device itself, to include the potentially harmful consequences of false negatives and false positives.
The RSS calls for evaluation of tests for each intended use, and for the ongoing monitoring of a test’s performance to be required by the regulator. During the pandemic, there have not been the necessary evaluations of tests for the different uses to which they are applied. For example, initial claims of the sensitivity of the Innova lateral flow test (LFT) were based on evaluations conducted in laboratory settings in patients with symptoms. However, when they were used by non-healthcare professionals in people without symptoms in mass testing in Liverpool, they missed many more cases. Now they are being used in school-children and as a self-test, for which no data are available at all.
The RSS report highlights the importance of the prevalence of a disease in deciding when to change or stop using tests. As a disease becomes rarer the usefulness of a test decreases, with increasing chances that positive results are false. When mass asymptomatic testing was first introduced for secondary school pupils in England, the RSS raised concerns that as disease rates were very low, the majority of positive results could be false. This led to pupils, their families and their contacts being forced by law to unnecessarily self-isolate as, at the time, the government would not allow a negative result from a more reliable PCR test to overrule the LFT result.
The RSS working group also calls for more transparency around testing. Their report stresses that those providing tests to patients and the public have a responsibility to ensure an informed choice can be made, and people are aware of both the advantages and disadvantages of testing. Clear, unbiased information must be provided.
During the pandemic, there has not been enough explanation to the public on the use of LFTs and the fact that negative test results do not rule out infection. The public must be given the full information on the likelihood of incorrect results, as otherwise the false reassurance provided may lead to greater risk taking and therefore increase transmission.
Professor Deborah Ashby, co-chair of the RSS Working Group on Diagnostic Tests, said: “Testing has been a key focus of many government’s strategies in fighting Covid, but the lack of statistical standards has caused issues, with tests coming to market without enough known on their effectiveness. We urge regulators to take on board our recommendations, to allow for more scrutiny of diagnostics more generally and for future pandemics.”
Professor Jon Deeks, co-chair of the RSS Working Group on Diagnostic Tests and Professor of Biostatistics at the University of Birmingham, commented: “While no one questions the need for an evidence-based approach to vaccines and treatments, the proper assessment of the suitability of Covid-19 tests has been neglected.
“Investment in well-designed studies evaluating tests in the real-world settings where they are used must become standard practice. We must learn from the mistakes made during the pandemic and put in place requirements for stronger science, better regulation and more transparency.”
Notes to editors:
