Patients set to benefit from new guidelines on Artificial Intelligence health solutions

SPIRIT AI 720
Artificial Intelligence could provide great advances in healthcare

Patients could benefit from faster and more effective introduction of artificial intelligence (AI) innovations to diagnose and treat disease – thanks to the first international standards for reporting of clinical trials for AI.

As evaluation of health interventions involving machine learning or other AI systems moves into clinical trials, an international group has developed guidelines aiming to improve the quality of these studies and ensure that they are reported transparently.

The use of these international guidelines will enable patients, health care professionals and policy-makers to be more confident on whether an AI intervention is safe and effective. This is a key step towards trustworthy AI in health.
Development of new reporting guidelines which expand on the current SPIRIT 2013 and CONSORT 2010 reporting frameworks will boost transparency and robustness for clinical trials evaluating AI health solutions.

Future clinical trials evaluating an AI intervention will be expected - and often required - to report their publications to the new standards. The guidelines will also help medical professionals, regulators, funders and other decision-makers assess the quality of planned clinical trials and assess whether the algorithm is safe and likely to bring about patient benefit.

Researchers from the University of Birmingham and University Hospitals Birmingham NHS Foundation Trust (UHB) worked with leading institutions from across the world and have published their findings and the new guidelines today in Nature Medicine, The BMJ and The Lancet Digital Health.

Researchers developed the additional guidance to tackle concerns that many studies of AI are of insufficient quality and are not transparent. This was highlighted in research published last September, led by several of the same researchers which highlighted that less than one per cent of 20,500 analysed studies relating to health AI were of sufficient quality that independent viewers could have confidence in their results.

Professor Alastair Denniston, Lead for AI at Birmingham Health Partners Centre for Regulatory Science and Innovation, and Consultant Ophthalmologist at UHB, commented: “Patients could benefit hugely from the use of AI in medical settings, but before we introduce these technologies into everyday practice we need to know that they have been robustly evaluated and proven to be effective and safe. Our previous work showed just how big a problem this can be and that we needed a way to cut through the hype surrounding AI in healthcare. 

“These new reporting guidelines – SPIRIT-AI and CONSORT-AI – provide a solution to the ‘hype’ problem. They provide a clear, transparent framework to support the design and reporting of AI trials that will help to improve quality and transparency. These extended guidelines will help to reduce wasted effort and deliver effective AI-led medical interventions to patients quicker.”

SPIRIT-AI extension is a new guideline for clinical trials protocols and CONSORT-AI extension is a new reporting guideline for clinical trial reports, for evaluating interventions with AI components. 

Professor Melanie Calvert, NIHR Senior Investigator and Director of Birmingham Health Partners Centre for Regulatory Science and Innovation commented: “There is growing recognition that interventions involving AI need rigorous evaluation to demonstrate their impact on health outcomes. Without this, we risk not generating sufficiently robust evidence to decide whether AI interventions should be commissioned in the real world.

“These new guidelines will help to identify and overcome research challenges associated with AI-led health innovation, but we could not have got to this exciting point without the help of patients involved in research.”

Elaine Manna, from London, has been living with age-related macular degeneration for 20 years and was one of a number of patient partners who helped to develop the new guidelines. She was asked to provide a patient perspective on developing the guidelines after taking part in an AI research study involving Moorfields Eye Hospital NHS Foundation Trust, in London, and British technology company DeepMind.

Elaine commented: “A super-fast algorithm was tested on my eye - diagnosing my condition as well as an expert ophthalmologist or optometrist. This was a development with significant implications for saving sight and reducing waiting times for appointments.

“It’s vital for patients to be equally involved in their healthcare - understanding how decisions are made, being informed and involved in decision making. Helping to develop the SPIRIT-AI and CONSORT-AI guidelines, I went from thinking of myself as someone with a degenerative eye disease to someone who felt empowered.” 

The SPIRIT-AI extension includes 15 new items and the CONSORT-AI extension includes 14 new items - all considered sufficiently important for clinical trial protocols of AI interventions to be routinely reported in addition to core items.

The work was funded by a Wellcome Trust Institutional Strategic Support Fund: Digital Health Pilot Grant, Research England (part of UKRI), Health Data Research UK and The Alan Turing Institute.

Notes to editors:

  • The University of Birmingham is ranked amongst the world’s top 100 institutions. Its work brings people from across the world to Birmingham, including researchers, teachers and more than 6,500 international students from over 150 countries.
  • University Hospitals Birmingham NHS Foundation Trust runs the Queen Elizabeth Hospital Birmingham, Birmingham Chest Clinic, Heartlands Hospital, Good Hope Hospital, Solihull Hospital and various community services across the region. The Trust has regional centres for trauma, burns, plastics, neurosciences, dermatology and cancer. It also has centres of excellence for vascular, bariatric and pathology services, as well as the treatment of MRSA and other infectious diseases. The Trust hosts the Institute of Translational Medicine (ITM) as well as the NIHR Birmingham Biomedical Research Centre, NIHR Trauma MIC, NIHR Surgical Reconstruction and Microbiology Research Centre and NIHR/Wellcome Trust Clinical Research Facility. UHB is also the lead organisation for two Health Data Research Hubs, INSIGHT and PIONEER, and is the substantive site for HDR UK Midlands.
  • ‘Reporting Guidelines for Clinical Trial Reports for Interventions Involving Artificial Intelligence: The CONSORT-AI extension’ - Xiaoxuan Liu, Samantha Cruz Rivera, David Moher, Melanie J. Calvert, Alastair K. Denniston and the SPIRIT-AI and CONSORT-AI Working Group is published by Nature Medicine, The Lancet Digital Health and The BMJ - paper available at https://doi.org/10.1038/s41591-020-1034-x
  • 'Guidelines for Clinical Trial Protocols for Interventions Involving Artificial Intelligence; The SPIRIT-AI extension’ - Samantha Cruz Rivera, Xiaoxuan Liu, An-Wen Chan, Alastair K. Denniston, Melanie J. Calvert and the SPIRIT-AI and CONSORT-AI Working Group is published by Nature Medicine, The Lancet Digital Health and The BMJ - paper available at https://doi.org/10.1038/s41591-020-1037-7
  • The Birmingham Health Partners Centre for Regulatory Science and Innovation (CRSI) supports the development and delivery of novel therapeutics and medical devices in the UK through advanced regulatory standards and tools. It leverages Birmingham Health Partners’ established profile and expertise in several key areas: medicinal science; health policy and management; clinical trial design; medical law; AI and patient-reported outcomes research. https://www.birminghamhealthpartners.co.uk/programmes/bhp-centre-for-regulatory-science-innovation/. The Centre recently published a major report: ‘Advancing Regulatory Science and Innovation in Healthcare’ https://bit.ly/3gNx3TX

  • The National Institute for Health Research (NIHR) is the nation's largest funder of health and care research. The NIHR:

  • Funds, supports and delivers high quality research that benefits the NHS, public health and social care
  • Engages and involves patients, carers and the public in order to improve the reach, quality and impact of research
  • Attracts, trains and supports the best researchers to tackle the complex health and care challenges of the future
  • Invests in world-class infrastructure and a skilled delivery workforce to translate discoveries into improved treatments and services
  • Partners with other public funders, charities and industry to maximise the value of research to patients and the economy

The NIHR was established in 2006 to improve the health and wealth of the nation through research, and is funded by the Department of Health and Social Care. In addition to its national role, the NIHR supports applied health research for the direct and primary benefit of people in low- and middle-income countries, using UK aid from the UK government.