̽��ֱ�� of Cambridge - Michael Roberts

New model improves accuracy of machine learning in COVID-19 diagnosis while preserving privacy

sc604 — Thu, 16 Dec 2021 10:48:16 +0000

̽��ֱ��international team, led by the ̽��ֱ�� of Cambridge and the Huazhong ̽��ֱ�� of Science and Technology, used a technique called federated learning to build their model. Using federated learning, an AI model in one hospital or country can be independently trained and verified using a dataset from another hospital or country, without data sharing.

̽��ֱ��researchers based their model on more than 9,000 CT scans from approximately 3,300 patients in 23 hospitals in the UK and China. Their results, reported in the journal Nature Machine Intelligence, provide a framework where AI techniques can be made more trustworthy and accurate, especially in areas such as medical diagnosis where privacy is vital.

AI has provided a promising solution for streamlining COVID-19 diagnoses and future public health crises. However, concerns surrounding security and trustworthiness impede the collection of large-scale representative medical data, posing a challenge for training a model that can be used worldwide.

In the early days of the COVID-19 pandemic, many AI researchers worked to develop models that could diagnose the disease. However, many of these models were built using low-quality data, ‘Frankenstein’ datasets, and a lack of input from clinicians. Many of the same researchers from the current study highlighted that these earlier models were not fit for clinical use in the spring of 2021.

“AI has a lot of limitations when it comes to COVID-19 diagnosis, and we need to carefully screen and curate the data so that we end up with a model that works and is trustworthy,” said co-first author Hanchen Wang from Cambridge’s Department of Engineering. “Where earlier models have relied on arbitrary open-sourced data, we worked with a large team of radiologists from the NHS and Wuhan Tongji Hospital Group to select the data, so that we were starting from a strong position.”

̽��ֱ��researchers used two well-curated external validation datasets of appropriate size to test their model and ensure that it would work well on datasets from different hospitals or countries.

“Before COVID-19, people didn’t realise just how much data you needed to collect in order to build medical AI applications,” said co-author Dr Michael Roberts from AstraZeneca and Cambridge’s Department of Applied Mathematics and Theoretical Physics. “Different hospitals, different countries all have their own ways of doing things, so you need the datasets to be as large as possible in order to make something that will be useful to the widest range of clinicians.”

̽��ֱ��researchers based their framework on three-dimensional CT scans instead of two-dimensional images. CT scans offer a much higher level of detail, resulting in a better model. They used 9,573 CT scans from 3,336 patients collected from 23 hospitals located in China and the UK.

̽��ֱ��researchers also had to mitigate for bias caused by the different datasets, and used federated learning to train a better generalised AI model, while preserving the privacy of each data centre in a collaborative setting.

For a fair comparison, the researchers validated all the models on the same data, without overlapping with the training data. ̽��ֱ��team had a panel of radiologists make diagnostic predictions based on the same set of CT scans, and compared the accuracy of the AI models and human professionals.

̽��ֱ��researchers say their model is useful not just for COVID-19, but for any other diseases that can be diagnosed using a CT scan. “ ̽��ֱ��next time there’s a pandemic, and there’s every reason to believe that there will be, we’ll be in a much better position to leverage AI techniques quickly so that we can understand new diseases faster,” said Wang.

“We’ve shown that encrypting medical data is possible, so we can build and use these tools while preserving patient privacy across internal and external borders,” said Roberts. “By working with other countries, we can do so much more than we can alone.”

̽��ֱ��researchers are now collaborating with the newly-established WHO Hub for Pandemic and Epidemic Intelligence, to explore the possibility of advancing the privacy-preserving digital healthcare frameworks.

Reference:
Xiang Bai et al. ‘Advancing COVID-19 Diagnosis with Privacy-Preserving Collaboration in Artificial Intelligence.’ Nature Machine Intelligence (2021). DOI: 10.1038/s42256-021-00421-z

Researchers in the UK and China have developed an artificial intelligence (AI) model that can diagnose COVID-19 as well as a panel of professional radiologists, while preserving the privacy of patient data.

By working with other countries, we can do so much more than we can alone

Michael Roberts

U.S. Department of Energy

Covid-19

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. Images, including our videos, are Copyright © ̽��ֱ�� of Cambridge and licensors/contributors as identified. All rights reserved. We make our image and video content available in a number of ways – as here, on our main website under its Terms and conditions, and on a range of channels including social media that permit your use and sharing of our content under their respective Terms.

Yes

Licence type:

Public Domain

Trinity Challenge announces inaugural winners

Anonymous — Fri, 25 Jun 2021 14:43:12 +0000

̽��ֱ��eight winners have been selected by an international panel of expert judges, out of a total of 340 applications from 61 countries. ̽��ֱ��competition has seen unprecedented collaborations between the private, public, charitable and academic sectors, and will drive a step-change in using data and analytics for pandemic preparedness.

̽��ֱ�� ̽��ֱ�� of Cambridge joined a coalition of some of the world’s leading businesses and academic and tech institutions to launch ̽��ֱ��Trinity Challenge in September 2020. ̽��ֱ��global challenge, convened by Dame Sally Davies, Master of Trinity College, provides a £10m prize fund for breakthrough solutions to make sure one billion more people are better protected against health emergencies.

Participatory One Health Disease Detection (PODD), which empowers farmers to identify and report zoonotic diseases that could potentially pass from animals to humans, has been named the grand prize winner at the inaugural awards ceremony. ̽��ֱ��organisation is being awarded £1.3 million (US$1.8 million) in pledged funding.

Led by Susumpat Patipow, General Director at OpenDream, PODD has developed a platform for livestock owners to report suspected animal illness, and in return receive veterinary care to improve animal health. If it appears a disease outbreak is likely, local health officials will quarantine the sick animals, saving the remaining livestock and possibly preventing the next COVID-19-type outbreak.

Having already achieved significant success in Thailand, with a network of 20,000 farmers helping to detect and control disease outbreaks, PODD is looking to expand its operations to Cambodia, India, Indonesia, Laos, Uganda and Vietnam over the next three years.

BloodCounts! - an international consortium of scientists, led by Professor Carola-Bibiane Schönlieb from Cambridge’s Department of Applied Mathematics and Theoretical Physics (DAMTP) that has developed an innovative infectious disease outbreak detection system, was one of two second-prize winners, each awarded £1 million in pledged funding.

Developed by Dr Michael Roberts and Dr Nicholas Gleadall, the BloodCounts! Solution uses data from routine blood tests and powerful AI-based techniques to provide a ‘tsunami-like’ early warning system for new disease outbreaks.

“Since the beginning of the pandemic I have been developing AI-based methods to aid in medical decision making for COVID-19 patients, starting with analysis of Chest X-ray data,” said Roberts, who is affiliated with DAMTP and the Cambridge Mathematics of Information in Healthcare (CMIH) Hub. “Echoing the observations made by the clinical teams, we saw profound and unique differences in the medical measurements of infected individuals, particularly in their full blood count data. It is these changes that we can train models to detect at scale.”

Unlike many current test methods, their approach doesn't require any prior knowledge of a specific pathogen to work, instead, they use full blood count data to exploit the pathogen detecting abilities of the human immune system by observing changes in the blood measurements associated with infection.

As the full blood count is the world’s most common medical laboratory test, with over 3.6 billion being performed worldwide each year, the BloodCounts! team can rapidly apply their methods to scan for abnormal changes in the blood cells of large populations - alerting public health agencies to potential outbreaks of pathogen infection.

This solution is a demonstration of how the application of AI-based methods can lead to healthcare benefits. It also highlights the importance of strong collaboration between leading organisations, as the development of these algorithms was only possible due the EpiCov data sharing initiative pioneered by Cambridge ̽��ֱ�� Hospitals.

“Hundreds of millions of full blood count tests are being performed every day worldwide, and this meant that we could apply our AI methods at population scale,” said Gleadall, from the ̽��ֱ�� of Cambridge and NHS Blood and Transplant. “Usually the rich measurement data are discarded after summary results have been reported, but by working with Cambridge ̽��ֱ��, Barts Health London, and ̽��ֱ�� College London NHS Hospitals we have rescued throughout the pandemic the rich data from 2.8 million full blood count tests.”

̽��ֱ��Sentinel Forecasting System is the other second-prize winner, and will explore the emergence of new infectious diseases in West Africa, beginning with Lassa fever. ̽��ֱ��system will combine data from ecology, social science, genomics and epidemiology to provide real-time disease risk for haemorrhagic fevers, such as Lassa and Ebola.

Lassa is a virus usually passed to humans through exposure to food or household items contaminated by infected rats. It is endemic in West African countries including Benin, Ghana, Guinea, Liberia, Mali, Sierra Leone, Togo and Nigeria.

Around 80% of people who become infected with Lassa virus have no symptoms, and the overall case-fatality rate is 1%. 1 in 5 infections can result in severe disease affecting the liver, spleen and kidneys.

̽��ֱ��UCL team will partner with the African Centre of Excellence for Genomics of Infectious Diseases in Nigeria, Nigeria Centre for Disease Control, Zoological Society of London, London School of Hygiene and Tropical Medicine, Microsoft, and Cambridge’s Laboratory of Viral Zoonotics (LVZ) to produce the system.

“This Trinity Challenge project brings new multidisciplinary technologies together to anticipate climatic, human, animal population, agricultural impacts on the likelihood of spill overs of infections from animals to humans,” said Professor Jonathan Heeney, who leads LVZ at Cambridge’s Department of Veterinary Medicine.

Additionally, five 3rd prize winners are each being awarded £480,000 (US$ 660,000) in pledged funding.

Dame Sally Davies said: “It was crystal clear at the beginning of this pandemic that the world had a lack of data, a lack of access to data, and a lack of interoperability of data, presenting a challenge. While others talked, we took action. ̽��ֱ��solutions we have discovered in the course of the Challenge will be a link between systems and countries.”

In addition to financial support, ̽��ֱ��Trinity Challenge will provide connections to the right organisations to maximise the impact of these solutions. Since its inception nine months ago, TTC has united early applicants with partners from the private, academic and social sectors to receive access to digital platforms, data, and technical advice, to scale-up the use of data and analytics to protect the world from future health emergencies. ̽��ֱ��Trinity Challenge has helped form over 200 connections between applicants and its members.

̽��ֱ��Trinity Challenge has announced the winners of its inaugural competition, and is investing a £5.7 million (US$8 million) charitable pledged prize fund into one grand prize winner, two 2nd prize winners, and five 3rd prize winners.

While others talked, we took action. ̽��ֱ��solutions we have discovered in the course of the Challenge will be a link between systems and countries

Dame Sally Davies

̽��ֱ��Trinity Challenge

Collage of Trinity Challenge finalists

Yes

Machine learning models for diagnosing COVID-19 are not yet suitable for clinical use

sc604 — Mon, 15 Mar 2021 15:07:12 +0000

Researchers have found that out of the more than 300 COVID-19 machine learning models described in scientific papers in 2020, none of them is suitable for detecting or diagnosing COVID-19 from standard medical imaging, due to biases, methodological flaws, lack of reproducibility, and ‘Frankenstein datasets.’

̽��ֱ��team of researchers, led by the ̽��ֱ�� of Cambridge, carried out a systematic review of scientific manuscripts – published between 1 January and 3 October 2020 – describing machine learning models that claimed to be able to diagnose or prognosticate for COVID-19 from chest radiographs (CXR) and computed tomography (CT) images. Some of these papers had undergone the process of peer-review, while the majority had not.

Their search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in the systematic review. None of the 62 models was of potential clinical use, which is a major weakness, given the urgency with which validated COVID-19 models are needed. ̽��ֱ��results are reported in the journal Nature Machine Intelligence.

Machine learning is a promising and potentially powerful technique for detection and prognosis of disease. Machine learning methods, including where imaging and other data streams are combined with large electronic health databases, could enable a personalised approach to medicine through improved diagnosis and prediction of individual responses to therapies.

“However, any machine learning algorithm is only as good as the data it’s trained on,” said first author Dr Michael Roberts from Cambridge’s Department of Applied Mathematics and Theoretical Physics. “Especially for a brand-new disease like COVID-19, it’s vital that the training data is as diverse as possible because, as we’ve seen throughout this pandemic, there are many different factors that affect what the disease looks like and how it behaves.”

“ ̽��ֱ��international machine learning community went to enormous efforts to tackle the COVID-19 pandemic using machine learning,” said joint senior author Dr James Rudd, from Cambridge’s Department of Medicine. “These early studies show promise, but they suffer from a high prevalence of deficiencies in methodology and reporting, with none of the literature we reviewed reaching the threshold of robustness and reproducibility essential to support use in clinical practice.”

Many of the studies were hampered by issues with poor quality data, poor application of machine learning methodology, poor reproducibility, and biases in study design. For example, several training datasets used images from children for their ‘non-COVID-19’ data and images from adults for their COVID-19 data. “However, since children are far less likely to get COVID-19 than adults, all the machine learning model could usefully do was to tell the difference between children and adults, since including images from children made the model highly biased,” said Roberts.

Many of the machine learning models were trained on sample datasets that were too small to be effective. “In the early days of the pandemic, there was such a hunger for information, and some publications were no doubt rushed,” said Rudd. “But if you’re basing your model on data from a single hospital, it might not work on data from a hospital in the next town over: the data needs to be diverse and ideally international, or else you’re setting your machine learning model up to fail when it’s tested more widely.”

In many cases, the studies did not specify where their data had come from, or the models were trained and tested on the same data, or they were based on publicly available ‘Frankenstein datasets’ that had evolved and merged over time, making it impossible to reproduce the initial results.

Another widespread flaw in many of the studies was a lack of involvement from radiologists and clinicians. “Whether you’re using machine learning to predict the weather or how a disease might progress, it’s so important to make sure that different specialists are working together and speaking the same language, so the right problems can be focused on,” said Roberts.

Despite the flaws they found in the COVID-19 models, the researchers say that with some key modifications, machine learning can be a powerful tool in combatting the pandemic. For example, they caution against naive use of public datasets, which can lead to significant risks of bias. In addition, datasets should be diverse and of appropriate size to make the model useful for different demographic group and independent external datasets should be curated.

In addition to higher quality datasets, manuscripts with sufficient documentation to be reproducible and external validation are required to increase the likelihood of models being taken forward and integrated into future clinical trials to establish independent technical and clinical validation as well as cost-effectiveness.

Reference:
Michael Roberts et al. ‘Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans.’ Nature Machine Intelligence (2021). DOI: 10.1038/s42256-021-00307-0

Systematic review finds that machine learning models for detecting and diagnosing COVID-19 from medical images have major flaws and biases, making them unsuitable for use in patients. However, researchers have suggested ways to remedy the problem.

Any machine learning algorithm is only as good as the data it’s trained on

Michael Roberts

NIAID

Novel Coronavirus SARS-CoV-2

Yes

Licence type:

Attribution