̽��ֱ�� of Cambridge - James Rudd

Machine learning models for diagnosing COVID-19 are not yet suitable for clinical use

sc604 — Mon, 15 Mar 2021 15:07:12 +0000

Researchers have found that out of the more than 300 COVID-19 machine learning models described in scientific papers in 2020, none of them is suitable for detecting or diagnosing COVID-19 from standard medical imaging, due to biases, methodological flaws, lack of reproducibility, and ‘Frankenstein datasets.’

̽��ֱ��team of researchers, led by the ̽��ֱ�� of Cambridge, carried out a systematic review of scientific manuscripts – published between 1 January and 3 October 2020 – describing machine learning models that claimed to be able to diagnose or prognosticate for COVID-19 from chest radiographs (CXR) and computed tomography (CT) images. Some of these papers had undergone the process of peer-review, while the majority had not.

Their search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in the systematic review. None of the 62 models was of potential clinical use, which is a major weakness, given the urgency with which validated COVID-19 models are needed. ̽��ֱ��results are reported in the journal Nature Machine Intelligence.

Machine learning is a promising and potentially powerful technique for detection and prognosis of disease. Machine learning methods, including where imaging and other data streams are combined with large electronic health databases, could enable a personalised approach to medicine through improved diagnosis and prediction of individual responses to therapies.

“However, any machine learning algorithm is only as good as the data it’s trained on,” said first author Dr Michael Roberts from Cambridge’s Department of Applied Mathematics and Theoretical Physics. “Especially for a brand-new disease like COVID-19, it’s vital that the training data is as diverse as possible because, as we’ve seen throughout this pandemic, there are many different factors that affect what the disease looks like and how it behaves.”

“ ̽��ֱ��international machine learning community went to enormous efforts to tackle the COVID-19 pandemic using machine learning,” said joint senior author Dr James Rudd, from Cambridge’s Department of Medicine. “These early studies show promise, but they suffer from a high prevalence of deficiencies in methodology and reporting, with none of the literature we reviewed reaching the threshold of robustness and reproducibility essential to support use in clinical practice.”

Many of the studies were hampered by issues with poor quality data, poor application of machine learning methodology, poor reproducibility, and biases in study design. For example, several training datasets used images from children for their ‘non-COVID-19’ data and images from adults for their COVID-19 data. “However, since children are far less likely to get COVID-19 than adults, all the machine learning model could usefully do was to tell the difference between children and adults, since including images from children made the model highly biased,” said Roberts.

Many of the machine learning models were trained on sample datasets that were too small to be effective. “In the early days of the pandemic, there was such a hunger for information, and some publications were no doubt rushed,” said Rudd. “But if you’re basing your model on data from a single hospital, it might not work on data from a hospital in the next town over: the data needs to be diverse and ideally international, or else you’re setting your machine learning model up to fail when it’s tested more widely.”

In many cases, the studies did not specify where their data had come from, or the models were trained and tested on the same data, or they were based on publicly available ‘Frankenstein datasets’ that had evolved and merged over time, making it impossible to reproduce the initial results.

Another widespread flaw in many of the studies was a lack of involvement from radiologists and clinicians. “Whether you’re using machine learning to predict the weather or how a disease might progress, it’s so important to make sure that different specialists are working together and speaking the same language, so the right problems can be focused on,” said Roberts.

Despite the flaws they found in the COVID-19 models, the researchers say that with some key modifications, machine learning can be a powerful tool in combatting the pandemic. For example, they caution against naive use of public datasets, which can lead to significant risks of bias. In addition, datasets should be diverse and of appropriate size to make the model useful for different demographic group and independent external datasets should be curated.

In addition to higher quality datasets, manuscripts with sufficient documentation to be reproducible and external validation are required to increase the likelihood of models being taken forward and integrated into future clinical trials to establish independent technical and clinical validation as well as cost-effectiveness.

Reference:
Michael Roberts et al. ‘Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans.’ Nature Machine Intelligence (2021). DOI: 10.1038/s42256-021-00307-0

Systematic review finds that machine learning models for detecting and diagnosing COVID-19 from medical images have major flaws and biases, making them unsuitable for use in patients. However, researchers have suggested ways to remedy the problem.

Any machine learning algorithm is only as good as the data it’s trained on

Michael Roberts

NIAID

Novel Coronavirus SARS-CoV-2

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. Images, including our videos, are Copyright © ̽��ֱ�� of Cambridge and licensors/contributors as identified. All rights reserved. We make our image and video content available in a number of ways – as here, on our main website under its Terms and conditions, and on a range of channels including social media that permit your use and sharing of our content under their respective Terms.

Yes

Licence type:

Attribution

Time travelling to the mother tongue

lw355 — Tue, 19 Jul 2016 08:00:51 +0000

No matter whether you speak English or Urdu, Waloon or Waziri, Portuguese or Persian, the roots of your language are the same. Proto-Indo-European (PIE) is the mother tongue – shared by several hundred contemporary languages, as well as many now extinct, and spoken by people who lived from about 6,000 to 3,500 BC on the steppes to the north of the Caspian Sea.

They left no written texts and although historical linguists have, since the 19th century, painstakingly reconstructed the language from daughter languages, the question of how it actually sounded was assumed to be permanently out of reach.

Now, researchers at the Universities of Cambridge and Oxford have developed a sound-based method to move back through the family tree of languages that stem from PIE. They can simulate how certain words would have sounded when they were spoken 8,000 years ago.

Remarkably, at the heart of the technology is the statistics of shape.

“Sounds have shape,” explains Professor John Aston, from Cambridge’s Statistical Laboratory. “As a word is uttered it vibrates air, and the shape of this soundwave can be measured and turned into a series of numbers. Once we have these stats, and the stats of another spoken word, we can start asking how similar they are and what it would take to shift from one to another.”

A word said in a certain language will have a different shape to the same word in another language, or an earlier language. ̽��ֱ��researchers can shift from one shape to another through a series of small changes in the statistics. “It’s more than an averaging process, it’s a continuum from one sound to the other,” adds Aston, who is funded by the Engineering and Physical Sciences Research Council (EPSRC). “At each stage, we can turn the shape back into sound to hear how the word has changed.”

Rather than reconstructing written forms of ancient words, the researchers triangulate backwards from contemporary and archival audio recordings to regenerate audible spoken forms from earlier points in the evolutionary tree. Using a relatively new field of shape-based mathematics, the researchers take the soundwave and visualise it as a spectrogram – basically an undulating three-dimensional surface that represents the shape of that sound – and then reshape the spectrogram along a trajectory ‘signposted’ by known sounds.

While Aston leads the team of statistician ‘shape-shifters’ in Cambridge, the acoustic-phonetic and linguistic expertise is provided by Professor John Coleman’s group in Oxford.

̽��ֱ��researchers are working on the words for numbers as these have the same meaning in any language. ̽��ֱ��longest path of development simulated so far goes backwards 8,000 years from English one to its PIE ancestor oinos, and likewise for other numerals. They have also ‘gone forwards’ from the PIE penkwe to the modern Greek pente, modern Welsh pimp and modern Englishfive, as well as simulating change from Modern English to Anglo-Saxon (or vice versa), and from Modern Romance languages back to Latin.

(Other audio demonstrations are available here)

“We’ve explicitly focused on reproducing sound changes and etymologies that the established analyses already suggest, rather than seeking to overturn them,” says Coleman, whose research was funded by the Arts and Humanities Research Council.

They have discovered words that appear to correctly ‘fall out’ of the continuum. “It’s pleasing, not because it overturns the received wisdom, but because it encourages us that we are getting something right, some of the time at least. And along the way there have also been a few surprises!” ̽��ֱ��method sometimes follows paths that do not seem to be etymologically correct, demonstrating that the method is scientifically testable and pointing to areas in which refinements are needed.

Remarkably, because the statistics describe the sound of an individual saying the word, the researchers are able to keep the characteristics of pitch and delivery the same. They can effectively turn the word spoken by someone in one language into what it would sound like if they were speaking fluently in another.

They can also extrapolate into the future, although with caveats, as Coleman describes: “If you just extrapolate linearly, you’ll reach a point at which the sound change hits the limit of what is a humanly reasonable sound. This has happened in some languages in the past with certain vowel sounds. But if you asked me what English will sound like in 300 years, my educated guess is that it will be hardly any different from today!”

For the team, the excitement of the research includes unearthing some gems of archival recordings of various languages that had been given up for dead, including an Old Prussian word last spoken by people in the early 1700s but ‘borrowed’ into Low Prussian and discovered in a German audio archive.

Their work has applications in automatic translation and film dubbing, as well as medical imaging (see panel), but the principal aim is for the technology to be used alongside traditional methods used by historical linguists to understand the process of language change over thousands of years.

“From my point of view, it’s amazing that we can turn exciting yet highly abstract statistical theory into something that really helps explain the roots of modern language,” says Aston.

“Now that we’ve developed many of the necessary technical methods for realising the extraordinary ambition of hearing ancient sounds once more,” adds Coleman, “these early successes are opening up a wide range of new questions, one of the central being how far back in time can we really go?”

Audio demonstrations are available here: www.phon.ox.ac.uk/jcoleman/ancient-sounds-audio.html

Inset image: Spectrograms showing how the shape of the sound of a word in one language can be morphed into the sound of the same word in another language; credit: John Aston.

̽��ֱ��sounds of languages that died thousands of years ago have been brought to life again through technology that uses statistics in a revolutionary new way.

As a word is uttered it vibrates air, and the shape of this soundwave can be measured and turned into a series of numbers

John Aston

Spectrogram showing the shape of the sound of a word

Medical imaging reshaped

̽��ֱ��statistics of shape are not just being used to show how different languages relate to each – they are also being used to improve the analysis of medical images.

Just as soundwaves have a shape that can be analysed using statistics, so do the patterns of neurons interacting with each other or the dimensions of the surface of a tumour. Now a new research Centre will develop tools that use the mathematics of the shapes found in medical images to improve diagnosis, prognosis and treatment planning for patients.

̽��ֱ��EPSRC Centre for Mathematical and Statistical Analysis of Multimodal Clinical Imaging, one of five ‘maths’ centres recently funded by £10 million from EPSRC, is co-led by Aston and Dr Carola-Bibiane Schönlieb from the Department of Applied Mathematics and Theoretical Physics in Cambridge.

“ ̽��ֱ��new methodologies will allow clinical medicine to move beyond one person reading single scans, to automated systems capable of analysing populations of images,” explains Schönlieb. “As a result, clinicians will have far greater scope to ask complex questions of the medical image.”

It’s already possible to extract statistical information from an image of a patient’s thigh bone, turn the data into a template for comparison with those from other people in the population, and then ask whether a particular shape of bone is more prone to being broken than others in the elderly.

Most organ scans split the image into many elements, which are then analysed voxel by voxel. “But complex structures like the heart and the brain should be analysed holistically,” explains Dr James Rudd, from the Department of Medicine, who leads the clinical interaction with the Centre. “ ̽��ֱ��tools we are developing will enable the analysis of organs like the brain as single objects with millions of connections.”

̽��ֱ��Centre brings together researchers and clinicians from applied and pure maths, engineering, physics, biology, oncology, clinical neuroscience and cardiology, and involves industrial partners Siemens, AstraZeneca, Microsoft, GSK and Cambridge Computed Imaging.

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. For image use please see separate credits above.

Yes

New research allows doctors to image dangerous ‘hardening’ of the arteries

cjb250 — Fri, 10 Jul 2015 08:21:49 +0000

̽��ֱ��technique, reported in the journal Nature Communications, could help in the diagnosis of these conditions in at-risk patients and in the development of new medicines.

Atherosclerosis – hardening of the arteries – is a potentially serious condition where arteries become clogged by a build-up of fatty deposits known as ‘plaques’. One of the key constituents in these deposits is calcium. In some people, pieces from the calcified artery can break away – if the artery supplies the brain or heart with blood, this can lead to stroke or heart attack.

“Hardening, or ‘furring’, of the arteries can lead to very serious disease, but it’s not clear why the plaques are stable in some people but unstable in others,” explains Professor David Newby, the BHF John Wheatley Professor of Cardiology at the Centre for Cardiovascular Science, ̽��ֱ�� of Edinburgh. “We need to find new methods of identifying those patients at greatest risk from unstable plaques.”

̽��ֱ��researchers injected patients with sodium fluoride that had been tagged with a tiny amount of a radioactive tracer. Using a combination of scanning techniques (positron emission tomography (PET) and computed tomography (CT)), the researchers were able to track the progress of the tracer as it moved around the body.

“Sodium fluoride is commonly found in toothpaste as it binds to calcium compounds in our teeth’s enamel,” says Dr Anthony Davenport from the Department of Experimental Medicine and Immunotherapeutics at the ̽��ֱ�� of Cambridge, who led the study. “In a similar way, it also binds to unstable areas of calcification in arteries and so we’re able to see, by measuring the levels of radioactivity, exactly where the deposits are building up. In fact, this new emerging technique is the only imaging platform that can non-invasively detect the early stages of calcification in unstable atherosclerosis.”

Following their sodium fluoride scans, the patients had surgery to remove calcified plaques and the extracted tissue was imaged, this time at higher resolution, using a laboratory PET/CT scanner and an electron microscope. This confirmed that the radiotracer accumulates in areas of active, unstable calcification whilst avoiding surrounding tissue.

Dr James Rudd, a cardiologist and researcher from the Department of Cardiovascular Medicine at the ̽��ֱ�� of Cambridge adds: “Sodium fluoride is a simple and inexpensive radiotracer that should revolutionise our ability to detect dangerous calcium in the arteries of the heart and brain. This will allow us to use current treatments more effectively, by giving them to those patients at highest risk. In addition, after further work, it may be possible to use this technique to test how well new medicines perform at preventing the development of atherosclerosis.”

̽��ֱ��Wellcome Trust provided the majority of support for this study, with additional contributions from the British Heart Foundation, Cancer Research UK and the Cambridge NIHR Biomedical Research Centre.

Reference
Irkle, A et al. Identifying active vascular microcalcification by 18F-sodium fluoride positron emission tomography. Nature Communications; 7 July 2015.

Researchers at the ̽��ֱ�� of Cambridge, in collaboration with the ̽��ֱ�� of Edinburgh, have shown how a radioactive agent developed in the 1960s to detect bone cancer can be re-purposed to highlight the build-up of unstable calcium deposits in arteries, a process that can cause heart attack and stroke.

Sodium fluoride is a simple and inexpensive radiotracer that should revolutionise our ability to detect dangerous calcium in the arteries of the heart and brain

James Rudd

̽��ֱ�� of Cambridge

Imaging atherosclerotic calcification or ‘hardening of the arteries’ using positron emission tomography

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. For image use please see separate credits above.

Yes

Licence type:

Attribution

Scientists develop new technique that could improve heart attack prediction

gm349 — Thu, 03 May 2012 14:59:30 +0000

Building on work pioneered in Cambridge 10 years ago, scientists have developed a new imaging approach that could help improve how doctors predict a patient’s risk of having a heart attack.

̽��ֱ��British Heart Foundation (BHF) funded project, a collaboration between scientists from the Universities of Cambridge and Edinburgh, is the first to demonstrate the potential of combined PET and CT imaging to highlight the disease processes causing heart attacks directly within the coronary arteries.

̽��ֱ��research, published last week in the Journal of the American College of Cardiology (JACC), involved imaging over 100 people with a CT calcium scan to measure the amount of calcified or hardened plaques in their coronary arteries. This is a standard test, which is commonly used to predict heart attack risk but cannot distinguish calcium that has been there for some time from calcium that is actively building up.

̽��ֱ��patients were also injected with two contrast agents that show up on PET imaging scans, and which can be used to track various metabolic pathways in the body. One of these tracers, 18F-sodium fluoride (18F-NaF), is a molecule taken up by cells in which active calcification is occurring. ̽��ֱ��18F-NaF can then be visualised and quantified during a PET scan.

̽��ֱ��researchers wanted to see if they could identify patients with active, ongoing calcification because these patients may be at higher risk of heart attack than patients in whom the calcium developed a long time ago. ̽��ֱ��results showed that increased 18F-NaF activity could be observed in specific coronary artery plaques in patients who had many other high-risk markers of cardiovascular disease.

Dr James Rudd, HEFCE Senior Lecturer at the Department of Medicine and joint senior author of the paper, said: “Our results show, for the first time, that certain areas of atherosclerosis within the coronary arteries, previously thought to be inert, are actually highly active and have the potential to cause heart attack. Once identified, they might be targeted with drug therapy more effectively.

“Additionally, we might be able to improve our ability to predict an individual person's future risk of heart attack using this fairly straightforward imaging test in selected people.

“This research exploits longstanding scientific links between my research team in Cambridge and Professor Newby's in Edinburgh, with core support from the Cambridge NIHR Biomedical Research Centre, HEFCE and the British Heart Foundation.”

Dr Marc Dweck, lead author on the research paper and a BHF Clinical Research Fellow at the ̽��ֱ�� of Edinburgh, said:

“Predicting heart attacks is very difficult and the methods we’ve got now are good but not perfect. Our new technique holds a lot of promise as a means of improving heart attack prediction although further ongoing work is needed before it becomes routine clinical practice.

“If we can identify patients at high risk of a heart attack earlier, we can then use intensive drug treatments, and perhaps procedures such as stents, to reduce the chances of them having a heart attack.”

Dr Shannon Amoils, Research Advisor at the (BHF), which funded the study, said:

“For decades cardiologists have been looking for ways to detect the high-risk plaques found in coronary arteries that could rupture to cause a heart attack, but it’s been difficult to develop a suitable imaging test that can focus in on these small vessels.

“This research is a technical tour de force as it allows us to assess active calcification happening right in the problem area – inside the wall of the coronary arteries and this active calcification may correlate with a higher risk of a heart attack.”

There are nearly 2.7 million people living with coronary heart disease (CHD) in the UK and it kills 88,000 people each year. Most of these deaths are caused by a heart attack. Each year there are around 124,000 heart attacks in the UK.

First use of PET and CT to look at disease processes leading to heart attack.

Our results show, for the first time, that certain areas of atherosclerosis within the coronary arteries, previously thought to be inert, are actually highly active and have the potential to cause heart attack. Once identified, they might be targeted with drug therapy more effectively.

Dr James Rudd, HEFCE Senior Lecturer at the Department of Medicine

Image courtesy of: James Rudd

From left to right, CT, PET and combined PET/CT images of the heart arteries.

This work is licensed under a Creative Commons Licence. If you use this content on your site please link back to this page.

Yes