̽��ֱ�� of Cambridge - Stephen Eglen

Codecheck confirms reproducibility of COVID-19 model results

sc604 — Mon, 08 Jun 2020 23:00:01 +0000

̽��ֱ��code, script and documentation of the 16 March report, which is available on Github, was subject to an independent review led by Dr Stephen Eglen, from Cambridge’s Department of Applied Mathematics and Theoretical Physics.

Eglen co-founded Codecheck last year to help evaluate the computer programs behind scientific studies. Researchers provide their code and data to Codecheck, who run the code independently to ensure the work can be reproduced.

Last week, Codecheck certified the reproducibility of arguably the most talked-about computational model of the COVID-19 pandemic, that of the Imperial College group led by Professor Neil Ferguson. ̽��ֱ��model suggested that there could be up to half a million deaths in the UK if no measures were taken to slow the spread of the virus, and has been cited as one of the main reasons that lockdown went into effect soon after. However, the Imperial group did not immediately make their code publicly available.

Codecheck.org.uk provided an independent review of the replication of key findings from Report 9 using CovidSim reimplementation. ̽��ֱ��process matches domain expertise and technical skills, taking place as an open peer review. ̽��ֱ��reviewer conducts the codecheck and submits the resulting certificate as part of their review.

̽��ֱ��results confirm that the key finding of Report 9 - on the impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand - are reproducible. Eglen did not review the epidemiology that went into the Imperial model, however.

In his analysis, Dr Eglen said: “Each run generated a tab-delimited file in the output folder. Two R scripts provided by Prof Ferguson were used to summarise these runs into two summary files... These files were compared against the values generated by Prof Ferguson... ̽��ֱ��results were found to be identical. Inserting my results into his Excel spreadsheet generated the same pivot tables.”

̽��ֱ��codecheck found that: “Small variations (mostly under 5%) in the numbers were observed between Report 9 and our runs.” ̽��ֱ��codecheck confirmed the trends and findings of the original report.

Building in part on code originally developed, published and peer-reviewed in 2005 and 2006, the code used for Report 9 continues to be actively developed to allow examination of the wider range of control policies now being deployed as countries relax lockdown. ̽��ֱ��Imperial team is sharing the code to enhance transparency and to allow others to contribute and make use of the simulation.

Refactoring the code has allowed changes to be made more quickly and reliably, including incorporating new data that has become available as the pandemic has progressed.

In addition to the features presented in Imperial Report 9, further strategies can now be examined such as testing and contact tracing, which was not a UK policy option in March.

Users also now have the ability to vary intensity of interventions over time and to calibrate the model to country-specific epidemic data.

Adapted from a piece originally published on the Imperial College London website

How you can support Cambridge's COVID-19 research effort

Donate to support COVID-19 research at Cambridge

Cambridge researcher confirms reproducibility of high-profile Imperial College coronavirus computational model.

Photo by Christian Wiediger on Unsplash

Closeup of computer keyboard

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. Images, including our videos, are Copyright © ̽��ֱ�� of Cambridge and licensors/contributors as identified. All rights reserved. We make our image and video content available in a number of ways – as here, on our main website under its Terms and conditions, and on a range of channels including social media that permit your use and sharing of our content under their respective Terms.

Yes

Pilot programme encourages researchers to share the code behind their work

sc604 — Fri, 02 Jun 2017 07:30:00 +0000

A new pilot project, designed by a Cambridge researcher and supported by the Nature family of journals, will evaluate the value of sharing the code behind published research.

For years, scientists have discussed whether and how to share data from painstaking research and costly experiments. Some are further along in their efforts toward ‘open science’ than others: fields such as astronomy and oceanography, for example, involve such expensive and large-scale equipment and logistical challenges to data collection that collaboration among institutions has become the norm.

Recently, academic journals, including several Nature journals, are turning their attention to another aspect of the research process: computer programming code. Code is becoming increasingly important in research because scientists are often writing their own computer programs to interpret their data, rather than using commercial software packages. Some journals now include scientific data and code as part of the peer-review process.

Now, in a commentary published in the journal Nature Neuroscience, a group of researchers from the UK, Europe and the United States have argued that the sharing of code should be part of the peer-review process. In a separate editorial, the journal has announced a pilot project to ask future authors to make their code available for review.

Code is an important part of the research process, and often the only definitive account of how data were processed. “Methods are now so complex that they are difficult to describe concisely in the limited ‘methods’ section of a paper,” said Dr Stephen Eglen from Cambridge’s Department of Applied Mathematics and Theoretical Physics, and the paper’s lead author. “And having the code means that others have a better chance of replicating your work, and so should add confidence.”

Making the programs behind the research accessible allows other scientists to test the code and reproduce the computations in an experiment — in other words, to reproduce results and solidify findings. It’s the “how the sausage is made” part of research, said co-author Ben Marwick, from the ̽��ֱ�� of Washington. It also allows the code to be used by other researchers in new studies, making it easier for scientists to build on the work of their colleagues.

“What we’re missing is the convention of sharing code or the tools for turning data into useful discoveries or information,” said Marwick. “Researchers say it’s great to have the data available in a paper — increasingly raw data are available in supplementary files or specialised online repositories — but the code for performing the clever analyses in between the raw data and the published figures and tables are still inaccessible.”

Other Nature Research journals, such as Nature Methods and Nature Biotechnology, provide for code review as part of the article evaluation process. Since 2014, the company has encouraged writers to make their code available upon request.

̽��ֱ��Nature Neuroscience pilot focuses on three elements: whether the code supporting an author’s main claims is publicly accessible; whether the code functions without mistakes; and whether it produces the results cited. At the moment this is a pilot project to which authors can opt in. It may be that in future it becomes mandatory and only when the code has been reviewed will a paper then be accepted.

“This extra step in the peer review process is to encourage ‘replication’ of results, and therefore help reduce the ‘replication crisis’,” said Eglen. “It also means that readers can understand more fully what authors have done.”

An open science approach to sharing code is not without its critics, as well as scientists who raise legal and ethical questions about the repercussions. How do researchers get proper credit for the code they share? How should code be cited in the scholarly literature? How will it count toward tenure and promotion applications? How is sharing code compatible with patents and commercialization of software technology?

“We hope that when people do not share code it might be seen as ‘having something to hide,’ although people may regard the code as ‘theirs’ and their IP, rather than something to be shared,” said Eglen. “Nowadays, we believe the final paper is the ultimate representation of a piece of research, but actually the final paper is just an advert for the scholarship, which here is the computer code to solve a particular task. By sharing the code, we actually get the most useful part of the scholarship, rather than the paper, which is just the author’s ‘gloss’ on the work they have done.”

Adapted from a ̽��ֱ�� of Washington press release.

New project, partly designed by a ̽��ֱ�� of Cambridge researcher, aims to improve transparency in science by sharing ‘how the sausage is made’.

Having the code means that others have a better chance of replicating your work.

Stephen Eglen

Lorenzo Cafaro

Close up code

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. For image use please see separate credits above.

Yes

̽��ֱ��amazing axon adventure

sc604 — Fri, 05 Feb 2016 09:45:21 +0000

To read these words, light is first refracted by the cornea, through the pupil in the iris and onto the lens, which focuses images onto the retina. ̽��ֱ��images are received by light-sensitive cells in the retina, which transmit impulses to the brain. These impulses are carried by a set of neurons called the retinal ganglion cells. Once the impulses reach the brain, the brain then has to piece together the information it receives into an understandable image. All of this happens in a fraction of a second.

Information travels from the retina to the brain via axons – the long, threadlike parts of neurons – sent out by the retinal ganglion cells. During embryonic development, axons are sent out to find their specific targets in the brain, so that images can be processed.

For an axon in a growing embryo, the journey from retina to brain is not a straightforward one. It’s a very long way for a tiny axon, through a constantly changing series of environments that it has never encountered before. So how do axons know where to go, and what can it tell us about how the brain is made and maintained?

Two ̽��ֱ�� of Cambridge researchers, Professor Christine Holt of the Department of Physiology, Development and Neuroscience, and Dr Stephen Eglen of the Department of Applied Mathematics and Theoretical Physics, are taking two different, but complementary, approaches to these questions.

With funding from the European Research Council and the Wellcome Trust, Holt’s research group is aiming to better understand the molecular and cellular mechanisms that guide and maintain axon growth, which in turn will aid better understanding of how nerve connections are first established.

“It’s an impressive navigational feat,” says Holt. “ ̽��ֱ��pathway between the retina and the brain may look homogeneous, but in reality it’s like a patchwork quilt of different molecular domains.”

On the pathway through this patchwork quilt, there is a set of distinct beacons, breaking the axon’s journey down into separate steps. Every time the growing axon reaches a new beacon, it has to make a decision about which way to go. At the tip of the axon is a growth cone, which ‘sniffs out’ certain chemical signals emitted from the beacons, helping it to steer in the right direction.

̽��ֱ��growth cones are receptive to certain signals and blind to others, so depending on what the axon encounters when it reaches a particular beacon, it will behave in a certain way. Holt’s research group uses a variety of techniques to determine what the signals are at the steering points where axons alter their direction of growth or their behaviour, such as the optic chiasm where certain axons cross to the opposite side of the brain, or at the point where they first leave the eye.

While Holt uses experiments to understand the development of the visual system, Eglen uses mathematical models as a complementary technique to try to answer the same questions.

“You’ve got much more freedom in a theoretical model than you do in an experiment,” he says. “A common experimental approach is to remove something genetically and see what happens. I think of that a little like taking the battery out of your car. Doing that will tell you that the battery is necessary for the car to function, but it doesn’t really tell you why.”

Theoretical models allow researchers to approach the questions around neural development from a different angle. To capture the essence of the neural system, they try to represent the building blocks of development and see what kind of behaviour would result.

But no model yet can fully capture the complexities of how the visual system develops, which Eglen views not only as a challenge for him as a mathematician, but also as a challenge back to the experimental community.

“It had been thought that if we built a model and took out all of the guidance molecules, there would be no topographic order whatsoever,” says Eglen. “But instead we found that there is still residual order in how the neurons are wired up, so there must be extra molecules or mechanisms that we don’t know about. What we’re trying to do is to take biology and put it into computers so that we can really test it.”

“In the past 15–20 years, there’s been a revolution in terms of being able to identify the specific molecules that act as guidance receptors or signals, but there’s still so much we don’t yet know, which is why we’re using both theoretical and experimental techniques to answer these questions,” says Holt. “And in addition to this question of wiring, we’re also looking at the problem of mapping – how do the terminal ends of the axons find their ultimate destination in the brain?”

Holt’s group has found that the same guidance molecule can have different roles depending on what aspect of growth is going on – but the question then becomes how do you wire the brain with so few molecules?

Adding to the complexity was another puzzling discovery – that the growth cones of axons can make proteins. Previous knowledge held that new proteins could be synthesised only within the main cellular part of each neuron, the cell body (where the nucleus is located), and then transported into axons. However, Holt’s group found that the growth cones of axons are also capable of synthesising proteins ‘on demand’ when they encounter new guidance beacons, suggesting that messenger RNA (mRNA) molecules play a role in helping axons to navigate to their correct destinations. mRNAs are the molecules from which new proteins are synthesised, and further experiments found that axons contain hundreds or even thousands of different types of this nuclear material.

In addition to their role in axon growth when the brain is wiring itself up during development, certain types of mRNA are also important in maintaining the connections in the adult brain, by keeping mitochondria – the energy-producing ‘batteries’ of cells – healthy, which, in turn, keeps axons healthy.

“It is a whole new view to the idea of degeneration in later life – a lot of different components have to work together to get local protein synthesis to work, so if just one of those components fails, degeneration can occur,” says Holt. “We’ve also found that many of the types of mRNA that are being translated in axons are the same ones that you see in diseases like Huntingdon’s and Parkinson’s, so basic knowledge of this sort is essential for the development of clinical therapies in nerve repair and for understanding these and other neurodegenerative disorders.”

How does the brain make connections, and how does it maintain them? Cambridge neuroscientists and mathematicians are using a variety of techniques to understand how the brain ‘wires up’, and what it might be able to tell us about degeneration in later life.

It’s an impressive navigational feat. ̽��ֱ��pathway between the retina and the brain may look homogeneous, but in reality it’s like a patchwork quilt.

Christine Holt

K-M. Leung

A growing axon tip exhibits polarised mRNA translation (red)

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. For image use please see separate credits above.

Yes

From Mexican wave to retinal wave: why sharing data is good for science

jfp40 — Mon, 07 Apr 2014 10:40:34 +0000

Now, researchers at Cambridge, York, Newcastle and Imperial College London have developed a system allowing neurophysiologists to share raw data with each other, something they hope will generate new discoveries in the field. ̽��ֱ��results are published in the journal GigaScience.

̽��ֱ��first type of data they collected and standardised are recordings of so called ‘retinal waves’. During early development, retinal neurons generate signals that rapidly spread across from one cell to another, much like a Mexican wave in a football stadium. These patterns of activity are thought to help forge the neural connections from the eye to the brain.

To record retinal waves, scientists use multielectrode arrays (tiny electrical devices). In this research, the team took 366 recordings from 12 different studies published between 1993 and 2014, converted them all to HDF5 – a standard open source format – and published them in a web-based ‘virtual laboratory’ called CARMEN.

According to lead author Dr Stephen Eglen from the Cambridge Computational Biology Institute: “Unlike other fields such as genomics, there hasn’t been much public data sharing in neuroscience, which could be because the data are heterogeneous and hard to annotate, or because researchers are reluctant to share data with a competitor.”

But Eglen believes there is much to be gained by a more cooperative approach. “There are two main benefits to sharing,” he said. “As well as leading to other collaborations and more interesting research, it also means that other people can check what you’ve done, which leads to more robust research. And if the taxpayer funds research, then I think it’s important for those results to be publicly available.”

CARMEN was a pilot project funded by the EPSRC, and is now supported by the BBSRC.

From the way we learn, to how our memories are made and stored, the workings of our brains depend on connections forged between billions of neurons, yet much about how our nervous system develops remains a mystery.

There are two main benefits to sharing. As well as leading to other collaborations and more interesting research, it also means that other people can check what you’ve done, which leads to more robust research.

Dr Stephen Eglen

Oyvind Solstad

Eye 9

This work is licensed under a Creative Commons Licence. If you use this content on your site please link back to this page.

Yes

Licence type:

Attribution