̽��ֱ�� of Cambridge - reproducibility

‘Robot scientist’ Eve finds that less than one third of scientific results are reproducible

sc604 — Tue, 05 Apr 2022 23:20:02 +0000

̽��ֱ��researchers, led by the ̽��ֱ�� of Cambridge, analysed more than 12,000 research papers on breast cancer cell biology. After narrowing the set down to 74 papers of high scientific interest, less than one-third – 22 papers – were found to be reproducible. In two cases, Eve was able to make serendipitous discoveries.

̽��ֱ��results, reported in the journal Royal Society Interface, demonstrate that it is possible to use robotics and artificial intelligence to help address the reproducibility crisis.

A successful experiment is one where another scientist, in a different laboratory under similar conditions, can achieve the same result. But more than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce some of their own experiments: this is the reproducibility crisis.

“Good science relies on results being reproducible: otherwise, the results are essentially meaningless,” said Professor Ross King from Cambridge’s Department of Chemical Engineering and Biotechnology, who led the research. “This is particularly critical in biomedicine: if I’m a patient and I read about a promising new potential treatment, but the results aren’t reproducible, how am I supposed to know what to believe? ̽��ֱ��result could be people losing trust in science.”

Several years ago, King developed the robot scientist Eve, a computer/robotic system that uses techniques from artificial intelligence (AI) to carry out scientific experiments.

“One of the big advantages of using machines to do science is they’re more precise and record details more exactly than a human can,” said King. “This makes them well-suited to the job of attempting to reproduce scientific results.”

As part of a project funded by DARPA, King and his colleagues from the UK, US and Sweden designed an experiment that uses a combination of AI and robotics to help address the reproducibility crisis, by getting computers to read scientific papers and understand them, and getting Eve to attempt to reproduce the experiments.

For the current paper, the team focused on cancer research. “ ̽��ֱ��cancer literature is enormous, but no one ever does the same thing twice, making reproducibility a huge issue,” said King, who also holds a position at Chalmers ̽��ֱ�� of Technology in Sweden. “Given the vast sums of money spent on cancer research, and the sheer number of people affected by cancer worldwide, it’s an area where we urgently need to improve reproducibility.”

From an initial set of more than 12,000 published scientific papers, the researchers used automated text mining techniques to extract statements related to a change in gene expression in response to drug treatment in breast cancer. From this set, 74 papers were selected.

Two different human teams used Eve and two breast cancer cell lines and attempted to reproduce the 74 results. Statistically significant evidence for repeatability was found for 43 papers, meaning that the results were replicable under identical conditions; and significant evidence for reproducibility or robustness was found in 22 papers, meaning the results were replicable by different scientists under similar conditions. In two cases, the automation made serendipitous discoveries.

While only 22 out of 74 papers were found to be reproducible in this experiment, the researchers say that this does not mean that the remaining papers are not scientifically reproducible or robust. “There are lots of reasons why a particular result may not be reproducible in another lab,” said King. “Cell lines can sometimes change their behaviour in different labs under different conditions, for instance. ̽��ֱ��most important difference we found was that it matters who does the experiment, because every person is different.”

King says that this work shows that automated and semi-automated techniques could be an important tool to help address the reproducibility crisis, and that reproducibility should become a standard part of the scientific process.

“It’s quite shocking how big of an issue reproducibility is in science, and it’s going to need a complete overhaul in the way that a lot of science is done,” said King. “We think that machines have a key role to play in helping to fix it.”

̽��ֱ��research was also funded by the Engineering and Physical Sciences Research Council (EPSRC), part of UK Research and Innovation (UKRI), and the Wallenberg AI, Autonomous Systems and Software Program (WASP)

Reference:
Katherine Roper et al. ‘Testing the reproducibility and robustness of the cancer biology literature by robot.’ Royal Society Interface (2022). DOI: 10.1098/rsif.2021.0821

Researchers have used a combination of automated text analysis and the ‘robot scientist’ Eve to semi-automate the process of reproducing research results. ̽��ֱ��problem of lack of reproducibility is one of the biggest crises facing modern science.

One of the big advantages of using machines to do science is they’re more precise and record details more exactly than a human can

Ross King

NIH Image Gallery

Breast Cancer Cell

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. Images, including our videos, are Copyright © ̽��ֱ�� of Cambridge and licensors/contributors as identified. All rights reserved. We make our image and video content available in a number of ways – as here, on our main website under its Terms and conditions, and on a range of channels including social media that permit your use and sharing of our content under their respective Terms.

Yes

Licence type:

Attribution-Noncommerical

Codecheck confirms reproducibility of COVID-19 model results

sc604 — Mon, 08 Jun 2020 23:00:01 +0000

̽��ֱ��code, script and documentation of the 16 March report, which is available on Github, was subject to an independent review led by Dr Stephen Eglen, from Cambridge’s Department of Applied Mathematics and Theoretical Physics.

Eglen co-founded Codecheck last year to help evaluate the computer programs behind scientific studies. Researchers provide their code and data to Codecheck, who run the code independently to ensure the work can be reproduced.

Last week, Codecheck certified the reproducibility of arguably the most talked-about computational model of the COVID-19 pandemic, that of the Imperial College group led by Professor Neil Ferguson. ̽��ֱ��model suggested that there could be up to half a million deaths in the UK if no measures were taken to slow the spread of the virus, and has been cited as one of the main reasons that lockdown went into effect soon after. However, the Imperial group did not immediately make their code publicly available.

Codecheck.org.uk provided an independent review of the replication of key findings from Report 9 using CovidSim reimplementation. ̽��ֱ��process matches domain expertise and technical skills, taking place as an open peer review. ̽��ֱ��reviewer conducts the codecheck and submits the resulting certificate as part of their review.

̽��ֱ��results confirm that the key finding of Report 9 - on the impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand - are reproducible. Eglen did not review the epidemiology that went into the Imperial model, however.

In his analysis, Dr Eglen said: “Each run generated a tab-delimited file in the output folder. Two R scripts provided by Prof Ferguson were used to summarise these runs into two summary files... These files were compared against the values generated by Prof Ferguson... ̽��ֱ��results were found to be identical. Inserting my results into his Excel spreadsheet generated the same pivot tables.”

̽��ֱ��codecheck found that: “Small variations (mostly under 5%) in the numbers were observed between Report 9 and our runs.” ̽��ֱ��codecheck confirmed the trends and findings of the original report.

Building in part on code originally developed, published and peer-reviewed in 2005 and 2006, the code used for Report 9 continues to be actively developed to allow examination of the wider range of control policies now being deployed as countries relax lockdown. ̽��ֱ��Imperial team is sharing the code to enhance transparency and to allow others to contribute and make use of the simulation.

Refactoring the code has allowed changes to be made more quickly and reliably, including incorporating new data that has become available as the pandemic has progressed.

In addition to the features presented in Imperial Report 9, further strategies can now be examined such as testing and contact tracing, which was not a UK policy option in March.

Users also now have the ability to vary intensity of interventions over time and to calibrate the model to country-specific epidemic data.

Adapted from a piece originally published on the Imperial College London website

How you can support Cambridge's COVID-19 research effort

Donate to support COVID-19 research at Cambridge

Cambridge researcher confirms reproducibility of high-profile Imperial College coronavirus computational model.

Photo by Christian Wiediger on Unsplash

Closeup of computer keyboard

Yes

Pilot programme encourages researchers to share the code behind their work

sc604 — Fri, 02 Jun 2017 07:30:00 +0000

A new pilot project, designed by a Cambridge researcher and supported by the Nature family of journals, will evaluate the value of sharing the code behind published research.

For years, scientists have discussed whether and how to share data from painstaking research and costly experiments. Some are further along in their efforts toward ‘open science’ than others: fields such as astronomy and oceanography, for example, involve such expensive and large-scale equipment and logistical challenges to data collection that collaboration among institutions has become the norm.

Recently, academic journals, including several Nature journals, are turning their attention to another aspect of the research process: computer programming code. Code is becoming increasingly important in research because scientists are often writing their own computer programs to interpret their data, rather than using commercial software packages. Some journals now include scientific data and code as part of the peer-review process.

Now, in a commentary published in the journal Nature Neuroscience, a group of researchers from the UK, Europe and the United States have argued that the sharing of code should be part of the peer-review process. In a separate editorial, the journal has announced a pilot project to ask future authors to make their code available for review.

Code is an important part of the research process, and often the only definitive account of how data were processed. “Methods are now so complex that they are difficult to describe concisely in the limited ‘methods’ section of a paper,” said Dr Stephen Eglen from Cambridge’s Department of Applied Mathematics and Theoretical Physics, and the paper’s lead author. “And having the code means that others have a better chance of replicating your work, and so should add confidence.”

Making the programs behind the research accessible allows other scientists to test the code and reproduce the computations in an experiment — in other words, to reproduce results and solidify findings. It’s the “how the sausage is made” part of research, said co-author Ben Marwick, from the ̽��ֱ�� of Washington. It also allows the code to be used by other researchers in new studies, making it easier for scientists to build on the work of their colleagues.

“What we’re missing is the convention of sharing code or the tools for turning data into useful discoveries or information,” said Marwick. “Researchers say it’s great to have the data available in a paper — increasingly raw data are available in supplementary files or specialised online repositories — but the code for performing the clever analyses in between the raw data and the published figures and tables are still inaccessible.”

Other Nature Research journals, such as Nature Methods and Nature Biotechnology, provide for code review as part of the article evaluation process. Since 2014, the company has encouraged writers to make their code available upon request.

̽��ֱ��Nature Neuroscience pilot focuses on three elements: whether the code supporting an author’s main claims is publicly accessible; whether the code functions without mistakes; and whether it produces the results cited. At the moment this is a pilot project to which authors can opt in. It may be that in future it becomes mandatory and only when the code has been reviewed will a paper then be accepted.

“This extra step in the peer review process is to encourage ‘replication’ of results, and therefore help reduce the ‘replication crisis’,” said Eglen. “It also means that readers can understand more fully what authors have done.”

An open science approach to sharing code is not without its critics, as well as scientists who raise legal and ethical questions about the repercussions. How do researchers get proper credit for the code they share? How should code be cited in the scholarly literature? How will it count toward tenure and promotion applications? How is sharing code compatible with patents and commercialization of software technology?

“We hope that when people do not share code it might be seen as ‘having something to hide,’ although people may regard the code as ‘theirs’ and their IP, rather than something to be shared,” said Eglen. “Nowadays, we believe the final paper is the ultimate representation of a piece of research, but actually the final paper is just an advert for the scholarship, which here is the computer code to solve a particular task. By sharing the code, we actually get the most useful part of the scholarship, rather than the paper, which is just the author’s ‘gloss’ on the work they have done.”

Adapted from a ̽��ֱ�� of Washington press release.

New project, partly designed by a ̽��ֱ�� of Cambridge researcher, aims to improve transparency in science by sharing ‘how the sausage is made’.

Having the code means that others have a better chance of replicating your work.

Stephen Eglen

Lorenzo Cafaro

Close up code

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. For image use please see separate credits above.

Yes

Opinion: ̽��ֱ��science ‘reproducibility crisis’ – and what can be done about it

cjb250 — Mon, 20 Mar 2017 09:57:15 +0000

A survey by Nature revealed that 52% of researchers believed there was a “significant reproducibility crisis” and 38% said there was a “slight crisis”.

We asked three experts how they think the situation could be improved.

Open Research is the answer

Danny Kingsley, head of the Office of Scholarly Communication, ̽��ֱ�� of Cambridge

̽��ֱ��solution to the scientific reproducibility crisis is to move towards Open Research – the idea that scientific knowledge of all kinds should be openly shared as early as it is practical in the discovery process. We need to reward the publication of research outputs along the entire process, rather than just each journal article as it is published.

As well as other research outputs – such as data sets – we should reward research productivity itself as well as the thought process and planning behind the study. This is why Registered Reports was launched in 2013, where researchers register the proposal and how the research will be conducted, before any experimental work commences. It allows editorial decisions to be based on the rigour of the experimental design and increases the likelihood that the findings could be replicated.

In the UK there is now a requirement from most funders that the data underpinning a research publication is made available. However, although there are moves towards open research, many argue against the sharing of data among the research community.

Questionable findings are often hidden. Shutterstock

Researchers often write multiple papers from a single data set and many fear that if this data is released with the first publication then the researcher will be “scooped” by another research group, who will publish findings from similar data sets before the original authors get the chance to publish follow up articles – to gain maximum credit for the work. If the publication of data itself could be recorded as a “research output”, then being scooped would no longer be such an issue, as such credit will have been given.

One benefit of sharing data could be an improvement in its quality – as previous research has shown. And there have been small steps towards this goal, such as a standard method of citing data.

We also need to publish “null” results – those that do not support the hypothesis – to prevent other researchers wasting time repeating work. There are a few publication outlets for this, and a recent press release from ResearchGate indicated that it supports the sharing of failed experiments through its “project” offering. It lets users upload and track experiments as they are happening – meaning no one knows how they will turn out.

Psychology is leading the way out of crisis

Jim Grange, senior lecturer in psychology, Keele ̽��ֱ��

To me, it is clear that there is a reproducibility crisis in psychological science, and across all sciences. Murmurings of low reproducibility began in 2011 – the “year of horrors” for psychology – with a high profile fraud case. But since then, ̽��ֱ��Open Science Collaboration has published the findings of a large-scale effort to closely replicate 100 studies in psychology. Only 36% of them could be replicated.

̽��ֱ��incentive structures in universities and the attitude that you “publish or perish” means that researchers prioritise “getting it published” over “getting it right”. It also means that some, implicitly or explicitly, use questionable research practices to achieve publication. These may include failing to report parts of data sets or trying different analytical approaches to make the data fit what you want to say. It could also mean presenting exploratory research as though it was originally confirmatory (designed to test a specific hypothesis).

However, many psychology journals now recommend or require the preregistration of studies which allow researchers to detail their predictions, experimental protocols, and planned analytical strategy before data collection. This provides confidence to readers that no questionable research practices have occurred.

Erasing data: a questionable research practice. Shutterstock

Registered Reports has taken this further. But of course, once results are produced, isolated findings don’t mean much until they have been replicated.

I make efforts to replicate results before trying to publish and you’d be forgiven for thinking that replication attempts are common in science, but this is simply not the case. Journals seek novel theories and findings, and view replications as treading over old ground which offers little incentive for career-minded academics to conduct replications.

This has also led to the introduction of Registered Replication Reports in Perspectives on Psychological Science. This is where teams of researchers each follow identical procedures independently and aim to replicate important findings from the literature. A single paper then collates and analyses them to establish the size and reproducibility of the original study.

Although psychology is leading the way for improvements with these pioneering initiatives, it is certainly not out of the woods. But it has started to move beyond a crisis and make impressive strides – more disciplines need to follow suit.

This is a publication bias crisis

Ottoline Leyser, director of the Sainsbury Laboratory, ̽��ֱ�� of Cambridge

Reproducibility is a fundamental building block of science. If two people do the same experiment, they should get the same result. But there are many good reasons why two “identical” experiments might not give the same result such as unknown differences that have not been considered – and some exciting discoveries have been made this way.

So if a lack of reproducibility is itself not necessarily a problem, why is everybody talking about a crisis? In some cases poor practice and corner cutting have contributed to lack of reproducibility, and there have been some high profile cases of out and out fraud. It’s a major concern, but what is causing it?

In 2014 I chaired a project on the research culture in Britain for the Nuffield Council on bioethics, which was motivated by concerns about research integrity including over-claiming, rushing prematurely to publication and incorrect use of statistics. ̽��ֱ��main conclusions were that poor practice is incentivised by hyper-competition with overly narrow rules for winning.

There is an excessive focus on the publication of groundbreaking results in prestigious journals. But science cannot only be groundbreaking, as there is a lot of important digging to do after new discoveries – but there is not enough credit in the system for this work and it may remain unpublished because researchers prioritise their time on the eye-catching papers, hurriedly put together.

̽��ֱ��reproducibility crisis is actually a publication bias crisis which is driven by the reward structures in the research system. Various approaches have been suggested to address problems, such as pre-registration of experiments. However, the research landscape is highly diverse and this type of solution is only sensible for some research types. ̽��ֱ��most widely relevant solution is to change the reward structures. In the UK there is a major opportunity to do this by reforming the Research Excellence Framework (REF). Through the REF, public money is allocated to universities based on the “quality” of the four best research outputs, usually papers, produced by each of their principal investigators over approximately six years and it disproportionately rewards groundbreaking research.

We need reward for a portfolio of research outputs, including not only the headline grabbing results, but also confirmatory work and community data sharing, which are the hallmarks of a truly high quality research endeavour. This would go a long way to shifting the current destructive culture.

Ottoline Leyser, Director of the Sainsbury Laboratory, ̽��ֱ�� of Cambridge; Danny Kingsley, Head, Office of Scholarly Communication, ̽��ֱ�� of Cambridge, ̽��ֱ�� of Cambridge, and Jim Grange, Senior Lecturer in psychology, Keele ̽��ֱ��

This article was originally published on ̽��ֱ��Conversation. Read the original article.

Reproducibility is the idea that an experiment can be repeated by another scientist and they will get the same result. It is important to show that the claims of any experiment are true and for them to be useful for any further research. However, science appears to have an issue with reproducibility.

NIAID

Study of Human Immune Response to HIV

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. For image use please see separate credits above.

Yes

Licence type:

Attribution