̽��ֱ�� of Cambridge - Cambridge Open Access Service

Opinion: ̽��ֱ��science ‘reproducibility crisis’ – and what can be done about it

cjb250 — Mon, 20 Mar 2017 09:57:15 +0000

A survey by Nature revealed that 52% of researchers believed there was a “significant reproducibility crisis” and 38% said there was a “slight crisis”.

We asked three experts how they think the situation could be improved.

Open Research is the answer

Danny Kingsley, head of the Office of Scholarly Communication, ̽��ֱ�� of Cambridge

̽��ֱ��solution to the scientific reproducibility crisis is to move towards Open Research – the idea that scientific knowledge of all kinds should be openly shared as early as it is practical in the discovery process. We need to reward the publication of research outputs along the entire process, rather than just each journal article as it is published.

As well as other research outputs – such as data sets – we should reward research productivity itself as well as the thought process and planning behind the study. This is why Registered Reports was launched in 2013, where researchers register the proposal and how the research will be conducted, before any experimental work commences. It allows editorial decisions to be based on the rigour of the experimental design and increases the likelihood that the findings could be replicated.

In the UK there is now a requirement from most funders that the data underpinning a research publication is made available. However, although there are moves towards open research, many argue against the sharing of data among the research community.

Questionable findings are often hidden. Shutterstock

Researchers often write multiple papers from a single data set and many fear that if this data is released with the first publication then the researcher will be “scooped” by another research group, who will publish findings from similar data sets before the original authors get the chance to publish follow up articles – to gain maximum credit for the work. If the publication of data itself could be recorded as a “research output”, then being scooped would no longer be such an issue, as such credit will have been given.

One benefit of sharing data could be an improvement in its quality – as previous research has shown. And there have been small steps towards this goal, such as a standard method of citing data.

We also need to publish “null” results – those that do not support the hypothesis – to prevent other researchers wasting time repeating work. There are a few publication outlets for this, and a recent press release from ResearchGate indicated that it supports the sharing of failed experiments through its “project” offering. It lets users upload and track experiments as they are happening – meaning no one knows how they will turn out.

Psychology is leading the way out of crisis

Jim Grange, senior lecturer in psychology, Keele ̽��ֱ��

To me, it is clear that there is a reproducibility crisis in psychological science, and across all sciences. Murmurings of low reproducibility began in 2011 – the “year of horrors” for psychology – with a high profile fraud case. But since then, ̽��ֱ��Open Science Collaboration has published the findings of a large-scale effort to closely replicate 100 studies in psychology. Only 36% of them could be replicated.

̽��ֱ��incentive structures in universities and the attitude that you “publish or perish” means that researchers prioritise “getting it published” over “getting it right”. It also means that some, implicitly or explicitly, use questionable research practices to achieve publication. These may include failing to report parts of data sets or trying different analytical approaches to make the data fit what you want to say. It could also mean presenting exploratory research as though it was originally confirmatory (designed to test a specific hypothesis).

However, many psychology journals now recommend or require the preregistration of studies which allow researchers to detail their predictions, experimental protocols, and planned analytical strategy before data collection. This provides confidence to readers that no questionable research practices have occurred.

Erasing data: a questionable research practice. Shutterstock

Registered Reports has taken this further. But of course, once results are produced, isolated findings don’t mean much until they have been replicated.

I make efforts to replicate results before trying to publish and you’d be forgiven for thinking that replication attempts are common in science, but this is simply not the case. Journals seek novel theories and findings, and view replications as treading over old ground which offers little incentive for career-minded academics to conduct replications.

This has also led to the introduction of Registered Replication Reports in Perspectives on Psychological Science. This is where teams of researchers each follow identical procedures independently and aim to replicate important findings from the literature. A single paper then collates and analyses them to establish the size and reproducibility of the original study.

Although psychology is leading the way for improvements with these pioneering initiatives, it is certainly not out of the woods. But it has started to move beyond a crisis and make impressive strides – more disciplines need to follow suit.

This is a publication bias crisis

Ottoline Leyser, director of the Sainsbury Laboratory, ̽��ֱ�� of Cambridge

Reproducibility is a fundamental building block of science. If two people do the same experiment, they should get the same result. But there are many good reasons why two “identical” experiments might not give the same result such as unknown differences that have not been considered – and some exciting discoveries have been made this way.

So if a lack of reproducibility is itself not necessarily a problem, why is everybody talking about a crisis? In some cases poor practice and corner cutting have contributed to lack of reproducibility, and there have been some high profile cases of out and out fraud. It’s a major concern, but what is causing it?

In 2014 I chaired a project on the research culture in Britain for the Nuffield Council on bioethics, which was motivated by concerns about research integrity including over-claiming, rushing prematurely to publication and incorrect use of statistics. ̽��ֱ��main conclusions were that poor practice is incentivised by hyper-competition with overly narrow rules for winning.

There is an excessive focus on the publication of groundbreaking results in prestigious journals. But science cannot only be groundbreaking, as there is a lot of important digging to do after new discoveries – but there is not enough credit in the system for this work and it may remain unpublished because researchers prioritise their time on the eye-catching papers, hurriedly put together.

̽��ֱ��reproducibility crisis is actually a publication bias crisis which is driven by the reward structures in the research system. Various approaches have been suggested to address problems, such as pre-registration of experiments. However, the research landscape is highly diverse and this type of solution is only sensible for some research types. ̽��ֱ��most widely relevant solution is to change the reward structures. In the UK there is a major opportunity to do this by reforming the Research Excellence Framework (REF). Through the REF, public money is allocated to universities based on the “quality” of the four best research outputs, usually papers, produced by each of their principal investigators over approximately six years and it disproportionately rewards groundbreaking research.

We need reward for a portfolio of research outputs, including not only the headline grabbing results, but also confirmatory work and community data sharing, which are the hallmarks of a truly high quality research endeavour. This would go a long way to shifting the current destructive culture.

Ottoline Leyser, Director of the Sainsbury Laboratory, ̽��ֱ�� of Cambridge; Danny Kingsley, Head, Office of Scholarly Communication, ̽��ֱ�� of Cambridge, ̽��ֱ�� of Cambridge, and Jim Grange, Senior Lecturer in psychology, Keele ̽��ֱ��

This article was originally published on ̽��ֱ��Conversation. Read the original article.

Reproducibility is the idea that an experiment can be repeated by another scientist and they will get the same result. It is important to show that the claims of any experiment are true and for them to be useful for any further research. However, science appears to have an issue with reproducibility.

NIAID

Study of Human Immune Response to HIV

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. For image use please see separate credits above.

Yes

Licence type:

Attribution

Ten thousand reasons to celebrate Open Access at Cambridge

sjr81 — Mon, 03 Oct 2016 08:58:45 +0000

̽��ֱ��10,000th submission, reporting on the impact of eating a Mediterranean diet on the risk of developing cardiovascular disease in a UK population, was deposited by Signe Wulund at the MRC Epidemiology Unit, on behalf of Dr Nita Forouhi, Programme Leader in Nutritional Epidemiology at the MRC Epidemiology Unit, and several co-authors.

̽��ֱ��Open Access movement has been growing in strength in academia for many years, and it is increasingly being mandated by funding bodies and government.

Dr Forouhi said: “Through open access our research can reach a worldwide audience. It would be a huge pity if interested researchers, practitioners or policy makers could not read about new research, such as our latest findings on the link between the Mediterranean diet and cardiovascular health in a non-Mediterranean setting, because of something as simple as lacking a journal subscription.

“Open access enables wider dissemination of research findings, and in turn, facilitates better research and evidence-based policy and clinical practice.”

̽��ֱ��Cambridge Open Access Service was established within the ̽��ֱ�� Library in 2013 in response to Research Councils UK (RCUK) making Open Access mandatory for anyone accepting their funding. Many other major funders, including the Wellcome Trust, Cancer Research UK and the British Heart Foundation, have similar policies.

In 2014, the Higher Education Funding Council for England announced that Open Access would be compulsory for any article included in the next Research Excellence Framework (REF) exercise. This policy came into force on April 1, 2016, effectively meaning that all research in UK institutions now has to be made freely available.

Since its inception in 2013, the Open Access service has processed 10,000 manuscripts, across all ̽��ֱ�� faculties and departments and worked with 3,000 different members of staff. 6,000 of the papers were covered by the HEFCE open access policy; 4,000 acknowledged RCUK funding and 1,900 COAF (many papers fall into multiple categories, and some into none). More than £5.4 million of Open Access grants from funding bodies have also been distributed.

Meeting these requirements is a major task for the ̽��ֱ��, and one it has tried to make as simple as possible for researchers. Authors are simply required to upload their manuscript to www.openaccess.cam.ac.uk when it’s accepted for publication, and the Open Access team advise them on what they need to do to comply with funder requirements, eligibility for any funding body grants, and handle depositing the article into Apollo, the ̽��ֱ��’s institutional repository.

Ten thousand manuscripts have now been received in this way, and the vast majority of them have been able to be made Open Access, free for anyone who wants to read and benefit from them.

The 10,000th article: ‘Prospective association of the Mediterranean diet with cardiovascular disease incidence and mortality and its population impact in a non-Mediterranean population: the EPIC-Norfolk Study’ in BMC Medicine. [DOI:10.1186/s12916-016-0677-4] can be seen here: http://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-016-0677-4

̽��ֱ��Open Access team at the ̽��ֱ�� of Cambridge is part of the Office of Scholarly Communication (OSC), within the ̽��ֱ�� Library. As well as assisting researchers with Open Access and Open Data compliance, it advises on scholarly communication tools, techniques, policies and practices, and provides training. For more details, visit www.osc.cam.ac.uk.

̽��ֱ�� ̽��ֱ�� of Cambridge has received its 10,000th Open Access submission – highlighting its commitment to making research freely available to anybody who wants to access it, without publisher paywalls or expensive journal subscriptions.

Through open access our research can reach a worldwide audience.

Nita Forouhi

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. For image use please see separate credits above.

Yes