̽��ֱ�� of Cambridge - Gabriel Recchia

How accurate were early expert predictions on COVID-19, and how did they compare to the public?

sc604 — Wed, 05 May 2021 18:00:00 +0000

Researchers from the Winton Centre for Risk and Evidence Communication surveyed 140 UK experts and 2,086 UK laypersons in April 2020 and asked them to make four quantitative predictions about the impact of COVID-19 by the end of 2020. Participants were also asked to indicate confidence in their predictions by providing upper and lower bounds of where they were 75% sure that the true answer would fall - for example, a participant would say they were 75% sure that the total number of infections would be between 300,000 and 800,000.

̽��ֱ��results, published in the journal PLOS ONE, demonstrate the difficulty in predicting the course of the pandemic, especially in its early days. While only 44% of predictions from the expert group fell within their own 75% confidence ranges, the non-expert group fared far worse, with only 12% of predictions falling within their ranges. Even when the non-expert group was restricted to those with high numeracy scores, only 16% of predictions fell within the ranges of values that they were 75% sure would contain the true outcomes.

“Experts perhaps didn’t predict as accurately as we hoped they might, but the fact that they were far more accurate than the non-expert group reminds us that they have expertise that’s worth listening to,” said Dr Gabriel Recchia from the Winton Centre for Risk and Evidence Communication, the paper’s lead author. “Predicting the course of a brand-new disease like COVID-19 just a few months after it had first been identified is incredibly difficult, but the important thing is for experts to be able to acknowledge uncertainty and adapt their predictions as more data become available.”

Throughout the COVID-19 pandemic, social and traditional media have disseminated predictions from experts and non-experts about its expected magnitude.

Expert opinion is undoubtedly important in informing and advising those making individual and policy-level decisions. However, as the quality of expert intuition can vary drastically depending on the field of expertise and the type of judgment required, it is important to conduct domain-specific research to establish how good expert predictions really are, particularly in cases where they have the potential to shape public opinion or government policy.

“People mean different things by ‘expert’: these are not necessarily people working on COVID-19 or developing the models to inform the response,” said Recchia. “Many of the people approached to provide comment or make predictions have relevant expertise, but not necessarily the most relevant.” He noted that in the early COVID-19 pandemic, clinicians, epidemiologists, statisticians, and other individuals seen as experts by the media and the general public, were frequently asked to give off-the-cuff answers to questions about how bad the pandemic might get. “We wanted to test how accurate some of these predictions from people with this kind of expertise were, and importantly, see how they compared to the public.”

For the survey, participants were asked to predict how many people living in their country would have died and would have been infected by the end of 2020; they were also asked to predict infection fatality rates both for their country and worldwide.

Both the expert group and the non-expert group underestimated the total number of deaths and infections in the UK. ̽��ֱ��official UK death toll at 31 December was 75,346. ̽��ֱ��median prediction of the expert group was 30,000, while the median prediction for the non-expert group was 25,000.

For infection fatality rates, the median expert prediction was that 10 out of every 1,000 people with the virus worldwide would die from it, and 9.5 out of 1,000 people with the virus in the UK would die from it. ̽��ֱ��median non-expert response to the same questions was 50 out of 1,000 and 40 out of 1,000. ̽��ֱ��real infection fatality rate at the end of 2020—as best the researchers could determine, given the fact that the true number of infections remains difficult to estimate—was closer to 4.55 out of 1,000 worldwide and 11.8 out of 1,000 in the UK.

“There’s a temptation to look at any results that says experts are less accurate than we might hope and say we shouldn’t listen to them, but the fact that non-experts did so much worse shows that it remains important to listen to experts, as long as we keep in mind that what happens in the real world can surprise you,” said Recchia.

̽��ֱ��researchers caution that it is important to differentiate between research evaluating the forecasts of ‘experts’—individuals holding occupations or roles in subject-relevant fields, such as epidemiologists and statisticians—and research evaluating specific epidemiological models, although expert forecasts may well be informed by epidemiological models. Many COVID-19 models have been found to be reasonably accurate over the short term, but get less accurate as they try to predict outcomes further into the future.

Reference:
Gabriel Recchia, Alexandra L.J. Freeman, David Spiegelhalter. ‘How well did experts and laypeople forecast the size of the COVID-19 pandemic?’ PLOS ONE (2021). DOI: 10.1371/journal.pone.0250935

Who made more accurate predictions about the course of the COVID-19 pandemic – experts or the public? A study from the ̽��ֱ�� of Cambridge has found that experts such as epidemiologists and statisticians made far more accurate predictions than the public, but both groups substantially underestimated the true extent of the pandemic.

Predicting the course of a brand-new disease like COVID-19 just a few months after it had first been identified is incredibly difficult, but the important thing is for experts to be able to acknowledge uncertainty and adapt their predictions as more data become available.

Gabriel Recchia

NIH Image Gallery

Novel Coronavirus SARS-Cov-2

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. Images, including our videos, are Copyright © ̽��ֱ�� of Cambridge and licensors/contributors as identified. All rights reserved. We make our image and video content available in a number of ways – as here, on our main website under its Terms and conditions, and on a range of channels including social media that permit your use and sharing of our content under their respective Terms.

Yes

Licence type:

Public Domain

Opinion: Climate change, pandemics, biodiversity loss – no country is sufficiently prepared

Anonymous — Fri, 01 Nov 2019 14:53:32 +0000

There’s little that the left and the right agree on these days. But surely one thing is beyond question: that national governments must protect citizens from the gravest threats and risks they face. Although our government, wherever we are in the world, may not be able to save everyone from a pandemic or protect people and infrastructure from a devastating cyberattack, surely they have thought through these risks in advance and have well-funded, adequately practiced plans?

Unfortunately, the answer to this question is an emphatic no.

Not all policy areas are subject to this challenge. National defence establishments, for example, often have the frameworks and processes that facilitate policy decisions for extreme risks. But more often than not, and on more issues than not, governments fail to imagine how worst-case scenarios can come about – much less plan for them. Governments have never been able to divert significant attention from the here and happening to the future and uncertain.

A recent report published by Cambridge ̽��ֱ��’s Centre for the Study of Existential Risk argues that this needs to change. If even only one catastrophic risk manifests – whether through nature, accident or intention – it would harm human security, prosperity and potential on a scale never before seen in human history. There are concrete steps governments can take to address this, but they are currently being neglected.

̽��ֱ��risks that we face today are many and varied. They include:

Tipping points in the environmental system due to climate change or mass biodiversity loss.
Malicious, or accidentally harmful, use of artificial intelligence.
Malicious use of, or unintended consequences from, advanced biotechnologies.
A natural or engineered global pandemic.
Intentional, miscalculated, or accidental use of nuclear weapons.

Each of these global catastrophic risks could cause unprecedented harm. A pandemic, for example, could speed around our hyper-connected world, threatening hundreds of millions – potentially billions – of people. In this globalised world of just-in-time delivery and global supply chains, we are more vulnerable to disruption than ever before. And the secondary effects of instability, mass migration and unrest may be comparably destructive. If any of these events occurred, we would pass on a diminished, fearful and wounded world to our descendants.

So how did we come to be so woefully unprepared, and what, if anything, can our governments do to make us safer?

A modern problem

Dealing with catastrophic risks on a global scale is a particularly modern problem. ̽��ֱ��risks themselves are a result of modern trends in population, information, politics, warfare, technology, climate and environmental damage.

These risks are a problem for governments that are set up around traditional threats. Defence forces were built to protect from external menaces, mostly foreign invading forces. Domestic security agencies became increasingly significant in the 20th century, as threats to sovereignty and security – such as organised crime, domestic terrorism, extreme political ideologies and sophisticated espionage – increasingly came from inside national borders.

Unfortunately, these traditional threats are no longer the greatest concern today. Risks arising from the domains of technology, environment, biology and warfare don’t fall neatly into government’s view of the world. Instead, they are varied, global, complex and catastrophic.

As a result, these risks are currently not a priority for governments. Individually, they are quite unlikely. And such low-probability high-impact events are difficult to mobilise a response to. In addition, their unprecedented nature means we haven’t yet been taught a sharp lesson in the need to prepare for them. Many of the risks could take decades to arise, which conflicts with typical political time scales.

Governments, and the bureaucracies that support them, are not positioned to handle what’s coming. They don’t have the right incentives or skill sets to manage extreme risks, at least beyond natural disasters and military attacks. They are often stuck on old problems, and struggle to be agile to what’s new or emerging. Risk management as a practice is not a government’s strength. And technical expertise, especially on these challenging problem sets, tends to reside outside government.

Perhaps most troubling is the fact that any attempt to tackle these risks is not nationally confined: it would benefit everyone in the world – and indeed future generations. When the benefits are dispersed and the costs immediate, it is tempting to coast and hope others will pick up the slack.

Time to act

Despite these daunting challenges, governments have the capability and responsibility to increase national readiness for extreme events.

̽��ֱ��first step is for governments to improve their own understanding of the risks. Developing a better understanding of extreme risks is not as simple as conducting better analysis or more research. It requires a whole-of-government framework with explicit strategies for understanding the types of risks we face, as well as their causes, impacts, probabilities and time scales.

With this plan, governments can chart more secure and prosperous futures for their citizens, even if the most catastrophic possibilities never come to pass.

Governments around the world are already working towards improving their understanding of risk. For example, the United Kingdom is a world leader in applying an all-hazard national risk assessment process. This assessment ensures governments understand all the hazards – natural disasters, pandemics, cyber attacks, space weather, infrastructure collapse – that their country faces. It helps local first responders to prepare for the most damaging scenarios.

Finland’s Committee for the Future, meanwhile, is an example of a parliamentary select committee that injects a dose of much-needed long-term thinking into domestic policy. It acts as a think tank for futures, science and technology policy and provides advice on legislation coming forward that has an impact on Finland’s long-range future.

And Singapore’s Centre for Strategic Futures is leading in “horizon scanning”, a set of methods that helps people think about the future and potential scenarios. This is not prediction. It’s thinking about what might be coming around the corner, and using that knowledge to inform policy.

But these actions are few and far between.

We need all governments to put more energy towards understanding the risks, and acting on that knowledge. Some countries may even need grand changes to their political and economic systems, a level of change that typically only occurs after a catastrophe. We cannot – and do not have to – wait for these structural changes or for a global crisis. Forward-leaning leaders must act now to better understand the risks that their countries face.

Gabriel Recchia, Research Associate, Winton Centre for Risk and Evidence Communication, and Haydn Belfield, Research Associate, Centre for the Study of Existential Risk.

This article is republished from ̽��ֱ��Conversation under a Creative Commons license. Read the original article.

Two Cambridge risk researchers discuss how national governments are still stuck on "old problems", and run through the things that should be keeping our leaders awake at night.

Risks arising from the domains of technology, environment, biology and warfare don’t fall neatly into government’s view of the world

Gabriel Recchia and Haydn Belfield

Markus Spiske

Banner from a climate strike in Erlangen, Germany

Yes