̽��ֱ�� of Cambridge - policymaker

Opinion: the learning of scientific advisers is the other curve to consider

sc604 — Fri, 05 Jun 2020 08:14:14 +0000

Despite years of experience advising the government, Professor Neil Ferguson couldn’t have anticipated that his private life would become a matter of public scrutiny last month, essentially ending his formal relationship with government. No less surprising is the move from a former Government Chief Scientific Adviser, Professor Sir David King, to set up a so-called ‘Independent SAGE’ - laying bare the kinds of deliberations that would have taken place behind closed doors during his appointment.

̽��ֱ��bottom line for high-profile scientists and scientific advisers is that the rules of the game have changed. They may have learned that discretion is highly valued by policymakers, and yet, calls for transparency continue to resound louder than ever. How are scientists dealing with these new circumstances? What are they learning?

In a recent paper, I argue that too little attention has been given to how experts learn to advise policymakers. Although there is no shortage of guidelines and fragments of wisdom for researchers who want to see their work (or the work of colleagues) inform policymaking, scientific advice to governments is largely a case of learning on the job.

In their role as scientific advisers, experts learn what is and isn’t appropriate behaviour, what is and isn’t politically acceptable, and to draw the line where the science ‘ends’ and the politics ‘begin’. Scientific advice is a tricky balancing act between making expert judgments on the best available evidence and calibrating those judgments to the politics of the issues at hand. Like a tightrope walker, the scientific adviser has to learn to get the balance just right.

̽��ֱ��journey from full-time academic to part-time scientific adviser can be a transformative one. Researchers might initially set out with expectations of how scientists and policymakers interact and have had to revisit those expectations in view of their various encounters and experiences. While their learning may not always be transformative, I suggest that it is always necessarily situated: different organisations and environments will influence and shape their learning in different ways. This includes discussions with peers on scientific advisory committees, for example. How and what advisers learn, then, is never quite divorced from where they learn.

So why should we care? Taking the long view, I see three reasons why we might want to put advisers’ learning under the microscope:

Compiling the know-how of experienced advisers can be helpful for less experienced or early-career researchers who wish to engage with policymakers. Because there is no universal roadmap for success, I think we should focus on coming up with some ‘warning signs’ - as opposed to ‘direction signs’ - by identifying and communicating common pitfalls, for instance.
Given the situated nature of their own learning, advisers can directly contribute to the institutional learning and memory of the science-policy organisations they are part of. Involving committee members in decision-making can help prevent needless reinvention of the wheel and improve organisational reforms, leading to more sustainable change.
As both academics and policy advisers, scientific advisers are particularly well-placed to understand how academic research informs (or fails to inform) policymaking, as well as how the scientific community works and is governed. Therefore, they are knowledgeable not only about science for policy, but also about policy for science. For those reasons, I think that research funding organisations – such as UKRI research councils - should more systematically consult experienced science advisers in the formulation of their policies, especially in relation to research impact. For instance, improved impact evaluation frameworks would have positive trickle-down effects on the wider academic community, especially for early-career researchers who tend to base their understanding of impact in large part on the existing guidelines for grant applications or job descriptions.

̽��ֱ��core message is that as the nature of both science and policymaking continues to change, the learning experiences of expert advisers is an abundant resource that has yet to be tapped into. This has become all the more evident with COVID-19, as scientific advisers’ learning curves are likely to be steep. In the aftermath of the pandemic, we’ll need an evaluation of ‘what happened’ and ‘what went wrong’.

For the whole picture, we can’t just rely on the loudest or the most visible voices. We’ll need to turn to those scientific advisers whose stories go largely untold. Importantly, we’ll need to understand why the acquired skillset of scientific advisers may not be suited for crisis situations. Only then can we ensure that lessons are learned and that our networks of science advice are prepared for future emergencies.

Policymakers around the world are relying on the expertise of scientists to help make decisions around the COVID-19 pandemic. But how do scientists learn to advise policymakers? Noam Obermeister from Cambridge’s Department of Geography argues that this has been overlooked in the past, and suggests how studying their learning might help us prepare for future emergencies.

Photo by Obi Onyeador on Unsplash

Phone with news headlines

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. Images, including our videos, are Copyright © ̽��ֱ�� of Cambridge and licensors/contributors as identified. All rights reserved. We make our image and video content available in a number of ways – as here, on our main website under its Terms and conditions, and on a range of channels including social media that permit your use and sharing of our content under their respective Terms.

Yes

Study identifies key challenges when communicating potential policies

fpjl2 — Thu, 14 Jun 2018 13:00:00 +0000

Researchers have trawled through what little evidence currently exists on effectively communicating policy options, and point out four communication challenges that are problematic and often overlooked – yet should be required information for those making decisions that affect the lives of millions.

These include the need to highlight both the “winners and losers” of any policy decision, and to find ways of representing trade-offs between, say, financial and ecological or health outcomes. ̽��ֱ��findings are published today in the Springer Nature journal Palgrave Communications.

Recent decades have seen significant progress in producing information summaries that allow people to better understand how personal health choices affect their lives, say researchers.

However, they argue that similarly clear and concise materials are rarely available for legislators – and all of us citizens – on the potential outcomes of policies with stakes far beyond the individual.

Aiming to create a new science for communicating policy options, a team based at Cambridge’s Winton Centre for Risk and Evidence Communication point out the difficulty of finding the optimal balance between “comprehensibility and coverage” of policy options when informing decision-makers.

“Too much complexity risks a lack of understanding or simply being ignored. However, a brief and easy-to-digest communication may well lack the depth and detail necessary for making an informed decision,” said the Winton Centre’s Dr Cameron Brick, lead author of the new study.

He describes this as the “core tension” at the heart of communicating any policy option. “We certainly see this with Brexit, for example: oversimplifications that don’t provide the full story competing with dense explanations that people struggle to understand.”

“ ̽��ֱ��ideal communication would provide appropriate detail in a quickly and easily understood format to help citizens and policymakers apply their own values to decisions. We want to find out if there is a template that can help achieve this balance.”

In this first analysis from the recently established Winton Centre, Brick and colleagues reviewed policy communications across a wide variety of areas – from taxes to health, climate change and international trade – as well as guidance and evidence for communication effectiveness.

̽��ֱ��spectrum of material ranged from a fairly impenetrable seventy-page report on the possibilities for the Heathrow third runway to colourful postcards emblazoned with a single statistic. All were trying to be balanced sources of information to support decision-making, yet none appear to have checked what effect their presentation had on their readers.

Policy decisions have enormous impacts, and citizens and voters need trusted and balanced sources of evidence. However, the team found surprisingly little evidence on effectively communicating policy options.

By comparing materials designed to inform personal choices with those covering policy choices, they identified four main characteristics that make communicating potential policies particularly difficult and are often neglected.

Policies almost inevitably create winners and losers, because some groups – whether demographic or regional – become better off than others. It is difficult to summarise the effects on different groups so that audiences can weigh those outcomes.
Policies are full of trade-offs – e.g. as financial costs go up pollution goes down – yet each is measured differently. Presenting multiple outcomes with different metrics that allow for easy comparison is a tricky communications problem.
Individual choices rarely go beyond our own lifespans. Yet some policy choices can affect generations, and even have different effects as time goes on – another challenge for a quick summary to capture.
Expected policy outcomes come with particularly large uncertainties from complex shifts of future social and political events and therefore generally cannot be predicted confidently.

Brick and colleagues point out that including more detail in policy options exacerbates the tension between in-depth coverage of the issues on the one hand, and the ability of audiences to get the gist of the communications on the other – and yet nobody appears to have worked on finding the sweet spot between amount of detail and ease of understanding.

“There is no standard model yet for how to tackle these four challenges, but we hope communicators devise effective strategies as the research progresses,” said Brick.

“We want to try and define that Goldilocks zone between too much information and not enough so that policymakers can see when key information is missing, and people can make choices that fit their values.”

As part of the current study, they used three pieces of policy communication from major organisations such as the UK’s Education Endowment Foundation and the International Panel on Climate Change to illustrate attempts to provide nonpartisan and detailed policy option summaries.

Brick and colleagues will be building on this initial work by conducting rigorous research on policy communications material, including one-on-one surveying with various demographics, and large-scale data collection through online surveys.

Professor Sir David Spiegelhalter, Chairman of the Winton Centre, added: “At the Winton Centre, we are interested in helping people judge the benefits and harms of alternative policies or regulations that are being suggested.”

“ ̽��ֱ��idea of our Centre is to help communicate evidence in a way that is balanced, transparent and doesn’t try to coerce people into thinking or acting in a particular way.”

Read a blogpost on this study by lead author Dr Cameron Brick here.

Reference:
Cameron Brick et al. 'Winners and losers: communicating the potential impacts of policies.' Palgrave Communications (2018). DOI: 10.1057/s41599-018-0121-9

Cambridge researchers set out to define a new science for policy communications, with ambitions of finding the “Goldilocks zone” between too much and not enough information when informing both legislators and the public on complex issues.

Too much complexity risks a lack of understanding or simply being ignored. However, a brief and easy-to-digest communication may well lack the depth and detail necessary for making an informed decision.

Cameron Brick

Priscilla Du Preez

Unsplash

Yes

Using experts ‘inexpertly’ leads to policy failure, warn researchers

fpjl2 — Wed, 14 Oct 2015 17:14:26 +0000

̽��ֱ��accuracy and reliability of expert advice is often compromised by “cognitive frailties”, and needs to be interrogated with the same tenacity as research data to avoid weak and ill-informed policy, warn two leading risk analysis and conservation researchers in the journal Nature today.

While many governments aspire to evidence-based policy, the researchers say the evidence on experts themselves actually shows that they are highly susceptible to “subjective influences” – from individual values and mood, to whether they stand to gain or lose from a decision – and, while highly credible, experts often vastly overestimate their objectivity and the reliability of peers.

̽��ֱ��researchers caution that conventional approaches of informing policy by seeking advice from either well-regarded individuals or assembling expert panels needs to be balanced with methods that alleviate the effects of psychological and motivational bias.

They offer a straightforward framework for improving expert advice, and say that experts should provide and assess evidence on which decisions are made – but not advise decision makers directly, which can skew impartiality.

“We are not advocating replacing evidence with expert judgements, rather we suggest integrating and improving them,” write professors William Sutherland and Mark Burgman from the universities of Cambridge and Melbourne respectively.

“Policy makers use expert evidence as though it were data. So they should treat expert estimates with the same critical rigour that must be applied to data,” they write.

“Experts must be tested, their biases minimised, their accuracy improved, and their estimates validated with independent evidence. Put simply, experts should be held accountable for their opinions.”

Sutherland and Burgman point out that highly regarded experts are routinely shown to be no better than novices at making judgements.

However, several processes have been shown to improve performances across the spectrum, they say, such as ‘horizon scanning’ – identifying all possible changes and threats – and ‘solution scanning’ – listing all possible options, using both experts and evidence, to reduce the risk of overlooking valuable alternatives.

To get better answers from experts, they need better, more structured questions, say the authors. “A seemingly straightforward question, ‘How many diseased animals are there in the area?’ for example, could be interpreted very differently by different people. Does it include those that are infectious and those that have recovered? What about those yet to be identified?” said Sutherland, from Cambridge’s Department of Zoology.

“Structured question formats that extract upper and lower boundaries, degrees of confidence and force consideration of alternative theories are important for shoring against slides into group-think, or individuals getting ascribed greater credibility based on appearance or background,” he said.

When seeking expert advice, all parties must be clear about what they expect of each other, says Burgman, Director of the Centre of Excellence for Biosecurity Risk Analysis. “Are policy makers expecting estimates of facts, predictions of the outcome of events, or advice on the best course of action?”

“Properly managed, experts can help with estimates and predictions, but providing advice assumes the expert shares the same values and objectives as the decision makers. Experts need to stick to helping provide and assess evidence on which such decisions are made,” he said.

Sutherland and Burgman have created a framework of eight key ways to improve the advice of experts. These include using groups – not individuals – with diverse, carefully selected members well within their expertise areas.

They also caution against being bullied or “starstruck” by the over-assertive or heavyweight. “People who are less self-assured will seek information from a more diverse range of sources, and age, number of qualifications and years of experience do not explain an expert’s ability to predict future events – a finding that applies in studies from geopolitics to ecology,” said Sutherland.

Added Burgman: “Some experts are much better than others at estimation and prediction. However, the only way to tell a good expert from a poor one is to test them. Qualifications and experience don’t help to tell them apart.”

“ ̽��ֱ��cost of ignoring these techniques – of using experts inexpertly – is less accurate information and so more frequent, and more serious, policy failures,” write the researchers.

Evidence shows that experts are frequently fallible, say leading risk researchers, and policy makers should not act on expert advice without using rigorous methods that balance subjective distortions inherent in expert estimates.

̽��ֱ��cost of ignoring these techniques – of using experts inexpertly – is less accurate information and so more frequent, and more serious, policy failures

William Sutherland and Mark Burgman

Ross Mayfield

Experts Only

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. For image use please see separate credits above.

Yes

Licence type:

Attribution-Noncommerical

Anyone for digital democracy?

fpjl2 — Mon, 02 Mar 2015 12:23:18 +0000

̽��ֱ��recent release of the report from the Speaker’s Commission on Digital Democracy attracted significant attention but really only for one of the 26 recommendations. ̽��ֱ��report contained much that was good, some that was pedestrian, but it has all been swept away because one recommendation was of a different stripe altogether – to have online secure voting as an option in the UK by 2020.

Before getting into the detail of the report we should get one thing out of the way. This is a report that was long overdue, whether it came from the Speaker, a Select Committee or another part of the mangle that is government. With the speed that online has come to dominate our personal and professional interactions, it was a glaring omission that the UK government and the Parliament in particular had not addressed the digital world, their role in it and how it can be used to improve democratic outcomes. Granted the Government Digital Service (GDS) is working hard to improve how government services are accessed, although it is best to not get into an argument about whether gov.uk is an improvement or not (as Chris Cook did with his Christmas wish for 2014). However, anything that starts a stronger debate on how the machinery of government interacts with the public and policymaking is a good thing.

Let’s get the obvious out of the way first. Many of the recommendations are not specifically about digital, they are about democracy. “By 2020, the House of Commons should ensure that everyone can understand what it does.” You would have hoped that this would at least have been as aspiration for some time now. Reducing the amount of jargon, making procedures clearer, having a communications strategy – it is frustrating that we are still at such an early point in the evolution of the House and its relationship to the outside world that these actions are pending.

Efforts to improve public participation in policy making depends on people wishing to be involved, as well as assuming we have a shared model of how representation works. On the first point, our recent survey highlighted an expressed desire to be involved, which we included as part of our submission to the Commission. According to the poll while 7% of the public feel engaged in decision making in Parliament, 53% want to be involved. Now that number may be soft, as it is easy to say you want to be involved but actually being involved is another thing. Assuming half of those who express the desire to be involved would actually get involved, that’s a 19 point difference between the current state of affairs and what people want. If new digital tools, clearer websites and better outreach help in that regard, fantastic.

̽��ֱ��concern here is raising expectations that cannot be met. This is the case with recommendation 18 to have a ‘cyber chamber’ for public debate on issues being discussed in Parliament. ̽��ֱ��desire to be involved has been translated by some as ‘getting my way’. To caricature, when I tell my MP what should be done I expect that to happen. If we promote engagement such as that involved in a ‘cyber chamber’ without clarifying the process of discussion, negotiation and compromise that happens in almost all policy areas, then we’ll be raising expectations that will be dashed, potentially worsening the situation further.

This leads to the second point on representation – how do we think representation works in the UK at the national level? Crudely the options have been characterised as having delegates (MPs who take our views on board as issues arise), trustees (MPs who can use their judgement on an issue), or party (where the party line is all). Is the approach in the report to tilt towards a delegate model of representation? It’s unclear reading the report. Without discussing how the link between the public and their representatives is construed, the report missed an opportunity to have a more fundamental conversation about the structure of the UK’s democratic system.

And then there is recommendation 26. Once the Commission’s report was released the news picked up on this to the exclusion of pretty much everything else in the report. From the BBC to the newspapers, the response was to acknowledge some of the good things in the report, but to be critical of the need for online voting. Is online voting necessary? Will having online voting improve turnout or improve the connectivity between the public and MPs? ̽��ֱ��answer to both questions is that we don’t know. ̽��ֱ��Commission says it is “… confident that there is a substantial appetite for online voting in the UK …” but does not present any evidence to back this statement up. ̽��ֱ��claim is that voting will become more accessible, but there is no evidence that even if it is more accessible that it will increase the rates of voting, or the subsequent engagement or satisfaction of the public with politics.

Overall the Commission should be regarded as successful in getting a conversation going on the modernisation of Parliament and its procedures. Letting in the sunlight should help increase trust and hopefully improve the speed and quality of policy making. However, this does seem to be an opportunity missed to have a deeper conversation on the nature of representation in the UK, and in a report with too many recommendations a mistake to allow one recommendation to overshadow all of the others.

Finbarr Livesey provided a submission to the Digital Democracy Commission on public engagement with policy making which can be accessed here.

This post was originally published on the blog site for the Cambridge Masters in Public Policy. Read more here.

Dr Finbarr Livesey – ̽��ֱ�� lecturer and Deputy Director of the MPhil in Public Policy – submitted research to Parliament’s recent report on digital democracy. Here, he discusses the report’s implications for the democratic process in the UK.

While 7% of the public feel engaged in decision making in Parliament, 53% want to be involved

Finbarr Livesey

Barney Brown

Online voting

̽��ֱ��text in this work is licensed under a Creative Commons Licence. If you use this content on your site please link back to this page. For image rights, please see the credits associated with each individual image.

Yes

Twenty top tips for interpreting scientific claims

sj387 — Thu, 21 Nov 2013 07:58:54 +0000

Scientists from the UK and Australia, concerned with the lack of scientific knowledge amongst key decision makers, have created 20 concepts to help those who interact regularly with science and scientists.

Recent issues such as nuclear power, bee declines, and the role of badgers in bovine tuberculosis have seen fierce debates and policy decisions being made without the support of the scientific community, something Professors William Sutherland, David Spiegelhalter and Mark Burgman have set out to change.

These scientists want to help people grasp the “imperfect nature of science” and enable policy-makers to interrogate their advisers and experts instead of simply accepting information as it is given. Though change will take time, it is their belief that “a wider understanding of these 20 concepts by society would be a marked step forward”, and could only lead to a better-informed future.

Professor Spiegelhalter said “These tips could be used as a checklist when confronted with scientific claims. Science is not just a body of facts – it’s important to have a grasp of the process by which conclusions are drawn, and the possible pitfalls on that path”

There is an obvious need to make sure that scientific policy is based on a sound understanding of science; this means making sure that policy-makers know the right things to ask, and how to interpret the answers they get. There have been many suggestions of how to increase the level of scientific knowledge in the political community, from encouraging more scientists to become politicians, to expanding the role of chief scientific advisors. However, none of these solutions fully address the fundamental issue of widespread “scientific ignorance” amongst those who have the ability to vote in parliament.

̽��ֱ��homogeneity of policy makers’ backgrounds shows just how far-reaching this problem is. No member of the current British cabinet has a scientific degree – the closest is Vince Cable, who initially read Natural Sciences at Cambridge before switching to Economics. Six cabinet members read Politics, Philosophy and Economics (PPE) at Oxford, including Ed Davey MP, the Secretary of State for Energy and Climate change. Of Britain’s 650 members of parliament, only Cambridgeshire MP Julian Huppert is a scientist; David Willets MP, the Minister of State for Universities and Science, read PPE.

This list of concepts will teach skills closely related to those that politicians already have, and will help people “understand the quality, limitations and biases of evidence”. This will, in turn, allow better interrogation of those communicating scientific information. By explaining the scientific process, these academics have helped to demystify science and make it accessible to those creating the country’s scientific policy. It is not a question of turning every policy-maker into a scientist, but of arming them with the tools to understand and question the scientific information they receive.

Some of the concepts seem common sense (“Scientists are human”), others less so (“Regression to the mean can mislead”, “Beware the base rate fallacy”). All contain practical advice and recommendations that, if followed, should help policy-makers better interact with science and scientists and understand the limitations of evidence. Though the authors acknowledge that improvements in policy will not happen instantaneously, and that uncertainty is inherent in the scientific method, they nonetheless feel that these concepts are the first step to take if we are to more closely integrate science into political decision-making.

Click here to read the Nature article in full

Aiming to improve policy-makers’ understanding of the imperfect nature of science, academics from the Universities of Cambridge and Melbourne have created a list of concepts that they believe should be part of the education of civil servants, politicians, policy advisers and journalists

Science is not just a body of facts – it’s important to have a grasp of the process by which conclusions are drawn, and the possible pitfalls on that path

Professor David Spiegelhalter

Sir Cam/ ̽��ֱ�� of Cambridge

Dr Pete Wothers giving a chemistry demonstration to an audience at the Cambridge Science Festival including David Willetts, Minister for Universities and Science.

This work is licensed under a Creative Commons Licence. If you use this content on your site please link back to this page.

Yes