Risk-aware policy evaluation using agent-based simulation

Risk-aware policy evaluation using agent-based simulation, Bruce Edmonds, November 2016. slide 1
Risk-aware policy evaluation using
agent-based simulation
Bruce Edmonds
Centre for Policy Modelling
Manchester Metropolitan University

Simple systems…
… may be complicated but behave in predictable
ways, allowing them to be represented by models...
•  where one can use them to numerically forecast
•  where uncertainty can be analytically estimated
•  where one can get rough estimates cheaply, and
better estimates with increasing investment
•  which one can sensibly plan and execute
systematically
•  where there is a basically one right way of doing it
•  so that one can fully understand the model

However…
Even with only two bits of wood the result can be complex
See video at: http://www.youtube.com/watch?v=czLIj-4suOk

The Main Point of the Talk…
…is that complex systems need to be dealt with in a
different way to that of simple systems...
...not only using different techniques but also how
models about complex systems are used in policy
development process needs to change including
moving away from prediction.
•  Simulation modelling will be increasingly important
as we try to develop better policies and deal with
complex and fast moving situations
•  But it can not be ‘business as usual’ – just doing
better modelling with the same modeller–policy
actor relationship will not work well

Structure of the (rest of the) Talk
1.  A bit about modelling context, purposes
and tensions
2.  Some of the underlying assumptions
and habits that need to change
3.  An eample model – A model of
Domestic Water Demand
4.  An example model – Stefano Picascia’s
Modelling of the Housing Rental Market
5.  Some suggestions as to ways forward

Tensions and difficulties for the
modeller
Part 1

The Complexity facing Modellers
•  Many of the situations or issues we need to
understand are mixtures of: technical, social,
behavioural and ecological factors
•  They are not only complicated, but also
unexpected outcomes can ‘emerge’ from the
interaction of the actors and internal processes
•  We do not have good general models for how
people behave (regardless of what economists
claim)
•  How to approach using models to understand
complex phenomena is not fully developed

Different modelling purposes
Models can be used for a wide variety of different
purposes, and these impact upon the kind of
techniques needed and its difficulties, e.g.
•  Forecasting – predicting unknown (e.g. future)
situations and outcomes
•  Explanation – understanding how known
outcomes might have come about
•  Theoretical Exploration – understanding a
complex model by exploring some of its properties
and behaviours
•  Analogy – using a model as a way of thinking
about something else

Model Scope
•  The scope of a model is the conditions under which it
is useful for its planned purpose
•  Whilst this is implicit and stable for many simple
systems, this is not the case for many complex ones
•  Thus trying to make scope explicit is important, and
these relate to model assumptions
•  A process not included in the model (and hence
outside its scope) can overwhelm the results…
•  ..but in complex systems internal processes of change
can also emerge, and some of these can be usefully
modelled (but only in more complex ways)

Possible modelling trade-offs
•  Some desiderata for
models: validity,
formality, simplicity and
generality
•  these are difficult to
obtain simultaneously
(for complex systems)
•  there is some sort of
complicated trade-off
between them (for each
modelling exercise)
simplicity
generality
validity
formality
Analogy
Solvable
Mathematical
Model
Data
What
Policy
Actors
Want

A picture of modelling
whatisobservedor
measured
themodel
themodellers
themodelusers

Assumptions and expectations from
Policy Actors
Part 2

Expectations of Scientists
•  What works well with simple systems does not
necessarily work well with complex ones
•  Many of the expectations of complexity scientists
by policy makers and the public come from:
–  What economists have claimed to be able to do
–  Or how physical scientists have been able to do
•  As I hope will be clear, complex simulation
modelling can usefully inform policy making
•  But these expectations can get in the way
•  So we will look next at some of these expectations

The Cost-Benefit Approach
•  Basically weighing the benefits – the costs
•  As if an economist had written a manual for policy
actors in how to think (i.e. as their theory states)
This assumes that one can:
1.  list the main alternative options
2.  forecast the results of these
3.  put meaningful numerical values on these
4.  decide on the best one, adopt that option
•  Allows policy optimisation…
•  ...if it were possible

Quantification
•  Makes life much easier for policy actors – choose
the one with the biggest (or smallest) number!
•  Especially when asked to justify an approach
•  But can be more misleading than helpful because
it gives a false impression of accuracy
•  And implicitly leads to a focus on the measurable
and that things will ‘average out’ etc.
•  Was a limitation of purely mathematical
approaches, but computer simulation does not
have to be focused on these aspects
•  1D quantification is often an inadequate
representation of what we need to understand

Planning and Managing Modelling
•  In a simple case one can apply an approach
where one carefully plans, manages and
evaluates models
•  As if this was like building a bridge!
•  But in complex cases complications about what
needs to be included or not requires a more
iterative approach…
•  ...where models are repeatedly built for a purpose
and the lessons learnt as you go along...
•  Becuase the difficulties can not be predicted in
complex cases!

No gradual approximation, but
scope-limited usefulness
It is often assumed that as time and effort increase
the accuracy of the results improve, but this is not
the case with complex systems and models
Rather in order for the outcomes to be within scope
enough iterative development has to occur
Before this the results are worse than nothing
Time and cost
Error

Compartmentalism
•  That some problems can be separated into
smaller sub-problems which can be modelled
more simply
•  Not true in many complex cases, where the scope
of modelling is dependent on having enough of
the key processes represented
•  Sometimes several different modelling
approaches with different (but overlapping)
assumptions can be more helpful
•  Just fiddling, incrementally expanding an existing
(and failing) model will probably not help here

An Example: A model of Domestic Water
Demand
Part 3

Context of model
•  As part of a broader model which sought to
understand the impact of climate change on the
domestic demand for water in the UK
•  For the UK government and water companies
•  Looked at the impact of some present and
extrapolated weather patterns under four different
future economic/cultural scenarios
•  Included sophisticated statistical models for
prediction of demand
•  Plus our agent-based model as a contrasting
approach

Monthly Water Consumption
REL_CHNG
.88
.75
.63
.50
.38
.25
.13
0.00
-.13
-.25
-.38
-.50
20
10
0
Std. Dev = .17
Mean = .01
N = 81.00

Relative Change in Monthly
Consumption in a small village
Date
FEB
2001
SEP
2000
APR
2000
N
O
V
1999
JU
N
1999
JAN
1999
AU
G
1998
M
AR
1998
O
C
T
1997
M
AY
1997
D
EC
1996
JU
L
1996
FEB
1996
SEP
1995
APR
1995
N
O
V
1994
JU
N
1994
REL_CHNG
1.0
.8
.6
.4
.2
-.0
-.2
-.4
-.6

Purpose of the Model
•  Not long-term prediction
•  But to begin to understand the relationship of
socially-influenced consumer behaviour to
patterns of water demand
•  By producing a representational agent model
amenable to fine-grained criticism
•  And hence to suggest possible interactions and
outcomes

Model Structure - Overall Structure
• Activity
• Frequency
• Volume
Households
Policy
Agent
• Temperature
• Rainfall
• Sunshine
Ground
Aggregate Demand

Model Structure - Microcomponents
•  Each household has a variable number of micro-
components (power showers etc.): bath
other_garden_watering shower hand_dishwashing
washing_machine sprinkler clothes_hand_washing
hand_dishwashing toilets sprinkler power_shower
•  Actions are expressed by the frequency and
volume of use of each microcomponent
•  Actions-Volume-Frequency distribution in model
calibrated by data from the Three Valleys

Model Structure - Household
Distribution
•  Households distributed randomly on a grid
•  Each household can copy from a set of
neighbours (those within a certain distance )
•  Households have different mixtures of
motivations: self, social, global
•  They decide which is the neighbour most similar
to themselves – this is the one they are most likely
to copy – but all neighbours have some influence
•  Depending on their evaluation of actions they
might adopt that neighbour’s actions
•  Or do the action they are used to (habit)
•  Or that suggested by the policy agent

An Example Social Structure (main
influence only)
- Global Biased
- Locally Biased
- Self Biased

Household Behaviour -
Endorsements
•  Action Endorsements: recentAction neighbourhoodSourced
selfSourced globallySourced newAppliance
bestEndorsedNeighbourSourced
•  3 Weights moderate effective strengths of
neighbourhoodSourced selfSourced globallySourced
endorsements and hence the bias of households
•  Can be summarised as 3 types of households
influenced in different ways: global-;
neighbourhood-; and self-sourced depending on
the dominant weight (though this is a
simplification, all three weights and factors can
play a part)

History of a particular action
from one agent’s point of view
Month 1: action 1330 used, endorsed as self sourced
Month 2: action 1330 endorsed as recent (from personal use) and
neighbour sourced (used by agent 27) and self sourced
(remembered)
Month 3: action 1330 endorsed as recent (from personal use) and
neighbour sourced (agent 27 in month 2).
Month 4: action 1330 endorsed as neighbour sourced twice, used by
agents 26 and 27 in month 3, also recent
Month 5: action 1330 endorsed as neighbour sourced (agent 26 in month
4), also recent
Month 6: action 1330 endorsed as neighbour sourced (agent 26 in month
5)
Month 7: replaced by action 8472 (appeared in month 5 as neighbour
sourced, now endorsed 4 times, including by the most alike
neighbour – agent 50)

Policy Agent - Behaviour
•  After the first month of dry conditions, suggests
AFV actions to all households (reducing water
usage)
•  These actions are then included in the list of those
considered by the households
•  If the household’s weights predispose it, it may
decide to adopt these actions
•  Some other neighbours might imitate these
actions etc.
•  Others, more self-sourced may not be influenced

Number of consecutive dry months in
historical scenario
0
1
2
3
4
5
6
7
8
9
J-73
J-74
J-75
J-76
J-77
J-78
J-79
J-80
J-81
J-82
J-83
J-84
J-85
J-86
J-87
J-88
J-89
J-90
J-91
J-92
J-93
J-94
J-95
J-96
J-97
Simulation Date
Numberofconsequativedrymonths

Simulated Monthly Water
Consumption
REL_CHNG
.075
.063
.050
.037
.025
.012
-.000
-.013
-.025
-.038
-.050
120
100
80
60
40
20
0
Std. Dev = .01
Mean = -.000
N = 325.00

Monthly Water Consumption (again)
REL_CHNG
.88
.75
.63
.50
.38
.25
.13
0.00
-.13
-.25
-.38
-.50
20
10
0
Std. Dev = .17
Mean = .01
N = 81.00

Simulated Change in Monthly
Consumption
Date
SEP
1997
APR
1996
N
O
V
1994
JU
N
1993
JAN
1992
AU
G
1990
M
AR
1989
O
C
T
1987
M
AY
1986
D
EC
1984
JU
L
1983
FEB
1982
SEP
1980
APR
1979
N
O
V
1977
JU
N
1976
JAN
1975
AU
G
1973
M
AR
1972
O
C
T
1970
REL_CHNG
.10
.08
.06
.04
.02
0.00
-.02
-.04
-.06

Relative Change in Monthly
Consumption (again)
Date
FEB
2001
SEP
2000
APR
2000
N
O
V
1999
JU
N
1999
JAN
1999
AU
G
1998
M
AR
1998
O
C
T
1997
M
AY
1997
D
EC
1996
JU
L
1996
FEB
1996
SEP
1995
APR
1995
N
O
V
1994
JU
N
1994
REL_CHNG
1.0
.8
.6
.4
.2
-.0
-.2
-.4
-.6

30% Neigh. biased, historical
scenario, historical innov. datesAggregate demand series scaled so 1973=100
0
20
40
60
80
100
120
140
160
180
200
J-
73
J-
74
J-
75
J-
76
J-
77
J-
78
J-
79
J-
80
J-
81
J-
82
J-
83
J-
84
J-
85
J-
86
J-
87
J-
88
J-
89
J-
90
J-
91
J-
92
J-
93
J-
94
J-
95
J-
96
J-
97
Simulation Date
RelativeDemand

80% Neigh. biased, historical
0
20
40
60
80
100
120
140
160
180
200
J-
73
J-
74
J-
75
J-
76
J-
77
J-
78
J-
79
J-
80
J-
81
J-
82
J-
83
J-
84
J-
85
J-
86
J-
87
J-
88
J-
89
J-
90
J-
91
J-
92
J-
93
J-
94
J-
95
J-
96
J-
97
Simulation Date
RelativeDemand

80% Neigh. biased, medium-high
0
20
40
60
80
100
120
140
160
180
200
Jan-
73
Jan-
74
Jan-
75
Jan-
76
Jan-
77
Jan-
78
Jan-
79
Jan-
80
Jan-
81
Jan-
82
Jan-
83
Jan-
84
Jan-
85
Jan-
86
Jan-
87
Jan-
88
Jan-
89
Jan-
90
Jan-
91
Jan-
92
Jan-
93
Jan-
94
Jan-
95
Jan-
96
Jan-
97
Simulation Date
RelativeDemand

What did the model tell us?
•  That it is possible that social processes within
communities:
–  can cause a high and unpredictable variety in patterns
of demand
–  can ‘lock-in’ behavioural patterns and partially ‘insulate’
them from outside influence (droughts only occasionally
had a permanent affect on patterns of consumption)
•  Thus identifying and taking measures at high-
usage areas at an early stage might be sensible
•  Also that the availability of new products could
dominate effects from changing consumptions
habits

An Example: A Model of the Rental
Housing Market
Part 4

The model
•  By Stefano Picascia, an PhD student of mine, now
at Sienna University, Italy
•  Is an agent-based simulation that represents both
tenants and developers co-adapting
•  Is geographically based with tenants making
decisions as where to move to based on location
as well as quality of housing and price
•  Developers put in captial to build/rennovate
housing for tenants
•  Rents are determined by the quality and prices of
surrounding housing

The Manchester Case
Waves of price
changes can
spread
Can have different
outcomes each
time it is run
Has also been
applied to London
and Beirut

Average prices in a run

Different Sectors of the City in a run

What it does and does not tell us
In the model (which is the private rental sector only):
•  That change is fundamentally internally driven as well
as due to outside events
•  Price oscillations are endemic to the system
•  That some regions of cities will be stuck as low quality
housing for long periods of time depending on the
state of neighbouring areas
•  The very high price regions stay that way
•  That under certain conditions sudden ‘gentrification’
may occur to some degree raising standards but
maybe also displacing existing functional communities
•  For poorer districts decline is gradual and continual
between any such periods

Concluding discussion and some ways
forward
Part 5

From Probabilistic to Possibilistic
•  When outcomes can not be sensibly forecast…
•  And especially numerically forecast…
•  …where even probability zones or 90% bounds
are misleading
•  Then moving to an approach that models and
understand (more of) underlying processes...
•  ...in terms of the different kinds of outcome might
be much more informative
•  Each outcome tagged with its own assumptions
and scopes (if they differ)

From Forecasting to Risk Analysis
•  However much one might like forecasting, often it is simply
not possible…
•  ...let alone in a way such that the outcomes from different
options can be compared!
•  Predicting outcomes can be more misleading than helpful
•  Rather it may be more approapriate to use models for risk
analysis – finding all the ways a policy might go wrong (or
right!)
•  Techniques are available to help discover and understand
how endogenous processes might result in different future
possibilities
•  Which can then inform the design of ‘early warning’
monitors giving the most immediate feedback to policy
makers

Informing the adaptive ‘driving’ of
policy
•  Complex models are no good for policy makers!
•  Because they have to make decisions on grounds
they understand and know the reliability of
•  They can not (and should not) delegate this to
‘experts’ and their inscrutable models
•  Rather modellers should use their modelling to
understand the key emergent kinds of outcome
•  To inform:
–  the consideration of these kinds of outcome
–  the design of appropriate data visalisations
–  the design of ‘earl warning indicators
•  …So that policy can adapt to changing trends and
events as quickly and fluidly as possible

Conclusions
•  Modelling of complex phenomena is not cheap or
quick and requires iterative development
•  It will not forecast the impact of potential policies
or events, but can anticipate possible future
outcomes in a way intuition can not
•  There will always be a ‘scope’ – a set of
conditions/assumptions a model depends upon
•  But a good model can repay its investment in
terms of cost and improving people’s lives many,
many times over

Summary
It is no good wishing that the world or
modelling is simple and trying to ‘force’ it to
be so, one has to adapt to suit reality…
…this includes how models and modelling
are used by the policy process

The End
The Centre for Policy Modelling:
http://cfpm.org
These slides will be available at: http://slideshare.net/BruceEdmonds
Stefano’s model of
housing was
developed under this
project, funded by the
EPSRC, grant
number EP/H02171X
Social Science Aspects
of Fisheries for the 21st
Century – with two
Icelandic partners:
MATIS and the
University of Iceland

Risk-aware policy evaluation using agent-based simulation

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (19)

Semelhante a Risk-aware policy evaluation using agent-based simulation

Semelhante a Risk-aware policy evaluation using agent-based simulation (20)

Mais de Bruce Edmonds

Mais de Bruce Edmonds (15)

Último

Último (20)

Risk-aware policy evaluation using agent-based simulation