The slides to accompany a discussion session at the ESSA Summer School in Aberdeen, June 2019

1. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 1
the Pitfalls of ABM
– intro and some cases
Bruce Edmonds
Centre for Policy Modelling
Manchester Metropolitan University

2. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 2
The Approach
• Start with the purpose of the simulation – which
derives from how the simulation can be justified
• E.g. Predict something, support an explanation,
describe, explore theoretical consequences of
some assumptions, as an illustration of process, a
way of thinking about some phenomena or a way
of mediating between people
• Then think of the ways in which using an ABM for
this purpose might go wrong – e.g. how we might
fool ourselves trying to do this
• Then do activities that mitigate against these risks

3. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 3
The Schelling 2D Model of Racial
Segregation
• Sakoda/Schelling’s 2D Model of segregation
based on a checkerboard space, with agents
moving from space to space.
• This showed that a relatively low level of racial
intolerance could result in spatial segregation
• It was a counter-example to the natural
assumption that the observed segregation was
due to high racial intollerance
• It has been extended and interpreted in many
different ways since

4. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 4
An Opinion Dynamics Model
• Previous models showed how opinions converge
but not how they may polarise
• No evidence or data is applied, this is just an
exploration of the general outcomes of some
abstract mechanisms
• So the space of behaviours is thoroughly explored
with many thousands of runs
• The results are then characterised in general
terms
• Deffuant, G., et al. (2002) How can extremism
prevail? jasss.soc.surrey.ac.uk/5/4/1.html

5. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 5
Water Distribution in Bali
• Lansing & Kramer’s (1993) model of water
distribution in Bali, explained how the system of
water temples act to help enforce social norms
and facilitate a complicated series of negotiations
• The implemented the system of rivers and how
norms about negotiation about water allocation
are medicated by a system of water temples
• It explained how this social system was effective
in facilitating an equitable and sufficient allocation
of water

6. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 6
Predicting US Presidential Elections
• Nate Silver’s team tries to predict the outcome of
elections using computational models.
• The structure of the electoral college is built in
• Opinion polls from each area are fed into the
simulation plus some noise
• The simulation is then run lots of times to give a
distribution of outcomes
• From this distribution one can extract probabilities of
the different outcomes
• The predicted a 1/3 chance of Trump being elected
and got ALL of the electoral colleges right in
Obama’s election
• (http://fivethirtyeight.com and Silver 2013)

7. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 7
Evolution of Cooperation
• Axelrod’s (1984 etc.) facilitated simulations and
competitions where different interaction strategies
competed against each other in tournaments
• This was also done within an evolutionary
framework where winners tend to be reproduced
into the next generation, so more successful
strategies proliferate
• It was found that “tit for tat” did better than other
strategies in both cases
• This was interpreted as showing how cooperation
could evolve even when individuals behave
selfishly

8. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 8
A Socio-Ecological Test Bed
• In this a ecology of ~100 interacting species is
evolved ‘from scratch’ in a complex food web within a
2D grid of locations
• This starts from plants, then herbivores appear, then
predators etc. Then the state saved as a consistent
starting point for experiments
• The balance between species is constantly changing
as new species emerge and interact
• This is used to do an uncertainty/risk analysis – what
might happen if humans do X
• When something unexpected happens, one can ‘drill
down’ into the details to understand how it occurred
• This does not say what will happen, but shows a
range of possibilities

9. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 9
A Model of Domestic Water Demand
• This combined
– input of weather data (rainfall, sun, wind etc.)
– a social network of households
– each household has a different statistic of using
different appliances (frequency, amount of water etc.)
– which were differently selfish, social influenced or
influenced by official advice
– new water-spending/saving devices are introduced
– during droughts the authority advises reduce water use
• The idea was to help inform policy for water
companies and government as to future water
supply and reaction to droughts

10. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 10
Summary of Purposes, features and risks
Modelling
Purpose
Essential features Particular risks (apart from that of lacking
the essential features)
Prediction Anticipates unknown data Conditions of application unclear
Explanation Uses plausible mechanisms to match
outcome data in a well-defined
manner
Model is brittle, so minor changes in the set-up
result in bad fit to explained data; bugs in the code
Description Relates directly to evidence for a set
of cases
Unclear provenance;
over generalisation from cases described
Theoretical
exposition
Systematically maps out or
establishes the consequences of some
mechanisms
Bugs in the code;
inadequate coverage of possibilities
Illustration Shows an idea clearly as a particular
example
Over interpretation to make theoretical or empirical
claims; vagueness
Analogy Provides a wayof thinking
about something; gives insights
Taking it seriouslyfor anyother purpose
Social learning Facilitates communication or
agreement
Lack of engagement; confusion with
objective modelling

11. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 11
There are a whole lot of other things
that can go wrong when dealing with
the policy world!
For example:
• Promising what one can not deliver (e.g. prediction)
• Model spread – when models are used outside the
context they were originally designed for
• People believe the model too much based on surface
plausibility and animations
• Narrowing the evidence used, by focusing attention
on what can be modelled
• Takes focus away from a discussion of values
(see extra slides in “Pitfalls – resources.pptx” in folder)

12. Pitfalls of ABM - cases, Bruce Edmonds, ESSA Summer School, Aberdeen, June 2019. slide 12
The End
Bruce Edmonds: http://bruce.edmonds.name
Centre for Policy Modelling: http://cfpm.org
A version of these slides will be in the shared dropbox
folder and at:
http://slideshare.com/BruceEdmonds