My presentation held at the 1st European Conference on Political Attitudes and Mentalities (ECPAM 2012) conference, Bucharest, Romania, September 3-5, 2012.
Electronic paper link:
http://mass.aitia.ai/images/publikaciok/2012-ecpam-replication_case_studies-camera_ready.pdf
Abstract: This paper examines model replication in the context of agent-based simulation through two case studies. Replication of a computational model and validation of its results is an essential tool for scientific researchers, but it is rarely used by modelers. In our work we address the question of validating and verifying simulations in general, and summarize our experience in approaching different models through replication with different motivations. Two models are discussed in details. The first one is an agent-based spatial adaptation of a numerical model, while the second experiment addresses the exact replication of an existing economic model.
Driving Behavioral Change for Information Management through Data-Driven Gree...
Model Replication in the Context of Agent-based Simulation
1. Richárd O. Legéndi, László Gulyás, Yuri Mansury
Eötvös Loránd University, AITIA International, Inc., Cornell University
rlegendi@aitia.ai, lgulyas@aitia.ai, ysm3@cornell.edu
This work was partially supported by the Hungarian Government (KMOP-1.1.2-08/1-2008-0002), the European Union
Seventh Framework Programme FP7/2007-2013 under grant agreement CRISIS-ICT-2011-288501 (CRISIS –
Complexity Research Initiative for Systemic InstabilitieS) and mOSAIC 2011-256910 (Open-Source API and Platform
for Multiple Clouds). These supports are gratefully acknowledged.
1st European Conference on Political Attitude and Mentality
ECPAM 2012, Bucharest, September 3-5, 2012
2. Layout
Motivation and Background
Replication? Why care?
Case Studies and Results
ABM approach for the New Economic Geography
Replication of the Bottom-up Adaptive Macroeconomics
Summary
2012.09.03. ECPAM 2012 - Replication case studies 2
3. 2012.09.03. ECPAM 2012 - Replication case studies 3
4. Replication? Why care?
Replication of experiments, validation of results are
essential
„Simulations as experiments”
If cannot be reproduced, its scientific value is in
question
Models never replicated - except a few classical ones
Helps us get a deeper understanding
Of relevant properties, key issues
Deploy simulation as a research tool
2012.09.03. ECPAM 2012 - Replication case studies 4
5. Validation?
Docking – alingment of different models
Different computational models for the same
phenomenon
Replication
W/o being able to replicate results of an artificial model,
how to target real-world systems?
Several problems, e.g. ambiguity
Different approaches exist (AgentUML, ODD, etc.)
But there’s no consensus on using them...
2012.09.03. ECPAM 2012 - Replication case studies 5
6. An Agent-Based Adaptation of the New Economic Geography
2012.09.03. ECPAM 2012 - Replication case studies 6
7. New Economic Geography
Paul Krugman’s city
formation model
Originally a numerical
model
Applied agent-based
approach
Masahisa Fujita, Paul Krugman, Anthony J.
Venables: „The Spatial Economy.” MIT Press,
Cambridge, MA, 1999.
2012.09.03. ECPAM 2012 - Replication case studies 7
9. Zipf’s Law in City Formation
City Population Rank
(2010)
New York 8,175,133 1
Los Angeles 3,792,621 2
Chicago 2,695,598 3
Houston 2,099,451 4
Philadelphia 1,526,006 5
Phoenix 1,445,632 6
San Antonio 1,327,407 7
San Diego 1,307,402 8
Dallas 1,197,816 9
San Jose, CA 945,942 10
2012.09.03. ECPAM 2012 - Replication case studies 9
10. Motivation
Previous works explains Zipf’s law successfully
But lacks micro-foundations
We extended th FKV model
General-equilibrium model
Excellent micro-foundations
But cannot generate a hierarchical system of cities
2012.09.03. ECPAM 2012 - Replication case studies 10
11. Why the Agent-Based approach?
Introduce heterogeneity
Noise
Agent-specific migration thresholds
Enables migration to proceed in a non ad-hoc way
Extensibility
2012.09.03. ECPAM 2012 - Replication case studies 11
12. Results
We proposed a spatial AB version of FKV
Applied an inherently different approach
Retains the key features of the original model
Including consumers’ love for varieties
Increasing returns in production
Tension between centripetal (agglomeration) and
centrifugal (dispersion) forces
2012.09.03. ECPAM 2012 - Replication case studies 12
13. Tomahawk-diagram
Population migration (λ)
vs. „freeness” of trade (φ)
Break and sustain point
φB and φS
Closed-form solution
and implicit function to
evaluate
2012.09.03. ECPAM 2012 - Replication case studies 13
14. Replication Results
Simulations replicates
expected results
t = 2000 / 5000 time
steps
φB and φS verified
2012.09.03. ECPAM 2012 - Replication case studies 14
15. Replication of the Macroeconomics from the Bottom-up
2012.09.03. ECPAM 2012 - Replication case studies 15
16. Macroeconomics from the Bottom-up
Agent-based macro model
Empirical external
validation
Using real-world data
Replication of the same
model
In a different environment
Gatti, Domenico Delli, Saul Desiderio, Edoardo
Gaffeo, Pasquale Cirillo, and Mauro Gallegati:
Macroeconomics from the Bottom-up. 1st ed.
Springer, 2011.
2012.09.03. ECPAM 2012 - Replication case studies 16
17. Model Structure
Source: Domenico Delli Gatti, personal communications
2012.09.03. ECPAM 2012 - Replication case studies 17
18. Agents
Households
Supply labor
Buy consumption goods
Hold deposits
Firms
Demand labor
Produce and sell consumption goods
Bank
Receive deposits from households
Extend loans to firms
2012.09.03. ECPAM 2012 - Replication case studies 18
19. Market Processes I
1. Fims compute net worth, production/price and
labour demand
2. Credit market:
1. Bank decides credit conditions
2. Firms decide to whether take loan or not
3. Job market:
1. Firms redefine labour demand, publish vacancies:
1. Excess workforce: fire workers
2. Insufficient workforce: hire if possible
2012.09.03. ECPAM 2012 - Replication case studies 19
20. Market Processes II
4. Consumption goods market:
1. Workers get wages and compute consumption budget
2. Firms post their price
3. Consumers contact z firms randomly
Ordered by price
4. Unspent money Involuntary savings
5. Unsold goods Sold at zero cost (non-durable)
5. Accounting
1. Firms calculate profits
2. Earnings are retained profits
Used to update net worth.
2012.09.03. ECPAM 2012 - Replication case studies 20
21. Why to replicate? Parameter sweeps
„[...] suppose that in a model there are just 10 relevant
parameters, and that each parameter can assume 10
different values (a rather simplifying assumption). As a
result, one obtains that the constellation of the
parameter space is given by 10^10 vectors. If we perform
20 different runs for each one of them to take into
account the possible effects of changing the random
seeds, the total number of simulations would
amount to 2*10^11!”
Gatti, Domenico Delli, Saul Desiderio, Edoardo Gaffeo, Pasquale Cirillo, and Mauro Gallegati:
Macroeconomics from the Bottom-up. 1st ed. Springer, 2011 (p. 76., section 3.10.1)
2012.09.03. ECPAM 2012 - Replication case studies 21
22. Why to replicate?
In a different environment?
Matlab Java/Mason
Efficiency
Reduce required time for a single simulation run
Tool support: MEME
Parameter sweep exploration
Being Strong
Exploiting Grid/Cloud systems
Being Smart
Design of Experiments
2012.09.03. ECPAM 2012 - Replication case studies 22
23. Background
“The CRISIS project addresses building a next generation
macroeconomic and financial system policymaking model: a
bottom-up agent-based simulation that fully accounts for the
heterogeneity of households, firms, and government actors. The
model will incorporate the latest evidence from behavioral
economics in portraying agent behavior, and the CRISIS team will
also collect new data on agent decision making using
experimental economics techniques. While any model must
make simplifying assumptions about human behavior, the CRISIS
model will be significantly more realistic in its portrayal of relevant
agent behavior than the current generation of policymaking
models.”
Crisis project description: https://www.crisis-economics.eu/
2012.09.03. ECPAM 2012 - Replication case studies 23
24. Replicated
Model
Modelling Economic Simulator
Framework (Cloud-Based
Parameter Sweep
Execution)
Models Web-based Game
(Participatory
Experiments)
2012.09.03. ECPAM 2012 - Replication case studies 24
25. Results I - Benchmarking
2012.09.03. ECPAM 2012 - Replication case studies 25
26. Result II – Verification
Scaled agents (w/o changing overall ratio)
Up to 7500 agents
Avg’d 40 runs
t = 1000 time steps
Included initial state
High oscillations
Until spontaneous
order emerges
(„equilibrium”)
2012.09.03. ECPAM 2012 - Replication case studies 26
27. 2012.09.03. ECPAM 2012 - Replication case studies 27
28. 2012.09.03. ECPAM 2012 - Replication case studies 28
29. Summary: Case Study 1
We created a replication of the FKV by using a
different approach
Retains hallmark of the original model
Introduced heterogeneity at several levels
Allows further studies
With different activation regimes
N-cities model
2012.09.03. ECPAM 2012 - Replication case studies 29
30. Project Info http://emergingcities.aitia.ai
2012.09.03. ECPAM 2012 - Replication case studies 30
31. Summary: Case Study 2
We created a replication of the MacroABM model in a
different environment
Identic output
Results are platform, environment-independent
Opens up the window of standardized simulation tools
Extensive parameter space explorations (MEME)
Performance speedup
By the factor 5x-10x
On the other hand, code length is increased similarly:
Matlab: ~300 LoC
Java: 1500 + 1000 LoC
2012.09.03. ECPAM 2012 - Replication case studies 31