Jeff Bohn, SSGX Chief Science Officer's 2016 Global Data Science Conference presentation.

1. GXN-1534
Balancing model performance and
complexity in real-world analytics
applications
Dr. Jeffrey R. Bohn
Chief Science Officer
Head of GX Labs
jbohn@statestreet.com
Limited Access

2. GXN-1534
Outline
• Introduction
• CDO “case study”: Balancing complexity and performance
• Defining complexity
• Driving complexity
• Simplicity and complexity
• Model performance
• Choosing a model
• Communicating complexity
• Recommendations
2

3. GXN-1534GXN-1534
The wisdom of Archilochus (Greek
lyric poet from Paros, lived 680
BCE to 645 BCE)
The fox knows many things, but
the hedgehog knows one big
thing.
3

4. GXN-1534GXN-1534
Thinking
Theories
4

5. GXN-1534
CDO “case study”
5

6. GXN-1534
Big Short: Collateralized debt obligations (CDOs)
• Simple in concept: Portfolio of many assets plus cash waterfall changes risk profile
• Complex in structure: Rules in cash waterfall; over-collateralization and interest-
coverage triggers
• Simple risk ratings: Diversity score (binomial expansion) or Gaussian copula plus
“shading” based on a committee
• Complex market structure: Borrowers, appraisers, mortgage brokers, commercial banks,
mortgage servicers, investment bankers, fund managers, rating agencies, institutional
investors, regulators (Fed, NAIC, SEC) lawyers and many consultants
• Simplicity in incentives: Money for nothing– or-- at the least-- mis-representation of
actual risk relative to fees and spreads
Communication regarding CDO risk to senior executives at financial institutions was
mostly not cognitively compelling.
“Real risk was not volatility; real risk was stupid investment decisions.” Lewis (2010)
6

7. GXN-1534
Failure of a simple CDO model
• Materially underestimated risk of mezzanine and senior tranches
• Often over-simplified waterfall rules– e.g., equity prioritized
• Unrealistic assumptions with respect to default rates
• Herd mentality on CDO ratings
• Correlation dynamics not well represented– i.e., correlation went toward 1 in crisis.
7

8. GXN-1534
Portfolio Simulation: A more complex, but better model
Sampling across the Latent Factors to simulation various risk conditions
N securities
Portfolio
Valuation
Model
…
Random draws
Single Position
K latent factors
Factor
draws
Factor realizationsFactor Betas
idiosyncratic
value of
ith security
K = 30 factors
N ≈ 10k ~ 500k positions/idiosyncratic draws
J ≈ 1m ~ 10m iterations
How much computing power needed?
J iterations
Distribution
Repeat J times
value of
portfolio for
one iteration
equity
fixed income
other
Repeat N times
Source: State Street Global Exchange℠
Limited Access
Figure provided for illustrative purposes only.
8

9. GXN-1534
Multi-asset-class portfolio-risk modeling
Use-objective focuses on incorporating risk strategy into portfolio management
probability
Start of TailExpected
Tail Loss
stress losses
Portfolio value
Expected
Value
Risk-appetite assessment
• Model guides executives as to whether allocation is prudent from a risk perspective
Sub-portfolio/manager/hedging evaluation
• Model assists in evaluating how well components of portfolio contribute to overall risk/return
Scenario analysis
• Model provide input into strategic discussions on portfolio construction/management
Figure provided for illustrative purposes.
9

10. GXN-1534
Interpreting simulated scenarios
Reverse Stress Testing
Values at horizon Search through
factor space
Inflation
S&P
Oil Price
GDP
Rates
Macro scenarios
What factor realizations
generate a portfolio value
in the specified confidence
interval of portfolio
distribution?
Source: State Street Global Exchange℠
Limited Access
Figure provided for illustrative purposes.
10

11. GXN-1534
Driving complexity
11

12. GXN-1534GXN-1534
Defining complexity
• Epistemic: Hard to understand conceptually
• Computational: Algorithm hard to understand and/or implement
• Dynamic: System changes over time– sometimes as a function of itself or its users
• Human minds are memory and predicting mechanisms: Does the complexity arise
from quantity of “insight chunks” that need to be remembered or from the quantity of
steps/inter-relationships in the predictive model?
• Simple heuristics may arise from highly complex models/systems. Complexity may arise
from incorporating nuances into interpretations/decisions based on a given heuristic.
12

13. GXN-1534GXN-1534
Consilience: A frequent driver
toward complexity
A "jumping together” of knowledge by the
linking of facts and fact-based theory across
disciplines to create a common groundwork of
explana=on.
Wilson (1999) p. 8
13

14. GXN-1534
What makes decision-support analytics complex?
• Multi-variate optimization problem– often with controls and results playing out over
different time horizons.
• Uncertainty– “risk” defined by known distributions and “Knightian uncertainty” defined by
unknown models/data-generating processes
• Interconnectedness– both explicit and implicit.
• Hierarchies of relationships and relevant assumptions
• Separation of point predictions/estimates and distribution estimates
14

15. GXN-1534GXN-1534
Drivers of model complexity
• Nonstationarity:
– Macro-economic regime changes
– Firms changing leverage
– Governments changing regulation
• Heterogeneous return distributions by asset class
– Non-normal
– Skewed
– Fat tails
– Liquidity
• Capturing obscure, but material risks
– Correlated exposure to a latent factor (e.g., US housing market)
– Sectorally diversified investment-grade bond portfolio may constitute material
concentration risk
– Wrong-way, counterparty risk to an over-hedged energy company
• Human reaction to changing
- Bank management reaction to stress environment by changing underwriting standards
- Not selling in a down market and distressed-selling in an up market not recognized
- Inability or unwillingness to understand a model and its implications may lead to value-
destroying behavior
- Traders gaming particularly sensitive aspects of a risk model
15

16. GXN-1534
Shift in risk modeling favors more model complexity
Normal Distribution
Assumptions
Linear estimation is
good enough
Current Mainstream
Paradigm (examples)
Focus on 2nd moment Linear Regressions
Bias toward tractable
“closed form”
solutions
Examples FastData™
BigComputation™
Machine Learning
Empirical orientation
Enablers
Skewed, Fat-tailed
distributions
Focus on Skewness
and Tail, generated by
simulations
Deep Learning
Need to recognize
non-linearities
Empirical approach
with recognition of
non-linearity
Examples
New Approaches
Source: State Street Global Exchange℠
16

17. GXN-1534
Simplicity and complexity
17

18. GXN-1534GXN-1534
Simple/Complex Enough
Albert Einstein displayed the
following aphorism in his office:
“Things that are difficult to do are
being done from the wrong centers
and are not worth doing.”
-- Diaconis (2003)
Albert Einstein may have said
something like “[Theories],
[Education] or [Things] should be as
simple as possible, but no simpler.”
18

19. GXN-1534GXN-1534
Simplicity may not always be the primary or best criterion
• Also attributed to Einstein: "Any intelligent fool can make things bigger, more complex, and more
violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction.”
• Simplicity should be one goal, not a hard criterion.
• Usefulness should also be a goal and should be a hard criterion: Thus, simple enough, but no
simpler. This thought can lead to…
• Occam’s Razor: “Among competing hypotheses, the one with the fewest assumptions should be
selected.”
– William of Ockham’s actual quote (I think): “Numquam ponenda est pluralitas sine
necessitate [Plurality must never be posited without necessity]”
– Bertrand Russell’s version: "Whenever possible, substitute constructions out of known
entities for inferences to unknown entities.”
• Simplicity in concept may belie complexity in reality– e.g., biological evolution, construct a
portfolio that finds highest return/risk
• Not likely to be a global principle– important that model “suitably” explains/predicts data
– “Suitably” implies matching model with use objective
– Over time, more complex models may explain data better and open new vistas e.g., atomic
theory, plate tectonics, Black-Scholes, light theory (Newton’s “simpler” particle’s versus
Huygen’s waves)– bottom-up, factor-based portfolio simulations?
19

20. GXN-1534GXN-1534
Model performance
20

21. GXN-1534
Hume’s problem: When will a model perform?
• When is it reasonable to think the future will be like the past?
• Rephrase– what part of a model’s/system’s/algorithm’s structure will operate in the future
in the same way it has in the past?
– Means are difficult to estimate and historical observation may be a poor guide
– Volatility may be easier (than means) to estimate, but still may change so much that
historical observation is likely to be a poor guide.
– Underlying correlation structure may change-- but not as much as volatility
– Co-skew?
– Co-kurtosis?
How much can/should non-linear-process-inducing feedback loops and tipping points be
included in a model and can historical observation help?
How does model complexification affect strategies for communicating to quantitatively-
informed business executives?
21

22. GXN-1534GXN-1534
How do we know a portfolio-risk system/model works?
“Science is no inexorable march to truth, mediated by the collection of
objective information and the destruction of ancient superstition.
Scientists, as ordinary human beings, unconsciously reflect in their
theories the social and political constraints of their times.”
-- Stephen Jay Gould
Karl Popper’s process applied to portfolio-risk analysis:
• Problem situation: “Risk-appetite consistent allocation”
• Tentative theories: “Hypothesize scenarios”
• Error elimination: “Stress the stress test i.e., simulate”
However, falsifying every scenario is not possible; further, Gould argues to
beware theory-laden analyses.
22

23. GXN-1534
Complex models perform better: Evolution
• Darwin (1859): Natural selection is a key mechanism of evolution based on the
differential survival and reproduction of individuals due to phenotype differences.
• Romanes (1895): Neo-darwinism refers to germ-plasm theory advocated by Wallace and
Weissmann.
• Huxley (1942): Modern synthesis drew together multiple fields of biology marrying
natural selection, genetics, natural population analysis, systematics, etc.
• Today’s models of evolution are much more complex:
– Genetic networks composed of tens to hundreds of genes interact
– “Regulatory” genes behave conditional on the environment
– Some hereditary variations are nonrandom in origin
– Some acquired information is inherited (epigenetic inheritance)
– Evolutionary change can result from instruction as well as selection
Jablonka and Lamb (2014)
23

24. GXN-1534
Complex models perform better: Risk analytics
• Capital-asset pricing model
• Markowitzian portfolio theory
• Arrow-Debreu
• Black-Scholes-Merton
• Continuous-time finance
• Value at risk
• Monte-Carlo simulation
• Expected-tail loss
Are these “advances” i.e., model complexification worth the investment?
24

25. GXN-1534GXN-1534
Choosing a model
25

26. GXN-1534GXN-1534
Criteria for deciding degree of model complexity
• Model performance
• Ease of communicating actionable model output to quantitatively-informed executives
• Ignores extraneous information
• Balances model uncertainty and credibility of priors
• Minimizes over-fitting risk: Sample size relative to number of parameters
• Resistant to manipulation as a consequence of incentives
“Because complexity generates uncertainty, not risk, it requires a regulatory response
grounded in simplicity, not complexity.” (Haldane, 2005, p. 19)
What does this mean in practice?
26

27. GXN-1534GXN-1534
Questioning the entire modeling enterprise
• Preproducibility is a prerequisite for attempting to reproduce a result: it involves providing
an adequate description of an experiment or analysis for the work to be re-undertaken. It
requires documentation, openness, and communication.
• Quantifauxcation is to assign a meaningless number, then pretend that since it’s
quantitative, it’s meaningful. Usually involves some combination of data, pure invention, ad-
hoc models, inappropriate statistics, and logical lacunae.
• Cost of most policy cost-benefit analyses is high: lost rationality
• Rates vs. probabilities
– Randomness created by taking a random sample, assigning subjects at random, etc.
– Probability model invented (assumed) for data that world generates– inferences are
only as good as the assumptions
– Aleatory: Coin toss, die roll, under some circumstances, behave “as if” random
– Epistemic: Stuff we don’t know
– Trials are random, have same chance of success and have known dependence– can
quantify estimate uncertainty.
– Ignorance does not equal randomness
– Tendency to treat haphazard as random
– Probability as metaphor
(From Stark 2015)
27

28. GXN-1534GXN-1534
Model risk
• Mis-specify data-generating process: Fail to estimate the “true” distribution due to the
fact that the underlying process differs materially from the model’s process assumptions.
– Second-order assessment of misspecification also important.
– What is range of wrong estimates?
– Are out-of-sample tests showing under-estimates in times of stress?
• Mis-estimate parameters: While the assumed data-generating process may be
reasonable, still fail to estimate “true” distribution based on parameterization problems.
– Are there enough data points to credibly estimate parameters?
– Can model structure be used to infer tail events without requisite data?
28

29. GXN-1534GXN-1534
Risk versus uncertainty
• Frank Knight (Knight, 1921): "Uncertainty must be taken in a sense radically distinct from
the familiar notion of risk, from which it has never been properly separated.... The
essential fact is that 'risk' means in some cases a quantity susceptible of measurement,
while at other times it is something distinctly not of this character; and there are far-
reaching and crucial differences in the bearings of the phenomena depending on which
of the two is really present and operating.... It will appear that a measurable uncertainty,
or 'risk' proper, as we shall use the term, is so far different from an unmeasurable one
that it is not in effect an uncertainty at all.”
• Risk is known unknowns i.e., data generating process, inter-relationships arising from
model structure, parameters imply known distributions
• Knightian uncertainty is unknown unknowns i.e., data generating process, model
structure and parameters are unknown
29

30. GXN-1534GXN-1534
Types of ignorance
Hansen and Sargent (2015)
Xt+1
=κ Xt
+ βUt
+αWt+1
Xt
≡ Observable state variable at time t
Ut
≡ Control variable
Wt+1
≡ Random shock to process
1. “Bayesian decision maker” does not know β, but trusts prior prob. distribution
2. “Robust Bayesian decision maker” does not trust prior distribution for response
coefficient β; but uses operators to twist prior distributions to generate conservative est.
3. “Robust decision maker” uses a multiplier or constraint preferences to express doubt
regarding probability distribution of W conditional on X and U.
4. “Robust decision maker” asserts ignorance as in 3 by adjusting an entropy penalty to
Make model robust to particular alternative probability models.
“The trouble with most folks isn’t so much their ignorance, as knowing so
many things that ain’t so.” Josh Billings related by Friedman (1965)
30

31. GXN-1534GXN-1534
Evaluating the adequacy of a theory/model
1. Accuracy [out-of-sample confirmation of estimated probability distribution and
contributions of underlying components to that distribution]
2. Consistency (both internal and external) [multi-asset-class, assets & liabilities]
3. Broadness in scope [granularity and comprehensiveness]
4. Simplicity [complex enough to capture dynamics, but simple enough to be diagnosed
and communicated to a quantitatively-informed business head]
5. Fruitfulness [output substantively contributes to impactful decisions]
Depending on the theory under evaluation, criteria may contradict each other so a relative
weighting may be needed i.e., given a particular circumstance, some criteria are more
important than others. Kuhn (1977)
In portfolio risk analysis, we typically add Timeliness to the evaluation process– a
successful theory/model/system that cannot provide timely output is useless.
31

32. GXN-1534
Communicating complexity
32

33. GXN-1534
Understanding the decision maker: Elephant and the rider
• Elephant: Automatic processes
• Rider: Controlled processes
• Modeling decision making
– Humean model: Reason is a servant
– Platonic model: Reason could and should rule
– Jeffersonian model: Head and heart are co-emperors
• “Seeing that” vs. “Reasoning why”
• Rationalist delusion: Maintaining healthy skepticism of reason– smarter people
rationalize better
People who devote their lives to studying something often come to believe that the object
of their fascination is the key to understanding everything. Location 58
Conscious reasoning functions like a press secretary who automatically justifies any
position taken by the president. p. 106
And as reasoning is not the source, whence either disputant derives his tenets; it is in vain
to expect, that any logic, which speaks not to the affections, will ever engage him to
embrace sounder principles. – David Hume
Haidt (2012)
33

34. GXN-1534
How individuals make decisions
Agree on values Disagree on values
Agree on facts Computational decision Negotiate
Disagree on facts Experiment Paralysis or chaos
34
From Koomey (2001) figure 19.1 p. 88.

35. GXN-1534
How to communicate in a cognitively compelling way?
• Research audience…
– Culture (values and biases)
– Education (general and specific to communicated analytics)
– Incentives (degree of departure from objectivity)
• Distill message...
– Empirical vs. rational/theoretical (where is the model in this iterative evolution?)
– Transparent assumptions
– Frank assessment of model uncertainty
– Link to audience’s narrative/values
• Use intuitive framing and strong visuals…
– Relevant metaphors
– Examples from different, but similar domains (e.g., health/medical)
– Design compelling graphs and figures
35

36. GXN-1534
Working memory and compelling communication
• Working memory
– Operates over a few seconds
– Temporary storage
– Manipulates attention
– Focuses attention
– Resists distractions
– Guides decision-making
• Can only process 5 to 9 “chunks” of information within working memory at any given
moment in time (Miller, 1955)
• Deviating from expectations typically causes the listener to disengage
• Working memory dis-fluently “chunks” instead of always focusing on what matters
• Working memory “calls” long-term memory to assist in processing; if nothing is there,
cognitive flow is broken– result is likely disengagement
Education is essential to build up long-term memory
36

37. GXN-1534
Building a narrative
1. Identify the [the set of] focal issue(s)
2. Determine the key micro or local forces impacting the focal issue(s)
3. Identify the key macro or global forces impacting the focal issue(s)
4. Rank by importance and uncertainty– distinguish parameterization of probability
distributions (known unknowns) from model uncertainty (unknown unknowns)
5. Select scenario logic in terms of the parameters (and maybe models) to adjust to show
range of possible outputs
6. Flesh out the scenarios of most importance and highest likelihood– drill into details of
micro/macro forces and nature of parameterized probabilities and uncertainty
7. Determine implications
8. Select leading indicators and signposts to monitor evolution of scenario in light of
decisions
37
From Koomey (2001)

38. GXN-1534
Communicating Uncertainty
• Explain signal and noise in specific terms
• Communicate how model disentangles signal and noise
• Identify and root out data biases
• Educate on error bars and confidence intervals
• Beware illusory precision– clarify how much precision is possible given data/model
Sampling error does not necessarily equal “uncertainty” in terms of implications of model
output
38

39. GXN-1534GXN-1534
Cognitive biases
• False dichotomy: Presenting two choices such that it seems they are the only possibilities.
– Simple vs. complex model
– Use no models vs. use only one model
• Perfect as the enemy of the good (or good enough)
• Red herrings and missing forest for the trees
• Biases
– Affect heuristic: Analyst or executive has “fallen in love with” a particular output so that
they minimize model problems and exaggerate model strengths.
– Groupthink
– Saliency bias: Overly influence by analogous, past success
– Confirmation bias
– Availability bias
– Anchoring bias
– Halo effect: Impression of model author, analyst or even model influences interpretation
– Sunk-cost fallacy: A particular model output has driven strategy/investment
– Overconfidence
– Disaster neglect
– Loss aversion
Traps arising from logical fallacies and cognitive biases
39

40. GXN-1534GXN-1534
Key points to remember for compelling communication
• Frame within a narrative
• Avoid “quantifauxcation”
• Contextualize (across time and across cohorts)
• Address biases: Highlight data selection concerns and explain assumptions & process
• Use transparency in model estimation process to spark questions and debate
• Compare output from multiple models (when possible)
• Visualize data– encourage interactive diagnostics and drill-down
• Emphasize actionable insight
• Educate
– Explain key components of analytical process
– Teach how to understand confidence intervals (noise vs. signal)
Build on understanding: Descriptive, prescriptive and cognitive
Move from analysis (breaking into components) to synthesis (re-assembling with insight)
40

41. GXN-1534
Principles of risk-data visualization
• Match output to use cases
– Concentration risk assessment
– Risk appetite assessment (stress testing)
– Position-level limits/allocation
• Prepare for multiple dimensions (e.g., region, sector, asset class, customer type, size)
• Incorporate drill-down capability
• Contextualize output (e.g., benchmarks, time series, scenario-based)
• Use robust statistics (e.g., median, inter-quartile, mean absolute deviation)
• Use techniques to address data difficulties (e.g., Winsorization, shrinkage)
• Target near-instantaneous rendering of decision-support output
Risk data tend to be defined by outliers
41

42. GXN-1534
Data are all figurative for illustrative purposes
Multiple attributes represented in two dimensions
Expected Returns vs. Volatility by Exposure Size – Sharpe Ratio as Color
High Sharpe Ratios,
but small positions
OK Sharpe Ratios,
and larger position
42

43. GXN-1534
Data are all figurative for illustrative purposes
So which is richer, from a data insight perspective?
43

44. GXN-1534
Improve visualization independent of model complexity
• Google study (Tuch, et al., 2012) found for websites:
– Visually complex websites are less appealing
– Prototypical websites (for a given category) are more appealing
– Simpler design is rated higher
• What makes analytical output compelling and credible?
– Prototypicality: Basic mental image one’s brain creates to categorize everything with
which you interact.
– Cognitive fluency: One’s brain prefers output that is easier to process.
– Mere exposure effect: Familiarity arising from repeated exposure.
– Metric balancing: Too many metrics equals no understanding.
44

45. GXN-1534GXN-1534
Recommendations
45

46. GXN-1534GXN-1534
Be transparent as to epistemic nature of model output
• What the executive doesn’t know, but is knowable: Model output is available and useful
e.g., credible metrics identify risk (in the technical sense.)
• What the executive or the analyst don’t know yet, but is knowable: Proof-of-concept
model is available; however, more investment (e.g., data, analysts, systems, tools) is
needed.
• What is knowable with uncertainty: Model output is available and potentially useful;
however, questions remain as to whether the model itself is specified correctly e.g.,
metrics reflect Knightian uncertainty. (Bayesian methods may be helpful.)
• What is unknowable: Model output is not available.
• What one chooses not to know: Incentives overpower model output.
46

47. GXN-1534GXN-1534
Contextualization: Describe regimes
• Business as usual (BAU) i.e., sustainable growth
• Cyclical (typical up and down growth– but same process and similar trend)
• Structural (move to a different growth path driven by a different process)
• Providing context is critical:
– Benchmark to competitors (cohorts)
– Benchmark to optimal, feasible outcome
– Show time series
– Drill into components on a consistent basis
47

48. GXN-1534GXN-1534
Compelling communication requires multiple interactions
• Educate as to model’s usefulness as a function of complexity
• Frame key performance indicators (KPIs)
• Prototype, socialize, productionize
• Avoid big-bang projects– include executives in discovery/iteration process
48

49. GXN-1534GXN-1534
Manage vagueness to communicate more compellingly
“Vagueness blurring and imprecision effectively provide a protective shell to guard [a]
statement against a charge of falsity.” (Rescher, p. 222)
• Distinguish overarching “truth claim” from (possibly inaccessible) “true details.”
• Avoid recommendation-rejection skepticism by…
– Combating cognitive myopia with context
– Clarifying certainty with respect to aspects of output
– Separating critical truth-claim components from color-commentary details
• Paradoxically, judicious omission of details (sometime unavoidable) may produce a
clearer, more compelling message. Finding the “goldilocks” balance is not always
straightforward.
• Examples
– Truth claim: I grew up in San Francisco. This vague statement masks the true detail
that I grew up in Danville– loosely part of metropolitan San Francisco.
– Truth claim: Portfolio is likely to lose 10% or more of its value in the next structural
recession. This vague statement masks details that a simulation algorithm was used
to determine the frequency of losses beyond 10% and the modeler assumes a
structural recession occurs 3% (1-in-33 years) of the time, which marks the amount
of loss– in this case, 10% of starting portfolio value.
49

50. GXN-1534
Moving from discovery to action
Persuade decision makers regarding…
• Credibility
• Likelihood
• Materiality
• Addressability
Chronic model weaknesses
• No feedback loops
• No thresholds
• Inadequate spill-over effects
50

51. GXN-1534GXN-1534
References 1
• Aikman, David, Piergiorgio Alessandri, Bruno Eklund, Prasanna Gai, Sujit Kapadia, Elizabeth
Martin, Nada Mora, Gabriel Sterne and Matthew Willison, 2009, “Funding liquidity risk in a
quantitative model of systemic stability,” Working Paper 372, Bank of England.
• Arrow, Kenneth J. and Gerard Debreu, 1954, "Existence of an equilibrium for a competitive
economy,” Econometrica 22 (3), pp. 265–290.
• Bohn, Jeffrey and Roger Stein 2009, Active Portfolio Management in Practice, Wiley.
• Diaconis, Persi, 2003, “The problem of thinking too much,” Bulletin American Academy of
Science, Spring, pp. 26-38.
• Fender, Ingo and John Kiff, 2004, “CDO rating methodology: Some thoughts on model risk and
its implications,” Monetary and Economic Department, BIS.
• Feynman, Richard P., 1974, “Cargo cult science,” Engineering and Science, June, pp. 10-13.
• Folinsbee, Kaila E; et al. 2007, "Quantitative approaches to phylogenetics; §5.2: Fount of
stability and confusion: A synopsis of parsimony in systematics,” In Winfried Henke, ed.
Handbook of Paleoanthropology: Primate evolution and human origins: Volume 2, Springer, p.
168.
• Gordy, Michael B., 2003, “A risk-factor model foundation for ratings-based bank capital rules,”
Journal of Financial Intermediation, 12, pp. 199-232.
• Gray, Dale and Samuel Malone, 2008, Macrofinancial Risk Analysis, Wiley.
• Haidt, Jonathan, 2012, The righteous mind: Why good people are divided by politics and
religion, Pantheon Books, New York, NY.
• Haldane, Andrew G., 2012, “The dog and the frisbee,” Bank of England speech.
• Hamilton, James, 1994, Time Series Analysis, Princeton University Press.
• Hansen, Lars P. and Thomas J. Sargent, 2015, “Four types of ignorance,” Journal of Monetary
Economics, 69, pp. 97-113.
51

52. GXN-1534GXN-1534
References 2
• Hansen, Lars P. and Thomas J. Sargent, 2007, “Recursive robust estimation and control
without commitment,” Journal of Economic Theory, 136, pp. 1-27.
• Jablonka, Eva and Marion J. Lamb, 2014, Evolution in four dimensions: genetic, epigenetic,
behavioral, and symbolic variation in the history of life, MIT Press, Cambridge, MA.
• Kalirai, Harvir and Martin Scheicher, 2002, “Macroeconomic Stress Testing: Preliminary Evidence for
Austria,” Financial Stability Report 3, Oesterreichische Nationalbank, 58-74.
• Knight, Frank H., 1921, Risk, Uncertainty, and Profit, Hart, Schaffner & Marx; Houghton Mifflin
Company: Boston, MA.
• Koomey, Jonathan, 2001, Turning numbers into knowledge: Mastering the art of problem solving,
Analytics Press, Oakland, CA.
• Kuhn, Thomas S., 1977, The essential tension: Selected studies in scientific tradition change,
University of Chicago Press, Chicago.
• Lewis, Michael, 2010, The big short: Inside the doomsday machine, W.W. Norton & Company: New
York, NY.
• Lucas, Robert, 1977, “Understanding business cycles,” in K. Brunner and A.H. Metzler, eds.,
Stabilization of the domestic and international economy, Carnegie-Rochester Conference Series on
Public Policy, 7729.
• Popper, Karl, 1972, Objective knowledge: An evolutionary approach, Clarendon Press, Oxford.
• Rescher, Nicholas, 2004, “Leibniz quantitative epistemology,” Studia Leibnitiana, 36(2), pp. 210-231.
• Simon, Herbert A., 1955, “A behavioral model of rational choice,” Quarterly Journal of Economics,
69(1), pp. 99-118.
• Stark, Philip B., 2015, “Pay no attention to the model behind the curtain,” Working paper, UC
Berkeley.
• Vasicek, Oldrich A., 1997, “The distribution of loan portfolio value,” Working paper, KMV.
• Wilson, Edward O., 1999, “Consilience: The unity of knowledge,” First Vintage Books, New York, NY.
52

53. GXN-1534
Disclaimers and Important Risk Information
State Street Global Exchange℠ is a trademark of State Street Corporation (incorporated in Massachusetts) and is registered or has registrations pending in
multiple jurisdictions. State Street Associates® is a research partnership between State Street Global Exchange and academia under which this document is
produced. State Street Associates is a registered trademark of State Street Corporation.
Use of report
This document and information herein (together, the “Content”) is for informational, illustrative and/or marketing purposes only and it does not constitute investment
research or investment, legal, or tax advice. The Content provided is not, nor should be construed as, any offer or solicitation to buy or sell any product, service, or
securities or any financial instrument, and it does not constitute any binding contractual arrangement or commitment for State Street Corporation and its
subsidiaries and affiliates (“State Street”) of any kind. The Content provided does not purport to be comprehensive nor intended to replace the exercise of a
client’s own careful independent review regarding any corresponding investment or other financial decision.
Distribution
The Content provided is not intended for retail clients, nor is intended to be relied upon by any person or entity, and is not intended for distribution to or use by any
person or entity in any jurisdiction where such distribution or use would be contrary to applicable law or regulation. No permission is granted to reprint, sell, copy,
distribute, or modify the Content in any form or by any means without the prior written consent of State Street.
Other Important Disclosures
The Content provided has been prepared and obtained from sources believed to be reliable at the time of preparation, however it is provided “as-is” and State
Street makes no guarantee, representation, or warranty of any kind including, without limitation, as to its accuracy, suitability, timeliness, merchantability, fitness for
a particular purpose, non-infringement of third-party rights, or otherwise. Views and opinions expressed herein are those of the author(s) and are subject to
change without notice based on market and other conditions and in any event may not reflect the views of State Street. State Street disclaims all liability, whether
arising in contract, tort or otherwise, for any claims, losses, liabilities, damages (including direct, indirect, special or consequential), expenses or costs arising from
or connected with the Content.
The Content provided may contain certain statements that could be deemed forward-looking statements; any such statements or forecasted information are not
guarantees or reliable indicators for future performance and actual results or developments may differ materially from those depicted or projected. Past
performance is no guarantee of future results.
Prospects, clients or counterparties should be aware of the risks of participating in trading foreign exchange, equities, fixed income or derivative instruments or in
investments in non-liquid or emerging markets. Prospects, clients or counterparties should be aware that products and services outlined may put their principal
and capital at risk and that diversification does not ensure a profit or guarantee against loss.
Australia: This communication is made available in Australia by State Street Bank and Trust Company ABN 70 062 819 630, AFSL 239679 and is intended only
for wholesale clients, as defined in the Corporations Act 2001.
53

54. GXN-1534
Brazil: The products in this marketing material have not been and will not be registered with the Comissão de Valores Mobiliários, the Brazilian Securities
and Exchange Commission ("CVM"), and any offer of such products is not directed to the general public within the Federative Republic of Brazil ("Brazil").
The information contained in this marketing material is not provided for the purpose of soliciting investments from investors residing in Brazil and no
information in this marketing material should be construed as a public offering or unauthorised distribution of the products within Brazil, pursuant to
applicable Brazilian law and regulations. To our knowledge, no analyst is related to an individual who works at the issuer of the securities subject to the
content of this website; no analyst or their spouse holds, directly or indirectly, securities subject to the reports on this website; no analyst or their spouse is,
directly or indirectly, involved in the acquisition, sale or intermediation of securities subject to the reports on this website; no analyst or their spouse has,
directly or indirectly, a financial interest in relation to the subject matter of the reports on this website; and the relevant analyst's compensation is not,
directly or indirectly, influenced by the revenues arising from the business and financial transactions carried out by the entity to which is associated or
otherwise related.
Canada: The products and services outlined in this document are generally offered in Canada by State Street Bank and Trust Company and/or by State
Street Global Markets Canada Inc.
Hong Kong: This communication is made available in Hong Kong by State Street Bank and Trust Company, which accepts responsibility for its contents,
and is intended for distribution to professional investors only (as defined in the Securities and Futures Ordinance).
Indonesia: This communication is made available in Indonesia by State Street Bank and Trust Company and its affiliates. Neither this communication nor
any copy hereof may be distributed in Indonesia or to any Indonesian citizens wherever they are domiciled or to Indonesian residents except in compliance
with applicable Indonesian capital market laws and regulations. This communication is not an offer of securities in Indonesia. Any securities referred to in
this communication have not been registered with the Capital Market and Financial Institutions Supervisory Agency (BAPEPAM-LK) pursuant to relevant
capital market laws and regulations, and may not be offered or sold within the territory of the Republic of Indonesia or to Indonesian citizens through a
public offering or in circumstances which constitute an offer within the meaning of the Indonesian capital market law and regulations.
Israel: This communication is made available in Israel by State Street Global Markets International Limited, which is not licensed under Israel’s Regulation
of Investment Advice, Investment Marketing and Portfolio Management Law, 1995. This communication may only be distributed to or used by investors in
Israel which are “eligible clients” as listed in the First Schedule to Israel’s Regulation of Investment Advice, Investment Marketing and Portfolio
Management Law 1995.
Japan: This communication is made available in Japan by State Street Global Markets (Japan) which is regulated by the Financial Services Agency of
Japan as a financial instruments firm.
Malaysia: This communication is made available in Malaysia by State Street Global Markets International Limited (“SSGMIL”) which is authorised and
regulated by the United Kingdom’s Financial Conduct Authority. SSGMIL is not licensed within or doing business within Malaysia and the activities that are
being discussed are carried out off-shore. The written materials do not constitute, and should not be construed as constituting: 1) an offer or invitation to
subscribe for or purchase securities or futures in Malaysia or the making available of securities or futures for purchase or subscription in Malaysia; 2) the
provision of investment advice concerning securities or futures; or 3) an undertaking by SSGMIL to manage the portfolio of securities or futures contracts
on behalf of other persons.
Mexico: This communication is distributed by State Street Bank and Trust Company and its affiliates from outside Mexico. State Street is not authorized to
act as an investment advisor in Mexico or as a regulated entity under Mexican law. Any products and services outlined herein will be provided from outside
Mexico exclusively.
New Zealand: This communication is made available in New Zealand by State Street Bank and Trust Company, which accepts responsibility for its
contents, and is intended for distribution only to “wholesale clients” as defined in the Financial Advisers Act 2008 (New Zealand). This communication is not
intended to constitute “financial advice” (as that term is defined in the Financial Advisers Act), or “advice or assistance” (in terms of section 37(5) of the
Securities Markets Act 1988 (New Zealand)), to any person.
54

55. GXN-1534
Oman: This communication is made available in Oman by State Street Bank and Trust Company and its affiliates. The information contained in this communication
is for information purposes and does not constitute an offer for the sale of foreign securities in Oman or an invitation to an offer for the sale of foreign securities.
State Street is neither a bank or financial services provider registered to undertake business in Oman and is neither regulated by the Central Bank of Oman nor the
Capital Market Authority. This document is confidential and is intended solely for the information of the person to whom it has been delivered. No representation or
warranty is given as to the achievement or reasonableness of any research material contained in this communication. Nothing contained in this communication
report is intended to constitute Omani investment, legal, tax, accounting, investment or other professional advice.
Qatar: This communication is made available in Qatar by State Street Bank and Trust Company and its affiliates. The information in this communication has not
been reviewed or approved by the Qatar Central Bank, the Qatar Financial Markets Authority or the Qatar Financial Centre Regulatory Authority, or any other
relevant Qatari regulatory body.
Singapore: This communication is made available in Singapore by State Street Bank and Trust Company, Singapore Branch (“SSBTS”), which holds a wholesale
bank license by the Monetary Authority of Singapore. In Singapore, this communication is only distributed to accredited, institutional investors as defined in the
Singapore Financial Advisers Act (“FAA”). Note that SSBTS is exempt from Sections 27 and 36 of the FAA. When this communication is distributed to overseas
investors as defined in the FAA, note that SSBTS is exempt from Sections 26, 27, 29 and 36 of the FAA. State Street Bank and Trust Company Limited, Singapore
Branch’s Unique Entity Number is T01FC6134G.
South Africa: The products and services outlined in this document are made available in South Africa through either State Street Global Markets International
Limited or State Street Bank and Trust Company, both of which are authorized in South Africa under the Financial Advisory and Intermediary Services Act, 2002 as
a Category I Financial Services Provider; FSP No. 42823 and 42671 respectively.
South Korea: This communication is made available in South Korea by State Street Bank and Trust Company and its affiliates, which accept responsibility for its
contents, and is intended for distribution to professional investors only. State Street Bank and Trust Company is not licensed to undertake securities business within
South Korea, and any activities related to the content hereof will be carried out off-shore and only in relation to off-shore non-South Korea securities.
Taiwan: This communication is made available in Taiwan by State Street Bank and Trust Company and its affiliates, which accept responsibility for its contents, and
is intended for distribution to professional investors only. State Street Bank and Trust Company is not licensed to undertake securities business within Taiwan, and
any activities related to the content hereof will be carried out off-shore and only in relation to off-shore non-Taiwan securities.
Turkey: This communication is made available in Turkey by State Street Bank and Trust Company and its affiliates. The information included herein is not
investment advice. Investment advisory services are provided by portfolio management companies, brokers and banks without deposit collection licenses within the
scope of the investment advisory agreements to be executed with clients. Any opinions and statements included herein are based on the personal opinions of the
commentators and authors. These opinions may not be suitable to your financial status and your risk and return preferences. Therefore, an investment decision
based solely on the information herein may not be appropriate to your expectations.
United Arab Emirates: This communication is made available in United Arab Emirates by State Street Bank and Trust Company and its affiliates. This
communication does not, and is not intended to, constitute an offer of securities anywhere in the United Arab Emirates and accordingly should not be construed as
such. Nor does the addressing of this research publication to you constitute, or is intended to constitute, the carrying on or engagement in banking, financial and/or
investment consultation business in the United Arab Emirates under the rules and regulations made by the Central Bank of the United Arab Emirates, the Emirates
Securities and Commodities Authority or the United Arab Emirates Ministry of Economy. Any public offer of securities in the United Arab Emirates, if made, will be
made pursuant to one or more separate documents and only in accordance with the applicable laws and regulations. Nothing contained in this communication is
intended to endorse or recommend a particular course of action or to constitute investment, legal, tax, accounting or other professional advice. Prospective
investors should consult with an appropriate professional for specific advice rendered on the basis of their situation. Further, the information contained within this
communication is not intended to lead to the conclusion of any contract of whatsoever nature within the territory of the United Arab Emirates. This communication
has been forwarded to you solely for your information, and may not be reproduced or passed on, directly or indirectly, to any other person or published, in whole or
in part, for any purpose. This communication is addressed only to persons who are professional, institutional or otherwise sophisticated investors.
55

56. GXN-1534
United Kingdom and European Union: The products and services outlined herein are only offered to professional clients or eligible counterparties through State
Street Bank and Trust Company, London Branch, authorised and regulated by Federal Reserve Board, authorised and subject to limited regulation by the Prudential
Regulation Authority and subject to regulation by the Financial Conduct Authority and/or State Street Global Markets International Limited, authorised and regulated
by the Financial Conduct Authority Details about the extent of our regulation by the Financial Conduct Authority and Prudential Regulation Authority are available from
us on request. Please note that certain foreign exchange business (spot and certain forward transactions) are not regulated by the Financial Conduct Authority.
United States and Latin America: The products and services outlined in this document are generally offered in the United States and in Latin America by State
Street Bank and Trust Company and/or by State Street Global Markets, LLC.
Please contact your sales representative for further information.
© 2016, State Street Corporation, All rights reserved.
56