1. The magazine of the Institute and Faculty of Actuaries
JUNE 2016
theactuary.com
In conversation:
Ben Kemp and
Ann Muldoon
Working together
to regulate the
profession
Life
Why the matching
adjustment is vital to
life insurance
Solvency II
Regulation and the
herd instinct
Risk
Reducing uncertainty
through expert
judgment
The battle against bacteria
RESISTANCE
IS FUTILE?
p01_cover_options 3.indd 1 19/05/2016 10:39

2. Raveem Ismail and
Scott Reid propose
that structured expert
judgment can be used
to significantly reduce
uncertainty in risk
appraisal when
considering areas such
as political violence
Inconsideringhow, as organisations and
individuals, we make decisions, we first refer to an
insightful quote: “There are no hard facts, just
endless opinions. Every day, the news media deliver
forecasts without reporting, or even asking, how
good the forecasters who made the forecasts really
are. Every day, corporations and governments pay for
forecasts that may be prescient, worthless, or
something in between. And every day, all of us –
leaders of nations, corporate executives, investors,
and voters – make critical decisions on the basis of
forecasts whose quality is unknown.”
Superforecasting: The Art and Science of Prediction,
Tetlock & Gardner, 2015
It would be preferable for all decision-aiding
models to be based on objective criteria such as
exhaustive data and sound physical principles.
This ideal situation rarely occurs, and (re)insurance
decisionmakers frequently act in data-poor
environments, relying heavily on expert judgment.
This occurs particularly in low-frequency high-
severity/loss areas, such as life and health and care,
as well as in unusual, rare and catastrophic risk
appraisal. Given that Solvency II requires assessment
of 1-in-200-year events, the regulatory capital regime
across the EU is also based on the application of
expert judgment.
Decisionmakers can and should demand the
most unbiased expert judgment procedures, with
experts
Askthe
Expert judgment
Risk
SHUTTERSTOCK June 2016 • THE ACTUARY
www.theactuary.com
25
p25_27_june_expert_FINAL•CT.indd 25 23/05/2016 09:19

3. objective criteria to appraise expert
performance. But how? Referencing one actual
study, we discuss one approach, used in other
fields but not yet in (re)insurance. This is
structured expert judgment (SEJ), which is an
auditable and objective combination of
multiple judgments, each weighted by its skill
in gauging uncertainty. This produces a
better overall judgment within a plausible
range of outcomes.
Expert opinion
Consulting 10 experts will yield 10 different
answers. Each answer is an (unknowable)
function of an expert’s previous experience,
grasp of data, judgmental capability, biases
or mood on the day. Without a method of
selecting between so many different
judgments, the customer (insurance company)
often simply sticks with what they know best:
a longstanding provider or market reputation.
None of these is any indicator of capability:
the client cannot know the quality, since no
performance-based appraisal of forecasting
ability has occurred.
While a single expert’s judgment might be an
outlier, any simple averaging leads to limited
gains. As each expert is weighted equally,
without regard for capability, the final answer
may actually be less accurate than some
individual answers, owing to outliers.
SEJ differs from, and extends, previous
opinion pooling methods. Each expert is first
rated with regard to prior performance by being
asked a set of seed questions to which the
answer is already known to the elicitation
facilitator but not necessarily to the expert. Each
expert’s performance on these seed questions
ascertains their weighting. They are then asked
the target questions; the actual judgments being
sought, to which answers are not known.
Weightings drawn from the seed questions are
then used to combine the experts’ judgments on
the target questions, producing one outcome
that truly combines different expert judgments
in a way that is performance-based, and is thus
potentially better than each individual answer.
The design of seed questions is critical: seed
questions must be chosen for their tight
alignment with the target questions, testing
THE ACTUARY • June 2016
www.theactuary.com
26
Figure 1: Seed question 11
0 200 400 600 800 1000 1200
Expert 1
Expert 2
Expert 3
Expert 4
Expert 5
Expert 6
Expert 7
Expert 8
Expert 9
Experts’ assessment of past SR&CC
event frequency in South East Asia,
1990-2000
Equal
weighted
combination
Performance
weighted
combination
0 500 1000 1500 2000
Expert 1
Expert 2
Expert 3
Expert 4
Expert 5
Expert 6
Expert 7
Expert 8
Expert 9
Experts’ assessment of future
SR&CC event frequency in
South East Asia, 2016
Equal
weighted
combination
Performance
weighted
combination
Figure 3: Target question 7
Structured expert
judgment is an auditable
and objective combination
of multiple judgments,
each weighted by its skill
in gauging uncertainty”
p25_27_june_expert_FINAL•CT.indd 26 23/05/2016 09:19

4. the same ability required for target questions
and thus maximising the utility of the
performance weighting.
Frequency of political violence
Cooke’s classical model for SEJ involves asking
each expert for two metrics: a confidence
interval between which they think the true
value lies (5% to 95%); and a central median
value. These are then used to calculate how
well the expert gauges uncertainty spreads
(information), and how reliably they capture
true values within their ranges (statistical
accuracy or calibration).
Under the European Cooperation in
Science and Technology framework, a
network was formed and first elicitation
performed in January 2016, with 18 seed and
eight target questions. This was for an
inherently unknowable future metric: the
2016 frequency of strikes, riots and civil
commotion (SR&CC) in blocs of countries
(Central Asia, Maghreb), with participants
drawn from across the (re)insurance
profession. An example of their
judgments on a single seed question
(related to prior SR&CC events in South-East
Asia) is shown in Figure 1. Experts produced a
variety of median values and ranges, some
having tightly bound ranges that captured
the true value (dotted line).
Figure 2 (above) shows information and
calibration scores across the full seed question
set. Two experts (experts one and four) emerge
with notably strong performance-based
weights. If all experts were weighted equally
(last column), this discovered capability would
SCOTT REID is head of
pricing and reinsurance
at AIG Life, UK
DR RAVEEM ISMAIL
is a specialty treaty
underwriter at Ariel Re,
Bermuda
Figure 2 Individual and weighted scores for nine experts across
18 seed questions
Expert Calibration
score
Information
score
Performance
weights
Equal
weights
1 0.2295 1.864 0.4280 0.1111
2 <0.0001 1.783 <0.0001 0.1111
3 0.0020 2.040 0.0041 0.1111
4 0.2274 1.665 0.3785 0.1111
5 0.0002 2.153 0.0006 0.1111
6 <0.0001 3.010 0.0002 0.1111
7 <0.0001 1.505 <0.0001 0.1111
8 <0.0001 2.495 <0.0001 0.1111
9 0.0001 0.734 <0.0001 0.1111
Equal-
weighted
combination
0.6286 0.869 (weighted average score)
0.4242
Performance
-weighted
combination
0.5173 1.701 0.7561
be diluted away (‘equal-
weighted’ row, table foot).
However, if the experts’
judgments are combined
using the weights from the
calibration exercise
(penultimate column),
then a combination
emerges that capitalises on
these high-performance
experts to produce better
results than all of them
(‘performance-weighted’
row at table foot).
When this performance-
weighted combination is
used for a target question,
the result can be seen in
Figure 3. For this forward-
looking question, there is
no known answer, yet we
see that the performance-
weighted process has
allowed the influence of experts one and four
to provide a much tighter and more
informative judgment than would most
individual experts, or the equal-weighted
combination (which is inflated by outliers).
For the performance-weighted
combination, outliers are ameliorated and
identified experts given more weight. Such a
final frequency, with associated range, could
now feed a pricing or catastrophe model
with greater assurance than customary
approaches. Structured expert judgment is
still judgment. But it is not guesswork. It is
a transparent method of pooling multiple
opinions, weighted according to
performance criteria aligned to the actual
judgments being sought. Where data or
models are lacking, it forms an objective and
auditable method of producing decision-
making judgments and inputs to models.
We have described a first SEJ elicitation in
our area of interest, where this method has
been shown to identify and outperform
uncalibrated methods.
It should be noted that SEJ is not a silver
bullet. Where there are science-based models
or suitable data, these should trump expert
judgment (or be used in tandem). But in their
absence, in classes of business such as political
violence, and for situations where tail risk is
being gauged, SEJ would look to naturally
provide significant enhancement to decision-
making and risk appraisal. a
Livelinks
onour
app!p!
Seed questions
are chosen for their
tight alignment with
target questions, thus
maximising the utility
of the performance
weighting”
p25_27_june_expert_FINAL•CT.indd 27 23/05/2016 09:19