1. Sensitivity Analysis for Cost-Benefit
Studies of Justice Policies: A Primer
September
13,
2012
Ben
Bryant,
Staff
Economist,
Millennium
Challenge
Corpora7on
Carl
Ma5hies,
Senior
Policy
Analyst,
Cost-­‐Benefit
Analysis
Unit,
Vera
Ins7tute
of
Jus7ce
Slide 1

2. Sensitivity Analysis for Cost-Benefit Studies
of Justice Policies: A Primer
Ben Bryant Carl Matthies
Economist Senior policy analyst
Department of Policy and Evaluation Cost-Benefit Analysis Unit
Millennium Challenge Corporation Vera Institute of Justice
Slide 2 • September 13, 2012

3. Agenda
Introductions and housekeeping 5 minutes
Sensitivity analysis as part of 10 minutes
cost-benefit analysis (CBA)
Perspectives included in a CBA 5 minutes
Deterministic sensitivity analysis 15 minutes
Probabilistic sensitivity analysis 15 minutes
Q&A and conclusion 10 minutes
Slide 3 • September 13, 2012

4. Housekeeping items
• Use the chat feature to send your questions at any time
during the webinar.
• If you need webinar support or for troubleshooting:
§ Type questions into the chat box.
§ Call 800-843-9166.
§ Send e-mail to help@readytalk.com.
• This webinar is being recorded.
• The recording and PowerPoint slides will be posted
on cbkb.org.
Slide 4 • September 13, 2012

5. Webinar preview
You will learn:
§ Why sensitivity analysis is a critical part
of cost-benefit analysis
§ Different techniques for conducting
sensitivity analysis
§ How to interpret the sensitivity analysis
results
Slide 5 • September 13, 2012

6. Sensitivity Analysis as
Part of Cost-Benefit Analysis
(CBA)
Slide 6 • September 18, 2012

7. Review: Basic steps of CBA
§ Specify the set of alternative projects.
§ Decide whose benefits and costs count.
§ Catalog the impacts and select measurement indicators.
§ Predict the impacts quantitatively over the life of the project.
§ Monetize all impacts.
§ Discount benefits and costs to obtain present values.
§ Compute the net present value (NPV) of each alternative.
§ Perform sensitivity analysis.
§ Recommend/choose a project.
Slide 7 • September 13, 2012

8. Review: Basic steps of CBA
 Specify the set of alternative
projects.
Can be political
 Decide whose benefits and
costs count.
 Catalog the impacts and Requires significant topical
select measurement indicators. knowledge
 Predict the impacts quantitatively
over the life of the project. Requires significant
quantitative skills and effort
 Monetize all impacts.
 Discount benefits and costs to
obtain present values. Involves only calculations,
 Compute the net present value in theory
(NPV) of each alternative.
 Perform sensitivity analysis. Varies from simple
 Recommend/choose a project. to an entire study
Slide 8 • September 13, 2012

9. Strengths and weakness of CBA
Some strengths:
§ CBA informs and fosters transparency in policy
process by making benefits and costs explicit.
§ It also uses a common unit of measurement for
potentially more efficient allocation of resources.
Some weaknesses:
§ Even with common measurement ($), people
may disagree on how to value outcomes.
§ Results may vary, subject to choices
about who counts and what counts.
Slide 9 • September 13, 2012

10. State initiatives to legalize marijuana
Decisions on
Net impact
tax rate and
on state and
regulatory
local
regime
budgets
Marijuana
consumption
Remove (quantity and
penalties patterns
for sale and of use)
possession
Source: Kilmer et al. (2010) Slide 10 • September 13, 2012

11. A web of uncertain variables
Decisions on
Tax Net impact
tax rate and
Tax evasion and revenues on state and
regulatory
tax-induced from legal local
regime
shift in mix of marijuana budgets
marijuana types sales
Changes in Marijuana prices
production and consumers face
distribution
costs
Other
Consumer factors that
price influence
sensitivity budgets
Marijuana (e.g., criminal
Non-price consumption justice and
Penalties effects on (quantity and treatment
removed consumption patterns costs; federal
for sale and of use) spending;
possession tourism)
Source: Kilmer et al. (2010) Slide 11 • September 13, 2012

12. Characterizing uncertainty in a CBA
The way we characterize uncertainty is related
to the methods we use:
• Best guess (percent change)
• Plausible bounds (deterministic, break-even)
• Probability distribution (Monte Carlo)
• Distinct stakeholder preferences (scenarios)
Slide 12 • September 13, 2012

13. Questions?
Use the chat feature to send us your questions.
Slide 13 • September 13, 2012

14. Perspectives
Included in a CBA
Slide 14 • September 18, 2012

15. Perspectives
• A program or policy’s net benefits can be
highly sensitive to the costs and benefits
included (who and what “counts”).
• Think of changing the perspective as a form of
sensitivity analysis.
Slide 15 • September 13, 2012

16. Which crime costs are included?
Total
Criminal Justice Crime Tangible
Offense Crime Victim Cost System Cost Career Cost Cost
Murder $744,239 $392,352 $148,555 $1,285,146
Rape/Sexual Assault $5,561 $26,479 $9,212 $41,252
Aggravated Assault $8,635 $8,641 $2,126 $19,472
Robbery $3,274 $13,827 $4,272 $21,373
Arson $11,453 $4,392 $584 $16,429
Motor Vehicle Theft $6,114 $3,867 $553 $10,534
Residential Burglary $1,654 $4,127 $681 $6,169
Larceny $479 $2,879 $163 $523
Source: McCollister, French, and Fang (2004)
Slide 16 • September 13, 2012

17. Which crime costs are included?
Offense Total Tangible Cost Total Intangible Cost
Murder $1,285,146 $8,442,000
Rape/Sexual Assault $41,252 $199,642
Aggravated Assault $19,472 $95,023
Robbery $21,373 $22,575
Arson $16,429 $5133
Motor Vehicle Theft $10,534 $262
Residential Burglary $6,169 $321
Larceny $3,523 $10
Source: McCollister, French, and Fang (2004)
Slide 17 • September 13, 2012

18. Perspectives may affect project choice
Annual
Annual Net Net Annual Net
Annual Benefit Benefit Benefit
Program Estimated Impact Program Cost CJS CJS + TV CJS + TV + IV
Prevents 100 motor
A vehicle thefts per year $300K +$87K +$700K +$730K
Prevents 10 sexual
B assaults per year $300K -$35K +$20K +$2.0M
CJS: Criminal justice system
TV: Tangible victim costs
IV: Intangible victim costs
Slide 18 • September 13, 2012

19. Questions?
Use the chat feature to send us your questions.
Slide 19 • September 13, 2012

20. Deterministic
Sensitivity Analysis
Slide 20 • September 18, 2012

21. Deterministic sensitivity analysis
This type of analysis examines relationships in specific,
carefully chosen ways:
• Partial sensitivity analysis varies model inputs
separately to see how sensitive the CBA result is to
each input’s value, all else remaining equal.
• Extreme scenario analysis defines worst and
best cases.
• Break-even analysis solves for model inputs
when the NPV equals zero.
Slide 21 • September 13, 2012

22. The base case as starting point
• A preliminary result derived from plausible
or best-estimate assumptions
• A reference point or benchmark for
sensitivity analysis
• A starting point—not necessarily the most
likely outcome!
Slide 22 • September 13, 2012

23. Example: Jail work-release program
• Inmates get three days off sentence for every two
days worked.
• The daily marginal cost is slightly higher than for jail
overall, but the program potentially reduces sentences
and recidivism.
• Assumptions:
• A six-year time horizon starts at Year Zero.
• No recidivism costs will occur until Year One.
• Only first-time offenders participate in the program.
• Non-compliers must serve out their entire sentence.
• The program has no effect on non-compliers.
Slide 23 • September 13, 2012

24. The work-release CBA model
Model Inputs Benefits Formula
Hiring two new corrections staff (s)
npq(SR)(JMC)(0.6)
Number of eligible participants (n)
Cost savings from reduced
jail time
Participation rate (p)
npq(SR)(BRR)(E)
Program compliance rate (q)
Cost savings from program
effect on recidivism
Program marginal cost/day (PMC)
Costs Formula
Jail marginal cost/day (JMC)
s
Initial sentence range for eligible Additional staff
individuals (ISR)
npq(SR)(PMC)(0.4)
Average program failure point for
non-compliers (f) Program cost for compliers
Baseline 5-year recidivism rate (BRR)
np(1-q)(SR)(f) (PMC)+
Change in recidivism rate among np(1-q)(JMC)(SR)
participants (E) Program cost
for non-compliers
Sentence range for recidivists (RSR)
Discount rate (d)
Slide 24 • September 13, 2012

25. Example: Jail work-release program
Model Inputs Base-Case Estimate
Hiring two new corrections staff (s) $80,000/year
Number of eligible participants (n) 1,000
Participation rate (p) 0.9
Program compliance rate (q) 0.8
Program marginal cost (PMC) $75/day
Jail marginal cost (JMC) $55/day
Initial sentence range for eligible individuals (ISR) 120
Average program failure point for
non-compliers (f) 0.1
Baseline 5-year recidivism rate (BRR) 0.5
Change in recidivism rate among participants (E) 0.2
Sentence range for recidivists (RSR) 180
Discount rate (d) 0.03
NPV for work-release program $6,520,950
The base case looks favorable, but what if there are departures
from the mean values?
Slide 25 • September 13, 2012

26. Partial sensitivity analysis
Slide 26 • September 18, 2012

27. Example: Jail work-release program
Model Inputs Base-Case Estimate Lower Bound Upper Bound
Hiring two new corrections staff (s) $80,000/year $ 70,000/year $90,000/year
Number of eligible participants (n) 1,000 700 1300
Participation rate (p) 0.9 0.8 1.0
Program compliance rate (q) 0.8 0.7 1.0
Program marginal cost (PMC) $75/day $65/day $80/day
Jail marginal cost (JMC) $55/day $50/day $60/day
Initial sentence range for eligible individuals (ISR) 120 15 220
Average program failure point for
non-compliers (f) 0.1 0.05 0.4
Baseline 5-year recidivism rate (BRR) 0.5 0.3 1.0
Change in recidivism rate among participants (E) 0.2 -0.2 0.6
Sentence range for recidivists (RSR) 180 120 365
Discount rate (d) 0.03 0.03 0.07
NPV for work-release program $6,520,950
Slide 27 • September 13, 2012

28. Partial sensitivity analysis
Change in recidivism rate among
participants (E)
Baseline 5-year recidivism rate (BRR)
Sentence range for recidivists (RSR)
Jail marginal cost (JMC)
Average program failure point in
non-compliers (p)
Program marginal cost (PMC)
Program compliance rate (q)
Number of eligible participants (n)
1,300
Initial sentence range for eligible
individuals (ISR)
Participation rate (p)
Hiring two new corrections staff (s)
90,000
70,000
Slide 28 • September 18, 2012

29. Worst-case and best-case scenarios
Model Inputs Base-Case Estimate Worst Case Best Case
Hiring two new corrections staff (s) $80,000/year $90,000/year $70,000/year
Number of eligible participants (n) 1,000 1300 1,300
Participation rate (p) 0.9 1.0 1.0
Program compliance rate (q) 0.8 0.7 1.0
Program marginal cost (PMC) $75/day $80/day $65/day
Jail marginal cost/day (JMC) $55/day $60/day $60/day
Initial sentence range for eligible
individuals (ISR) 120 15 220
Average program failure point for
non-compliers (f) 0.1 0.4 0.05
Baseline 5-year recidivism rate (BRR) 0.5 1.0 1.0
Change in recidivism rate among
participants (E) 0.2 -0.2 0.6
Sentence range for recidivists (RSR) 180 120 365
Discount rate (d) 0.03 0.07 0.03
NPV for work-release program $6.5M -$43.3M $104.0M
Slide 29 • September 13, 2012

30. Break-even analysis
Model Input Estimate
Hiring two new corrections staff (s) $80,000
Number of eligible participants (n) 1,000
Participation rate (p) 0.9
Program compliance rate (q) 0.8
Program marginal cost (PMC) $91/day
Jail marginal cost/day (JMC) $55/day
Initial sentence range for eligible individuals (SR) 120
Average program failure point for non-compliers (f) 0.1
Baseline 5-year recidivism rate (BRR) 0.5
Change in recidivism rate among participants (E) 0.2
Sentence range for recidivists (SR) 180
Discount rate (d) 0.03
NPV for work-release program $0
The program would still break even if its marginal cost were $91/day,
all else remaining equal.
Slide 30 • September 13, 2012

31. Break-even analysis
Model Input Estimate
Hiring two new corrections staff (s) $80,000
Number of eligible participants (n) 1,000
Participation rate (p) 0.9
Program compliance rate (q) 0.8
Program marginal cost (PMC) $75/day
Jail marginal cost/day (JMC) $55/day
Initial sentence range for eligible individuals (SR) 120
Average program failure point for non-compliers (f) 0.1
Baseline 5-year recidivism rate (BRR) 0.5
Change in recidivism rate among participants (E) 0.0
Sentence range for recidivists (SR) 180
Discount rate (d) 0.03
NPV for work-release program $0
The program would break even if it had no effect on recidivism,
all else remaining equal.
Slide 31 • September 13, 2012

32. Questions?
Use the chat feature to send us your questions.
Slide 32 • September 13, 2012

33. Probabilistic Sensitivity Analysis:
Monte Carlo Simulations
Slide 33 • September 18, 2012

34. Probabilistic sensitivity analysis
This type of analysis changes all model inputs
at once.
• The other methods tell us little about the likelihood
of various program outcomes.
• Monte Carlo simulation repeatedly draws random
values for each model input to create a probability
distribution of outcomes.
Slide 34 • September 13, 2012

35. MS Excel add-ons
for Monte Carlo simulation
Slide 35 • September 13, 2012

36. Step 1: Define assumptions.
Slide 36 • September 13, 2012

37. Step 1: Define assumptions.
Slide 37 • September 13, 2012

38. Define assumptions for all uncertain inputs.
Slide 38 • September 13, 2012

39. Step 2: Set the forecast cell.
Slide 39 • September 13, 2012

40. MC simulations provide the distribution
of outcomes.
Slide 49 • September 18, 2012

41. Distribution of outcomes
Slide 50 • September 18, 2012

42. Questions?
Use the chat feature to send us your questions.
Slide 42 • September 13, 2012

43. Wrap-Up
Slide 36 • September 18, 2012

44. Review
We covered:
§ Why sensitivity analysis is a critical part
of CBA
§ Different techniques for conducting
sensitivity analysis
§ How to interpret the sensitivity analysis
results
Slide 44 • September 13, 2012

45. Takeaway points
§ Base-case CBA is just a starting point.
§ Sensitivity analysis is a way of managing
uncertainty, not eliminating it.
§ Computation and simulation are
relatively easy.
§ Defining input boundaries and probability
distributions is harder.
Slide 45 • September 13, 2012

46. Reminders
§ Please complete the evaluation form before you
leave this webinar.
§ To receive information and notifications about
upcoming webinars and other events:
• Visit the Cost-Benefit Knowledge Bank
for Criminal Justice at cbkb.org.
• Follow us on Twitter at twitter.com/CBKBank.
Slide 46 • September 13, 2012

47. Contact information
Ben Bryant
bryantbp@mcc.gov
Carl Matthies
cmatthies@vera.org
213-223-2445
E-mail: cbkb@cbkb.org
Website: cbkb.org
Slide 47 • September 13, 2012

48. The Cost-Benefit Knowledge Bank for Criminal Justice
(CBKB) is a project of the Vera Institute of Justice
funded by the U.S. Department of Justice’s Bureau of
Justice Assistance.
• Website (cbkb.org)
• CBA toolkit
• Snapshots of CBA literature
• Podcasts, videocasts, and webinars
• Roundtable discussions
• Community of practice
Slide 41

49. This project is supported by Grant No. 2009-MU-BX K029 awarded by the
Bureau of Justice Assistance. The Bureau of Justice Assistance is a
component of the Office of Justice Programs, which also includes the
Bureau of Justice Statistics, the National Institute of Justice, the Office of
Juvenile Justice and Delinquency Prevention, and the Office of Sex
Offender Sentencing, Monitoring, Apprehending, Registering, and
Tracking. Points of view or opinions in this presentation are those of the
authors and do not represent the official position or policies of the
Millennium Challenge Corporation or the United States Department of
Justice.
Slide 42

50. Thank you.
Slide 43 • September 18, 2012