1.
Developing Forward-Looking Metrics and Reporting Jeff Horon, Mike Yiu University of Michigan Medical School Grant Review and Analysis Office May 2011

2.
About UsBoth – Business Analysts, U-M Medical School – Engage in metric and reporting design, econometric and financial modeling, design ad hoc and standardized reporting describing the Medical School’s research enterpriseJeff – MBA w/Emphases in Strategy and Finance, Ross School of Business – Formal training in decision supportMike – Bachelors in Economics, U-M

3.
OutlineWhat are metrics anyway?So, what’s the problem?Oh great, a problem, what’s the solution?Case study

4.
What are metrics?Quantitative valuesMeasure, distill real-world a.k.a. Key Performance Indicators (KPI’s), performance measures, etc.Connotation of monitoring, control Good Ok Bad

5.
Why engage in analysis and reporting?Decision Support!Ad hoc analysis explicitly supports a decisionMetrics and reporting often implicitly supportdecisions, specifically:How are we doing with respect to _____?

6.
Why engage in analysis and reporting?Metrics and reporting repeatedly draw and refocusmanagerial attention over time:How are we doing with respect to _____? [Really] Good Good Good Ok Ok Ok Bad BadHow much attention do I need to pay to _____?

7.
So, what’s the problem?Most metrics and reporting describe the past or present, soyou may miss opportunities for ‘course correction’ or youmay find yourself in trouble before you detect it! [Really] Good Good Normal Good Ok ? Ok Bad Bad Only clear view is backward-looking

8.
Reporting MaturityBackward-looking reporting asks “How did we do?”and might imply corrective action in situations where ‘it paysto correct your mistakes.’Better reporting provides context about the present andassists in decision supportThe best reporting includes forward-looking views thatenable proactive decision-making (Decision Support!Decision Support! D-e-c-i-s-i-o-n S-u-p-p-o-r-t!)

9.
ProblemEvery business unit wants to be forward-lookingMost reporting only provides a backward-lookingview; We only have data about events that havealready happenedHow do you break out of this cycle?

10.
SolutionBreaking out requires creating and developing forward-looking views, using available data to create expectationsabout the future Past Today Future

11.
SolutionForward-looking views may allow for ‘course correction’ [Really] Possible ‘course correction? Good Good Normal Good ? Ok ? ? Ok Bad Bad

12.
Implementation – How?Unfortunately, it’s not easy, but we hope to make it tractableIt will require sustained commitment across several criticalsteps in the development life cycle: Create good metrics Diagnose ‘bad’ metrics Build awareness and acceptance Standardize and refine Integrate with existing reporting

13.
Create good metricsAsk: What do we want to achieve / reward? (Not: What data do we have available?)Set good goals (a topic unto itself – SMART Framework:Specific, Measurable, Achievable, Relevant, Time-Bound)Case Study: The Grant Awards ‘Pipeline’Diagnose ‘bad’ metricsBuild awareness and acceptanceStandardize and refineIntegrate with existing reporting

14.
Case Study: Grant Awards ‘Pipeline’Designed to be forward-looking view of financial sustainabilityOriginal construction: All future award commitments for the next fiscal year or multiple years divided by historical commitments Historical | Future Historical | Future Time

15.
Create good metricsWhat do we want to achieve / reward?Diagnose ‘bad’ metricsTest, test, test!Do the metrics reward desired outcomes?Are the metrics stable? Balanced?Build awareness and acceptanceStandardize and refineIntegrate with existing reporting

16.
Case Study: ‘Flip-Flop’ PatternUpon testing, the metric proved to be idiosyncratic –departments tended to meet the goal in alternating‘flip-flop’ patterns Pipeline Ratio - Current Fiscal Year vs Prior Fiscal Year 1.20 1.00 FY2008 Pipeline Metric 0.80 0.60 0.40 0.20 0.00 0 0.2 0.4 0.6 0.8 1 1.2 FY2007 Pipeline Metric

17.
Case Study: ‘Flip-Flop’ PatternUpon testing, the metric proved to be idiosyncratic – departments tended to meet thegoal in alternating ‘flip-flop’ patternsDiagnosis By stating the goal relative to the prior year, the Pipeline Ratio could reward poor performance in the prior year and punish good performance in the prior yearMechanics When a department performs well, the goal for next year becomes more difficult When a department performs poorly, the goal for next year becomes easierThese effects combine to flip-flop likely payout structures each year (confirmed bystatistical testing): Meets Meets Coefficient P-value Year 1 Year 2 Goal? Goal? One Year -0.6896 0.0099 Department Above Goal Yes Easier Goal More Likely Department Below Goal No More Difficult Goal Less Likely Multiple Years -0.6768 0.0014

18.
Case Study: Sawtooth PatternAnd there was an additional idiosyncrasy:Diagnosis Underlying event patterns will cause the pipeline metric to move in cyclical patternsMechanics Awards are often funded for multiple years, with abrupt endpoints Renewal does not occur until an award is expiringThese effects produce sawtooth patterns in the pipeline metric, with rapid increases driven bylarge new awards or renewals, followed by gradual decline over the life of the awards: New Award Renewal Pipeline Metric Linear Decline Linear Decline In Pipeline In Pipeline Years

19.
Case Study: Rewarding Desired OutcomesProblematically, the highest performing departmentby absolute measures fell below goal with respect tothe ratio: Pipeline Ratio - Current Fiscal Year vs Prior Fiscal Year 1.20 1.00 FY2008 Pipeline Metric 0.80 0.60 Departments Outperformed 0.40 0.20 0.00 0 0.2 0.4 0.6 0.8 1 1.2 FY2007 Pipeline Metric

20.
Create good metricsWhat do we want to achieve / reward?Diagnose ‘bad’ metricsTest, test, test!Do the metrics reward desired outcomes?Are the metrics stable? Balanced?Build awareness and acceptance‘Sell’ the ideaStandardize and refineIntegrate with existing reporting

21.
Case Study: ‘Selling’ a Refined Pipeline‘Sold’ by presentation, quasi-white paperPackaged with a solution – the pipeline metric was refined to: -Utilize one future year of data to mitigate the sawtooth pattern -Compare this one future year against the same observation one year earlier (to recognize single- year growth) -Balanced with a longer-term growth measure (to recognize multiple-year growth, mitigating the effect of punishing good historical performance).

22.
How much refinement? High Impact -High per-unit stakes High Value -High volume Repeated Low Impact -Low per-unit stakes Low Value -Low volume Not Repeated

23.
Create good metricsWhat do we want to achieve / reward?Diagnose ‘bad’ metricsTest, test, test!Do the metrics reward desired outcomes?Are the metrics stable? Balanced?Build awareness and acceptance‘Sell’ the ideaStandardize and refineIntegrate with existing reportingInto existing reportingAlongside existing reporting

24.
Practical Implementation Cost Value Added Complexity

25.
Practical Implementation Cost Value Added Value Added Cost ‘Back of Envelope’ ‘Sketching’ ‘Low-Fi Prototyping’

26.
Case Study: Related Metrics and ReportingSustained commitment and insights from metrictesting, combined with forecasting techniques,awareness-building, refinement, and standardizationhave led to related new metrics and reporting… Historical and Probable Future Commitments Trend Analysis Time Time

27.
Case Study…. available ‘on demand’ in a production reportingenvironment Historical and Probable Future Commitments with Gap-to-Trend Prescriptive Analysis

28.
Recap Create good metrics What do we want to achieve / reward? Diagnose ‘bad’ metrics Test, test, test! Do the metrics reward desired outcomes? Are the metrics stable? Balanced? Build awareness and acceptance ‘Sell’ the idea Standardize and refine Integrate with existing reporting Into existing reporting Alongside existing reporting

29.
Case Study: FutureContinue to advance reporting maturity of additionalmetricsImprove visualization tactics to describe uncertainty

30.
Q&AJeff Horon – jhoron@umich.edu – http://jeffhoron.com Mike Yiu – myiu@umich.eduForecasting: http://jeffhoron.com/2011/02/forecasting/