O slideshow foi denunciado.

The Ultimate Metric

7

Compartilhar

Próximos SlideShares
A Programmer's Guide to Humans
A Programmer's Guide to Humans
Carregando em…3
×
1 de 112
1 de 112

The Ultimate Metric

7

Compartilhar

Baixar para ler offline

Since the dawn of software development, we've struggled with a huge disconnect between the management world and the engineering world. We try to explain our problems in terms of "technical debt", but somehow the message seems to get lost in translation, and we drive our projects into the ground, over and over again.

What if we could detect the earliest indicators of a project going off the rails, and had data to convince management to take action? What if we could bridge this communication gap once and for all?

In this session, we'll focus on a key paradigm shift for how we can measure the human factors in software development, and translate the "friction" we experience into explicit risk models for project decision-making.

Since the dawn of software development, we've struggled with a huge disconnect between the management world and the engineering world. We try to explain our problems in terms of "technical debt", but somehow the message seems to get lost in translation, and we drive our projects into the ground, over and over again.

What if we could detect the earliest indicators of a project going off the rails, and had data to convince management to take action? What if we could bridge this communication gap once and for all?

In this session, we'll focus on a key paradigm shift for how we can measure the human factors in software development, and translate the "friction" we experience into explicit risk models for project decision-making.

Mais Conteúdo rRelacionado

Livros relacionados

Gratuito durante 30 dias do Scribd

Ver tudo

Audiolivros relacionados

Gratuito durante 30 dias do Scribd

Ver tudo

The Ultimate Metric

  1. 1. The Ultimate Metric Janelle Klein @janellekz
  2. 2. How do we measure a human system that’s constantly learning and changing, and becoming something different? What are we trying to measure and why?
  3. 3. What problem are we trying to solve?
  4. 4. 1. Define the Problem 2. Shared Language 3. Metrics: Paper & Pencil 4. Summary & Discussion
  5. 5. What problem are we trying to solve? 1. Define the Problem PAIN
  6. 6. PAIN: Why should I care? Business Leader
  7. 7. Across the Industry… Start% Over% Unmaintainable% So0ware%
  8. 8. We Start with the Best of Intentions High Quality Code Low Technical Debt Easy to Maintain Good Code Coverage
  9. 9. Then This Happens!
  10. 10. Stages of Escalating Risk Product Owner: “We’ve got more important things to do.” Deferring( Problems(
  11. 11. Deferring( Problems( Painful( Releases( Manager: “Good job everyone! Keep up that great work ethic!” Stages of Escalating Risk
  12. 12. Deferring( Problems( Painful( Releases( Thrashing) Manager: “We need to go faster! Let’s hire more developers.” Stages of Escalating Risk
  13. 13. Deferring( Problems( Painful( Releases( Thrashing) Project( Meltdown( Developer: “I give up. I don’t care anymore if the project fails.” Stages of Escalating Risk
  14. 14. RESET “A description of the goal is not a strategy.” -- Richard P. Rumelt What’s wrong with our current strategy?
  15. 15. Our “Strategy” for Success High Quality Code Low Technical Debt Easy to Maintain Good Code Coverage
  16. 16. RESET “A good strategy is a specific and coherent response to— and approach for overcoming—the obstacles to progress.” -- Richard P. Rumelt The problem is we don’t have a strategy...
  17. 17. Obstacle: Breakdown in Communication Visibility No Control Control No Visibility What happens?
  18. 18. Imagine Organization as a Robot Arm Controller Hand PAIN Sensor
  19. 19. Organization = Robot driving a car Planning = Aiming the car
  20. 20. Investment = Push the gas pedal Organization = Robot driving a car
  21. 21. Risk Management - Steering in response to emerging risks Organization = Robot driving a car
  22. 22. Reality is more like an “Agile Barge” We are failing to learn from our mistakes
  23. 23. What’s the Strategy? Visibility No Control Control No Visibility
  24. 24. What’s the Strategy? Team 1 Team 2 Team 3 Support System Shared Language
  25. 25. Strategy: Build a Learning Organization Shared Language
  26. 26. 1. Define the Problem 2. Shared Language 3. Metrics: Paper & Pencil 4. Summary & Discussion
  27. 27. PAIN is a stress signal in our bodies that tells us we ought to move away from something What is PAIN?
  28. 28. What is PAIN? Stress Wave What if we could make this visible? Happy Engineers Run Away from Misery!
  29. 29. Define a Shared Language How to Measure the PAIN in Software Development Janelle Klein leanpub.com/ideaflow
  30. 30. Great Team Disciplined with Best Practices Constantly Working on Improvements+ Project FAILURE About 10 Years Ago…
  31. 31. “What’s the best opportunity for improvement? “The awful email template engine code!” Our biggest problem The Retrospective
  32. 32. “Fill in missing unit tests!” Our biggest problem The Retrospective “What’s the best opportunity for improvement?
  33. 33. “We should clean up the database code!” Our biggest problem The Retrospective “What’s the best opportunity for improvement?
  34. 34. “Let’s improve maintainability of our test framework!” Our biggest problem The Retrospective “What’s the best opportunity for improvement?
  35. 35. Just because a problem comes to mind, doesn’t mean it’s an important problem to solve. Our biggest problem The Retrospective “What’s the best opportunity for improvement?
  36. 36. Our biggest problem What do I feel the most intensely about? Daniel Kahneman Thinking Fast and Slow The Retrospective “What’s the best opportunity for improvement?
  37. 37. “The awful email template engine code!” Recency Bias Our biggest problem The Retrospective “What’s the best opportunity for improvement?
  38. 38. Guilt Bias “Fill in missing unit tests!” Our biggest problem The Retrospective “What’s the best opportunity for improvement?
  39. 39. Known Solution Bias Our biggest problem The Retrospective “What’s the best opportunity for improvement? “We should clean up the database code!”
  40. 40. Sunk Cost Bias “Let’s improve maintainability of our test framework!” Our biggest problem The Retrospective “What’s the best opportunity for improvement?
  41. 41. The Retrospective “What’s the best opportunity for improvement?
  42. 42. The Retrospective “What are we supposed to do?!” Our biggest problem “The problem is all the technical debt is causing us to make mistakes!”
  43. 43. “Without data you’re just another person with an opinion.” — W. Edwards Deming
  44. 44. Technical Debt Mistakes Hypothesis: Technical Debt increases likeliness of Mistakes
  45. 45. ? Data: Most of our mistakes were in the most well-written parts of the code Mistakes
  46. 46. Rules Engine Looks okay. Alert! Measurements Config UI code Charting UI Code System Architecture SPC Monitoring and Alert System Tools
  47. 47. Beautiful Looks okay. Alert! Measurements UGLY UGLY System Architecture SPC Monitoring and Alert System Tools
  48. 48. Data: We made significantly more mistakes in code that we didn’t write ourselves Lower Familiarity More Mistakes= There had to be more to the story...
  49. 49. Complex( So*ware( This is what I knew... What makes development feel painful?
  50. 50. Unexpected Observation Problem Resolved Tracking Painful Experience with the Code Troubleshooting Progress 5 hours and 18 minutes of troubleshooting... PAINFUL
  51. 51. The amount of PAIN was caused by… Likeliness(of(( Unexpected( Behavior( Cost(to(Troubleshoot(and(Repair( High(Frequency( Low(Impact( Low(Frequency( Low(Impact( Low(Frequency( High(Impact( PAIN(
  52. 52. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger Most of the pain was caused by human factors. What causes PAIN?
  53. 53. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger What causes PAIN? Most of the pain was caused by human factors. Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm
  54. 54. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger What causes PAIN? Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm Most of the pain was caused by human factors.
  55. 55. PAIN occurs during the process of understanding and changing the software Complex( So*ware( Not the Code. Optimize “Idea Flow”
  56. 56. My team spent tons of time working on improvements that didn’t make much difference. We had tons of automation, but the automation didn’t catch our bugs.
  57. 57. My team spent tons of time working on improvements that didn’t make much difference. We had well-modularized code, but it was still extremely time-consuming to troubleshoot defects.
  58. 58. “What are the specific problems that are causing the pain?” The hard part isn’t solving the problem, it’s identifying the right problem to solve
  59. 59. “What are the specific problems that are causing the pain?” Continuous Acceleration
  60. 60. PAIN is a stress signal in our bodies that tells us we ought to move away from something What is PAIN? What do I feel the most intensely about?
  61. 61. Our PAIN Sensors are Terribly Miscalibrated! Ugliness bothers us a LOT Moderate Difficulty Is Enjoyable We can recalibrate our sensors with data!!!
  62. 62. Run Experiment Our Perception of Time is WAY OFF Setup Experiment Analyze Results & Decide Next Experiment Execute waiting - time goes slow doing - time zooms by
  63. 63. Language of Idea Flow How to Measure the PAIN in Software Development Janelle Klein leanpub.com/ideaflow
  64. 64. Make Sense? The process of communicating an idea according to an Intention, and validating that the idea was understood What is Idea Flow? Intention Steering Loop
  65. 65. The process of communicating an idea according to an Intention, and validating that the idea was understood What is Idea Flow? Intention Steering Loop Software System
  66. 66. What is Idea Flow? Process Output Target Intention Steering Loop
  67. 67. Software Sculpture Friction “In the Wires” Idea Flow Network Not “in the thing”
  68. 68. What is Friction?
  69. 69. What is Friction? Time to recover understanding when observable behavior doesn’t match expectations WTF?! YAY! Friction measures the Frequency & Duration of Cognitive Dissonace
  70. 70. Likeliness(of(( Unexpected( Behavior( Cost(to(Troubleshoot(and(Repair( High(Frequency( Low(Impact( Low(Frequency( Low(Impact( Low(Frequency( High(Impact( PAIN( What is Friction? Friction measures the Frequency & Duration of Cognitive Dissonace
  71. 71. Friction is a result of our Decision Habits Write a little code at a time and work out the kinks 4:02 Skip the tests… 11:35
  72. 72. Difficulty increases Exponentially Difficulty of Work Increases Ultimate Constraint Human Limitations Once we lose our ability to understand cause and effect, we lose our ability to deliver software
  73. 73. Quality Target Lower Control Limit Upper Control Limit X" Perfect Quality Upper Control Limit Lower Control Limit Process Control in Lean Manufacturing Lower Variability => Better Control Out of Control (OOC)
  74. 74. Optimal Friction Upper Control Limit X" “Out of Control” 20min 0m 50m 0m I understand Perfectly Confusion Limit X Out of Control (OOC) Lower Variability => Better Control Idea Flow in Software Development
  75. 75. Product( Requirements( So1ware(Task( Possible(Decision(Paths( So1ware( Software is a Reflection of our Decision Habits
  76. 76. Product( Requirements( So1ware(Task( High%Risk% Decision%Habits% So1ware( (and(all(our(problems)( ( RESET Software is a Reflection of our Decision Habits
  77. 77. Product( Requirements( So1ware(Task( High%Risk% Decision%Habits% So1ware( (and(all(our(problems)( ( Technical Debt is just a Symptom of the Problem Software is a Reflection of our Decision Habits
  78. 78. Time% Pressure% Compromise% Safety% for% Speed% Increase% Number%&% Severity%of% Hazards% % More%Pain% and%Higher% Task%Effort% Constant' Urgency' Cycle of Chaos High-Risk Decision Habits
  79. 79. Time% Pressure% Compromise% Safety% for% Speed% Increase% Number%&% Severity%of% Hazards% % More%Pain% and%Higher% Task%Effort% Constant' Urgency' Cycle of Chaos High-Risk Decision Habits
  80. 80. Time% Pressure% Compromise% Safety% for% Speed% Increase% Number%&% Severity%of% Hazards% % More%Pain% and%Higher% Task%Effort% Constant' Urgency' Cycle of Chaos High-Risk Decision Habits
  81. 81. Time% Pressure% Compromise% Safety% for% Speed% Increase% Number%&% Severity%of% Hazards% % More%Pain% and%Higher% Task%Effort% Constant' Urgency' Cycle of Chaos High-Risk Decision Habits
  82. 82. Time% Pressure% Compromise% Safety% for% Speed% Increase% Number%&% Severity%of% Hazards% % More%Pain% and%Higher% Task%Effort% Constant' Urgency' Cycle of Chaos High-Risk Decision Habits
  83. 83. Cycle of Chaos Escalates Risk Likelihood)of)) Unexpected) Behavior) Cost)to)Troubleshoot)and)Repair) High)Frequency) Low)Impact) Low)Frequency) Low)Impact) Low)Frequency) High)Impact) PAIN)
  84. 84. Fewer% Problems%to% Fix% Stop%% and% Think% Mi8gate% the% Risk% Increased% Produc8vity% and% Innova8on% Safety' Cycle of Safety Low-Risk Decision Habits
  85. 85. Fewer% Problems%to% Fix% Stop%% and% Think% Mi8gate% the% Risk% Increased% Produc8vity% and% Innova8on% Safety' Cycle of Safety Low-Risk Decision Habits
  86. 86. Fewer% Problems%to% Fix% Stop%% and% Think% Mi8gate% the% Risk% Increased% Produc8vity% and% Innova8on% Safety' Cycle of Safety Low-Risk Decision Habits
  87. 87. Fewer% Problems%to% Fix% Stop%% and% Think% Mi8gate% the% Risk% Increased% Produc8vity% and% Innova8on% Safety' Cycle of Safety Low-Risk Decision Habits
  88. 88. Fewer% Problems%to% Fix% Stop%% and% Think% Mi8gate% the% Risk% Increased% Produc8vity% and% Innova8on% Safety' Cycle of Safety Low-Risk Decision Habits
  89. 89. Likelihood)of)) Unexpected) Behavior) Cost)to)Troubleshoot)and)Repair) High)Frequency) Low)Impact) Low)Frequency) Low)Impact) Low)Frequency) High)Impact) PAIN) Cycle of Safety Reduces Risk
  90. 90. Breakdown in Communication Message isn’t getting across the wall
  91. 91. Technical Debt is a Broken Metaphor
  92. 92. Explained the problem of Technical Debt Business Coaching “That doesn’t sound so bad.” The Response: ? ? ? ? WHAT?!
  93. 93. Loans are a Predictable Financial Tool Revenue - Cost Profit + 10% Increase Price? Increase Sales? Reduce Cost? What makes investment decisions harder isn’t higher costs, it’s lower predictability. Investment Strategy
  94. 94. Likelihood)of)) Unexpected) Behavior) Cost)to)Troubleshoot)and)Repair) High)Frequency) Low)Impact) Low)Frequency) Low)Impact) Low)Frequency) High)Impact) PAIN) Better: Explain Dynamics of Risk Once we lose our ability to understand cause and effect, we lose our ability to deliver software
  95. 95. “Technical Debt” “Escalating Risk” Predictable increase in cost over time Can add resources to compensate Bias around problems in the code Probabilistic model captures loss of predictability Adding resources increases risk Problems don’t have to be nouns Likelihood)of)) Unexpected) Behavior) Cost)to)Troubleshoot)and)Repair) High)Frequency) Low)Impact) Low)Frequency) Low)Impact) Low)Frequency) High)Impact) PAIN) How is this different?
  96. 96. 1. Define the Problem 2. Shared Language 3. Metrics: Paper & Pencil 4. Summary & Discussion
  97. 97. 2.#Record#an#Idea#Flow#Map# So#ware(Task( Strategy(( Design( Experience( Review( Iden:fy(the(Pa>erns( 1.#Clarify#the#Strategy# So#ware(Task( 3.#Reflect#on#Decisions# Focus( Mee:ng( 4.#Share#Discoveries# Continuous Acceleration Framework Strategy Walk Reflection Walk Mastery Circle Flow Metrics Measure WTFs Look for Patterns Identify Team Pains The hard part is identifying the right problem to solve
  98. 98. 1. What’s the goal of this task? 2. What changes will be the most mistake-prone? 3. What changes will be most difficult to debug? 4. What can we do to mitigate those risks? Talk through the Risks before the Task
  99. 99. Reflect on Decisions after the Task 1. What were the biggest WTFs? 2. What made troubleshooting take so long? 3. What decisions would you make differently if you could do this task again? 4. What could we do to reduce risk on future tasks?
  100. 100. Measure WTFs during the Task WTF?! YAY! Jot down start time Jot down end time Write a note about what behavior you observed, and the discovered cause
  101. 101. 1. Define the Problem 2. Shared Language 3. Metrics: Paper & Pencil 4. Summary & Discussion
  102. 102. RESET “A good strategy is a specific and coherent response to— and approach for overcoming—the obstacles to progress.” -- Richard P. Rumelt Think in terms of Obstacles & Strategy
  103. 103. The hard part isn’t solving the problem, it’s identifying the right problem to solve Continuous Acceleration is about optimizing for Leverage
  104. 104. Just because a problem comes to mind, doesn’t mean it’s an important problem to solve. Our biggest problem The Retrospective “What’s the best opportunity for improvement?
  105. 105. Our PAIN Sensors are Terribly Miscalibrated! Ugliness bothers us a LOT Moderate Difficulty Is Enjoyable We can recalibrate our sensors with data!!!
  106. 106. Product( Requirements( So1ware(Task( High%Risk% Decision%Habits% So1ware( (and(all(our(problems)( ( RESET Optimize the Quality of Decisions, not the Code Better Steering
  107. 107. Time% Pressure% Compromise% Safety% for% Speed% Increase% Number%&% Severity%of% Hazards% % More%Pain% and%Higher% Task%Effort% Constant' Urgency' Cycle of Chaos High-Risk Decision Habits
  108. 108. Fewer% Problems%to% Fix% Stop%% and% Think% Mi8gate% the% Risk% Increased% Produc8vity% and% Innova8on% Safety' Cycle of Safety Low-Risk Decision Habits
  109. 109. Human Understanding is the Ultimate Constraint Difficulty of Work Increases Ultimate Constraint Human Limitations Reduce Troubleshooting Difficulty Once we lose our ability to understand cause and effect, we lose our ability to deliver software
  110. 110. WTFs are the Ultimate Metric
  111. 111. Teach Everyone the Language of Idea Flow Team 1 Team 2 Team 3 Support System Shared Language Get the book: leanpub.com/ideaflow
  112. 112. The Ultimate Metric Optimize for Happy Humans! Thank you! janelle@dreamscale.io @janellekz Sign up for Newsletter: news.teamscience.io

×