You get what you measure
•Download as PPT, PDF•
7 likes•1,417 views
This is a presentation given by Harriet Wakelam and Ash Donaldson at Next Bank Hong Kong looking at how we find metrics that reflect evolution of product and service experiences.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to You get what you measure
Similar to You get what you measure (20)
More from Harriet Wakelam
More from Harriet Wakelam (11)
Recently uploaded
Recently uploaded (20)
You get what you measure
- 1. You get what you measureHarriet Wakelam @hwakelam Ash Donaldson @ashdonaldson
- 2. Every lifecycle follows a similar process and sometimes metrics change behaviorin the wrong way.
- 3. By Sun14916 (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons
- 4. self determination + feedback + that’s weird moment = curiosity
- 5. Evolution .
- 6. Science vs Bacteria Adaptation Science Bacteria Change Observation sparks curiosity. Hypotheses are formed. Environmental change create the pressure to adapt. Variety Conduct a series of experiments to test the hypotheses. Mutations during replication results in a variety of genetic traits. Selection Most fail. Some may be successful. Most die. Some may survive. Heredity Draw conclusions. Publish results for other scientists to build on. Surviving organisms reproduce. Their genetics are passed on for further generations to build on.
- 7. US = accountability, standardisation Finland = responsibility autonomy Finland = responsibility autonomy US = accountability, standardisation Finland = responsibility autonomy Finland = responsibility autonomy
- 8. Finland vs USA 2009 PISA Scores
- 9. “A measurement is used to gauge some quantifiable component of a company's performance”
- 11. “Companies are dying as kids” numbers + incentives + accountability = death
- 13. Can’t change the Person But can change Behaviour by changing the Environment B = f( )EP Organism Product Organisation Industry
- 14. Intent not incentive Goodquestions create responsibility for understanding good problems
Editor's Notes
- Let us tell you some stories about how metrics can help and hinder and about adaptive expertise,
- H: Our stories are about lifecycles and human behaviour, and how we can learn from them.
- A: Let’s start with a story of science. The story of penicillin. science, evolution and how they are similar Alexander Fleming Staphylococcus culture - “That’s weird.” Observation fueled curiosity. Hypothesis. Series of experiments. Most failed. One succeeded. Penicillin- the birth of antibiotics - save millions of lives; revolutionise medicine. Fleming published his results. Other scientists built on his discovery. Refined other antibiotics. Pharmaceutical industry was born.
- H: Explore, understand and fail in my own right Self determination, think, feel and do - Measures create frameworks, and curiosity arises when things fall outside frameworks. the thats weird moment -that’s the basis of science
- A: But this was the start of a war with the most successful organism on earth: bacteria. Nature’s killer app. When the environment becomes hostile, it quickly adapts. High mutation and reproduction rates. As Christel called it - the Fecundity of bacteria Penicillin may kill 99.9% of bacteria Superbugs have evolved to be resistant to everything we throw at them. But it’s an even battle. Evolution and the Scientific method follow the same pattern.
- A: Everything big started small. Change; Variety; Selection / Survival; Heredity Biological, cultural, technological, or financial. We can see it also in all design processes: human-centred design, design thinking, lean startup, etc In design we often test one thing instead of many slight variations of the same concept Neal Cross’s home loan form button could be tested on both the left and right
- H: Finland started with intent, education- autonomy and responsibility to schools US started with accountability reach #1 with value for money what happens if you incentivise systems And is started from a Question - a powerful question and powerful leadership delivering the ability to collaborate creatively around a problem. The numbers tell the story - US vs Finland, standardisation vs domains of enquiry. US education has become standardised. Standardised curriculum, lessons, and tests. Made schools and teachers accountable by making federal funding dependent on annual test scores. The number became the goal. Teachers teach to the test. Finnish education is decentralised. The school and its teachers feel responsible for the education of their students. Student education is the goal. Testing is small and frequent, providing feedback to both the students and the teacher, so both can adapt their methods and continuously improve. ----- standardisation evolution assessment embedded in the learning process so both teachers and learners evolve. even when doing the presentation all we could find was point in time. Finland education is decentralised. The school is responsible for the education of their students US education became standardised. The school is accountable for the test scores they achieve. This is linked to
- H: And a system which spends 30% less than the US - teachers 15 years from graduation get 105% of the graduate salary us 62% Program for international student assessment: Finland #1 for science, 2nd in reading and maths. US #25 in Maths, #17 in science, #12 in reading. 93% graduate from high school vs 74% in US. Program for international student assessment: Finland #1 for science, 2nd in reading and maths. US #25 in Maths, #17 in science, #12 in reading. 93% graduate from high school vs 74% in US. Finish students take one standardised test at 16. retention at high school
- H: Good metrics measure that which is evolving as well as that which is real right now. Csat NPS CES VOC Harriet What are metrics anyway How do you know that adaptation has happened, progress has occurred or success/failure - you measure it. Point in time What if you can measure IMPACT and evolving behaviour Real metrics do not introduce competition between say online banking and branch, they encourage people to collaborate across real problems to innovate. What STORY are CSat, NPS, CES telling us? Are they telling us a spreadsheet, or are they giving us excitement evolution and temptation - We have touched on measurement through the structures that enable at a macro level the Finnish education system to survive, at a meta level evolution to function - Scientific method supports us to evaluate evolution, metrics support us in evaluating organisational, business progress.. But does it (CSat, NPS, CES, etc) help us measure and effect evolutionary process in banking? PFM - point in time vs period in time. --- What metrics should do is to show performance against
- A: Behavioural economics has uncovered plenty about how we really behave - how our Triune brain makes us irrational Loss aversion - We feel the pain of loss around 2.5 times the satisfaction of gain. Impact bias - We tend over exaggerate the impact of future events. Over-justification effect - People pay more attention to the reward for an activity rather than the inherent enjoyment.... If you’ve linked a KPI to an incentive - people get scared - loss aversion, impact bias crushing innovation.
- H: Digital darwinism, Steve Monaghan People tied to numbers creates accountability for a number, and competition over an outcome. “ the systematic gathering and analysis of anecdote . Kevin von Appen We've linked numbers to incentives (Neal Cross, Steve Monaghan - told us yesterday about digital darwinism - and acquired innovation defiency syndrome - companies are pouring money into the wrong thing - should be variety and selection and heredity. Moores law doubling in speed every year Acquired innovation deficiency syndrome. CONVERT unstructured data to numeric representations…
- A: Customers are chasing numbers too More importantly we have made numbers the incentives to our customers - come with us for a better rate, not come with us for a better outcome. This has generated the arguments circulating around banking evolution such as ‘is the branch dead’ let’s build an app for that - it’s all about transaction, and has wholeheartedly failed to build consensus, curiosity and excitement about what banking could be. Behaviour is a function of a person and their environment - Kurt Lewin
- Behaviour is a function of a person in their environment - Lewin Ash first then H story Ash’s: During research for a recent project, I was chatting with a lady around my age - we’ll call her Julia - about her finances. She told me that she was a customer with her bank for 10 years. They started advertising a new high interest savings account with an introductory rate that was better than her term deposit, so she went into the bank and asked about it. The nice teller told Julia “I’m sorry, this offer is only available to new customers.” “ But I’ve been a loyal customer for 10 years!” she replied. The teller said she couldn’t do anything, so Julia closed her account and moved to another bank. The new bank made it very easy to open a high interest savings account. It had an introductory offer for 4 months. Julia told me that she’ll be moving that money again in four months. “ You get penalised for being loyal to banks” she told me. “Every few months I can just move my money and I’ll get more flexibility and a better rate than my old term deposit.” This has been an increasing trend in my research: customers are adapting to the changing environment. No longer do they have a relationship with their bank. They’re actively chasing introductory offers - whether they be high interest savings, or 0% credit card balance transfers. In an effort to attract new customers (to meet someone’s KPI), the banks are making it easier to originate products, providing more competitive incentives, and in turn, making their customers feel as though they’ll be penalised for being loyal. It’s a vicious circle of banks cannibalising each other’s business in a race to lowering profit margins. Tim Kobe quoting Steve Jobs - not about speeds and feeds - that’s a race to the bottom Harriet's:
- Ash & Harriet A: We start with a hypothesis - the intent of where we want to be. H: We frame good questions which excite curiosity A: People and teams become responsible for understanding and resolving juicy problems, not chasing numbers. this happens everywhere
- H: We’ve shown how linking incentives to metrics can be a dangerous thing A: We know that we can build a more efficient framework for design H: Curiosity is the seed for collaboration, discovery and innovation. A: We’d like to leave you with a question - how do you want to build responsibility in YOUR organisation.