Avoiding the Pitfalls of Vanity Metrics
•Transferir como PPTX, PDF•
3 gostaram•2,633 visualizações
The document discusses the pitfalls of using "vanity metrics" which look good on paper but don't provide meaningful information. It provides examples of vanity metrics like release timelines filled with unrealistic assumptions. Large companies often focus on vanity metrics that prioritize predictability over innovation, taking small safe steps rather than risks. This can lead them to get stuck at local maxima rather than reaching their true maximum potential. The document advocates for focusing on learning and experimentation to drive growth rather than vanity metrics.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Avoiding the Pitfalls of Vanity Metrics
Semelhante a Avoiding the Pitfalls of Vanity Metrics (20)
Mais de Janna Bastow
Mais de Janna Bastow (14)
Último
Último (20)
Avoiding the Pitfalls of Vanity Metrics
- 1. Avoiding the Pitfalls of Vanity Metrics @simplybastow
- 2. Vanity metrics Numbers or stats that look good on paper, but don’t really mean anything important.
- 4. Hello! I’m Janna Bastow You can talk to me about product or say hi at @simplybastow Get the slides: bit.ly/vanity-metrics
- 5. (really big product management community)
- 6. (really nice product management tool)
- 8. JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC Designed to fail (and not in the good way) Slides: bit.ly/vanity-metrics
- 9. Time Thingstodo Designed to fail (and not in the good way) Slides: bit.ly/vanity-metrics
- 10. Time Feature A Thingstodo Designed to fail (and not in the good way) Slides: bit.ly/vanity-metrics
- 11. Feature A Thingstodo Time Feature B Designed to fail (and not in the good way) Slides: bit.ly/vanity-metrics
- 12. Feature A Thingstodo Feature B Time Feature C Designed to fail (and not in the good way) Slides: bit.ly/vanity-metrics
- 13. Feature A Thingstodo Feature B Time Feature C Designed to fail (and not in the good way) Slides: bit.ly/vanity-metrics
- 14. Thingstodo Time Due dates & duration ALL THE THINGS! Designed to fail (and not in the good way) Slides: bit.ly/vanity-metrics
- 15. Thingstodo Time ALL THE ASSUMPTIONS! Full of assumptions Slides: bit.ly/vanity-metrics
- 16. You assume you know how much work and how long each feature is going to take. Assumption #1 Time Feature A Feature B Feature C Slides: bit.ly/vanity-metrics
- 17. You assume that nothing else is going to disrupt your timeline. Assumption #2 Time Slides: bit.ly/vanity-metrics
- 18. You assume that each feature will work as soon as it is launched. Assumption #3 ✔ ✔ ✔ Slides: bit.ly/vanity-metrics
- 19. You assume that each of these features actually deserves to exist! Assumption #4 ✔ ✔ ✔ Slides: bit.ly/vanity-metrics
- 20. The grand assumption: Nothing’s going to change.
- 21. Made up release dates Development death marches Mismanaged expectations Missed market opportunities Building the wrong thing Sad team What could possibly go wrong? Slides: bit.ly/vanity-metrics
- 24. What kind of organisation are you? Nimble and lean Slow and risk-averse Slides: bit.ly/vanity-metrics
- 25. What kind of organisation are you? Discovery led Capitalizing on market Slides: bit.ly/vanity-metrics
- 26. What kind of organisation are you? Experimentation- driven Ruled by vanity metrics Slides: bit.ly/vanity-metrics
- 27. So what’s going on in these companies?
- 29. Just keep growing... Slides: bit.ly/vanity-metrics
- 31. Best route to predictability = Small manageable chunks
- 34. Counterintuitive Business Decisions Slides: bit.ly/vanity-metrics
- 36. Profit Center vs Cost Center Slides: bit.ly/vanity-metrics
- 38. Mis-incentivised Product development / R&D
- 40. JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC Slides: bit.ly/vanity-metrics
- 42. No room for discovery
- 43. Mis-incentivised Sales / Marketing
- 45. Have a North Star Metric
- 47. Less is more
- 49. It’s a trap! Slides: bit.ly/vanity-metrics
- 50. Product Life Cycle You are here Introduction DeclineMaturityGrowth Salesvolume/Usage Slides: bit.ly/vanity-metrics
- 51. Large corporations struggle to grow Walmart <1% Ford <0% Coca Cola <0% Dow Chemical <0% Via Janice Fraser - @clevergirl Slides: bit.ly/vanity-metrics
- 52. Local maximum True maximum Slides: bit.ly/vanity-metrics
- 53. Find maximum value Break out of local maxima Slides: bit.ly/vanity-metrics
- 54. Innovation requires taking risks Break out of local maxima
- 55. Chart that shows companies from 20 years ago vs today
- 56. Via Suzie Prince - @pm_suzie
- 57. “Every company that ever has or does exist, will one day go bust. It’s just a matter of when.”
- 59. Be a Netflix in a world of Blockbusters
- 60. Further reading on ditching your timeline roadmap: www.bit.ly/growing-up-lean
- 64. Thank you! I’m Janna Bastow Let’s go for or and talk product to @simplybastow or janna@prodpad.com Slides: bit.ly/vanity-metrics
Notas do Editor
- Get to the root: Caused by mismanagement, Competition internally Totally depend on your product and business
- You might know me from my work with Mind the Product, a huge global community of product people that I co-founded.
- Or from my work on ProdPad, a b2b SaaS company that I also co-founded. ProdPad is a tool that 1000s of companies around the world use to manage their products. It allows teams to gather insights, suggestions and feedback, and prioritise product changes on a roadmap. It connects to tools like Jira, Trello, Slack and many others that I’m sure you’re all familiar with. Give it a try! We’ve got a promotion available on this via the Product-Led Summit site, so check that out after you finish watching today.
- You might know me as the person who talks about why timeline roadmaps are such a bad idea. And as a matter of fact, they are related to one of the most vain of your vanity metrics. Let me explain.
- I know how this format makes you look great today… your board and your bosses certainly love when you can give them this level of certainty. But it’s setting you up for failure!
- See, if you deconstruct this format of roadmap, you basically get a chart that maps out ‘time’ versus ‘things to do’. You have time on the x-axis, creating this timeline.
- And at first, it’s seems pretty easy and intuitive to use,
- especially in the short term,
- but the further out you plan, and the more you put on there,
- the harder it becomes to manage.
- And because that timeline sits at the top, always marching forward, no matter what you put on the roadmap, they always include a due date and a time estimate – just by the format of this roadmap!
- And as a result, you end up with a big pile of features, a big pile of deadlines, all based on this big pile of assumptions.
- First assumption that you’ve made with this format is that you know how long each of these is going to take. This might be easy for the stuff you’ve already broken down into more detail and had the developers give some estimates, but the further down the list you go, the less clarity you’ve actually got.
- You’re also assuming that nothing else is going to come in to mess up your timeline. No changes in the market, no new competitors, no fresh ideas coming from customers, no need for iteration.
- You’re also assuming that each of these features will work as soon as they launch. You put 3 weeks to build that new checkout page? Then at the end of three weeks, it’ll be converting exactly as well as expected, and you’re free to move on to the next thing.
- And by explicitly adding these features to the timeline, you’re assuming that each of these features should definitely exist! That they form part of the strategy and should therefore be codified.
- At the end of the day, you’re making one big, dangerous assumption: That nothing is going to change
- So what could go wrong? made up release dates forcing your developers on stressful marches to launch on time sales and customers get expectations you can’t meet missing opportunities in the market downright building the wrong things and this leads to you, having a sad and stressed out team
- Individuals get it, but somehow organisations as a whole don’t
- Not just large enterprises, though they are most susceptible Holds true for smaller companies too. Thinks about things on a continuum
- So I end up fielding a bunch of questions from people who are absolutely right - they CAN’T just rock up to work tomorrow having ditched their timeline roadmap. I get that this new way isn’t going to just slot comfortably in with their regular processes. Their company isn’t ready for that yet. Here’s what’s really going on at those companies, and how to get past this.
- Fiduciary duty to increase shareholder value
- CEOs are incentivised to show quarter on quarter growth. If they don't, they are turfed. And so management style lends itself well to slow improvements, driving costs down or revenues up, but rarely given opportunity to invest in proper future growth.
- Predicablility is a huge factor, the upward growth doesn’t even have to be that much (even in the sub 1% per year), so long as the outcomes match the forecasts. Unpredictability spooks investors, which can have a bigger knock on effect than anything operational happening within the company.
- Break the business up and understand it in smaller, more manageable chunks. And this sort of makes sense too… except what that basically leaves you with is Silos
- Silos
- And that’s how vanity metrics are born. Which leads us into dangerous territory, where vanity metrics rule. Division leads actually find themselves competing to max their own proprietary metrics, instead of collaborating for the good of the business.
- Which lead to silo’d decisions which make sense for the bottom line, but seem counter intuitive.
- Let’s dive into those silos
- Sales = revenue (rev cener) Marketing = ROI campaigns (Revenue center) Development = story points, velocity, burn down charts, man hours. (cost center) Innovation? (cost center) It results in divisions being created, each measured as either a cost center or a profit center, and each measured independently. Squeeze just a little more revenue out of sales and marketing… though if that doesn’t work, push down costs in support, tech, and operations. And you know where product, customer service, and innovation sit? We’re seen as a cost center.
- Vanity metric for product development becomes delivery focused: Talk about shipping features Cadence Story points
- Which basically gets us trapped in this old pattern, of writing finer and finer grained specs, breaking them into ‘points’ which are really just a rough measure of time, and then measuring things like velocity: how quickly your team can burn down a stack of points. At the end of the day, this model of working simply optimises for building the most features as fast as possible.
- It’s output focused when it should be outcome focused. All this pointsing and burning down at high velocity rarely leaves room for discovery, and instead usually leads to team burnout and tech debt
- And if they want to deliver this roadmap, surely the answer is to just throw more developers at it!
- But everyone knows that building a bunch of features isn’t the answer to building a good product.
- Removes ability to be lean No time for discovery Discovery so key to your success
- You know what you get when you mis-incentivise sales and marketing? Shady acquisition tactics that only work in the short term. It bolsters numbers for now, but hinders the company in the long run,
- The best defense against vanity metrics: Have a north Star metric for the whole company What one figure should everyone contribute to in order to enable the health of the business? Maximize revenue from 12+month customers - sales/marketing not incentivised to come up with cheap leads - development incentivised to create a product that’s powerful and performant enough to impress users for the long-run Other objectives should flow into here.
- Problem: Vanity metrics feel like work, so must be work How do you prove your effort? Less is more
- Marie Kondo your stats, and figure out which ones actually lead to success. Anything else, just stop reporting on, and see if it comes up. Each team has one objective per quarter and reports on progress of that.
- One of the reasons companies keep reporting their vanity metrics is when everything’s seeming to go in the right direction.
- Actually _looks_ like it's working, because the big buffers and certainty in deliverables _do_ do a couple things: make costs and timelines more predictable, makes scope more predictable, which is an easier pill to swallow for execs who are, frankly, just trying to hit their numbers for the quarter. And so they get really good at operationalising this process: They get their sliver of growth year on year, and get their bonuses.
- The top of this graph, it plateaus.
- Large companies are growing by fractions of a percentage, if at all.
- BUT, in the long run, these companies are doomed. By optimising for short term, consistent growth, they leave behind the option to be flexible and solve new problems in the future. The industry will pass them by. Smaller startups will always be nipping at their heels. Look what's happening in banking! HSBC vs. a hundred fintech startups.
- The only way to break out of this local maxima is to test areas of operation outside of a comfort zone. And this experimentation can be costly, and for many reasons, can be awkward and uncomfortable for the organisation. Especially an organisation with poor product culture, who are used to deliving results in their silos, but aren't incentivised or 'psychologically safe' enough to test the boundaries. Even if it does save the company in the long run.
- It’s at this point that the CEO starts sweating around the collar.
- May be nihilistic, but also realistic.
- Blockbuster had plenty of chances. They had people internally who advocated for new modern models, but couldn’t get the organisation as a whole to embrace it. They actually passed on the option to buy Netflix for pennies on what it’s worth now, but doing so would have represented a risk; a dip in their tidy and predictable quarter on quarter growth, and their executive team wasn’t ready to stomach it. And we all know how that ended up for Blockbuster, who couldn’t keep up with the times. Kodak is another example. They had the patents and the technology for digital photography, but they didn’t dare cannibalize their high margin old-school film-based product lines. As a result, they were left in the dust because technology and consumer desires change quickly. Another example is Blackberry. Did you know that their most profitable years were actually the 2 years after the iphone was released? They famously didn’t react to the obvious changes in the market, and the comfort of the nice up-and-to-the-right graph gave them all a false sense of security. By the time the numbers started to fall, they were so far behind in technology they simply couldn’t compete on any real front.
- So along with telling you things like ditch your timeline roadmap, I’m also going to give you some extra tricks:
- Objectives and key results - designed to align teams around common, company-wide goals, while giving individual teams autonomy find their own best route: Ask how your metrics are going to solve problems for the business, ask how they contribute to the longevity and continuation of the business. Vanity metrics shouldn’t be able to be gamed by one division at the expense of another division. Each division or teams objectives should roll up to tell part of the larger story of the companies vision. Tie those objectives into your roadmap so that they become part of your company strategy
- Change how you’re measured. The old way is to be measured by how many story points you and your dev team can crank out, as a rubric for how much value the company is getting for their investment in the product development division. Your new way is to start reporting on how many experiments you can run. It changes the mindset from delivery of specific features by dates, and instead allows you room for discovery and experimentation, while still giving the exec team an idea of what they actually get out of their investment. After all, all they want is for this smart team that they’ve hired to do the best they can to maximise their investment. It’s up to us, the product people, to point out that counting stories is counter intuitive, and that counting experiments is a much better indicator of long term success.
- The best way to get through to any person is to speak their language. PMs can speak about customer journey mapping and discovery until the cows come home, but we often struggle to communicate upwards to our execs. Our execs want to know what’s going to affect the bottom line, not what the sentiment was on that last user test. Shortcut: show the money, make a case in the language that makes sense to the business. Talk about Risk. Talk about how their investment in the company is at risk if they don’t invest in the right areas to defend against competition and market forces. Talk about the product portfolio the same way a hedge fund manager would talk about their investments. They wouldn’t put everything in purely safe slow moving bonds, nor would they put everything in high risk startups. Explore your company’s appetite for risk, and propose micro investments in initiatives that make sense in the wider picture. Google might be able to justify 20% of their time in R&D, but that’s probably not reasonable for everyone. Can you convince your team to start with 2%, or even just 0.2%? This is the remit of our product leaders; to help their organisations become product led.
- And so with that, I thank you! I’m sure many of you have questions, so feel free to connect with me on LinkedIn, find me on Twitter as simplybastow, or drop me an email. If we’re ever in the same city, let’s go for a coffee or a beer and chat! I look forward to hearing about the experiments you’ll be running next, and I’m happy to share my notes on what we’re working on too.