O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Improving AI products with Analytics

758 visualizações

Publicada em

Artificial Intelligence is here to stay and drastically improve our lives. However as with any emerging tech, there is been a FOMO rush to get something (AI-As-A-Brand) out which led to creation of AI products first and then looking for customers and problems to solve. Creating products that drive real impact at scale requires loving your "customers and their problems" instead of loving the "product that you created". It means commitment, persistence and humility to identify real customer needs, give your everything to meet it and learn & improve along the way. The framework of "Learn-Listen-Test" is perfectly to do this at scale and effectiveness by marrying together Reporting to monitor KPIs, Analytics to explain the reasons behind things, User Research to contextualize it and Experimentation to pick the best solution. AI Product leaders today became who they are by going back to the basics and learning their way to become integral part of our lives and we should emulate them as we think of our own products.

Publicada em: Dados e análise
  • Entre para ver os comentários

Improving AI products with Analytics

  1. 1. www.productschool.com Part-time Product Management, Coding, Data Analytics, Digital Marketing, UX Design and Product Leadership courses in San Francisco, Silicon Valley, New York, Santa Monica, Los Angeles, Austin, Boston, Boulder, Chicago, Denver, Orange County, Seattle, Bellevue, Washington DC, Toronto, London and Online
  2. 2. FREE INVITE Join 35,000+ Product Managers on
  3. 3. +10,000 alumni graduated across 17 campuses · San Francisco · Silicon Valley · New York · Los Angeles · Santa Monica · Orange County · Austin · Boston · Boulder · Chicago · Denver · Seattle · Washington DC · Toronto (Canada) · London (UK) · Bellevue · Online
  4. 4. COURSES Product Management Learn the skills you need to land a product manager job
  5. 5. COURSES Coding for Managers Build a website and gain the technical knowledge to lead software engineers
  6. 6. COURSES Data Analytics for Managers Learn the skills to understand web analytics, SQL and machine learning concepts
  7. 7. COURSES Digital Marketing for Managers Learn how to acquire more users and convert them into clients
  8. 8. COURSES UX Design for Managers Gain a deeper understanding of your users and deliver an exceptional end-to-end experience
  9. 9. COURSES Product Leadership for Managers For experienced Product Managers looking to gain strategic skills needed for top leadership roles
  10. 10. COURSES Corporate Training Level up your team’s product management skills
  11. 11. Abhishek Guglani & Ramkumar Ravichandran TONIGHT’S SPEAKER
  12. 12. Improve AI using Analytics Ramkumar Ravichandran – Data science TLM, Google Shopping Abhishek Guglani – Director of Product, Visa Inc., Ecommerce Disclaimer: Participation in this summit is purely on personal basis and is not meant to represent VISA or Google’s position on this or any other subject and in any form or matter. The talk is based on learning from work across industries and firms. Care has been taken to ensure no proprietary or work related information of any firm is used in any material.
  13. 13. “Service” mindset in Leadership Simon Sinek - Do you love your wife? 13
  14. 14. AI at peak of inflated expectations last year… https://twitter.com/chucknicecomic/status/744550059447050244 14
  15. 15. •Clear Value Prop & “Need based” User Flow FOMO led to companies missing out on, •Commercializable •Policy Compliant •Scalable & “agile” architecture •Data Driven Optimization 15
  16. 16. 16 https://imgflip.com/i/2uw8bj
  17. 17. What separates the “leader” from the pack? In 2017... ...and now https://www.dietworkoutfitness.com/tag/alexa-photos.html 17
  18. 18. Product Leadership Checklist! • “Design” thinking- anchored to “User-Needs-Context” • “Engagement” focus right from start • “Learn-Listen-Test” launch & ramp • “Partner” ecosystem & support • “Loyalty” pricing 18
  19. 19. 19 https://imgflip.com/i/2uwarz
  20. 20. ExtendManageBuildPrototypeDesignPlan Going from left to right on PDLC, instead of first creating product then searching for Customers, • Critical Review of Existing solutions (optimization or differentiator) • Use Cases (Opp, Impact, RoI) • Goals, Success/Stop Criteria • Readiness (Customer, Provider, Regulator, Creator) • Design iteration decisions (by user, by needs, by context) • Tactical: Platform, Program, Process • Use Case Scoring & Prioritization • POC- Success/Lessons, RoI • Optimization/Customization • Review, Stress Test, UAT • Plan & Timelines - Milestones • Evangelize & Engage • Data Driven Optimization • Support, Operations & Distribution • Refine, Revamp or Retire? • ”Learn-Listen-Test” launch • Deploy, Monitor & Iterate • Usage Protocols : Guide & Comply • Innovation & Upgrade 20
  21. 21. Strategy Data Tagging Data Platform Reporting Analytics Research Scale Iterative Loop Optimization …iterating to User-Product-Market fit with “Learn-Listen-Test” framework ✓ Reporting for “monitoring” KPIs ✓ Analytics provides insights into “user behavior” ✓ Research context on “motivations” ✓ Testing helps verify the “tactics” 21
  22. 22. …appreciating that AI product will have it’s own analytics maturity curve https://www.intel.com/content/dam/www/public/us/en/documents/guides/analytics-planning-guide.pdf 22
  23. 23. Optimize Product Lifecycle • Strategy • Experience • Development • Management Metrics: Click Through Rate, Conversion, %Happy Path, Speed, Distribution Minimize Risk • Decrease in Standard Risk • Successful Prevention Rate • New Risk Detection Efficiency • Rule efficiency: FPR/FNR, Agent Reviews, Reported • Implementation cost: CXM, CSS Metrics: Bad Rate Changes, %bad prevented, %leak through, business KPI impact Optimize User Journey • Campaign Strategy • Performance Attribution • Funnel Management (Omni) • Cost Optimization • Brand Management Metrics: Awareness, Sentiment, Adoption, CPE/CPM/CPC, Engagement, NPS, LTV Optimize Marketing and/or Sales Process • Goal Setting, Monitoring & Tweaking • Prospect Scoring & Prioritization • Lead Funnel Management: Rate, Speed, Cost • Retention & Growth • Turnover Metrics: Topline, Time to Live, Cost of Acquisition & Retention, Account growth/ NPS Optimize Strategy & Operations • Strategy Development & Execution • Innovation Delivery • Performance Tracking & Intervention • Business Operations • Resource Investment Decisions (Finance) • Strategic Research: Competitive Monitoring, Regulatory, Policies, Legal Metrics: Earnings Growth, Guidance delivery, Investor Confidence Optimizely Technology Delivery Cycle • Development Prioritization • Delivery Quality & Monitoring • Cost of Development • Platform Management • Scalability: Compatibility, Detection, Pre- emption & Prevention Metrics: Uptime, Performance, #Story points to Develop/Scale/Iterate, #Bugs/Bug Rate ..and that Analytics will span multiple aspects of business critical to overall success 23
  24. 24. Let’s make all this real with an illustrative use case... 24
  25. 25. HealthBot: How are you feeling today? Adam: Stressed. And I drove too fast. Was feeling awful HealthBot: Is there anything I can help you with? Adam: Yes. And I feel like a couple of extra drinks will take the edge off Day 7 of chat HealthBot: How are you feeling today? Adam: I feel doing something extreme HealthBot: Did you try exercising? Adam: Yes I did but that does not seem to help anymore Day 14 of chat Adam is suffering from depression and found a chatbot for support... 25
  26. 26. HealthBot: How are you feeling today? Adam: Stressed. And I drove too fast. Was feeling awful HealthBot: Is there anything I can help you with? Adam: Yes. And I feel like a couple of extra drinks will take the edge off Day 7 of chat HealthBot: How are you feeling today? Adam: I feel doing something extreme HealthBot: Did you try exercising Adam: Yes. I did but that does not seem to help anymore Day 14 of chat Natural Language Processing Conversation Service Machine Learning & Deep Learning built into Chat Bot Chat Related Services Pre-built conversation trees Connectivity with other services Machine Learning – Models trained by data & experts External/3rd party services Internal services Instant solutioning without truly “understanding” the needs & problem causes attrition... 26 Day 21 of chat HealthBot: How are you feeling today? Adam: I am suicidal HealthBot: I’m not a therapist. But dear human, I think you should seek one. Please, call your doctor, emergency or suicide hotline. I wish you well.
  27. 27. Testing Monitoring HealthBot: How are you feeling today? Adam: Stressed. And I drove too fast. Was feeling awful HealthBot: Is there anything I can help you with? Adam: Yes. And I feel like a couple of extra drinks will take the edge off Day 7 of chat HealthBot: How are you feeling today? Adam: I feel doing something extreme HealthBot: Did you try exercising Adam: Yes. I did but that does not seem to help anymore Day 14 of chat Natural Language Processing Conversation Service Machine Learning & Deep Learning built into Chat Bot Chat Related Services Pre-built conversation trees Connectivity with other services Machine Learning – Models trained by data & experts External/3rd party services Internal services Analytics CX Research Effective Product = Problem to V1 Solution followed by continuous Test & Learn optimization 27
  28. 28. Focus Area Use Cases • Core Learning Algorithm Performance • Model Performance: Metrics, Drifts, Alerts • Performance Analytics: Load Time, Errors, Bugs, Alerts • Abuse Monitoring & Alerts: “Warning” phrases • Escalation paths: Offline reviews/feedback loop • Product/UX Analytics • UX Monitoring (Professionals & Individuals): Funnel & Pathing • Feature Sizing, Launches & Ramp • Integration issues • Marketing/Consumer Engagement Analytics • Brand Monitoring • Adoption & Engagement: Downloads, Launches, Opens->Engage->Loyal • Reviews & Ratings • Partner Adoption: Referrals, Ratings, Engagement, Monitoring • Strategy & Operations • Impact Sizing: Consumer Value Prop monitoring, Benchmarked lift on KPIs • Portfolio Management, Expansion/Retirements/Pivots • Feature Set Augmentation, Professional certifications, Research Studies • Investor & Regulator Relationship Management • Sales • Target Audience Sizing & Segment Penetration • Integration with the ecosystem: Pricing, Referrals, Adoption, Management Illustrative analytics use cases across the board 28
  29. 29. …which depends on integrity across the layers DATA ANALYTICS PLATFORM 29
  30. 30. …especially Data Governance & Standards • Coverage & reliability of data feed: timely, quick, real-time • Privacy concerns and residence of data (local or cloud) • Guard against getting overwhelmed with unnecessary or noisy data • Guard against irrationality & bias • Data homogenization, integration, transformation & lineage: Multiple data forms, sources, signal processing 30
  31. 31. …& progression along the Analytics Maturity Curve Cognitive Prescriptive Predictive Diagnostic Descriptive o Dashboards by Functions (e.g., #Checkins, Time Spent, Emotions) o Real Time Monitoring & Alerts (Flagged phrases, Escalations) o Visualizations & Storyboarding o Drilldowns/Segment Insights (Analytics+Research+Testing) o Investigations, Deviations, Sizing & Opportunity Assessment o UX Personalization: Individuals & Professional use cases o Tracking: Sentiment, Themes, Entity, Clustering o Scoring & Actions: Conversations, Individuals, Seasons, Events o Preventive Escalations & Interventions: Causations o Medical Studies, Professional reviews, Intervention Effectiveness o Grid Strategy for the business & domain o Core Algorithm augmentation, expansion or retirement o Scalability: Geo, User Segments, Needs, Domain 31
  32. 32. …& anchored on a platform that support “agile” development & management (DevOps) Data & Analytics (Customer, UX, Mktg, Sales, Finance) OPTIMIZATION Platform Monitoring (Logs, Anomaly, Data Quality) VoC & CX Management Experimentation Layer 32 Feedback
  33. 33. The parting words 33
  34. 34. Key takeaways • AI products need “external” optimization above and beyond core algo based learning. • Pillars of a successful AI products: Design+Data+Analytics+Algo+Policy • “Agile” AI product platform hinges on DevOps mindset & architecture • Compliance & policy crucial for success • AI product success hinges on the ecosystem improvements: Partners, Affiliates, Industry bodies, Regulation. So be integration friendly. 34
  35. 35. Thank you! We would love to hear from you... https://twitter.com/decisions_2_0 http://www.slideshare.net/RamkumarRavichandran https://www.youtube.com/channel/UCODSVC0WQws607clv0k8mQA/videos http://www.odbms.org/2015/01/ramkumar-ravichandran-visa/ https://www.linkedin.com/pub/ramkumar-ravichandran/10/545/67a RAMKUMAR RAVICHANDRAN ABHISHEK GUGLANI https://www.linkedin.com/in/abhi-guglani-40016a5/ 35 https://twitter.com/abhishekguglani
  36. 36. UPCOMING EVENTS Wednesday, March 6 Thursday, March 7 How to Build Great Consumer Products How to Execute Like a PM Rockstar
  37. 37. UPCOMING Product Management Courses March 23 - May 18 9:30am - 3:30pm 4 Spots left SaturdaysTuesdays & Thursdays March 19 - May 9 6:30pm - 9:00pm 3 Spots left
  38. 38. Mondays & Wednesdays 3 Spots left March 18 - May 8 6:30pm - 9:00pm UPCOMING Digital Marketing for Managers Courses

×