Learning's Big Data Problem: Measuring & Analyzing Impact Part 2
•Transferir como PPTX, PDF•
2 gostaram•712 visualizações
A recent study by Brandon Hall Group found nearly 80% of L&D departments have little to no direct impact on improving skills or affecting business performance. So how's the other 20% able to do it? With planning, measurement, and the right technology.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Learning's Big Data Problem: Measuring & Analyzing Impact Part 2
Semelhante a Learning's Big Data Problem: Measuring & Analyzing Impact Part 2 (20)
Mais de Watershed
Mais de Watershed (7)
Último
Último (20)
Learning's Big Data Problem: Measuring & Analyzing Impact Part 2
- 1. Learning’s Big Data Problem: Measuring & Analyzing Impact Part 2 David Wentworth, Principal Analyst, Brandon Hall Group @DavidMWentworth Mike Rustici, CEO, Watershed @mikerustici Michael Rochelle, Chief Strategy Officer Brandon Hall Group @MichaelRochelle
- 2. HCM Research and Advisory Firm © Brandon Hall Group 2017 2 FOUNDED IN 1993 Community of 300,000+ 10,000 Global Clients Leading Independent HCM Research & Analyst Firm Publishing 2+ Pieces of Research Each Day Sample of Our Clients
- 3. How We Help You © Brandon Hall Group 2017 3 MEMBERSHIP STRATEGIC CONSULTING EXCELLENCE AWARDS PROFESSIONAL DEVELOPMENT Learning & Development Talent Management Leadership Development Talent Acquisition Workforce Management
- 4. Open Surveys © Brandon Hall Group 2017 4 Visit www.brandonhall.com click Open Surveys If you would like to join a panel of survey takers, please contact us at success@brandonhall.com • Extended Enterprise Learning 2017 • State of Talent Acquisition 2017 • HCM Technology 2017
- 5. How To Ask Questions © Brandon Hall Group 2017 5 • Submit your questions or comments about the discussion to our presenters using the Questions tab on your control panel. • Presentation slides and giveaways can be found in the Handouts tab on your control panel. Recording & Slides will also be sent out after the webinar.
- 6. Agenda The State of Learning Measurement Moving and Measuring Beyond Formal How Do We Get Better? The Role of Learning Standards © Brandon Hall Group 2017
- 7. What Got Us Here © Brandon Hall Group 2017 7
- 8. 8 Learning Measurement Impact Model © Brandon Hall Group 2017
- 9. Drivers for Measurement 76% 55% 62% 53% Organizational Performance Individual Performance Levels 4 & 5 Levels 1-3 © Brandon Hall Group 2017 9
- 10. Measuring Outcomes © Brandon Hall Group 2017 10 63% 54% 44% 41% 41% 31% 30% 33% 31% 24% 16% 39% 36% 14% 18% 13% 14% 15% 12% 9% 13% 10% Individual performance Employee engagement Team effectiveness Business process improvement Voluntary turnover/retention Ability to respond to market Revenue growth Rate of knowledge transfer Leadership promotion rates Profitability Revenue per employee Levels 1-3 Levels 4 & 5
- 11. Model Takeaways • Organizations at all levels measure transactional learning outcomes: completion, grades, etc. • Organizations at levels 4 and 5 are more focused on behavioral and organizational outcomes • More than half (54.9%) of Level 1-3 companies say they don’t have the proper metrics to measure beyond Kirkpatrick Level 2, while only 32% of Levels 4 & 5 say the same. 11© Brandon Hall Group 2017
- 12. How Do We Measure Today? 12© Brandon Hall Group 2017
- 13. Kirkpatrick • Does not measure learning duration and uptake • Learner sentiment does not correlate with change in behavior and performance • Most organizations are at lower/lowest level but that doesn’t mean that something is wrong with learning measurement • Requires additional level (level 5) which is enterprise return, hardest to quantify and attain 13© Brandon Hall Group 2017
- 14. Kaufman • More like Kirkpatrick plus • Focuses on inputs and processes which should be broken out and evaluated separately • Does shine a light on how the organization is doing with developing and delivering learning not just the learner reaction • Does embody the idea of immediate yet small benefits for learners • States training does not encompass all forms of learning • Similar approaches to evaluating the application of training • Similar in approach to measuring performance impact • Societal output is a stretch but how your customers benefit is not 14© Brandon Hall Group 2017
- 15. Brinkerhoff Success Case Method • Does focus on connecting learning objectives to business outcomes • Learner surveys can be effective if constructed properly • Analyzing data is great if it is the right data • Communicating what works and what doesn’t work is always good but it needs to be accompanied with a solution 15© Brandon Hall Group 2017
- 16. Anderson’s Model for Learning Evaluation • Does emphasize the need for a strong connection between learning and organizational performance • More of an evaluation model for the overall learning strategy of the organization not an individual program • Program does measure stakeholder returns compared to overall organizational returns for the long and short term • Contemplates different metrics for measuring organizational and stakeholder requirements 16© Brandon Hall Group 2017
- 17. Key Takeaways • All four models fall short in taking into consideration the following: • Efficiency and effectiveness coefficients for learning • Multimodal learning approaches • Learning audience variability • Learning environments • Learning content impact vs. learning delivery impact • Require individual learning assessments • Link closely to performance management processes • Regular and frequent measurement intervals for behavioral change and application of learning • Correlation of learning data to performance data • Manager vs. contributor learning • Team development learning • Unstructured learning or experiential learning • Quality of data being collected © Brandon Hall Group 2017
- 18. 18 A Broader Perspective © Brandon Hall Group 2017
- 19. “Scalable learning is the new reason for large organizations to exist” John Hagel Founder Deloitte Center for the Edge Innovation A world of exponential change
- 20. “Without data you’re just another person with an opinion” W. Edwards Deming Management and Systems Legend Use the data
- 21. So much data! Where do I start? © Watershed 2017 21 categories + complexity learning analytics =
- 22. Categories of data © Watershed 2017 22
- 23. Complexities of analytics © Watershed 2017 23
- 24. Bringing it all together © Watershed 2017 24
- 25. 25 What Do I Need to Get Started? © Brandon Hall Group 2017
- 26. The role of learning standards © Watershed 2017 26 all data Learning is so much bigger than the LMS. Track informal and experiential activities. xAPI and the LRS present a transformative technology for learning analytics. Seamlessly bring all your data together in one platform and perform real-time analytics tailor made for L&D. business impact Include data about behaviors and performance in your learning analytics real time Create, view and distribute reports seamlessly in real-time.
- 27. © Watershed 2017 27 A next generation learning ecosystem brings this all together. This example from Visa shows that the LMS is just one small part of a modern learning ecosystem. A new ecosystem
- 28. Getting started © Watershed 2017 28 01 Gather your data Start by collecting data in one place and in a common format. Make it low friction and ensure you have access. 04 Explore your data Starting to understand not just “what” is happening but “why”. Form questions and see if your data can answer them. Often the best answer leads to another another question. 02 Get to know your data Understand what you have. What is out there, what is reliable and what is missing? Do some simple evaluations and create baselines. 03 Operationalize your data Automate your way out of “Excel hell”. Define some interesting metrics and KPIs, start monitoring them routinely. What trends do you notice? 05 Experiment with your data Start a new learning program with analytics in mind. Set up well controlled experiments and A/B tests to validate a hypothesis. Create a culture of continuous improvement. 06 Show off your data We’ve long known that learning is vital, not we can probe it. Go forth and show the world!
- 29. Questions? Submit your questions to the presenters via the questions tab on your control panel 29 Type your questions here © Brandon Hall Group 2017
- 30. Thank you for joining us today! If you have any additional questions, please email us at success@brandonhall.com.
- 31. Learn More About Us Products and services that we offer © Brandon Hall Group 2016 31
- 32. Membership Offers Tailored Support RESEARCH ACCESS & EVENTS • Reports • Case Studies, Frameworks & Tools • DataNow® & TotalTech® • Webinars and Research Spotlights • Annual HCM Conference ADVISORY SUPPORT • Ask the Expert • 1 on 1 Consultations • Research Briefings • Benchmarking CLIENT SUCCESS PLAN • Your Priorities • Executive Sponsor • Client Associate • Monthly Meetings © Brandon Hall Group 2016 32 To learn more about becoming a member or to sign up for a free trial, please email us at success@brandonhall.com
- 33. Strategic Consulting Offers Expert Solution Development BENCHMARKING • Competitive/Comparative • Maturity Model • Custom Research STRATEGY • Business Case • Planning • Organization & Governance TECHNOLOGY SELECTION • Vendor Selection • Architecture Design • Systems Evaluation DEVELOPMENT & INTEGRATION • Program Design • Assessment • Survey • Process Integration © Brandon Hall Group 2016 33 To learn more about consulting, please email us at success@brandonhall.com
- 34. Why Brandon Hall Group © Brandon Hall Group 2016 34 1 2 3 4 5 We offer flexible, high-value memberships that offer everything you need at a cost you can afford. Our research is better – more targeted, easily digestible, focused on business results, predictive, prescriptive. Our solutions gets better results – research-driven, focused on the business, simple, targeted, flexible approach. Our case studies are unique for their diversity and focus on business results. Our professional development is practical, tailored to your needs, and equips you to take action.
- 35. We Do This Through… Research and Analysis that Define New HCM Standards Success Stories from Global Organizations that Show those Standards in Action Professional Development that Assesses Current State and Enables Innovation Flexible Services to Design and Implement Transformative Solutions Recognition that Validates the Transformation © Brandon Hall Group 2016 35
- 36. What Our Clients Have to Say Brandon Hall Group’s Excellence Awards allows you to take a deeper dive and evaluate the programs that you’ve created at a level that you might not have the opportunity to do… They have always been a cornerstone for me. - Katrina Williams, Director, Sales Capability CDW https://youtu.be/OgdA_tpM55U © Brandon Hall Group 2016 36
- 37. What Our Clients Have to Say I describe the attention and that customer service that I’ve experienced as being excellent. They are very attentive and wherever I need to be connected to whomever, they are there to help me. - Joanne Veech, Global Talent Leader PwC https://youtu.be/YkY-v8gfW5k © Brandon Hall Group 2016 37
- 38. HCMx Radio © Brandon Hall Group 2016 38 THE ONLY PODCAST IN THE HCM ARENA THAT WEAVES CURRENT MARKET RESEARCH, HR TECHNOLOGY AND INDUSTRY LEADERS INTO EACH EPISODE To catch up on the latest episodes, visit www.brandonhall.com/hcmxradio or subscribe on iTunes
Notas do Editor
- There are so many things required to achieve scalable learning We’re not just talking about dissemination of information we know, but also about the acquisition and discovery to new knowledge in a scalable way How do we get there. To achieve scale, we need measurement, we need a culture of constant improvement and we need alignment with business objectives Because this isn’t just about scaling learning, it’s about scaling an organization’s ability to adopt to the exponential rate of change the digital economy is creating To achieve scale, we need analytics, we need data
- There are so many things required to achieve scalable learning We’re not just talking about dissemination of information we know, but also about the acquisition and discovery to new knowledge in a scalable way How do we get there. To achieve scale, we need measurement, we need a culture of constant improvement and we need alignment with business objectives Because this isn’t just about scaling learning, it’s about scaling an organization’s ability to adopt to the exponential rate of change the digital economy is creating To achieve scale, we need analytics, we need data
- You can’t manage what you can’t measure. You can’t measure what you can’t define. Learning and development analytics don’t have to be complicated, as long as you stay organized. Start with the basics and work your way out from there. The simplest way to think of learning analytics is to divide it up into categories and complexities.
- Learning experience analytics seek to help you understand more about a particular learning experience or set of learning experiences. Learner analytics seeks to understand more about a specific person or group of people engaged in activities where learning is one of the outputs. Learning program analytics seeks to understand how an overall learning program is performing.
- The most basic form of analytics is measurement—the simple act of tracking things and recording values. Evaluation is the process of trying to make meaning out of the data you’ve measured. And does the data mean something good or bad? Advanced evaluation techniques—such as exploratory and inferential analytics— search for relationships within your data and discover powerful insights. The first three levels help you understand what’s already happened. Predictive and prescriptive analytics help you understand what will happen in the future.
- When categories and complexities are combined, it forms the matrix (or pyramid) of learning analytics. Many organizations master the lowest level of complexity within a specific category and then work their way outwards. Some organizations may reach high levels on one category, but remain in the lower levels for the others. As long as you can identify where you fall within the pyramid, you’ll know where you’re going, which helps you to set goals, determine metrics, and evaluate the maturity of your program. So doesn’t this contradict the evaluation methods Michael discussed before? Nope—they fall nicely in the middle. Evaluation is result of measuring data and the cause for predictive and prescriptive analysis.
- During the last webinar we talked more in depth about xAPI and LRS. As a quick recap, xAPI is a technical specification that makes it easier and cheaper to connect learning technologies and an LRS is the place where information from all of those systems can easily be analyzed to drive a culture of continuous improvement.
- During the last webinar we talked more in depth about xAPI and LRS. As a quick recap, xAPI is a technical specification that makes it easier and cheaper to connect learning technologies and an LRS is the place where information from all of those systems can easily be analyzed to drive a culture of continuous improvement.
- HOW WE DO IT We Put Your Business First We Listen and Respond to Your Needs We Keep Our Approach Simple, Flexible, Principled We Communicate