Faster, Better, Cheaper: How Fortune 500 Companies are Seeing Massive ROI from Crowdsourced Testing
•Transferir como PPTX, PDF•
2 gostaram•188 visualizações
In 2017, software failures impacted an estimated $1.7 trillion in assets, and more than 70% of these failures were the direct result of software bugs or usability glitches. These failures are unacceptable in a digital-first world — yet given the rapid pace of innovation, device fragmentation and increased global app usage, software quality issues are increasingly difficult to avoid. To combat these challenges, crowdsourced testing has emerged as a valuable complement to traditional QA, but how do you know which resource to invest in? It all comes down to ROI. Applause recently commissioned a survey with Hobson & Company, a research firm focused on Return on Investment (ROI) studies, to better understand the key metrics used to evaluate and measure the business impact of their crowdsourced testing solution. In just 1.7 months, Hobson found that Applause clients can receive a full return on their investment, and a 753% ROI after 3 years. Join Applause and Hobson on June 7th at 1:00 PM ET / 10:00 AM PT to learn the study's results and find out how crowdtesting can improve your organization's QA strategy.
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a Faster, Better, Cheaper: How Fortune 500 Companies are Seeing Massive ROI from Crowdsourced Testing
Semelhante a Faster, Better, Cheaper: How Fortune 500 Companies are Seeing Massive ROI from Crowdsourced Testing (20)
Mais de Applause
Mais de Applause (20)
Último
Último (20)
Faster, Better, Cheaper: How Fortune 500 Companies are Seeing Massive ROI from Crowdsourced Testing
- 1. MEASURING THE BUSINESS VALUE OF CROWDSOURCED TESTING Webinar with Hobson & Company
- 2. 2 CATE RODRIGUEZ Partner Hobson & Company LUKE DAMIAN VP, Customer Strategy Applause
- 3. 3 Agenda 1. Intro to Applause and Hobson 2. Measuring the BusinessValue 3. ROI Case Study 4. Applause Key Capabilities 5. Q & A
- 5. Helping companies uncover, quantify and validate their key sources of value driving the adoption of new and emerging technologies.
- 8. ENABLING YOU TO DELIVER FLAWLESS EXPERIENCES
- 9. 9 Applause Solutions DIGITAL TESTING Deliver experiences that work every time for everyone OMNICHANNEL FEEDBACK & TESTING Provide engaging experiences for your customers PAYMENT TESTING Ensure successful and accurate payments across the globe Get results in hours with our community of over 300,000 passionate, skilled digital experts located around the globe. //////////////////////////////////////////////////////////// MANAGED SERVICES Our industry-leading SaaS platform helps you deliver a high quality digital experience to your customers, no matter who they are or where they are from.
- 10. 300 Thousand community members 2.4 Million devices 200+ countries and territories 1+ Million vetted submissions/year • Access to skilled and highly vetted global talent • Real people, real environments • Team engaged within hours on demand • Curated based on required skills, demographics, locations, devices • Engaged through gamification and meritocracy Largest Global Community Of Software Testers 10
- 11. Leading Brands Rely On Applause 11
- 12. MEASURING THE BUSINESS VALUE
- 13. What Is The Driving Need For Crowdsourced Testing? Software failures impacted an estimated $1.7 trillion in assets in 2017 and more than 70% of failures were a direct result of bugs or usability glitches. glitches. 13
- 14. Today’s QA Challenges 14 Rising Customer Expectations Increased importance of App score ratings and product reviews Increasing Competition In an increasingly competitive environment, the need to innovate faster to stay ahead is growing Balance of Speed, Quality & Cost Faster release cycles almost never come with increased resources increasing the chance of critical bugs escaping into production Lack of Adequate Testing Coverage There are more than 45,000 device/OS/network/location mobile device profiles in the world today
- 15. 15 How Crowdtesting Addresses These Issues IN THE WILD Real-world testing and feedback solutions REAL PEOPLE World’s largest community of over 300,000 testers REAL DEVICES More than 2.4 million mobile, desktop, and IoT devices
- 16. Delivering Value Where It Counts Hobson & Company explains howApplause clients receive a full return on their investment in less than two months. 16
- 17. ROI Framework for Crowdsourced Testing 17 All statistics verified by third-party research firm: REDUCE TESTING EFFORT & COSTS INNOVATE FASTER WITH LOWER RISK IMPROVE USER EXPERIENCE We have found that customers see the most value from crowdsourced testing in three key areas.
- 18. Top 5 Benefits and Metrics of Crowdtesting 18 1. Increase Capacity of ExistingTeam Metric: Capacity of existing testing team 2. Reduce hard costs of test lab Metric: Average cost per year of maintaining a device lab 3. Reduce costly bugs by discovering them earlier in the SDLC Metric: Average number of critical bugs detected 4. Release higher quality products in pace with demand Metric: Average number of major code releases per year 5. Protect loyalty and retention Metric: Customer retention rate
- 19. To try and replicate the testing Applause is doing for us would have not only been prohibitively expensive, but also logistically impossible to organize” SVP, PLATFORM AND PRODUCT DELIVERY
- 20. ROI CASE STUDY
- 21. 21 Customer’s Goals 1. Launch new products and features with confidence through pre-release testing and feedback. 2. Ensure the highest quality digital experience possible through new and existing feature testing. 3. Ensure flawless experiences by performing real-world user feedback.
- 22. 22 The Applause Capabilities in the Partnership 1. Usability UI/UX feedback to assist in application redesign efforts 2. Continually testing of iOS and Android apps to protect against feature degradation 3. Expanded testing for real-world scenarios, leveraging numerous device configurations and teams to scale out internal capabilities 4. Specific use case driven testing with highly targeted demographic and geography coverage
- 23. 23 Key Customer Inputs 1. Annual gross revenue $600,000,000 2. Percent of revenue from digital property (in-scope) 30% 3. Total number of existing digital customers (in-scope) 500,000 4. Average number of test cycles per month (in-scope) 3 5. Capacity of existing ManualTesting team (in Full Time Equivalents) 5
- 24. The ROI of The Partnership 3-Year ROI 753% 3-Year NPV $6,853,184 Payback Period 1.7 months Investment (Year 1) $360,000 InvestmentOngoing (AnnualAvg) $376,981 SUMMARY CUMULATIVE INVESTMENT/RETURN 0 1,000,000 2,000,000 3,000,000 4,000,000 5,000,000 6,000,000 7,000,000 8,000,000 9,000,000 10,000,000 Year 1 Year 3 Investment Return
- 25. The ROI of the Partnership BENEFITS BYVALUE DRIVER 22% Improve User Experience 41% Innovate Faster with Lower Risk 38% ReduceTesting Effort and Cost BENEFITS BYTYPE 60% Revenue Gains 1% Productivity Improvements 39% Cost Savings
- 26. Customer’s Three Biggest ROI Drivers 26 Release Higher Quality Products in Pace with Demand Increase Capacity of InternalTeam Protect Loyalty & Retention Applause helps its customer generate $3.2 million per year by helping to ReduceTesting Effort and Cost, Innovate Faster with Lower Risk, and Improve User Experience. 39% 35% 14%
- 27. QUALITY Directly attributable increase in quality that has helped retain current subscribers while adding new ones each day. SCALABILITY/COVERAGE Effectively scale extremely challenging device and configuration testing requirements that tie directly to worldwide subscriber base. VELOCITY Rapidly increase the speed in which new features are released.
- 28. 28 Where Crowdtesting Can Deliver Your ROI MOBILE DEVICES WEB RETAILHOME AUTO DIGITAL TESTING Deliver experiences that work every time for everyone Manual Functional Testing Automated Functional Testing Accessibility Assessments OMNICHANNEL FEEDBACK & TESTING Provide engaging experiences for your customers Digital Customer Journeys In-Field Customer Journeys Usability Studies PAYMENT TESTING Ensure successful and accurate payments across the globe Transaction Validation Digital Wallet Testing Applause Solutions Your Digital Strategy Security Testing
Notas do Editor
- <SARA>
- <SARA> Quick background/Bio Started in hardware and software engineering in a company that believed everyone started in a QA role and then moved into software development, then hardware development VP 5x with experience with all forms of QA on a variety of SMB and Enterprise class products and platforms Informal audience polls How many Engineering heads do we have? How many Quality heads do we have?
- <CATE> Thanks, Sarah for the kind introduction. I’m looking forward to sharing the results of my research with you today. We have a lot of great material to cover, so let’s jump right into the agenda we have for this session on “Measuring the Business Value of Crowdsourced testing solutions”. It is my hope in our next 30 minutes together that we can share some information regarding what’s driving the increasing need for more comprehensive testing and what your peers care about when evaluating crowdsourced testing solutions. [Intro] - We will start with a brief introduction to Applause and Hobson [Measuring the Business Value] - Luke will then walk you through some trends in the industry that are evolving and really establishing the market need for these Solutions. I will then share a framework for measuring the value of a crowdsourced testing solution. Our objective is to leave you with a set of skills you can walk away with and apply right away. My goal here is two-fold: 1) to provide you with a value framework to use when evaluating crowdsourced testing solutions for your organization, and 2) to have you Walk away with the 5 key metrics of concern when creating your own value assessment of the categories that impact your business [ROI Case Study] - At the end as a practical example, I will walk through a REAL life ROI example that shares the concepts of our findings within the context of a real firm that many of you should be able to identify with. [Applause Key Capabilities] – We will wrap up with Luke walking you through some of the key capabilities from Applause. [Q&A] - Then we have allotted some time for Q&A on either the process suggestions or the content itself.
- <CATE> As Sarah mentioned earlier, I am a Partner with the consulting firm Hobson and Company.
- <CATE> We are an independent 3rd party research firm that specializes in helping companies uncover, quantify and validate their key sources of value, driving the adoption of new and emerging technologies.
- <CATE> We have over 16 years of experience working with companies across a wide range of industries. Applause commissioned our team at Hobson & Company to help them uncover the key areas of their value proposition that were most important to their existing client base. I will walk you through our findings from the research after Luke gives you a brief introduction to Applause and an overview of the current trends in the market.
- <LUKE> Thanks Cate – I see some familiar attendee names out there today , but for those less familiar with Applause, let me share a quick overview
- <LUKE> Applause is the creator and pioneer of real-world testing and feedback solutions,… and the only crowd-testing company to ever appear in Gartner’s Magic Quadrant. Since 2008, Applause has become the go-to partner for the world’s leading digital brands – including innovators like Google, MasterCard, Uber, Michael Kors, and FOX – to help them create and deliver flawless digital experiences to their customers and users alike.
- <LUKE> Applause offers a portfolio of fully managed crowdtesting solutions ranging from Functional Testing, User Feedback and Omni-Channel Testing, as well Payment testing to support our clients throughout their software development lifecycle. Through the Applause Platform and our managed and curated global community of over 300,000 professional, digital experience experts, Applause empowers our clients to create engaging digital experiences and deliver unrivaled software quality for websites, mobile apps, IoT products and omni-channel experiences…in a way no other approach can. With Applause, you receive impactful testing results - fast. What many accomplish in days and weeks, we are able to accomplish in sometimes hours…
- <LUKE> We are able to consistently deliver these types of solutions so fast by employing the largest, most diverse community of software testers and digital experts in the world. Our global community of over 300k provide real world feedback, from real testers, on real devices, under real market conditions in real time… That is a lot of REAL!!!! This profiled, certified, rated and curated group provides access to over 2.4 million unique devices across 200 countries and territories to ensure you gain the critical insight required to satisfy the depth and breadth of today’s dynamic testing requirements.
- <LUKE> Over the last 10 years, we have had the pleasure to work with and learn from the thousands upon thousands of the most innovative, brand conscious organizations as they travel down their own digital paths. Along the way, we have tested hundreds of thousands of applications and digital experiences over a wide range of vertical industries, and helped our clients identify millions upon millions of defects before they ever reach their customer end users.
- <Luke> – For many line of business executives in these organizations, crowdsourced testing has been somewhat of a foreign concept. Consequently, in recent years we have seen crowdtesting champions struggle to articulate the full value of the approach to their executives – driving them to better define and measure the true benefits that their organizations can gain with the unique approach.
- <LUKE> I think what has made the biggest impact in helping them better measure the business value of crowd sourced testing has been directly tied to the connection between customer and user experiences becoming, predominantly, digital experiences. As software defects creep into these digital customer experiences, the business impact can be more readily seen across every major industry. In 2017 alone, it was estimated that more than $1.7 trillion in assets were negatively impacted by defects alone… and more easily exposing the benefits of crowdtesting.
- <Luke> Today’s challenges in building and releasing digital customer experiences have never been greater. At a time when customer expectations continuously rise Competition compete on speed Trade off of quality/cost/speed Scaling across highly fragmented device configurations
- <LUKE> Crowdtesting offers great way to centrally address these challenges
- <CATE> Thank you Luke, for walking us through such a vivid description of the challenges that people are facing in the market today. As I mentioned earlier, Hobson specializes in helping companies find uncover, quantify, and validate their key sources of value. The core of our work is done through interviewing existing users of the technology. For the Applause research, we conducted 18, in-depth, one-on-one interviews with key clients across 5 vertical markets. From these interviews we walked away with a very deep sense of the key business pains that crowd-sourced testing solutions such as Applause help solve and how to dimensionalize the magnitude of the pain. The benefits that made the cut into the final model were essentially validated for both “importance and believability” (the Applause solution has made a difference) and one key metric was tied to each benefit so it’s super transparent what has improved and therefore what to track. Let’s walk through the key value drivers, benefits and metrics that were uncovered during our research process.
- <CATE> Regardless of the vertical market, we found three consistent key value drivers emerge from our research. The first Key Value Driver is Reduce Testing Effort and Cost which covers both the costs of scaling your internal testing team and device lab as well as the costs associated with finding bugs very late in the Software Development Lifecycle. The first key benefit here speaks to the ability to increase the capacity of your existing team by augmenting your testing coverage. The Director of ecommerce at one customer said “Applause has increased the capacity of our internal team by 2-3x at a minimum.” We spoke to many testing groups who could previously only cover testing that represented 20-30% of their customer base and now they have the ability to execute tests that cover over 80% of customers. Other benefits in this area include the ability to ‘reduce hard costs of maintaining a test lab’ as well as ‘reduce costly bugs by discovering them earlier in the Software development lifecycle’. The second Key Value Driver is ‘Innovate Faster with Lower Risk.’ It is very rich from an ROI perspective since it helps with the area of protecting revenue through ‘releasing higher quality products in pace with demand’. This is not only supporting a faster release cadence, but also a higher quality product. One SVP of Platform and Delivery said “We have quadrupled our ability to release products with lower incidents of bugs” The third key value driver is Improve User Experience. This was important across the board as companies worked to improve the quality of their digital properties. Many of the interviewees referenced their positive experience with reducing the number of bugs found by customers. One IT Director said “We have seen a more than 50% reduction in the number of customers that experience a defect with our properties.” We heard through our conversations that this is having a direct impact on customer loyalty and retention.
- <CATE> When evaluating Crowdsourced testing solutions there could be an infinite number of metrics that you might wish to track and control. By using the value framework presented earlier, I would suggest you look at YOUR top 1-2 pains and needs across each of the 3 key value drivers to focus on the ROI benefits that will have the greatest impact in your organization. Hopefully the benefits reviewed here as well as these corresponding metrics will help bring some focus to this exercise in terms of both real metrics to consider and general benefit areas to evaluate. Here are the top 5 benefits and metrics that drive either top-line or bottom-line ROI value in the area of cost savings, productivity gains, or revenue enhancement that we uncovered during our research. In addition, I’d like to share some specific data and quotes gathered during the customer interview process that help dimensionalize the magnitude of impact one could expect. #1: Benefit = Increase the Capacity of your existing team. Metric = Capacity of your existing team. (cost) Prior to using Applause, we had many customers discuss the challenges of a fixed amount of resources struggling to keep up with the ever increasing workload. One SVP of Delivery and Platform said “We are limited to 6-7 FTEs. Their effectiveness has been quadrupled since we started working with Applause”. By augmenting the existing team with testers from Applause, customers are able to experience a 150- 200% increase in the capacity of their existing team. #2: Benefit = Reduce the cost of maintaining a Device lab. Metric = Avg cost per year of maintaining a device lab (cost) Depending on the number of devices that are being maintained, we learned that companies are spending anywhere from 10’s of thousands of dollars up to hundreds of thousands per year maintaining a device lab. With testing augmented with Applause, companies are experiencing a 50-75% reduction in these testing lab costs. We had one Sr. Mgr of IT share with us that “In order to match the testing that Applause is performing, we would need approximately 100 additional devices, and it would be a nightmare to manage” #3: Benefit = Reduce costly bugs by discovering them earlier in the SDLC. Metric = Avg number of critical bugs detected (productivity) There is a lot of research out there that has shown the cost to fix a but in production is significantly higher than one found earlier in the development process. Some studies suggest it is up to 10 times more expensive. The broader coverage that is offered by Applause allows companies to find bugs earlier, resulting in fewer bugs requiring critical fixes. On average, customers are requiring 30-50% fewer critical fixes. On Sr Dir of Global Verification said “Applause has really provided a lot of value by allowing us to find substantially more defects ealier in the the development cycle, which results in fewer bugs finding their way to the field.” #4: Benefit =Release higher quality products in pace with demand. Metric = Avg number of major code releases per year (revenue) Businesses increasingly view the digital ecosystem as the prevailing field of competition. A Harvard Business Review (HBR) Analytic Services global survey characterizes the pace of digital change as a “steepening trajectory,” and notes a “tightening digital race across all industries” . In this increasingly competitive environment, many organizations feel pressure to find new innovative ways to engage with customers and to do it faster than their competitors. We found that Applause customers have been able to experience a 100-150% increase in release velocity. One [SVP Platform and Delivery] said “We have massively picked up the cadence of release in the last 6-9 months and those performance metrics around the amount of time people spend on the site went up by over 50%” #5: Benefit = Protect Loyalty and Retention. Metric = Customer retention rate (revenue) Mobile users are finicky and they can quickly and easily delete a poorly performing application from their device. One Sr Dir of IT that I spoke with said that “The testing that Applause team provides really helps us better understand the normal user”. In fact the theme of ‘getting closer to the customer’ is one that I heard consistently in my conversations. This closeness to the customer has resulted in an improvement in customer loyalty and retention. We learned that customers are experiencing a 10% improvement in customer retention and attributing 5% of that improvement to the work with Applause.
- <CATE> I’d like to conclude with this great quote that we heard during our research - “To try and replicate the testing Applause is doing for us would have not only been prohibitively expensive, but also logistically impossible to organize”. I think this does a fantastic job of capturing the sentiment about the value customers are experiencing.
- <CATE> We have developed a representative ROI Case study to share with you. Luke is going to kick us off by describing the business challenges and objectives and I will jump in to discuss the ROI findings.
- <LUKE> We were fortunate that one of our innovative clients recently shared the value they are experiencing from their Crowdtesting engagement with Applause As a leading media company, they are always searching for innovative ways to bring news and information to the masses. With a number of media outlets and digital experiences, they focus on creating and maintaining a trusted partnership with all their media consumers. With that said, they had identified several key goals that were important for their team to focus on : Launching new products and features fast and with confidence Establish the highest possible standards for digital experiences And ensure these experiences would operate the same – from their internal release to when it hits the hands of its consumers
- <LUKE> Over the course of the engagement, Applause has provided broad capabilities to help this client execute its digital strategy: Usability feedback Continuous Functional testing of their mobile apps to protect against degradations as new features are rolled out True in-the-wild testing across a wide range of device configurations to prove the applications would operate properly in the hands of the on-the-go users… as well as use case driven testing that validated that core user paths functioned in every core media geography Cate will further discuss how the Hobson helped define the full business value of our crowdtesting engagement.
- <CATE> Thanks, Luke. It’s important to remember that this example highlights the key benefits that were most important to this client based on a customized business impact analysis leveraging the ROI calculator from our research. Benefits can be turned on/off on the fly, inputs changed and assumptions challenged to best reflect a given organization, but this example will give you a good sense for what kind of ROI expectations an organization should have of Applause in terms of payback period and specific ROI targets. There are 5 key customer specific sizing inputs for the ROI calculator: Annual gross revenue Percent of revenue from a digital property Total number of existing digital customers Average number of test cycles per month Capacity of existing testing team In addition to these 5 sizing inputs, there are a number of other assumptions and inputs that can be adjusted in order to closely align the analysis with your business environment. We worked with this client to customize their inputs to accurately reflect the scope and magnitude of the work they are doing with their crowd sourced testing solution.
- <CATE> Once we walked through all of the assumptions and benefits with the internal team and customized the inputs and assumptions to align with their business (as well as turning off benefits that were not applicable), we reviewed the ROI analysis. The overall ROI figures are strong at a 1.7 month payback period and a 753% 3-Year ROI. The model includes 10 potential business benefits that can easily be turned on and off once the most important benefits for a business environment are identified.
- <CATE> Now let’s take a look at how the benefits split by Value Driver. If you remember earlier in our discussion, we reviewed the 3 Key Value Drivers for Crowd sourced testing. The first of our 3 benefits fall into Reduce Testing Effort and Cost, The 4th into Innovate Faster with Lower Risk, and the 5th into Improve User Experience. Here is how they are each contributing to the Value Creation: Reduce Testing Effort and Cost – 38% Innovate Faster with Lower Risk – 41% Improve User Experience – 22% Secondly we categorize each of our benefits into either revenue gains, cost savings or productivity improvements. You can see here that the majority of the Value is being creating through revenue enhancing benefits which are tied to the incredible gains they have been able to achieve around release velocity improvements. Productivity improvements were somewhat limited here because they are linked to reducing critical bugs and in this environment, they had so few to begin with, the improvements were on the smaller side. Here is how the Benefits split by Benefit Type (revenue gains, cost savings and productivity improvements) Revenue gains – 60% Cost Savings – 39% Productivity Improvements – 1%
- <CATE> Earlier we discussed the top 5 benefits of Crowdsourced testing, but we found with this example there were really 3 that drove the majority of the benefit they are experiencing. The number one benefit in this example is ‘Release Higher Quality Products out in Pace with Demand’ at 39%, followed by ‘Increase capacity of internal team’ at 35%, ‘Protect Loyalty and Retention’ at 14% Although this example touches on some key universal areas of value, it is quite likely that some of these benefits would apply to your business but that there are others that are equally important. As mentioned earlier, our model allows Applause to configure the analysis to speak more directly to your specific business challenges -- you can turn on/off factors and assumptions to effectively create a custom business case of your own. It really needs to be credible at the specific benefit level first before anything is rolled up and communicated on an overall ROI level. I am now going to pass the mic back over to Luke to summarize how some of the key business themes expressed in our research and today are playing out in the market and which specific Applause capabilities help address these key needs
- <LUKE> Cate just shared some very strong financial return our partnership has produced, in addition, we are also happy to say this client feels great about their progress against their team goals of launching new products and features faster, that are of the highest quality, and function flawlessly in the hands of its worldwide users. QUALITY Since using Applause, our app score ratings have dramatically improved… from 1.5-2 stars up to 4.5 stars today.” SCALABILITY/COVERAGE To try and replicate the testing Applause is doing for us would have not only been prohibitively expensive, but also logistically impossible to organize. This is just a capability we wouldn’t have had.” VELOCITY We have massively picked up the cadence of release in the last 6-9 months and the amount of time people spend on the site went up by over 50%.”
- <LUKE> Close the session highlighting the broad range of crowdsourced testing solutions and how the similar ROIs can be expected.
- <LUKE to close and transition to Sara for Q&A> You mentioned that some customers have seen an increase in customer retention. We just completed a major redesign of all of our digital properties, so how would I know that improvements aren’t coming from the work that we did with the redesign (where improving retention was a key objective? We certainly recognize that major strategic initiatives such as the one you described will likely drive the majority of the improvements that you may see. However, most of the customers that we spoke with felt that these types of initiatives were more successful due to the work they are doing with crowdsourced testing and therefore, felt very comfortable attributing a conservative 5% of the benefit to the crowdsourced testing solution. One of the larger benefits you discussed was around capacity improvements of the existing team, but we do not have plans to reduce our headcounts. Will we still experience a benefit here? Absolutely. What came through in our research is that folks really see this as a capacity gain and NOT a head count reduction opportunity. We heard that the ability to have the highly skilled existing team focused on more strategic initiatives is not only a win from a cost perspective, but also a softer benefit of having a positive impact on team morale. People wish they could more and support growth without increasing headcount. Why did Applause decide to engage with Hobson and make the investment in the Business Impact Analysis approach? We wanted to help our customers more deeply understand the value to help justify the investment because: Applause created a new market and folks are still getting their heads around crowdtesting and the breadth of benefits, Our champions know the value we provide but don’t have experience in measuring it and building business case, In many of our client relationships we are viewed as a strategic partner and a true extension of the in-house team, and in that vein, we are dealing with senior leadership frequently. Companies don’t have a line item for crowdtesting, and, This support is representative of the white glove fully managed service we provide…making our partners lives easier while helping them improve their end users’ experiences Is this kind of ROI / business justification support something Applause competitors or many other vendors offer today? We’re not aware of any other crowdsource testing companies who offer this kind of support For other types of vendors who have an ROI tool, most don’t go to the extent of having measurable sources of value that have been validated by 18 of the best brands (fortune 500 and household brands) in the world through an independent consultant. This is the secret sauce to this capability as it brings credibility to the model. Is the primary benefit to Applause clients and prospects the summarized benefits from the 18 customer interviews, or the ability to build a custom Business Impact Analysis for the proposed engagement, or both? The answer is both. While we expect the details on the sources of value as outlined in the whitepapers will provide enough insight and evidence of the value of a partnership with Applause, we want to be working with our clients to measure the specific value we’re delivering by helping them get closer to their customers through the Applause community. In the example you shared today, the analysis period was 5 years. Does it take that long to realize the returns at that level of magnitude? No. While we do expect the ROI to maintain or even improve over time (as we become more strategically aligned), our customers are typically recovering their investment in a matter of months and typically see an investment of 400%+ in year 1. Again, we’re very conservative in the model and account for a ramping adoption by the client and we typically model in a higher level of adoption/optimization in years 2-5. As a result of launching this new capability, do you anticipate any changes in customer behavior in terms of the types of engagements they want to pursue with Applause? It’s early to say if there’s a direct correlation to the Business Impact Analysis/ROI capability, but we are seeing an increasing number of customers looking for multi-year subscriptions with price lock and entitlements that include multiple solutions including functional testing, customer journey feedback, Accessibility assessments, payment testing, etc. Based on the number of benefits and magnitude of the returns we’ve validated to date, I fully expect our multi-year, multi-solution subscriptions to grow more aggressively. ROI is a language that spans organizations like Product Management, Engineering, QA, Operations, etc., so we also hoping that this capability will help us expand Applause awareness and engagement across teams sooner which will only allow us to provide more value sooner to all teams involved in building and delivering digital and omnichannel experience.