3 Reasons You Need Real User Measurement
•
2 likes•543 views
Learn how real user measurements give you the the real time information you need to reach potential customers & communicate with existing customers.
Recommended
More Related Content
Viewers also liked
Viewers also liked (12)
More from Neustar, Inc.
More from Neustar, Inc. (12)
Recently uploaded
Recently uploaded (20)
3 Reasons You Need Real User Measurement
- 1. There’s Money in the Metrics 3 Reasons why you need Real User Measurements Connie Quach, Neustar David Cumberland, Shopatron
- 2. Speakers Connie Quach Sr. Product Manager, Web Performance Management David Cumberland Vice President of Engineering and IT
- 3. The World is Online 2 Billion Users Ubiquitous broadband Uncompromising expectations Performance is differentiator AND mandate
- 4. Overwhelming majority agree: Revenue at stake N=117 Neustar/InfoWorld Market Pulse Research Study: Website Performance/Monitoring and the User Experience, April 2013 Poor online experience = lost revenue Reach potential customers Communicate with existing customers
- 5. A growing dependency N=117 Neustar/InfoWorld Market Pulse Research Study: Website Performance/Monitoring and the User Experience, April 2013 % Average of Annual Revenue - Across a dozen industries - eCommerce % much higher - Biggest growth source
- 6. The revenue risk is real N=376 Neustar Web Performance Research Study, May 2013 $1k-$10k per hour More than $10,000 per hour Don’t Know Up to $1 Million per hour
- 7. Real User Measurements How we got here Why they matter Collect : Correlate : Commit
- 8. Collect Collecting baseline and real time information Baseline requirements - Foundational - Repeatable control - Representative of infrastructure Real User Experience - Know the experience of every user - Act on problem information in real time - Improve against competitive websites Necessities at work
- 9. Correlate Essential to interpret ALL data - Filling in all the piece for a clear picture Relationship of data is vital - View baseline AND real user - Hone in on “first causes” - Take correct action Understanding the “How To’s” - Targeted improvement areas - Measurable impact - Ongoing testing and management
- 10. Commit Gaining confidence from consistent practices - Meeting service level agreements (SLAs) - Commit to business objectives Confidence must be built - Proper diagnosis - Fast time to resolution - Demonstrable ability to meet targets Stepping into the business - Transparency to customer expectations - Service level assurance - Meeting growth objectives
- 11. How Shopatron gets it done
Editor's Notes
- Connie My name is Connie Quach and I am the product manager for Neustar’s Web Performance product line responsible for performance tools like: external and internal synthetic monitoring, load testing, and Real User Measurements. I am also one of the organizers for the Web Performance meetup in the San Diego area. I am excited to be here, hosting this discussion with David Cumberland, VP of Engineering at Shopatron.David My name is David Cumberland. I run Engineering at Shopatron, which means I am responsible for the application development, qa, devops, and IT teams. One of my responsibilities is the performance and reliability of our systems. Through this responsibility, I have spent a lot of time looking at application monitoring and management.Shopatron is the world’s leading provider of cloud-based, eCommerce order management solutions. Our patented Shopatron Order Exchange allows manufacturers and multi-channel retailers to seamlessly leverage the available inventory of every retail storefront and distribution center when fulfilling online orders. Unlike legacy order management software or custom-built solutions, Shopatron makes it easy and affordable to deploy advanced capabilities like in-store pickup, ship-from-store, inventory lookup and vendor drop-ship. With Shopatron, retail channels join forces to increase sales, turn inventory, and deliver a superior purchase experience to online shoppers.
- [CONTEXT: The consumer has gone online, business followed and is increasingly relying upon the user experience for growth]Connie The world is online – and that’s a fact. It’s easy to enter a market but to be online, we have to work hard to generate revenue and attracting new customers while keeping the ones we have. A few years ago when Dave and I started coming to Velocity, messaging was quite different about web performance. The level of awareness has heightened since then – necessity is now widely understood. However, times are changing again and again and today, the message is “Web Performance drives Revenue.” Quite honestly, imagine your company 5+ years ago, what were you worried about? Perhaps, focusing on SEO, making sure you had the right infrastructure in place, racing against your competitors to launch an online storefront. Our environment is naturally competitive. Companies can quickly set up a business and can quickly market and attract customers. Fast forward to now, we have an infinite supply of tools, bandwidth and support, therefore anyone can deliver a premium online experience. Frankly, Consumers are now empowered with more choices and Competition is everywhere.We need to institute best practices for Web Performance as a differentiator and mandate which all ties back to dollars/revenue – and your business operations.
- [CONTEXT: Organizations are getting it, not only is revenue at stake, but websites are growing as critical user exchange points]Connie Organizations are getting it, not only is revenue at stake, but websites are growing to handle new/existing business.We recently looked at survey of companies like yourself … approximately 85% of companies agree that poor user experience and poor online presence puts company revenue at risk. Majority of them use websites to generate new business and retain current ones. This raises the criticality of the role of websites in business.Connie Here we see how organizations use their websites – they are essential tools developing new business and communicating with existing customersConnie David, how does this industry response stand up with what you see at Shopatron?David Shopatron is e-commerce. Everything is critical since customers are online. And one of our core values is being World Class. To me, world class e-commerce means that the Consumer experience is everything. In addition, we have our Shopatron Promise where we clearly state goals for availability and performance of our systems because we know that poor experiences result in poor business.
- [CONTEXT: The numbers become more real. Neustar research shows substantial reliance upon websites for revenue – they MUST perform]In the same market study, on average, respondents (companies) attribute about 1/3 of total annual revenue to website transactions. This study is across a dozen industries and within specialized industries, such as eCommerce, that rev % increases to average more that 80%.With the reliance of the website as another avenue for revenue growth, we have to take a deeper dive in the metrics.Connie David, I am sure this applies to you, can you describe?David This absolutely applies to us. You have to know your market space because the it varies. For us, in the eCommerce space website transactions can be a lot more than 32% of revenue. It can be the majority of your revenue. This probably applies to anyone selling consumer goods and services online.
- [CONTEXT: Organizations are recognizing the hard costs and revenue loss, but many are still not making the connection putting those organizations at risk]<Add a little more content on this slide re: industry stats. Reference:http://money.cnn.com/2013/01/31/technology/amazon-down/index.htmlhttp://www.wired.com/insights/2012/06/cloud-downtime-cost/>Connie We see companies getting a better understanding of the revenue risks incumbent to their websites and knowing is a good thing. But looking at these results from nearly 400 companies spread across industries that include Financial Services, eCommerce, Retail, and Media, there is still far too much that is not known when it comes to performance impact.1 out of 5 companies attribute approx $5K of costs as it relates to website unavailability.And more surprisingly, 1 out 2 don’t even know that they are incurring costs.Connie David, what is your opinion about this stat?David You have to know your costs. You should figure out your costs, both loss revenue and the opportunity cost. Downtime can have ripple effects beyond the incident. Because of this ripple effect, you need to have eyes on the system. You might be leaving money on the table due to poor performance.
- [CONTEXT: traditional performance monitoring is maturing, but RUM is evolving. This important key to ensure the user experience needs a deeper examination]Connie Let’s see how we can figure out the unknowns and get a good gauge on how to avoid unavailability. The impact is real…Most companies use external monitoring to track website performance; generating synthetic traffic from beyond the firewall to see how your sites performs. This provides an important, structured perspective of the website experience, but it isn’t based on measurements of real user activity. In the last couple of years, we were introduced to the idea Real User Measurement (or Monitoring) but from a technical perspective, we needed the modern browser to support the W3C Navigation Timing API, therefore, collecting more accurate data/metrics. In 2010, Chrome released their version followed by IE and Firefox in late 2011. So with that, implementing RUM now within your operations allows you to get more detailed metrics about your customers/users. It’s a combination of Need and Technology that brings us to the use of RUM.Connie David, how did your evolution of need and technology occur in your business?David Our monitoring solutions have followed the evolution of tools available in the marketplace. We started with synthetic monitoring on key pages. We selected pages the represented large classes of use cases to cast a wide but general net. However, we have many clients and the performance of any one client may vary based on factors not expressed in the general case. This was best illustrated by our board asking me about specific, favorite clients during the holiday season. I started to look for ways to measure every client, and that led me to RUM.Real User Measurements (RUM) reveals more. When user activity falls into gaps not covered by synthetic monitoring, RUM sees and reports it in real time. With a holistic understanding of performance from the user perspective which includes synthetic and RUM, this will help you find and fix website problems faster. Because RUM data eliminates blind spots in external monitoring, you’re more confident in knowing what customers are experiencing. RUM completes the monitoring spectrum. It provides a clearer understanding of web performance, enabling you to take targeted action and remove performance inhibitors. This puts website operators in a better position to commit to customer demands, marketing strategies and business revenue objectives.
- [CONTEXT: Collecting information means sealing the gaps – RUM is necessary]Let’s begin with “Collect”Hopefully, you are collecting your baseline metrics that comes with no surprises based on your business operations. Frankly, your synthetic monitoring should give you the same metrics day in day out; therefore your operations are normal and SLAs are met. Connie So, Dave, Shopatron has a large and complex set of clients to support. When did you implement Synthetic Monitoring in your daily operations and what did you look for? Dave As I mentioned earlier, there can be a lot of variations. While I can watch the server side response time for all clients through logs, it doesn’t tell me who has an image heavy site or who has a demographic that is skewed towards low bandwidth connections. Synthetic monitoring gives me a very stable baseline to always come back to. It tells me if there is something changing in my infrastructure that is positively or negatively impacting performance. It is somewhat of a proxy to the end user experience, but it is also off the mark in many cases. RUM allows me to see the end user experience and collect data on pages that I cannot watch all the time with Synthetic monitoring.Connie And now with the availability of RUM, how has that impacted you? How does RUM help meet your client needs and expectations?Dave With RUM, I have data already collected for pages. So, when I get questions about slow performance I have a ready repository of information to go to. This greatly improves my ability to respond to questions. Of course, the next level of operation is proactively watching performance across all clients and pages. This is more difficult to get to. Apply RUM for targeted customers - fill the gapDescribe the decision that you decided to add RUM to targeted customers? Direct ties the business - IT objectives and rev goals?Your acting on real-time troubleshooting, let’s talk about How you avoid chasing false positives – wild goose chases? Large Ecosystem and need to identify quickly “who has the time?” [3 strike policy and “Find and Fix”]Using RUM is a competitive advantage – reference the stats about 5% of customers are much better than competitors. We are competing against performance we can’t see but we have to assume that are competitors are working as hard as I am. It’s a necessity. What is your necessity you had to address when implementing RUM?Connie How has applying RUM helped with the Shopatron Promise?Dave I had one client recently ask me about page load times in China. His China office called and reported that the pageload times were significantly slower for the China site than for his US site. I already had RUM data ready, so I could dive in immediately. It used to be that I would have to setup a monitor to watch and tell the client that I would get back to him in a few weeks. This was a big step up. Then, the data was really interesting. The performance of the pages had a lot to do with the page content – the media being rendered on the site. This is something you don’t see just by looking at web logs.
- [CONTEXT: Collecting is one thing, making sense of it is another and requires correlation]Now that we have Collected the metrics, Correlating the metrics is equally important. Having a good understanding of what is happening with the site and the ability to interpret the metrics can set your operational and action plans for the business.Connie Dave, you had mentioned to me that you had a customer issue that he related to web performance. How were you able to help your customer solve the issue using RUM in conjunction with Synthetic monitoring? Dave I mentioned the China case where I dove into the data. Getting to the data quickly was very important. It was also essential that you need to be able to slice and collect the data along the correct lines. I see interesting behaviors when diving into the data – a lot of traffic to some sites from regions other than the intended audience. Lots of slow bandwidth or mobile traffic. All of this is interesting but you have to be able to focus on the problem you are trying to solve… who is the target audience? Where are they? What kind of connectivity and devices are they using?Correlating data is coming up with the conclusion/assessment and then taking action.The relationship is vital – with NSR WPM, see data in common view/overlay and not having to go to multiple tools. Honing in on root cause of the problem. And then take “CORRECT” action. Beyond corrective action, what else do you do with data? Clients has customers – protecting his clients business. Avoid crisis mode – how to minimize the impact of fire drills. Money = Service Availability, Customer Experience, Competitive threats
- [CONTEXT: Closing the gaps, making sense of the information, and executing decisive action with target investments builds confidence to perform]Connie So now, you have a holistic view of the metrics you are collecting (both synthetic and RUM) and you have correlated the web performance data, where does this leave you?Dave Earlier I mentioned the need to go to the next level. You need to get to the point where you are being proactive. For us, we commit to delivering specific qualities in performance. This means continually keeping an eye on how that performance is changing. This commitment is more than just marketing gloss – it is valued throughout the organization. Meeting the commitment is where marketing and technology connect in meaningful ways. Confidence Consistent practice and discipline and business objectives (Client SLAs)Managing a website, make commitment – find and fix, reduce false positive -- shy to commit and over estimate level of commitment (intelligent alerting?)Intelligent Alerting – NSR knows the importance and how to quickly find and fix the issue. Set up and Automated decision making – script out contingencies. Problems can start fixing themselves before you know that there is an issue. You can now commit to your own organization and stakeholders (marketing) and show the confidence and ability to perform on your growth objectives. Crossing into the business
- Dave’s thoughts about tools to use – Splunk, Neustar Web Performance, etc. How he uses load testing then moves to monitoring
- [CONTEXT: Summary of key points and offering of advice - potential Q&A]We know that there is money in the metrics and having both synthetic and RUM can drive Business, customers are watching you and competitors are benchmarking you, and we are in new territory with consumers having more choices. You have to add RUM to your tool box. Invest in the tools to help you find and fix performance issues. Connie Are you glad you did it? And what are the key suggestions/advice you would give to your colleagues?Dave Yes, it was great to dive into another tool to help solve the problems of today. This is one step on the long road of developing maturity around performance analytics. Key word of advice is to collect all the data you can. Start simple with internal monitoring, then get external monitoring going, then get RUM. All the data you collect today will help you solve tomorrow’s problem faster.Thanks Dave for your insight and sharing your experience with us.So, with that, this concludes the presentation and thanks again for joining us! I will open up the floor for Questions.