RUM discussion at NYC Web Performance Meetup

1. RUM First

2. Who I Am
Cliff Crocker
VP Product
SOASTA, Inc.
@cliffcrocker

3. http://www.flickr.com/photos/luc/
Real Users

4. http://www.flickr.com/photos/spullara/
Happy Users

5. http://www.flickr.com/photos/sybrenstuvel/
Not So
Happy

6. RUM 101
Real User Measurement (RUM) is a technology for collecting
performance metrics directly from the browser of an end user.
• Involves instrumentation of the page with JavaScript
• Measurements are fired across the network to a collection point
through a small request object (beacon)
<JS> MySite.com
<beacon>

7. This helped…

8. C’mon Safari, be cool…
http://caniuse.com/nav-timing
69.34%
Browser
Support

9. Why RUM is Awesome
• Reduces the level of convincing we have to do within our
organization that ‘faster is better’
• The only way to measure perceived performance from your
end user
• Easy to maintain
• Only getting better…
– Check out: http://blog.patrickmeenan.com/2013/07/measuring-
performance-of-user-experience.html
– User Timing for RUM and Synthetic

10. What gives??
No RUM
RUM • According to a study
done in 2012* only 1
in 5 of the top
ecommerce sites are
using some form of
RUM
*http://www.webperformancetoday.com/2012/11/13/real-user-monitoring-rum-ecommerce-sites/

11. Challenges with RUM
1. It’s a “data” problem
2. People don’t like change
3. Hard to “sell” internally

12. For a modest sized site, in a given day
1-3 Million Page Views
Perf Timing
User Agent
OS/Device
Geography
Page
Segmentation
Bandwidth
Classification
A/B Bucket
‘Other’

14. Many have advocated sampling…

15. 1% sample, looking at a period of about 3 hours for a given page segment (Checkout)
Filtered to the United States…

16. Using a sampling approach
Looking only at California…
…and IE 9 traffic only

17. Now look at a 100% sample for the same time duration
All the same filters applied – AND looking specifically at ‘Back-End’ time

18. Sampled at 1%
Not sampled

19. Many use a batch solution…

20. • Batch processing large amounts of data has
been a traditional approach for RUM
– Great for looking at historical data, trending, etc.
– Big Data is not necessarily Fast Data
• Not ‘Operational’

21. In order for RUM to be operationally relevant,
you need the data in real-time

22. Synthetic vs. RUM
http://www.flickr.com/photos/myvector/

23. It is Not Either Or
• Synthetic and RUM coexist in any
performance strategy
• Great use cases for both

24. Synthetic Monitoring
• Pros
– Technology is great
– Real Browsers (IE,
Chrome, FF)
– Built in Alerting
– Waterfall analysis
– Screen shots & headers
– Object level detail
• Cons
– False Positives
– Impossible/impractical
to measure everything
– Fixed number of
browser/OS
combinations
– Simulated bandwidth
constraints at best
– Too few data points for
statistical relevance

25. RUM
Cast a wide net
• Identify key areas of concern
• Understand real user impact
• Tie this back to $$ (next)
Synthetic
Diagnostic tool
• Identify issues in a ‘lab’
• Reproduce a problem found with
RUM
http://www.flickr.com/photos/84338444@N00/with/3780079044/
http://www.flickr.com/photos/ezioman/

26. RUM in the
Workplace

27. Studies show…

28. However, you have to look at your own
data

29. Understand your KPIs and How to
Track Them
• Retail: Order Conversion, Add to Cart
Conversion, Revenue
• Media: In Line Display Ad Views, Comments
• Other: Facebook likes/Social Media sharing

30. Bottom Line, if you want your business
to care – then make it relevant for
them (speak their language)

31. Strategy for Success – RUM First
RUM First
• All the data
• Real-time and
historical
Supplement
with
Synthetic
• Reduce
measurements
based on RUM
• Use for adhoc
diagnostic
Map to
business
KPIs
• YOUR goals
• YOUR data

32. Free and Open Toolsets
RUM JS Collection Visualization Synthetic
boomerang.js Stats D Graphite WebPagtest
Episodes ShowSlow SpeedCurve (beta is
free?)
SOASTA mPulse, Torbit, Google Analytics, Pingdom, New Relic

33. mPulse Lite
http://www.soasta.com/free/