From Velocity 2012 in Santa Clara, CA. Buddy Brewer, Philip Tellis, and Carlos Bueno talk about real user measurement collection, analysis, and insights.

1. Velocity 2012 / 2012-06-26 RUM for Breakfast 1

2. RUM for Breakfast
Buddy Brewer, Carlos Bueno, Philip Tellis
Velocity 2012 / 2012-06-26
Velocity 2012 / 2012-06-26 RUM for Breakfast 2

3. Real Users
Velocity 2012 / 2012-06-26 RUM for Breakfast 3

4. Real Browsers
Velocity 2012 / 2012-06-26 RUM for Breakfast 4

5. https://github.com/lognormal/boomerang/
Velocity 2012 / 2012-06-26 RUM for Breakfast 5

6. Carlos built the dns plugin
Velocity 2012 / 2012-06-26 RUM for Breakfast 6

7. Buddy built the navtiming plugin
Velocity 2012 / 2012-06-26 RUM for Breakfast 7

8. tl;dr
1 Measure a bunch of stuff in the browser
2 Use high school stats that we vaguely remember
3 Randomly invent insights
Velocity 2012 / 2012-06-26 RUM for Breakfast 8

9. 1
Measure
Velocity 2012 / 2012-06-26 RUM for Breakfast 9

36. 2
Analyze
Velocity 2012 / 2012-06-26 RUM for Breakfast 10

37. Log-Normal Distribution
Velocity 2012 / 2012-06-26 RUM for Breakfast 11

38. Log-Normal Distribution
The logarithm of the x-axis follows a Normal distribution
Velocity 2012 / 2012-06-26 RUM for Breakfast 11

39. Log-Normal Distribution
Use the Geometric Mean for pure Log-Normal distributions
Velocity 2012 / 2012-06-26 RUM for Breakfast 12

40. Log-Normal Distribution
Performance data does not always follow a "pure" Log-Normal
distribution
Velocity 2012 / 2012-06-26 RUM for Breakfast 13

41. Look at the entire spread
...
Velocity 2012 / 2012-06-26 RUM for Breakfast 14

42. Look at the entire spread
which often approaches an infinite width
Velocity 2012 / 2012-06-26 RUM for Breakfast 14

43. Distill
Velocity 2012 / 2012-06-26 RUM for Breakfast 15

44. • 0.8% of hits are fake/abusive
• 0.2-0.5% of hits are from a stale cache
• 0.1% of hits are absurd
• Timestamps in the future (or past depending on how you
interpret it)
• Bots ignore robots.txt across domains
• "Interesting" caches/copies
Velocity 2012 / 2012-06-26 RUM for Breakfast 16

45. Even with beacons, you need to sanitize your input
Velocity 2012 / 2012-06-26 RUM for Breakfast 17

46. Band-pass filtering
Velocity 2012 / 2012-06-26 RUM for Breakfast 18

47. Band-pass filtering
• Strip everything outside a reasonable range
• Bandwidth range: 4kbps - 4Gbps
• Page load time: 0ms - 600s
• You may need to relook at the ranges all the time
Velocity 2012 / 2012-06-26 RUM for Breakfast 18

48. IQR filtering
Velocity 2012 / 2012-06-26 RUM for Breakfast 19

49. IQR filtering
Derive the range from the data
Velocity 2012 / 2012-06-26 RUM for Breakfast 19

50. Sampling
Velocity 2012 / 2012-06-26 RUM for Breakfast 20

51. Margin of Error
σ
±1.96 √n
Velocity 2012 / 2012-06-26 RUM for Breakfast 21

52. MoE & Sample size
There is an inverse square root correlation between sample size
and margin of error
Velocity 2012 / 2012-06-26 RUM for Breakfast 22

53. How big a sample is representative?
Select nsuch that
σ
1.96 √n ≤ 5%µ
Velocity 2012 / 2012-06-26 RUM for Breakfast 23

54. This needs to be at your lowest drilldown level
Velocity 2012 / 2012-06-26 RUM for Breakfast 24

55. 3
Insight
Velocity 2012 / 2012-06-26 RUM for Breakfast 25

56. How does performance
impact human behavior?

57. 8 million pages
1.5 million visits
50 different dimensions

58. very fast sessions had high bounce rates
70.00%
52.50%
35.00%
17.50%
0%
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

59. bounce rate vs. load time
70.00%
52.50%
35.00%
17.50%
0%
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

60. bounce rate vs. DOM interactive
70.00%
52.50%
35.00%
17.50%
0%
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5 11 11.5 12 12.5

61. bounce rate vs. front end time
80.00%
60.00%
40.00%
20.00%
0%
0.5 2 3.5 5 6.5 8 9.5 11 12.5 14 15.5 17 18.5 20 21.5 23 24.5 26 27.5 29

62. is my web site performance toxic to my
users?
http://www.flickr.com/photos/21560098@N06/3796822070

63. LD50 - when do half the users bounce?
http://www.flickr.com/photos/thecosmopolitan/6117530924

64. Bounce rate =50%
Back end time 1.7 sec
DOM Loading 1.8 sec
DOM Interactive 2.75 sec
Front end time 3.5 sec
DOM Complete 4.75 sec
Load event 5.5 sec

65. Future directions
What is the LD50 for your site?
Other bounce rates? 40%? 30%?
Other variables? (critical content
visible, etc)
Other behaviors? Conversions,
revenue, pages per session, actions,
when do people make tea?

66. Numbers don’t lie
Velocity 2012 / 2012-06-26 RUM for Breakfast 26

67. Questions?
Buddy Brewer @bbrewer Carlos Bueno @archivd
Philip Tellis @bluesmoon

68. Thank you
Velocity 2012 / 2012-06-26 RUM for Breakfast 27