Slides from Velocity NYC 2013 presentation on measuring and tracking web performance.

1. Tracking Performance
Patrick Meenan
@patmeenan

2. Slides will be available
http://www.slideshare.net/patrickmeenan

3. Introducing shop-de-nile.com

4. System Architecture
Web Server (App)
Web Server (App)
Load Balancer
Load Balancer
Product
DB
Reviews
DB
Web Server (App)
Web Server (App)
SOLR FT Search
Web Server (Static)
Web Server (Static)
Product
Images

6. Us
Them

7. http://www.memegen.com/meme/dbuj21

8. New System Architecture
Product
DB
Web Server (App)
Web Server (App)
Load Balancer
Web Server (App)
Load Balancer
Web Server (App)
Varnish
Varnish
Memcache
Memcache
Memcache
Reviews
DB
SOLR FT Search
Web Server (Static)
Web Server (Static)
Product
Images

12. Testing Plan
● 14 Locations
● 3 “Representative” Pages
○
○
○
Landing Page
Search Results Page
Product Listing Page
● 1 Browser
● First View Only
● Every 5 Minutes (12 samples per hour)

13. 12,096 Tests Per Day

17. http://www.memegen.com/meme/ek1v79

19. CDN test plan
● Test 3 CDN’s
● Over 2 Weeks
● Compare against:
○ Each Other
○ Local Performance
○ No CDN

20. 36,288 Tests Per Day

22. Re-Run Competitive Comparison
●
●
●
●
Globally
20 Page Loads Per Day
3 Representative Pages
1,680 Page Loads

24. http://www.memegen.com/meme/duqjfs

25. Real User Monitoring (RUM)

26. 5.28
1.38

27. http://www.memegen.com/meme/9vyywz

28. Instrument back-end
http://www.slideshare.net/mikebrittain/metrics-driven-engineering-at-etsy
http://codeascraft.com/
Access Logs -> logster -> statsd -> graphite

29. Application Performance Measurement (APM)

31. Cache Hits

32. Not no much

34. Testing Plan
● 14 Locations
● 3 “Representative” Pages
○
○
○
Landing Page
Search Results Page
Product Listing Page
● 1 Browser
● First View Only
● Every 5 Minutes (12 samples per hour)

35. Go buy an SSD
@crucially
http://www.youtube.com/watch?v=H7PJ1oeEyGg

36. New New System Architecture
Product
DB (SSD)
Web Server (App)
Web Server (App)
Load Balancer
Web Server (App)
Load Balancer
Web Server (App)
Varnish
(SSD)
Web Server (Static)
Memcache
Reviews
DB (SSD)
SOLR FT Search
(SSD)
Web Server (Static)
Varnish
(SSD)
Memcache
Memcache
Product
Images
(SSD)

38. 4.58

39. Not matching synthetic testing
● Backbone vs last-mile
● Long-tail images

40. http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/

41. Our
Testing
http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/

42. Our Users
http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/

43. Testing Plan
Images always in edge cache
● 14 Locations
● 3 “Representative” Pages
○
○
○
Landing Page
Search Results Page
Product Listing Page
● 1 Browser
● First View Only
● Every 5 Minutes (12 samples per hour)

44. New Experience Testing Plan
● 14 Last-Mile Locations
● 3 “Representative” Pages
○
○
○
Landing Page
Random Search Results Page
Random Product Listing Page
● 2 Browsers
○ Desktop
○ Mobile (actual device)
● First View Only
● 20 Samples per day

46. Measured Load Times still too slow
●
●
●
●
Social Sharing Widgets
Ads
Retargeting pixels
Analytics
All already Async and using best practices

48. Ummm….
● Large drop in revenue
● Loss of tracking
● Doesn’t “feel” significantly faster

49. Speed Index FTW
● Measures visual experience
● Synthetic Only (WebPagetest)

50. Speed Index 3rd Party Evaluation

51. Speed Index FTW
● Measures visual experience
● Synthetic Only (WebPagetest)

52. We got this...it’s only code
onload=”performance.mark(‘aft.Image Loaded’)”...
window.onload:
● performance.getEntriesByType(“mark”)
● Report latest aft.* as custom time
http://blog.patrickmeenan.com/2013/07/measuring-performance-of-user-experience.html

53. Roll-your-own
● Send http beacon to beacon server
○ All timings as query params
○ 204 or transparent 1px gif
○ Log requests to access log
● IP, User Agent and Timings all in each
record
● access log -> logster -> statsd -> graphite
● Profit!

54. Google Analytics
_gaq.push(['_trackTiming', 'UserTimings', 'aft', measuredTime);

55. Boomerang/Soasta mPulse
BOOMR.plugins.RT.setTimer(‘custom0’, measuredTime);
https://gist.github.com/pmeenan/5902672

56. AFT vs onload
AFT
Median
0.932
95th Percentile
4.141
98th Percentile
7.262
onLoad
Median
2.235
95th Percentile
11.787
98th Percentile
26.72
http://www.soasta.com/products/mpulse/

57. Custom timings in WebPagetest

58. Where does that leave us?
● Competitive testing with synthetic last-mile
○
After builds pushed to staging/prod
● Performance tracking with RUM custom measurements
○
○
Data can be noisy
No competitive capability (opportunity)

59. Where does that leave us?
● Competitive testing with synthetic last-mile
○
After builds pushed to staging/prod
● Performance tracking with RUM custom measurements
○
○
Data can be noisy
No competitive capability (opportunity)

60. Synthetic testing in CI
● PhantomJS + CI
● WebPagetest Private Instance + CI
○
https://github.com/marcelduran/webpagetest-api
● Graphing/Trending artifacts
○
○
○
○
httparchive
graphite
Show Slow
tsviewdb

61. tsviewdb
●
●
Trending
Regression Detection
https://github.com/google/tsviewdb

62. People will optimize for what you measure….
So measure what you want optimized.