Speed and reliability are key for Mobile. But how do we know what the performance is we actually deliver to our end users? At what point do they actually drop off or how does the device or carrier factor in for end user performance? How do we collect the right type of data that allows us to a) measure end user performance and b) analyze it for improvement and business impact. Various options are available; sniffing vs tagging, real user monitoring vs synthetic monitoring, real devices vs emulation, and so on…What’s the difference, when to use the one or the other or what works best for your situation. This presentation, presented at the Hyves Mobile Developers Summit, will provide an overview and will cover: Why do we need to measure mobile performance? Overview of various methods for measuring mobile performance; Which type of tools are available and how to use them? How to interpret the data and make it actionable?

2. @jeroentjepkema
www. .nl

3. Mobile Performance Testing
Framework
Hyves Mobile Developers Summit,
2 March 2012

4. “”

5. “”
Your users spend most of their
time on other sites!
Jacob Nielsen

6. Putting things in perspective....

7. It’s about the Mobile Web....

8. It’s about the Mobile Web....
NOT

9. It’s your product....

10. ....and there are touchpoints

11. ....and there are touchpoints

12. ....and there are touchpoints

13. ....and there are touchpoints

14. Performance matters....

15. Abandonment"Rate"Across"200+"Sites"/"177+"Million"Page"Views"
Over"2"weeks"/"All#Browsers#
30"
25"
Abandonment"Rate"(%)"
20"
Abandonment*Rate*,*
15" All*Browsers*
10"
5"
0"
0" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15"
Page"Load"Time"Band"(sec.)"
Source: Gomez real user monitoring

16. Abandonment"Rate"Across"200+"Sites"/"177+"Million"Page"Views"
Over"2"weeks"/"All#Browsers#vs.#iPhone#Safari#
30"
25"
Abandonment"Rate"(%)"
20" Abandonment*Rate*,*
All*Browsers*
15"
Abandonment*Rate*,*
iPhone*Safari*
10"
5"
0"
0" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15"
Page"Load"Time"Band"(sec.)"
Source: Gomez real user monitoring

17. 58
40 40
20
Loads as fast as desktop Abandonment % above 5sec % Will not retun? % that goes to competition
2009 2011
Source: Ovum Research, 2011

18. 71
58
40 40
20
Loads as fast as desktop Abandonment % above 5sec % Will not retun? % that goes to competition
2009 2011
Source: Ovum Research, 2011

19. 74
71
58
40 40
20
Loads as fast as desktop Abandonment % above 5sec % Will not retun? % that goes to competition
2009 2011
Source: Ovum Research, 2011

20. 74
71
58
46
40 40
20
Loads as fast as desktop Abandonment % above 5sec % Will not retun? % that goes to competition
2009 2011
Source: Ovum Research, 2011

21. 74
71
58
46
40 40
33
20
Loads as fast as desktop Abandonment % above 5sec % Will not retun? % that goes to competition
2009 2011
Source: Ovum Research, 2011

22. The Mobile User

23. Under time pressure
The Mobile User

24. While on the move
Under time pressure
The Mobile User

25. Often one handed
While on the move
Under time pressure
The Mobile User

26. “”

27. “”
Mobile Consumers are willing to
trade in functionality, but are not
willing to give up on comfort
Ovum Research, 2011

28. Things will break...

29. Quality

30. Connection

31. Screen size

32. Security?

33. Design for disaster...

34. Design for disaster...be ready for failure

35. Mobile Performance Testing
Framework

36. Develop &
Optimize

37. Real Devices

38. 3G and WiFi connections

39. Real usage...

40. tekst
WEINRE
http://phonegap.github.com/weinre/
http://calendar.perfplanet.com/2011/i-see-http/

42. Critical Path Analysis:
http://velocityconf.com/velocityeu/public/schedule/detail/22313

43. Webpagetest.org

44. Per object
!
!
Webpagetest.org
Optimization tips

45. Webpagetest.org
Film strip

46. Mobitest / Blaze.io
!

47. Mobitest / Blaze.io
!

48. Cross Browser

49. Multi Device

50. Multi Browser

51. Multi Browser
Multi Browser

52. Touch & Gesture

53. Gomez
http://
www.smashingmagazine.com/
2011/08/07/a-dozen-cross-
browser-testing-tools/

54. Gomez

55. !
Gomez

56. DeviceAnywhere
http://www.soasta.com/mobile-app-testing/

57. Performance
Testing

59. Good
campaign

60. Good
gone
campaign

61. Good
gone BAD!
campaign

62. Find your ceilings...

63. Source: John Allspaw, Flickr.com
what you have left
The End
....know what your limits are

64. Web%traffic%%(Pageviews)!
1.000.000!
1.200.000!
200.000!
400.000!
600.000!
800.000!
0!
Mei;09!
jun;09!
jul;09!
aug;09!
sep;09!
okt;09!
nov;09!
dec;09!
jan;10!
feb;10!
mrt;10!
apr110!
mei10!
jun;10!
jul;10!
aug;10!
sep;10!
okt;10!
nov;10!
dec;10!
Jan;11!
feb;11!
Max.%number%of%pageviews%per%hour%!
mrt;11!
apr;11!
Christmas/EOY%
mei;11!
MAX!capacity!
Safety!capacity!
Traffic!Realized!
Traffic!Forecast!
jun;11!
jul;11!
aug;11!
sep;11!
okt;11!
Capacity planning
nov;11!
dec;11!

65. http://www.soasta.com/cloudtest/
performance-and-load-testing-tools/
Gomez

66. http://www.soasta.com/cloudtest/
performance-and-load-testing-tools/
http://www.soasta.com/cloudtest/
http://keynote.com/products/
performance-and-load-testing-tools/
web_load_testing/index.html
Gomez

67. http://www.soasta.com/cloudtest/
performance-and-load-testing-tools/
http://keynote.com/products/
web_load_testing/index.html
Gomez

68. Uptime &
Alerting

69. Not everybody is a
mastermind...

70. ...we need some protection

71. 3rd&Party/& Local&ISP& Browsers&and& End@Users
&
Cloud&Services& devices
&
Load&&
Balancers!
Web&&
Servers!
Mobile&&
Components!
App&&
Servers! Internet&
DB&&
Servers!
Storage!
Mainframe!
Network&
Major& Content&Delivery& Mobile&&
ISP& Networks& Carrier
&
Web Application Delivery Chain

72. 3rd&Party/& Local&ISP& Browsers&and& End@Users
&
Cloud&Services& devices
&
Load&&
Balancers!
Web&&
Servers!
Mobile&&
Components!
Is it my (IT)
App&&
Servers! Internet&
datacenter?
DB&&
Servers!
Storage!
Mainframe!
Network&
Major& Content&Delivery& Mobile&&
ISP& Networks& Carrier
&
Web Application Delivery Chain

73. 3rd&Party/& Local&ISP& Browsers&and& End@Users
&
Cloud&Services& devices
&
Load&&
Balancers!
Web&&
Servers!
Mobile&&
Components!
Is it my (IT)
App&&
Servers!
Is it my Internet&
datacenter?
DB&&
Servers!
ISP?
Storage!
Mainframe!
Network&
Major& Content&Delivery& Mobile&&
ISP& Networks& Carrier
&
Web Application Delivery Chain

74. 3rd&Party/& Local&ISP& Browsers&and& End@Users
&
Cloud&Services& devices
&
Load&&
Balancers!
Web&&
Servers!
Mobile&&
Components! Is it my 3d
Is it my (IT) Is it my Internet&
party
App&&
Servers!
datacenter?
DB&& ISP?
Servers!
Storage!
provider?
Mainframe!
Network&
Major& Content&Delivery& Mobile&&
ISP& Networks& Carrier
&
Web Application Delivery Chain

75. 3rd&Party/& Local&ISP& Browsers&and& End@Users
&
Cloud&Services& devices
&
Load&&
Balancers!
Web&&
Servers!
Mobile&&
Components! Is it my 3d Is it a
Is it my (IT) Is it my Internet&
party browser or
App&&
Servers!
datacenter?
DB&& ISP?
Servers!
Storage!
provider? device?
Mainframe!
Network&
Major& Content&Delivery& Mobile&&
ISP& Networks& Carrier
&
Web Application Delivery Chain

76. 3rd&Party/& Local&ISP& Browsers&and& End@Users
&
Cloud&Services& devices
&
Load&&
Balancers!
Web&&
Servers!
Mobile&&
Components! Is it my 3d Is it a
Is it my (IT) Is it my Internet&
Measure from the party perspective or
End User browser
App&&
Servers!
datacenter?
DB&& ISP?
Servers!
Storage!
provider? device?
Mainframe!
Network&
Major& Content&Delivery& Mobile&&
ISP& Networks& Carrier
&
Web Application Delivery Chain

77. Synthethic Monitoring:
‣ Simulate users by scripting pages and/or
Synthetic Monitoring business transactions;
‣ Continuous measurements (heartbeat) from
outside the firewall where your end users are;
‣ Best used for incident, problem and SLA
management.

78. Watchmouse
Alertsite
Keynote
Gomez

79. Gomez

80. Gomez

81. Gomez

82. Gomez

83. Real User
Experience

84. But, Synthetic testing isn’t enough

85. But, Synthetic testing isn’t enough
Gomez Asia
Gomez US
Gomez UK

86. Synthethic Monitoring:
‣ Simulate users by scripting pages and/or
Synthetic Monitoring business transactions;
‣ Continuous measurements (heartbeat) from
outside the firewall where your end users are;
‣ Best used for incident, problem and SLA
management.
Real User Monitoring:
Real User Monitoring ‣ Measures traffic of ALL users, transactions
and pages
‣ Passive solution which gives information
about the user experience
‣ Best usef for performance analytics and
capacity management

87. Gomez

88. New Relic

89. New Relic

90. Localytics
http://www.compuware.com/
application-performance-
management/real-user-
monitoring-mobile.html

91. Localytics
http://www.compuware.com/
application-performance-
management/real-user-
monitoring-mobile.html

92. 0,00%
2,00%
4,00%
6,00%
8,00%
01
-1
1-
11
02
-1
1-
11
03
-1
1-
11
04
-1
1-
11
05
-1
1-
11
06
-1
1-
11
07
-1
1-
11
08
-1
1-
11
09
-1
1-
11
10
-1
1-
11
11
-1
1-
11
12
-1
1-
11
13
-1
1-
11
14
-1
1-
11
15
-1
1-
11
16
-1
Conversion rate
1-
11
17
-1
1-
11
18
-1
1-
11
Availability
19
-1
1-
11
20
-1
1-
11
21
-1
1-
11
Conversion rate vs. Availability
22
-1
1-
11
23
-1
1-
11
24
-1
1-
11
25
-1
1-
11
26
-1
1-
11
27
-1
1-
11
28
-1
1-
11
29
-1
1-
11
30
-1
1-
11
90,0%
92,0%
94,0%
96,0%
98,0%
100,0%

93. 0,00%
2,00%
4,00%
6,00%
8,00%
0,00%
0,50%
1,00%
1,50%
01 01
-0 -1
1- 1-
12 11
02 02
-0 -1
1- 1-
12 11
03 03
-0 -1
1- 1-
12 11
04 04
-0 -1
1- 1-
12 11
05 05
-0 -1
1- 1-
12 11
06
-0 06
1- -1
12 1-
11
07
-0 07
1- -1
12 1-
11
08
-0 08
1- -1
12 1-
11
09
-0 09
1- -1
12 1-
11
10
-0 10
1- -1
12 1-
11
11
- 01 11
-1 -1
2 1-
11
12
-0 12
1- -1
12 1-
11
13
-0 13
1- -1
12 1-
11
14
-0 14
1- -1
12 1-
11
15
-0 15
1-
12 -1
1-
11
16
Conversion rate
-0 16
1-
-1
Conversion rate
12
1-
11
17
-0
1- 17
12 -1
1-
11
18
-0
1- 18
12 -1
1-
11
19
-0
Availability
1- 19
12 -1
loadtime (in sec)
1-
11
20
-0
1- 20
12 -1
1-
11
21
-0
1- 21
12 -1
1-
11
22
Conversion rate vs. Availability
-0
Conversion rate vs. Performance
1- 22
12 -1
1-
11
23
-0
1- 23
12
-1
1-
11
24
-0
1- 24
12
-1
1-
25 11
-0
1-
12 25
-1
1-
26 11
-0
1-
12 26
-1
1-
27 11
-0
1-
12 27
-1
1-
28 11
-0
1-
12 28
-1
1-
29 11
-0
1-
12 29
-1
1-
30 11
-0
1-
12 30
-1
1-
31 11
-0
1-
12
90,0%
92,0%
94,0%
96,0%
98,0%
0,0
1,0
2,0
3,0
4,0
5,0
6,0
100,0%

94. 0,00%
2,00%
4,00%
6,00%
8,00%
0,00%
0,50%
1,00%
1,50%
0
200.000
400.000
600.000
800.000
1.000.000
1.200.000
1.400.000
01 01
01 -0 -1
-0 1- 1-
6- 12 11
11
02 02
02 -0 -1
-0 1- 1-
6- 12 11
11
03 03
03 -0 -1
1- 1-
-0 12 11
6-
11
04 04
-0 -1
04 1- 1-
-0 12 11
6-
11
05 05
-0 -1
05 1- 1-
-0 12 11
6-
11
06
-0 06
06 1- -1
12 1-
-0 11
6-
11 07
-0 07
1- -1
07 12 1-
-0 11
6-
11 08
-0 08
1- -1
08 12 1-
-0 11
6-
11 09
-0 09
1- -1
09 12 1-
-0 11
6-
11 10
-0 10
1- -1
10 12 1-
-0 11
6-
11 11
- 01 11
-1 -1
11 2 1-
11
- 06
-1 12
1 -0 12
1- -1
12 1-
12 11
-0
6- 13
11 -0
1- 13
12 -1
1-
13 11
-0
6- 14
11 -0 14
1- -1
12 1-
14 11
-0 15
6-
11 -0 15
1-
12 -1
1-
Pageviews
15 11
-0 16
Conversion rate
6- -0
11 1- 16
-1
Conversion rate
12
1-
16 11
-0 17
6- -0
11 1- 17
12 -1
1-
17 11
-0 18
6-
11 -0
1- 18
12 -1
1-
18 11
-0 19
-0
Availability
6-
loadtime (in sec.)
11 1- 19
12 -1
loadtime (in sec)
1-
19 11
-0 20
6- -0
11 1- 20
12 -1
1-
20 11
-0 21
6- -0
1- 21
Pageviews vs. Performance
11 12 -1
1-
11
21 22
Conversion rate vs. Availability
-0 -0
6-
Conversion rate vs. Performance
1- 22
11 12 -1
1-
11
22 23
-0 -0
6- 1- 23
11 12
-1
1-
11
23 24
-0 -0
6- 1- 24
11 12
-1
1-
25 11
24 -0
-0 1-
6- 12 25
11 -1
1-
26 11
25 -0
-0 1-
6- 12 26
11 -1
1-
27 11
26 -0
-0 1-
6- 12 27
11 -1
1-
28 11
27 -0
-0 1-
6- 12 28
11 -1
1-
29 11
28 -0
-0 1-
6- 12 29
11 -1
1-
30 11
29 -0
1-
-0 12 30
6-
11 -1
1-
31 11
-0
30 1-
-0 12
6-
11
90,0%
92,0%
94,0%
96,0%
98,0%
0,0
1,0
2,0
3,0
4,0
5,0
6,0
100,0%
0
1
2
3
4
5

95. Would you recommend me?

96. Performance vs. Sentiment

97. Mobile Performance Testing Framework

98. Develop &
Optimize
Test code
Weinre
ICY
Dragonfly
Optimize
performance
Webpagetest
Mobitest
Firebug lite
Dynatrace Ajax
Mobile Performance Testing Framework

99. Develop & Cross Browser
Optimize testing
Test code Multi Browser
Weinre Browserstack
ICY Gomez
Dragonfly Selenium
Optimize Touch &
performance Gesture
Webpagetest Soasta
Mobitest DeviceAnywhere
Firebug lite
Dynatrace Ajax
Mobile Performance Testing Framework

100. Develop & Cross Browser Performance
Optimize testing Testing
Test code Multi Browser Capacity
Weinre Browserstack Gomez
ICY Gomez Soasta
Dragonfly Selenium
Optimize Touch &
performance Gesture
Webpagetest Soasta
Mobitest DeviceAnywhere
Firebug lite
Dynatrace Ajax
Mobile Performance Testing Framework

101. Develop & Cross Browser Performance Uptime &
Optimize testing Testing Alerting
Test code Multi Browser Capacity Mobile Website
Weinre Browserstack Gomez Watchmouse
ICY Gomez Soasta Gomez
Dragonfly Selenium Keynote
Optimize Touch &
Native App
performance Gesture
Webpagetest Soasta Gomez
Mobitest DeviceAnywhere Keynote
Firebug lite
Dynatrace Ajax
Mobile Performance Testing Framework

102. Develop & Cross Browser Performance Uptime & Real User
Optimize testing Testing Alerting Experience
Performance
Test code Multi Browser Capacity Mobile Website
Analytics
Weinre Browserstack Gomez Watchmouse Gomez
ICY Gomez Soasta Gomez Watchmouse
Dragonfly Selenium Keynote New Relic
Localytics
Google Analytics
Optimize Touch &
Native App
performance Gesture
Webpagetest Soasta Gomez
Mobitest DeviceAnywhere Keynote
Firebug lite
Dynatrace Ajax
Mobile Performance Testing Framework

103. Recommended...

104. Maximiliano Firtman: http://firt.mobi/
Luke Wroblewski: Mobile First
Ethan Marcotte: Responsive Web Design
Steve Souders: http://www.stevesouders.com/
Estelle Whyle: http://www.standardista.com h
Dutch Web Performance meetup:
http://www.meetup.com/Dutch-Web-Operations-Meetup/

105. Questions?

106. Thanks! More questions?
M: jtjepkema@measureworks.nl
T: @jeroentjepkema
W: www.measureworks.nl

107. Mobile Context

108. Who’s driving?
Mobile Context context!
What we need is

109. Why like this?
Who’s driving?
Mobile Context context!
What we need is

110. Why like this?
Who’s driving?
Where is he driving to?
Mobile Context context!
What we need is

111. Accounting:
Web Analytics = What did
they do?

112. Accounting:
Web Analytics = What did
they do?
Context:
Performance Analytics = Could they
do it?

113. Collect Mobile Experience data....

114. Where they use it?

115. When they use it?

116. How they use it?

117. And correlate...