‘Tis the Season – Holiday 2014 eCommerce Quality Checklist Past Webinar Archived (originally presented June 26th, 2014) This year, your holiday traffic will increase 15% or more, and 50% of the users will be mobile. Recent research shows 71% of your revenue comes from multi-channel users, so if you haven’t started planning, you’re already behind. Leading retailers are preparing for Holiday “14 and testing their production sites for multi-channel access to 115% capacity, or beyond! If you’re not one of them, your plans are incomplete. Cover your risks. Join Tenzing and SOASTA experts as they discuss the must-do checklist for peak performance. In this webinar you’ll learn: Align your Marketing and Quality plans Cover the multichannel user experience Test early in the lab and fully in production Optimize end-to-end site speed and performance When to freeze for the winter Don’t miss this opportunity to “shop early” and see how the leading retailers are already beating the odds with cloud testing.

1. ‘Tis the Season
A Holiday 2014 Ecommerce Checklist
Preparing for best case retail success

2. Holiday
2013 was
BIG

4. 2014 will be
BIGGER

6. mobile
matters

8. your site
needs to
PERFORM

9. 40 % abandon a
website that takes
more than 3
seconds to load
A 1 second delay
in page response
can result in a 7%
reduction in
conversions
79% of shoppers who
are dissatisfied with
website performance
are less likely to buy
from the same site
again.

10. If an ecommerce site is making
$10,000 a day, a 1 second
delay could cost you
$250,000 in lost sales every
year

11. Start Now
Prepare for the holiday rush

12. ‘Tis the Season
A Holiday 2014 Ecommerce Checklist
Preparing for best case retail success

13. Elizabeth Scott, Tenzing
Director of Technical Services
@BethxScott
Brad Johnson, SOASTA
VP Product Marketing
@bradjohnsonsv
Your Hosts

14. Infrastructure and Application
Monitoring
Load Testing
Marketing
Agenda

15. Poll Question

17. Infrastructure and
Application

18. Assess Your Infrastructure
Current infrastructure
performance and capacity
baseline:
 Server
Hardware
 Storage
 Network
 Database
 Application

19. Assess Your Infrastructure
 Know how many users can be on
your site: Do a load test
 Understand your vendor’s capacity
planning process
 Develop an emergency capacity plan

20. Ensure your
environment is up to date
on latest patch levels
Develop an emergency
patch and security plan
Secure Your Environment

21. Ensure ports on your
servers are not open
unnecessarily
Limit access
Secure Your Environment

22. Develop and test your
Denial of Service
response strategy
Run a vulnerability scan
Secure Your Environment

23. Tune queries,
search results and
session limits
Leverage a CDN
Tune Your Application

24. Find free capacity:
 Adjust your caching
 Adjust ‘type ahead’ search
 Remove/adjust view all options
 Adjust other features which are resource
hogs
Tune Your Application

25. Monitoring

26. Application Performance
End User Experience
Traffic Levels
Measure, Analyze, Act

27. •Upstream Connections
Payment Gateway
Shipping and Fulfillment
Measure, Analyze, Act

28. Plan for the
Inevitable

29. Assess your risk …
WHAT WILL GO
DOWN AND WHY?
Plan for the Inevitable

30. •Create a plan
Website overload plan
Incident response plan
Security/DoS response plan
Plan for the Inevitable

31. COMMUNICATE
Plan for the Inevitable

32. Modern Load
Testing

33. True Story

34. • Division of $6B Company
True Story

35. • Division of $6B Company
• Cyber Monday & last Free Shipping Day
True Story

36. • Division of $6B Company
• Cyber Monday & last Free Shipping Day
• “We wouldn’t disrupt the plan”
True Story

37. • Division of $6B Company
• Cyber Monday & last Free Shipping Day
• “We wouldn’t disrupt the plan”
• Crashed on Cyber Monday
True Story

38. • Division of $6B Company
• Cyber Monday & last Free Shipping Day
• “We wouldn’t disrupt the plan”
• Crashed on Cyber Monday
• No answers on Thursday
True Story

39. • Division of $6B Company
• Cyber Monday & last Free Shipping Day
• “We wouldn’t disrupt the plan”
• Crashed on Cyber Monday
• No answers on Thursday
• Production testing on Friday
reproduced the issue
True Story

40. Is there anything
missing from the
plan?

41. Prerequisite:
Know Your
Real Users
Modern Load Testing
Stop guessing
about what to test

42.  Most accurate models of real user
behavior
 Meaningful scale & appropriate
concurrency
 Control to start, stop & modify tests
without delays
 Test on your schedule
Modern Load Testing
Test for spikes – using the cloud

43.  Failover Scenarios
 Success Disasters
 End to End
Test what you don’t think needs to be
tested
Modern Load Testing

44. Measure,
Test and Act
in Real Time
Modern Load Testing

45. Don’t Wait until it’s too late
Compress processes for more test iterations
Traditional
Approach
TIME
Testdevelopment Infra.setup Fix Repeat?RetestTest Analyze
Modern Load Testing

46. Don’t Wait until it’s too late
Compress processes for more test iterations
/
Traditional
Approach
TIME
Testdevelopment Infra.setup Fix Repeat?RetestTest Analyze
CloudTest
Modern Load Testing

47. Don’t Wait until it’s too late
Compress processes for more test iterations
/
Traditional
Approach
TIME
Testdevelopment Infra.setup Fix Repeat?RetestTest Analyze
/ / / / Extra time!CloudTest
Modern Load Testing

48. Use real user data to
iterate dev & test plans
Run iterative load
tests via Jenkins
Daily (or more frequent) updates
keep teams quality focused
Embedded failure
details & bottleneck
data isolate issues
Manage to a
performance
baseline
Mobile? Run functional
automation on real devices
for functional & performance
validation
Spin up suitable, consistent
test environments in the
cloud or the lab
Jenkins runs any
unit tests as usual
Test Continuously – including performance!
5. Don’t Wait

49. s
Memory leaks
Test Scale
Stage/Team
Dev
& Test
Release
& Deploy
Network
& Ops
CDN file placement
Load Balancer configuration
Network bandwidth
Network configuration
DNS routing
Inadequate server resources
Default configuration settings
Unbalanced web serversAuto-scaling failures
Latency between systems
Slow third-party plug-ins
Garbage collection
Database thread counts
Inefficient database queriesSlow pages
Conflict with other apps
Test Lab Staging Production (100% +++)
Search technology limits
Method-level tuning
Max sockets exceeded
Firewall max capacity
Global latency variance
Security bottlenecks
Continuous tuning for full-scale…and well beyond
Start early, end in the real environment
6. Test in production and at Production scale

51. • Tests for flash sales with
unknown spikes
• mPulse analyzes real user
performance
• Continuous tests of mobile
and web traffic
“SOASTA was very easy to work with and could
accommodate our requirements in a four-week
timeframe.” Gavin Cliffe, OZSale CTO.
OZSale

52. NO
ISSUES
“SOASTA was very easy to work with and could
accommodate our requirements in a four-week
timeframe.” Gavin Cliffe, OZSale CTO.
OZSale
160%
more
visitors
$5M
AUD in
one sale
50%
mobile

53. Critical Factors for
Performance Testing Retail Sites
 Test for spikes
 Test what you think
doesn’t need to be
tested
 Test expecting failure
 Analyze & act in real
time
 Don’t forget about the
user experience
 Monitor everything
 Don’t wait until just
before the holidays
 Test in production

54. Marketing

55. Communicate your campaign plan
internally and with stakeholders
Establish a campaign change
process
Approval
Communication
Campaign Management

56. Establish a change
freeze on application
and infrastructure
Limit publishing and
catalogue changes
Campaign Management

57. Campaign Best Practices

58. Segment your list into
smaller groups
Trickle your email
Manage your media content
Email Best Practices

59. Don’t get too fancy
Develop test cases
for all of your discount
codes
Promotions and Discounts

60. Avoid using discount
structures that require a
database search
Don’t use new discount
structures
Promotions and Discounts

61. TEST, TEST, TEST!
Promotions and Discounts

62. Are you interested
in a FREE Retail
Readiness
Performance Test

63. Contact SOASTA:
www.soasta.com
info@soasta.com
866.344.8766
Follow us:
Twitter.com/cloudtest
Facebook.com/cloudtest
Retail Readiness Page:
soasta.com/retail-readiness/
Case studies:
soasta.com/customers/ecommerce/
Webinars future and past:
soasta.com/knowledge-center/webinars/
CloudTest Lite is FREE (forever)!
soasta.com/FREE
Resources from SOASTA
Thank you

64. www.SOASTA.com
more than managed hosting. commerce anywhere.
www.tenzing.com