n 2016, Shutterfly, the leader in photo services and personalized gifts in the United States, embarked on a major program to accelerate every aspect of its website while building a PerfOps culture spanning across engineering, operation and product teams. The goal? Offer the fastest experience to customers while increasing revenue, conversion and SEO traffic. Some of the largest retailers and marketplaces went through this journey: As an example, Pinterest announced that their initiatives resulted in a 40% decrease in wait time, a 15% increase in SEO traffic and a 15% increase in conversion rate to signup. There is a strong correlation between faster web performance and higher business KPIs and Shutterfly saw a big opportunity for their customers and business. For the past 18 months, with strong support from Shutterfly’ senior staff, Fred Beringer, VP Digital Customer Experience is acting as champion for performance and with help from his product and engineering peers, crafted a strategy to rip the benefits of FAST web performance. They together kicked off a number of major initiatives, all targeted at faster performance. These initiatives include: • Revamped NFR (Non-functional Requirement) process to address performance and scalability requirement during the design phase. • Continuous testing on pre-production using a Gatling based framework to identify speed degradation during sprints. • Back-end optimization via profiling (using YourKit) to identify CPU and Memory inefficiencies (aka hotspots). • Implemented the top 10 front-end optimization industry best practices. • Benchmarked CDNs providers to find the best suited for Shutterfly performance requirements. • Leveraged AppDynamics to understand, troubleshoot and optimize key business transactions. • Implemented a RUM solution (SOASTA mPulse) to provide real-time visibility into each customer experience. • Implemented Synthetic Monitoring (NeuStar) for continuous testing and performance analysis of key user journeys. • Implemented a comprehensive Production Load and Performance program leveraging SOASTA CloudTest to prepare for Q4 peak days.

1. The PerfOps
Revolution
Shutterfly’s Journey
To FAST
Fred Beringer – VP, Digital Customer Experience

2. 2
WHO IS FRED?

3. 3

4. 4
WHY?

5. Source: Forrester Research
CUSTOMER SATISFACTION

6. Source: Forrester Research
Rebuilding Pinterest pages for performance
resulted in a 40% decrease in wait time, a
15% increase in SEO traffic and a 15%
increase in conversion rate to signup.

7. Source: Forrester Research
BBC has seen that they lose an additional 10%
of users for every additional second it takes
for their site to load

8. Source: Forrester Research
Staples reduced median homepage load time
by 1 second and reduced load time for the
98th percentile by 6 seconds. As a result,
they saw a 10% increase in their conversion
rate.

9. Source: Forrester Research
MORE STATS
https://wpostats.com/

10. 10
PerfOps Program

11. 11
Synthetic
Monitoring
Real User Monitoring
NFRs
Back-End Optimization
Performance Budget
Automated Load Test
Automated
WebPageTestBack-End Monitoring
Front-End
Optimization
Production Load Test

12. Non-Functional Requirements
Bring Engineering, Operations and Product together

13. class ProjectServicePatchSimulation extends Simulation {
val users = Configuration.getNumUsers(100)
val ramp = Configuration.getRampDuration(60 seconds)
val simulationDuration = Configuration.getDuration(StandardDurations.EXTRA_LONG)
val httpConf = newProjectServicesConfig(Configuration.ssapiProjectService.httpsUrl)
val numRepeats = 500;
val projectPatchScenario = scenario(this.getClass.getSimpleName)
.during(simulationDuration) {
exitBlockOnFail {
exec(clearCookies)
.exec{session=>
println("Simualtion duration: " + simulationDuration)
session
}
.exec(NonVisualAuthenticationService.signInUniqueUser)
.exec(ProjectService.createProject("photobook"))
.repeat(numRepeats) {
exec(ProjectService.patchProject("photobook", "${userid}",
"${photobook_project_guid}"))
}
.exec(ProjectService.createProject("calendar"))
.repeat(numRepeats) {
exec(ProjectService.patchProject("calendar", "${userid}", "${calendar_project_guid}"))
}
.exec(signOut)
}
}
setUp(projectPatchScenario.inject(rampUsers(users) over (ramp)))
.protocols(httpConf)
.assertions(simulationAssertions)
private def simulationAssertions: List[Assertion] = {
val baseAsserts = List(
details(ProjectService.REQUEST_PATCH_PROJECT + "-
calendar").failedRequests.percent.lessThan(1),
details(ProjectService.REQUEST_PATCH_PROJECT + "-
photobook").failedRequests.percent.lessThan(1),
details(ProjectService.REQUEST_PATCH_PROJECT + "-
calendar").responseTime.percentile3.lessThan(12000),
details(ProjectService.REQUEST_PATCH_PROJECT + "-
photobook").responseTime.percentile3.lessThan(1411),
details(ProjectService.REQUEST_PATCH_PROJECT + "-
calendar").requestsPerSec.greaterThan(10),
details(ProjectService.REQUEST_PATCH_PROJECT + "-
photobook").requestsPerSec.greaterThan(30)
)
baseAsserts
}
AUTOMATED LOAD AND PERFORMANCE

14. WEB PERFORMANCE BEST PRACTICES

15. 15
AUTOMATED WEBPAGE TESTContinuous Monitoring of Performance Improvements

16. PERFORMANCE BUDGET

17. BACK-END OPTIMIZATIONAPPDYNAMICS

18. BACK-END OPTIMIZATIONCPU HOTSPOTS WITH YOURKIT

19. BACK-END OPTIMIZATION
MEMORY HOTSPOTS WITH YOURKIT

20. BACK-END OPTIMIZATION
During non-peak
1. P80 - Backend time for this page is around 1.5Secs (from mpulse)
2. Around 30 slow transactions every 10 mins (from appD)
3. Around 4 very slow transactions ( > 900ms) every 10 mins
SAMPLE: ONE SFLY CRITICAL PAGES

21. BACK-END OPTIMIZATION
JDBC Connection Objects are created rather than
obtaining from pool. Database connections are
often expensive to create because of the overhead
of establishing a network connection and initializing
a database connection session in the back end
database.
Recommendation : Use Connection pool
SAMPLE: DB CONNECTION

22. 22
SYNTHETIC MONITORINGNEUSTAR

23. 23
REAL-USER MONITORINGMPULSE

24. PRODUCTION LOAD AND PERFORMANCESOASTA CLOUDTEST

25. PRODUCTION LOAD AND PERFORMANCESOASTA CLOUDTEST

26. Q4 - GAME ON!

27. Q4 - GAME ON!

28. 28
Get in Touch!
fberinger@shutterfly.com
twitter.com/fredberinger
linkedin.com/in/fredberinger