Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2xtgyJH. Michael Bryzek explores what it’s like to build quality software with no development, QA, or staging environments. He includes a deep dive into “verification in production” and what it really takes to build software that can safely be tested continuously in production. Filmed at qconnewyork.com. Michael Bryzek is the CTO and co-founder of Gilt Groupe, an innovative online shopping destination offering its members special access to the most inspiring merchandise, culinary offerings, and experiences every day, many at insider prices. He has built Gilt's technology platform to support massive bursts of traffic as sales start every day at noon.

1. Production: Designing
for Testability
Michael Bryzek
mike@flow.io / @mbryzek
Cofounder / CTO Flow
Cofounder / ex-CTO Gilt

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
quality-production

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. Think about something you deployed
recently to production…

5. Is It Working?

6. Right now in production?

7. How do you know?

8. Feeling anxious?

9. 7
Let’s remove that anxiety

10. About Me
www.flow.io
www.gilt.com

11. Software Quality is Hard
Think end to end for entire lifecycle of code
Verification in Production is a powerful technique
to help us build quality software

12. True continuous delivery
No staging environments
Don’t run code locally
Life at Flow: Delivering Quality Software

13. True Continuous Delivery
Automated tests / No safeguards
1 way to do something
Assume Continuous Delivery in Design Process
github.com/flowcommerce/delta

14. “I love my staging environment”
Said Nobody Ever

15. No Staging Environments
Bottlenecks
Fragile
Difficult to understand failure
Expensive (30-40% of budget common)
Create the wrong incentives

16. Don’t run code locally
If unsure, write the test!
Learn to trust your tests

17. Quality Through Architecture
Extreme Isolation
• Rich event streams
• Own DNS, load balancer
• Private database
• No consul/zk/shared state
• Stop cascading failures
• “Delay” not “Outage”

18. Let’s look at real examples
Successfully “testing in production”

19. Example: Know That Checkout Works
Bot places an order every few minutes
Identify test orders and immediately cancel

20. Example: Support “Sandbox” Accounts
”SaaS” – even for internal accounts
Mark individual accounts as sandbox
One API Key for all sandbox accounts
"every service is a third party"

21. Example: End to End Integration Tests
Create Sandbox Org
Run tests
Delete Sandbox Org
“Safe and Repeatable”

22. Example: Using Sandbox Account for Test Orders

23. Example: Verifying Proxy Server Works as Expected
https://github.com/flowvault/proxy

24. Operating As Expected

25. But sometimes things go wrong
Even to the best of teams

26. Considerations
Make production access explicit (not the default)
Use defined paths (e.g. API calls)
Restrict sensitive data
Design for side effects

27. Unexpected Benefits

28. Perfect Documentation

29. Capture request/response of API Calls

30. Tooling: API | Builder
(formerly known as apidoc)
28
www.apibuilder.io
Version control for APIs
Backwards Compatibility
High Quality Mocks

31. High Quality Mocks – From Specs
Full Mock Generated
Implement Only What
You Need To Test
www.apibuilder.io

32. Tooling: Real Time DB Monitoring
https://www.vividcortex.com/blog/2014/04/22/visualizing-impact-index-change/

33. Tooling: Super Simple Alerts from Log
Log a prefix
Schedule a real time alert

34. Key Takeaways – Design Production to be Testable
Trust your tests, run subset in production
Invest in continuous delivery
Sandbox accounts are powerful
High quality, trustworthy mocks
Real-time feedback from production

35. Thank You!
We’re hiring: https://www.flow.io/careers
Michael Bryzek
mike@flow.io / @mbryzek
Cofounder / CTO Flow
Cofounder / ex-CTO Gilt

36. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/quality-
production