When shipping new features, do you optimize for speed or quality? What if you could have more of both? In this presentation, Emily will describe how the use of feature flags, observability, and the principles of code ownership come together to help Honeycomb engineers ship quickly and safely. Adopting these tools can help engineers more rapidly iterate on new features, reduce deployment risk, and quickly resolve incidents, all while keeping customers happy. When you attend this webinar, you will: Learn how to incorporate feature flags into your new feature release process Understand how observability can help your engineers ship new features quickly and safely Learn about the principles of code ownership and how to adopt them on your team Find out how these three components will help your team increase safety, speed, and user happiness

1. Emily Nakashima
Director, Engineering
Honeycomb.io
Building & Deploying Resilient Products: How Feature Flags
and Observability Can Help Teams Ship Faster & Safer
Tune in on Twitter: #testandlearn19

2. Building & Deploying Resilient Products
@eanakashima

3. about://
Emily Nakashima
dir. engineering @ honeycomb.io

6. “Resilient deploys with
feature flags”

7. What is resilience?
• A system that can adjust its functioning
• … in response to events (changes, disturbances, opportunities)
• … and continue to sustain required operations
• … in both expected or unexpected conditions
http://erikhollnagel.com/ideas/resilience-engineering.html

8. What is resilience?

9. What is resilience?
resilience
!=
reliability

10. Resilient Products

11. Remember Chaos Monkey?

12. As software engineers, we make our own chaos!

13. As software engineers, we make our own chaos!
unique Honeycomb web application releases deployed over the past week

14. What is resilience for application code?
• A resilient software application continues to sustain required operations
• … even when we deploy code with logical errors
• … or UI with an imperfect design
• … to a heterogenous, changing client environment

15. What is resilience for application code?
And why do we care? If we have a resilient application, we can:
• spend less time in QA and deliver faster
• offer a higher-quality experience to customers
• reduce support & ops pain

16. What is resilience for application code?
either customers don’t notice issues
or
we fix them really, really fast

17. Feature Flags

18. What is a feature flag?
/* Feature flag for new home page experience */
if FeatureFlags.instantGratification {
return <NewHomeView team={team} datasetTypes={datasetTypes} />;
} else {
return (
<div>
<h2>Honeycomb Home</h2>
<LegacyGraphs {...this.props} />
</div>

19. What is a feature flag?

20. FeatureFlagly
What is a feature flag?

21. FeatureFlagly
New home page experience

22. Old page layout

23. New page layout (work in progress!)

24. Find issues early

25. Find issues early

26. Solicit feedback

27. Benefits
• Avoid long-running branches
• Allow early feedback (from employees AND customers)
• Find bugs before customers do (using production data)
• Enable quick fixes
• Componentize design & code and allow iteration

28. Feature Flags:
Not just for UI changes

29. Varstring compression change

30. Varstring compression change

31. Varstring compression change

32. Varstring compression change

33. Varstring compression change

34. Varstring compression change

36. Varstring compression change

37. Varstring compression change

38. Observability

39. Observability != Dashboards & Graphs

40. Instrumenting your code so that
you can deeply understand the
state of your system just by
observing its outputs
Observability

42. What is an event?
1 event
==
1 unit of work
~=
1 request

43. What’s in an event?
{
"GojiPattern": "/user_event/:event_type",
"Header.Content-Type": "["application/json"]",
"Header.Cookie": "["_ga=GA1.2.2033006133.1516389900;",
"Header.User-Agent": "["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1)…”]”,
"Host": "127.0.0.1:8080",
"IsXHR": true,
"Method": "POST",
"RequestURI": "/user_event/page-unload",
"ResponseContentLength": 443,
"ResponseHttpStatus": 200,
"ResponseTime_ms": 123,
"Timestamp": "2018-03-02T06:14:57.206349701Z",
"UserEmail": "nathan@honeycomb.io",
"UserID": 18,
"availability_zone": "us-east-1b",
"build_id": "6552",
"env": "dogfood",

44. Observability != Dashboards & Graphs

45. Where do I start instrumenting?
Start at the edge.
Instrument the surfaces
customers touch first.

46. Observability: ask any question

47. Observability: ask any question

48. Observability: ask any question

49. “Nines don’t matter if users
aren’t happy.”
- Charity Majors, all the time

50. Code Ownership

51. Everyone works with production

52. Everyone’s on call now

53. Everyone has autonomy to make the product
better

54. Deploy on Friday afternoons!
https://charity.wtf/2019/05/01/friday-deploy-freezes-are-exactly-like-murdering-puppies/

55. Thank you!
@eanakashima
honeycomb.io

56. Next Session
Creating a Data-Informed Culture on Your
Product Team
Moderated by Claire Vo, Optimizely
Lauren Schuman, Sr. Director Product Insights, Mailchimp
Tune in on Twitter: #testandlearn19