Designing a perfect, failproof software delivery system is impossible. Failures will happen. What's more important is the speed and reliability of your recovery. Shipping with feature flags helps you limit your risk in the first place and recover faster when the unexpected happens. Today, with Optimizely Agent, companies that build their apps using service-oriented architectures can achieve production-scale faster with their feature delivery and experimentation platform.

1. 1
Failure is an Option:
Scaling Resilient Feature Delivery
Jamie Connelly
Sr. Product Manager
Mike Davis
Sr. Staff Engineer

2. ● Customize the widgets on your page
to your preference
● This webinar is recorded and you will
receive the link with the slides in the
next few days
● We will have time for questions! Please
submit them in the Q&A box on the left
side of the screen
Housekeeping

3. 3
Agenda
1. Why progressive delivery and experimentation?
2. How Optimizely helps you deliver more resilient features
3. Scaling resilience: Introducing Optimizely Agent
4. Benefits of deploying Optimizely as a microservice
5. Deep-dive into Optimizely Agent architecture
6. Dogfooding Case Study on Optimizely Agent
7. Q&A

4. 4
Why progressive delivery

7. Ship happens

8. 8
There's a better way….

10. “Our success at Amazon is a
function of how many
experiments we do per year,
per month, per week, per day.”
“Our company culture
encourages experimentation
and free flow of ideas.”
“We use experimentation
and testing to inform as
much of the business as we
can."
Culture of
experimentation
Ship every
11 Seconds
Ship every
22 seconds
Continuous
Code Deploys
Progressive Delivery and Experimentation
Greg Peters / Netflix CPO
Jeff Bezos / Amazon Founder
Larry Page / Google Founder
Ship every
3.5 Seconds

11. 11
Progressive Delivery

12. Am I building the right thing?
Can we quickly build it in the right way?
Can we catch bugs before our users do?
Will this change positively or negatively impact our KPI’s?
There is a guesswork gap
between your idea and getting it
into the hands of users
Ideation
Team
Guesswork Gap
Everyone

13. Define
& Design
Code
& Release
Ideation
Research
participants
Internal testersTeam
Iterative design & development
processes help, but don’t reflect
production environments or users
Guesswork Gap
Everyone

14. Define
& Design
Code
& Release
Ideation
Research
participants
Internal testersTeam
Iterative design & development
processes help, but don’t reflect
production environments or users
Guesswork Gap
Everyone$6B market cap loss
for Twitter in 1 day
in ad platform bugs

15. Define
& Design
Code
& Release
Measure &
Iterate
Optimize &
Grow
Ideation
Optimizely eliminates guesswork
across your entire lifecycle

16. Define
& Design
Code
& Release
Measure &
Iterate
Optimize &
Grow
Ideation
Rapid Prototyping &
Painted Door Tests
A/B/n tests
Feature Flagging
Phased & Targeted Rollouts
Omnichannel A/B/n testing
CRO & Messaging A/B/n Testing
Personalization
Our Progressive Delivery & Experimentation
Platform enables you to deliver better code,
better apps and growth faster

17. More frequent code depoys
208x
Prioritize experimentation
Elite product development teams have 208 times more
frequent code deployments and are twice as likely to
meet or exceed performance goals, like profitability.
Organizations that significantly outperform their
peers are almost twice as likely to prioritize
testing and experimentation.
Industry research confirms
2x

18. 18
Scaling Resilient Feature Delivery

19. Starts with feature flags

20. Key Feature Flagging Use Cases with Optimizely
Develop features concurrently by implementing
flags set to “off”. Deploy to master at any time
while avoiding merge conflicts.
Trunk-based development
Use a flag to send a small percentage of traffic
to a new build of your application to discovery
any unforeseen issues prior to rolling out to all
users.
Canary Releases
Give business users the power to grant
exclusive access to new features or beta
programs for your best customers or
prospects in a sales cycle.
Beta or Exclusive Access
With feature flags, you can quickly turn off
problem features without rolling back an entire
release.
Kill Switch
Deploy features behind “off” flags, then give
marketers or PMs the ability to turn on the
feature independent of a new deploy.
Dark Launches
Add parameterized variables to feature flags to
update features in real time, without a code
push or app update. Variables can also be
used to A/B test variations of features.
Remote Configuration

21. Optimize your search results, recommendations,
promotions, or any other algorithms to deliver the
most relevant content to your customers
Full Stack Experiment Use Cases
Algorithms
Test entirely new site or app designs before you
launch them, without any performance impact to
your customers
Front-End & Redesigns
Test features as you roll them out, to validate your
hypothesis and monitor your key business
metrics
Feature Experiments & Rollouts
Experiment with different prices, plans, or fees
personalized to your customer segments to drive
revenue
Price Testing
Don’t just test the customer experience: test
APIs, microservices, clusters, and architecture
designs to improve performance & reliability
Backend Performance
Test experiences on mobile devices, over-the-top
apps, IoT, conversational interfaces, and any
other experience that runs in code, and test
across devices.
Omnichannel

22. Rise of microservices
Monolith

23. Rise of microservices
Monolith Microservices

24. Rise of microservices
Monolith Microservices
● Faster development
velocity
● Isolated services are more
resilient
● Purpose-built

25. Rise of microservices
Monolith Microservices
50% of companies say that more than 50% of their new development utilizes microservices
-Source: 2018 O'Reilly The State of Microservices Maturity
● Faster development
velocity
● Isolated services are more
resilient
● Purpose-built

26. 27
Alternative: Build a Service around SDK

27. 28
Alternative: Build a Service around SDK
Drawback: Requires months of engineering time

28. Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Introducing
Optimizely Agent
Scale feature delivery by implementing Optimizely
as a highly available, distributed microservice

29. Implementation: Monolith vs. SOA
Monolith Microservices

30. Implementation: Monolith vs. SOA
Monolith Microservices
SDK

31. Implementation: Monolith vs. SOA
Monolith Microservices
SDK
SDK SDK
SDKSDK

33. 34
At production scale,
customers maintain
1000s of services

34. 35

35. 36
Deployment Options
Deploy as a container Compile from source
> make run

36. 37
Benefits of Using Optimizely Agent

37. 38
Faster implementation time vs. building a service

38. 39
Faster implementation time vs. building a service
● Get started in minutes
● Reach production scale in days vs.
months
● Customizable out-of-the-box service
● Open source, extend Agent to meet
custom needs

39. 40
Maintenance and monitoring

40. 41
Maintenance and monitoring
100s to 1000s of
SDK instances
1 Service

41. 42
Maintenance and monitoring
100s to 1000s of
SDK instances
1 Service
● Single location for
upgrades
● Fits microservices pattern
● Built-in admin API to track
health, application
performance

42. 43
Governance and Onboarding
Team A Team B Team C
Optimizely Agent

43. 44
Security
● One implementation for
security review
● Decreased network surface
area with less external network
requests
● Predictable outbound requests

44. 45
Technical Deep dive on Optimizely Agent

45. 46
Agent APIs
ADMIN APIs
● GET /health
● GET /metrics
● GET /info
● POST /webhooks/
CLIENT APIs
● POST /v1/activate
● POST /v1/track
● GET /v1/config
● POST /v1/override

46. 48
Activate API
POST /v1/activate
Activate iterates over all qualifying features and
experiments and returns decisions for the given user.
Activate returns variables where applicable.
REQUEST BODY
● userId — string
● userAttributes — object
QUERY PARAMETERS
● featureKey — Array of strings
● experimentKey — Array of strings
● disableTracking — boolean
● type — enum array (“feature”, “experiment”)
● enabled — boolean
Example Request
{
"userId": "string",
"userAttributes": {
"country": "string",
"customer": false
}
}
Example Response
[
{
"featureKey": "string",
"experimentKey": "string",
"variationKey": "string",
"type": "feature",
"enabled": true,
"variables": {
"property1": "string",
"property2": "string"
}
}
]

47. 49
Track API
POST /v1/track
Example Request
{
"userId": "string",
"userAttributes": {
"country": "string",
"customer": false
}
"eventTags": {
"revenue": int,
"product": "string"
}
}
Example Response
n/a
Track sends event and user details to Optimizely’s
analytics backend for experiment analysis.
REQUEST BODY
● userId — string
● userAttributes — object
● eventTags — object
QUERY PARAMETERS
● eventKey (required) — string

48. 50
Config API
GET /v1/config
The config API allows you to access
the contents of your datafile — the
experiments and features required by
your application.
Example Response
{
"revision": "string",
"experimentMap": {
"experiment-key1": {
"id": 0,
"key": "string",
"variationsMap": {...}
},
"experiment-key2": {
"id": 0,
"key": "string",
"variationsMap": {...}
}
},
"featureMap": {
"feature-key1": {
"id": 0,
"key": "string",
"variablesMap": {...},
"experimentsMap": {...}
},
"feature-key2": {
"id": 0,
"key": "string",
"variablesMap": {...},
"experimentsMap": {...}
}
}
}

49. 51
Override API
POST /v1/override
Override an experiment or feature test decision. The
override is stored in local memory. Should be used
debugging and testing purposes. Should not be
used for production overrides.
REQUEST BODY
● userId — string
● experimentKey — string
● variationKey — string
Example Request
{
"userId": "string",
"experimentKey": "string",
"variationKey": "string"
}
Example Request
n/a

50. 52
Built for Enterprise Scale
● Native OAuth2 support
● Optional TLS encryption
● Network isolation between client and
admin APIs
● Support for multiple environments
● Open source software

51. 54
Optimizely Dogfooding Case Study:
The Results Page

53. 56
Optimizely Results API: SDK Implementation

54. 57
Optimizely Results API: SDK Implementation
= 70 servers

55. 58
Optimizely Results API: SDK Implementation
= 70 x10 = 700 Server Instances

56. 59
Optimizely Results API: SDK Implementation
= 700 SDK Instances

57. 60
Optimizely Results API: Agent Implementation
= 12 clustered
Agent instances

58. 61
Optimizely
Agent
Instance 1
Load Balancer
Optimizely
Agent
Instance N...
Customer
Service
/activate
Customer
Service
/track
Customer
Service
/activate
Optimizely CDN
Optimizely
Backend
datafiles
events

59. 62

60. 63

61. 64
Agent and the Optimizely Results API
How it’s deployed:
● 12 containers
● 6 vCPUs
● 12GB RAM
What it’s doing:
● >700 clients
● 2.5K requests per second
● P95 latency* ~1ms per request
*All performance data depends on the connection speed, network, device, web page, and many other factors; actual results will vary.

62. 65
How to get started with Agent

63. 66
Recap on Optimizely Agent
1. Faster implementation in service-oriented architectures
2. Easier to maintain, monitor, and upgrade
3. Better governance and easy onboarding
4. Easier security and centralized networking

64. 67
Getting Started with Agent
1. Check out the Github Repo today! https://github.com/optimizely/agent
2. Not using Optimizely? Get free feature flags with optimize.ly/agent-rollouts
3. Docs will be published next week
4. GA launch end of month

65. 68
Q&A
Sign-up today for Optimizely Rollouts account:
optimize.ly/agent-rollouts