Embracing service-level-objectives of your microservices in your Cl/CD This presentation discusses defining service level objectives (SLOs) for microservices and using them to automate quality gates in continuous integration/continuous delivery (CI/CD) pipelines. It presents the open-source Keptn framework, which can provide quality gates for microservices running on Kubernetes clusters based on defined SLOs. The presentation covers how SLOs and service level indicators (SLIs) can help prevent bad code from reaching production and configure alerts, and demonstrates Keptn's automated SLO evaluation and a use case for performance as a self-service tool.

1. Embracing service-level-objectives
of your microservices in your Cl/CD
Rob Jahn
Technical Partner Manager
Dynatrace
https://www.linkedin.com/in/robjahn/
https://events.nebulaworks.com/lightningtalk-april

2. Unfortunately, a robust CI/CD pipeline with well-defined tests does not guarantee a failure-safe
application in production. That is since many microservices are interwoven and dependent on each
other. Some of them have to handle hundreds of thousands of requests, while others are merely for
batch processing of data. Therefore, it is crucial to define the objectives and quality metrics of
individual services in terms of service level objectives as code.
• How service level indicators and service level objectives can help you set up automated quality gates in
a Cl/CD system to prevent bad code changes to ever reach production
• How quality criteria can be used to configure your alerts in monitoring solutions to be alerted on any
issues in your environments
• Present the open-source framework Keptn to provide quality gates for your microservice applications
running on your Kubernetes cluster.
More information about Keptn can be found on www.keptn.sh or on https://github.com/keptn
Embracing service-level-objectives of your microservices in your Cl/CD

3. Agenda
• Motivation / Inspiration
• Use Cases we provide in Keptn
• How it works
• How you can leverage & contribute
Embracing service-level-objectives of your microservices in your Cl/CD

4. API Testing Integration Testing Application Testing
Test a Component Test a System Test Real World
Continuous Testing Embedded in CI/CD Pipelines
Shift Left Performance
Continuous performance testing strategy

5. Challenge: Lengthy manual analysis and approval
Build Deploy to
„Test“
Run Test
In „Test“
Manual Approval
Promote to
„Staging“
Deploy to
„Test“
Functional: Test Result Trend Not Enough Performance: Manual Comparison Is Slow Monitoring: Too much unstructed data
Which build is better Which data comes from my testIs this a real problem
~30-60min

6. Solution - Automate the analysis
Error rate
Throughput + Response Time versus
objectives
Resources check
Test Failure

7. • An SLI is a service level indicator—a carefully defined
quantitative measure of some aspect of the level of
service that is provided.
• An SLO is a service level objective: a target value or
range of values for a service level that is measured by
an SLI. A natural structure for SLOs is thus SLI ≤
target, or lower bound ≤ SLI ≤ upper bound.
Inspiration #1 - Google SRE Handbook – Free online!
https://landing.google.com/sre/books/

8. Concept: Automated performance analysis by scoring
Spec files define:
• Service Level Indicators (SLI)
• Service Level Objectives (SLO)
Pass / Fail / Warning
=
Read the Spec
Retrieve
data
Calculate
Score
Datasource
1
2
3
4

9. Inspiration #2 – Mark Tomlinson
1 Create JIRA Ticket for Performance Feedback
Next Continuous Test Cycle picks up Request
Performance Feedback back on JIRA Ticket
2
3
1 2
3

10. Inspiration #3 – Thomas Steinmaurer
“Performance Signature”
for Build Nov 16
“Performance Signature”
for Build Nov 17
“Performance Signature”
for every Build
“Multiple Metrics”
compared to prev
Timeframe
Simple Regression Detection
per Metric

11. Define application delivery and
operations processes
declaratively
Use predefined CloudEvents to
separate the process from the
tools
Easy way to integrate and
switch between different tools
Blue/Green Deployments
Automated Quality Gates
Standardized communication protocol Keptn’s uniform
www.keptn.sh
an event-based control plane for continuous delivery and
automated operations for cloud-native applications

12. Use Case #1 – Automate SLO Evaluation
1 $ keptn start-evaluation timeframe service sli slo
Keptn pulls data from different data sources and Evaluates SLIs
Keptn delivers values and Score based on SLOs
2
3
1 2
3
7/8
(87.5%)
4/8
(50%)

13. Use Case #2 – Performance as a Self-Service
1 $ keptn deployment-complete my-service
Keptn
Keptn Delivers Score based on SLO
2
3
1 2
and Evaluates SLIsRuns Tests
3
7/8
(87.5%)
4/8
(50%)

14. Demo
Automate SLO Evaluation

15. Build and push
image
Deploy
Code
SLO Quality
Gate
Performance
Test
Automate SLO Evaluation
1 2 3 4
Pass or fail
pipeline
5
Deploy
Code
1
If PASS, then
deploy to
production
6
Production Environment
Request SLO Evaluation &
View in Keptn UI
SLO Quality
Gate
2
Rollback
yes/no??
3
Developer Checks in
code & SLO / SLI files

16. How it works
Keptn underneath the hood

17. SLIs & SLOs detailed example
Overall Failure Rate
Source: Dynatrace
Query: builtin:service.errors.total
Response Time LOGIN Tx
Source: Dynatrace
Query: calc:service.responsetime:filter(tx, LOGIN)
# DB Calls LOGIN Tx
Source: Dynatrace
Query: calc:service.dbcalls:filter(tx, LOGIN)
<= 5%
<= 2%
<=150ms & <=+10%
<= 400ms
<= +0%
<= +1
Build 1
0%
80ms
100ms
SLO: Overall Score Goal 90% 75%
Response Time 95th Perc
Source: Dynatrace
Query: builtin:service.responsetime(p95)
<=100ms
<= 250ms
SLOSLIs (Service Level Indicators) warn
pass
3
100%
Build 2
4%
120ms
90ms
3
75%
Build 3
1%
90ms
120ms
6
62.5%
Build 4
0%
95ms
95ms
3
100%
Build 1 Build 2 Build 3 Build 4

18. • YAML based definition of SLIs
• SLI Provider specific queries, e.g: Response Time
Keptn Quality Gate: SLIs & SLOs
Quality Gates
...
Dynatrace Prometheus Neoload
Scores SLIs
Queries SLI
Providers with
SLI Definitions &
Timeframe
• YAML based definition of SLOs:
• List of objectives with fixed or relative pass & warn criteria
indicators:
error_rate: "builtin:service.errors.total.count:merge(0):avg"
count_dbcalls: "calc:service.toptestdbcalls:merge(0):sum"
jvm_memory: "builtin:tech.jvm.memory.pool.committed:merge(0):sum"
objective:
- sli: error_rate
- sli: jvm_memory
- sli: count_dbcalls
pass:
- criteria:
- "=+2%" # We allow a 2% increase in DB Calls to previous runs
warning:
- criteria:
- "<=10" # We expect no more than 10 DB Calls per TX
total_score:
pass: "90%"
warning: "75%"
0.5 1.0 0.0 info
7/8
(87.5%)
4/8
(50%)
$ keptn start-evaluation timeframe service sli slo
5 DB Calls 360MB 4.3% 123SLI Value:
SLI Score:
Total Score
2
3
4
Tool X
1

19. SLI / SLO Validation Example
SLI #1
SLO
Scoring
Objective
Response Time (ms)
<500: 0.5
<200: 1
<210: 1
<204: 2
# SQLs / min
Value: 400
Score: 0.5
Value: 0% 2
2.5/3 (83%)
>190: 1
Criteria per SLI Weighted
Grading
SLI Values
Total
spec_version: "1.0"
comparison:
compare_with: "single_result”
aggregate_function: avg
objectives:
- sli: response_time_95
pass:
- criteria
- "<200"
warning:
- criteria
- "<500"
- sli: sql_statements
weight: 2
pass:
- criteria:
- "<+2%"
warning:
- criteria:
- "<+5%"
- ">-5%"
total_score:
pass: "90%"
warning: "75%"
SLI #2
SLO
500
200
Strategy
400ms
202/min
500
200
Base: 200
204 (+2%)
190 (-5%)
210 (+5%)
>500: 0
<190: 0
>210: 0
Quality Gates

20. Demo
Performance as a self-service

21. As an engineer I can request performance
feedback on demand without having any
dependency to other teams or environments
Allow everyone to get performance
feedback from their latest builds
and production in an easy way
Performance Center of Excellence
Teams enable Self-Service to
Engineers (more scale)
Elevate every engineer to
become a Performance Expert
(right tools and guard rails)
Scale through automation, best
practices and standards
What is performance as a self-service?

22. Notification
Service
Run test
Performance as a self-service with Jira & Keptn
Performance Feedback
back on JIRA Ticket for
team review
1
2
7
From a JIRA story, invoke Keptn
via automation call (REST)
Dynatrace
service
Jmeter
Service
Jira
Service
SLO Validation
Service
Evaluate
targets
3
Push test context
to Dynatrace
4
Send result
notifications
5
Add Jira issue
with test results
6

23. Thank you
https://keptn.sh https://tutorials.keptn.sh Build a new SLI provider
Rob Jahn
Technical Partner Manager Dynatrace
https://www.linkedin.com/in/robjahn
& how you can leverage & contribute