Keptn: Unbreakable
Continuous Delivery
Jürgen Etzlstorfer
@jetzlstorfer

Confidential 2
Agenda
• Why we have to build keptn
• What problems keptn solves
• What keptn is and how it works

3Confidential
Why we have to build !
Because Cloud Native Continuous Delivery is a big challenge!

Confidential 4
Your App / Container
Cloud Native: Deliver
High Quality Faster

Confidential 5
< 1 day < 1 week 1 to 2 weeks 2 to 3 weeks 3 to 4 weeks > 4 weeks
Small (11-100 employees) Medium (101-1000 employees)
Large (1001-5000 employees) Extra large (over 5000 employees)
3%
20%
35%
42%
2 days 12.5 days
95th Percentile Median
Commit Cycle Time: From Dev to Pro
Goal: 1h to Production
1 hour

Confidential 6
3 out of 10
Business Impacting
Deployments
3 hotfixes
Per Production
Deployment
4.8 days
MTTR
(Mean Time to Repair)
Evaluate for yourself: https://dynatrace.ai/acsurvey
12.5 days
Code to Production
(Commit Cycle Time)
1 out of 10 0 hotfixes ~4 hours2 days
Median
95th
Percentile

Confidential 7
How do we elevate an organization from Median to 95th percentile!
• Automated Quality (Shift-Left)
• Goal: Stop Bad Code Changes Early & Automated
• Automated Testing & Quality Gates
• Automated Multi-Stage Deployments (Shift-Right)
• Goal: Increase deployments into stable environments
• Dark, Shadow, Blue/Green Deployments with Auto-Validation
• Automated Operations (Self-Healing)
• Goal: NoOps & Zero-Impact on End Users
• Automated Remediation & (On-Demand) Scaling
97%deployment lead time
Faster
More Frequently
12 26releases per year
to
75%production incidents
Better

Confidential 8
What problems solves!
Let‘s developers build code vs maintaining pipelines & k8s clusters
Let‘s operations automate run books vs manual firefighting

Confidential 9
What we have seen organizations actually do!
• Building Custom Integrations with OpenSource & Commercial tools for
• Deployment Pipelines
• Testing Pipelines
• Auto-Remediation
• Notifications
• Auditing

Confidential 10
What we have seen organizations struggle with
Quote: „Pipelines seem to become our new future unmanagable legacy code!“
• Teams want to stick with existing tools to protect investement
• Containing lots of custom code for tool integration, error handling, logging, ...
• Getting harder to maintain the more tools get integrated
• Pipelines becoming more complex requiring dedicated teams
• Uncoordinated deployments between pipelines resulting in unstable Environments

Confidential 11
What is and how it works!
Enterprise-grade framework for shipping and running cloud native apps

Confidential 12
from 10000ft
GitOps-based collaboration
Operator patterns for all logic components
Monitoring and operations as code
Built on and for Kubernetes
Event-driven and serverless
Pluggable tooling
Automated multistage unbreakable delivery pipelines
Self-healing blue/green deployments
Event-driven runbook automation
Design Principles
Core capabilities

Confidential 13
keptn – conceptual architecture
Autonomous Cloud Control Plane
GitOps
Container
Registry
Continuous
Delivery AIOps
Operations
Automation
Test
Automation
Environment Definition (shipyard file)
ChatOps
Dev Namespace Staging Namespace Production Namespace
CoreServicesPlatform
Data
Provider

Confidential 14
keptn – Shipping through Unbreakable Continuous Delivery Pipelines
Autonomous Cloud Control Plane
prodstagedev
1: push
2: deploy (dark)
3: test (functional)
4: evaluate (architecture)
6: deploy (shadow)
7: test (performance)
8: evaluate (scalablity)
10: deploy (blue/green)
11: evaluate (business impact)
12: operate (NoOps)
5: promote 9: promote
Dev

Confidential 15
Keptn: Phase 1 – Creating a Project
$ keptn create project sockshop shipyard.yaml
prodstagedev prodstagedev
1
1 Create GitOps Config Repos
2
2 Create k8s Namespaces
shipyard.yaml
stages:
- name: "dev"
deployment_strategy: "dark"
test_strategy: "functional"
- name: "stage"
deployment_strategy: "shadow"
test_strategy: "performance"
- name: "prod"
deployment_strategy: "blue_green"

Confidential 16
Keptn: Phase 2 – Onboarding a service
$ keptn onboard service --project=sockshop --values=values_carts.yaml
prodstagedev
1
1 Setup Registry Hooks
2
2 Apply Configuration + Istio
2
prodstagedev
2 2
values_carts.yaml
replicaCount: 1
service:
name: carts
type: LoadBalancer
externalPort: 80
container:
name: carts
resources:
limits:
cpu: 100m
memory: 128Mi
...

Confidential 17
Keptn: Phase 3 – Pushing an artifact and deploy to dev
$ docker push yourregistry/carts:0.2
prodstagedev
1
prodstagedev
1
1 New Artifact: carts:0.2 dev
2
2 Start Deployment: carts:0.2 (dev, dark)
3
3 Deployment Finished: Success or Failure
4
4 Start Tests: Functional, Dev, carts
5
5 Tests Finished: Test Result & State
6
6 Start Deployment Evaluation: Perfspec
7
7 Evaluation Finished: Deployment Score
8
"indicators": [ {
"id" : “Prometheus:go_memstats_alloc",
"metricScore": 20
},{
"id": “Dynatrace:Avg_ActionDuration",
"metricScore": 20
} ],
"objective" : { "pass" : 90, “warning” : 75 }
6
8 Send new Artifact Event: carts:0.2 stage

Confidential 18
Keptn: Phase 4 – Promoting an Artifact to Staging
$ keptn send event newartifact carts:0.2 stage
prodstagedev
1
prodstagedev
1
1 New Artifact: carts:0.2 stage
2
2 Start Deployment: carts:0.2 (stage, shadow)
3
3 Deployment Finished: Success or Failure
4
4 Start Tests: Performance, Stage, carts
5
5 Tests Finished: Test Result & State
6
6 Start Deployment Evaluation: Perfspec
7
7 Evaluation Finished: Deployment Score
8
6
8 Send new Artifact Event: carts:0.2 prod
"indicators": [ {
"id": “Dynatrace:RT_Real",
"metricScore": 20
},{
"id": “Dynatrace:RT_Shadow",
"metricScore": 20
} ],
"objective" : { "pass" : 95 }

Confidential 19
Keptn: Phase 4 – Promoting an Artifact to Production: Blue/Green no issues!
$ keptn send event newartifact carts:0.2 production
prodstagedev
1
prodstagedev
1
1 New Artifact: carts:0.2 prod
2
2 Start Deployment: carts:0.2 (prod, bluegreen)
3
3 Deployment Finished: Success or Failure
4
4 Start Deployment Evaluation: Perfspec
5
5 Evaluation Finished: Deployment Score4
6
6 Blue/Green Switch
"indicators": [ {
"id": “Dynatrace:ConversionRate",
"metricScore": 100
}],
"objective" : { "pass" : 95 }

Confidential 20
Keptn: Phase 4 – Promoting an Artifact to Production: Blue/Green with problem!
$ keptn send event newartifact carts:0.2 production
prodstagedev
1
prodstagedev
1
1 New Artifact: carts:0.2
2
2 Start Deployment: carts:0.2 (prod, bluegreen)
3
3 Deployment Finished: Success or Failure
4
4 Start Deployment Evaluation: Perfspec
5
5 Evaluation Result: Dynatrace Problem Notify
4
6
6 Remediation: Back to Blue
6 Remediation: Trigger ServiceNow Workflow
6
66
6 Remediation: Update Configuration: carts:0.1

21Confidential
Demo Time

Confidential 22
Want to learn more?
• Learn how to use keptn: https://keptn.sh/docs/
• Clone, fork, star, contribute to keptn: https://github.com/keptn/
• Develop your own keptn-services: https://github.com/keptn/keptn-service-template
• Join our community (slack invite, meetings): https://github.com/keptn/community
• Follow us @keptnproject
• Fill out our survey: https://dynatrace.ai/acsurvey
• Reach out with any requests:
• juergen.etzlstorfer@dynatrace.com
• @jetzlstorfer

Confidential 23
dynatrace.com