Building Autonomous Operations for
Kubernetes with keptn

2Confidential
Agenda
• Why we have to build keptn
• The problems companies are currently facing
• What keptn is and how it works
• Demo – Onboarding a Service (throughout the talk)
• Demo – Runbook Automation & Self-healing

3Confidential
Why we have to build
Because Cloud Native Continuous Delivery is a big challenge.

Confidential 4

Confidential 5
Collecting more evidence: https://dynatrace.ai/acsurvey
< 1 day < 1 week 1 to 2 weeks 2 to 3 weeks 3 to 4 weeks > 4 weeks
Small (11-100 employees) Medium (101-1000 employees)
Large (1001-5000 employees) Extra large (over 5000 employees)
3%
20%
35%
42%
2 days 12.5 days
95th Percentile Median
Commit Cycle Time: From Dev to Pro
Goal: 1h to Production
1 hour

Confidential 6
Median
95th
Percentile

Confidential 7
How do we elevate an organization from Median to 95th percentile
• Automated Quality (Shift-Left)
• Goal: Stop Bad Code Changes Early & Automated
• Automated Testing & Quality Gates
• Automated Multi-Stage Deployments (Shift-Right)
• Goal: Increase deployments into stable environments
• Dark, Shadow, Blue/Green Deployments with Auto-Validation
• Automated Operations (Self-Healing)
• Goal: NoOps & Zero-Impact on End Users
• Automated Remediation & (On-Demand) Scaling

Confidential 8
Why companies can‘t delivering high
quality faster

Confidential 9
What we have seen organizations actually do
• Building Custom Integrations with OpenSource & Commercial tools for
• Deployment Pipelines
• Testing Pipelines
• Auto-Remediation
• Notifications
• Auditing

Confidential 10
What we have seen organizations struggle with
Quote: „Pipelines seem to become our new future unmanagable legacy code!“
• Teams stick with existing tools to protect investement
• Containing lots of custom code for tool integration, error handling, logging, ...
• Getting harder to maintain the more tools get integrated
• Pipelines becoming more complex requiring dedicated teams
• Uncoordinated deployments between pipelines resulting in unstable environments

Confidential 11
What is and how it works
Enterprise-grade framework for shipping and running cloud native apps

Confidential 12
from 10000ft
GitOps-based collaboration
Operator patterns for all logic components
Monitoring and operations as code
Built on and for Kubernetes
Event-driven and serverless
Pluggable tooling
Automated multistage unbreakable delivery pipelines
Self-healing blue/green deployments
Event-driven runbook automation
Design Principles
Core capabilities

Confidential 13
keptn – Conceptual Architecture
Autonomous Cloud Control Plane
GitOps
Container
Registry
Continuous
Delivery AIOps
Operations
Automation
Test
Automation
Environment Definition (shipyard file)
ChatOps
Dev Namespace Staging Namespace Production Namespace
CoreServicesPlatform
Data
Provider

Confidential 14
Autonomous Cloud Control Plane
event-
broker
authenticator
controlcli
keptn-channel
event-
broker-ext
configuration
deployment
evaluation
test
problem
new-artefact
Services
jenkins-
service
github-
service
service
now-
service
keptn: Phase 3 – Pushing an artifact and deploy to dev

Confidential 15
Autonomous Cloud Control Plane
prodstagedev
1: push
2: deploy (dark)
3: test (functional)
4: evaluate (architecture)
6: deploy (shadow)
7: test (performance)
8: evaluate (scalablity)
10: deploy (blue/green)
11: evaluate (business impact)
12: operate (NoOps)
5: promote 9: promote
Dev
keptn – Shipping through Unbreakable Continuous Delivery Pipelines

Confidential 16
keptn: Phase 1 – Creating a Project
$ keptn create project sockshop shipyard.yaml
prodstagedev prodstagedev
1
1 Create GitOps Config Repos
2
2 Create k8s Namespaces
shipyard.yaml
stages:
- name: "dev"
deployment_strategy: "dark"
test_strategy: "functional"
- name: "staging"
deployment_strategy: "shadow"
test_strategy: "performance"
- name: "production"
deployment_strategy: "blue_green"

Confidential 17
keptn: Phase 2 – Onboarding a Service
$ keptn onboard service --project=sockshop --values=values_carts.yaml
prodstagedev
1
1 Setup Registry Hooks
2
2 Apply Configuration + Istio
2
prodstagedev
2 2
values_carts.yaml
replicaCount: 1
service:
name: carts
type: LoadBalancer
externalPort: 80
container:
name: carts
resources:
limits:
cpu: 100m
memory: 128Mi
...

Confidential 18
keptn: Phase 3 – Pushing an artifact and deploy to dev
$ docker push yourregistry/carts:0.2
prodstagedev
1
prodstagedev
1
1 New Artifact: carts:0.2 dev
2
2 Start Deployment: carts:0.2 (dev, dark)
3
3 Deployment Finished: Success or Failure
4
4 Start Tests: Functional, Dev, carts
5
5 Tests Finished: Test Result & State
6
6 Start Deployment Evaluation: Monspec
7
7 Evaluation Finished: Deployment Score
8
"indicators": [ {
"id" : “Prometheus:go_memstats_alloc",
"metricScore": 20
},{
"id": “Dynatrace:Avg_ActionDuration",
"metricScore": 20
} ],
"objective" : { "pass" : 90, “warning” : 75 }
6
8 Send new Artifact Event: carts:0.2 stage

Confidential 19
keptn: Phase 4 – Promoting an artifact to staging
$ keptn send event newartifact carts:0.2 stage
prodstagedev
1
prodstagedev
1
1 New Artifact: carts:0.2 stage
2
2 Start Deployment: carts:0.2 (stage, shadow)
3
3 Deployment Finished: Success or Failure
4
4 Start Tests: Performance, Stage, carts
5
5 Tests Finished: Test Result & State
6
6 Start Deployment Evaluation: Monspec
7
7 Evaluation Finished: Deployment Score
8
6
8 Send new Artifact Event: carts:0.2 prod
"indicators": [ {
"id": “Dynatrace:RT_Real",
"metricScore": 20
},{
"id": “Dynatrace:RT_Shadow",
"metricScore": 20
} ],
"objective" : { "pass" : 95 }

Confidential 20
keptn: Phase 4 – Promoting an artifact to production: Blue/Green no issues
$ keptn send event newartifact carts:0.2 production
prodstagedev
1
prodstagedev
1
1 New Artifact: carts:0.2 prod
2
2 Start Deployment: carts:0.2 (prod, bluegreen)
3
3 Deployment Finished: Success or Failure
4
4 Start Deployment Evaluation: Monspec
5
5 Evaluation Finished: Deployment Score4
6
6 Blue/Green Switch
"indicators": [ {
"id": “Dynatrace:ConversionRate",
"metricScore": 100
}],
"objective" : { "pass" : 95 }

Confidential 21
keptn: Phase 4 – Promoting an artifact to production: Blue/Green with problem
$ keptn send event newartifact carts:0.2 production
prodstagedev
1
prodstagedev
1
1 New Artifact: carts:0.2
2
2 Start Deployment: carts:0.2 (prod, bluegreen)
3
3 Deployment Finished: Success or Failure
4
4 Start Deployment Evaluation: Monspec
5
5 Evaluation Result: Dynatrace Problem Notify
4
6
6 Remediation: Back to Blue
6 Remediation: Trigger ServiceNow Workflow
6
66
6 Remediation: Update Configuration: carts:0.1

22Confidential
Demo – Onboarding a Service

Confidential 23

Confidential 24
Use case in a nutshell
• Create project sockshop
• Onboard carts service and carts database
• Build a new artefact for the carts service and watch keptn doing the deployment
stages:
- name: "dev"
deployment_strategy: "direct"
test_strategy: "functional"
- name: "staging"
deployment_strategy: "blue_green_service"
test_strategy: "performance"
- name: "production"
deployment_strategy: "blue_green_service“

25Confidential
Demo – Runbook Automation &
Self-Healing

Confidential 26

Confidential 27
Use case in a nutshell
• Dynatrace for Monitoring & ServiceNow for Runbook Automation
• Both tools a properly configured meaning that Dynatrace can send problem notifications and ServiceNow
can trigger runbooks
• Load generation to simulate production traffic
• Configuration change at runtime
• Problem detection by Dynatrace
• Incident creation & Workflow execution by ServiceNow

28Confidential
Curious …
- how to use keptn?
- how to contribute to keptn?

Confidential 29
You want to try out keptn?
• Setup keptn: https://keptn.sh/docs/0.2.0/installation/setup-keptn/
• Instructions to install keptn on a GKE cluster
• Instructions to install keptn on a AWS EKS cluster (coming soon)
•
•
•
•
•

Confidential 30
Contribute to the keptn project with your custom service
1. Identify the channel you want to subscribe to:
2. Write your knative service
• A reference service in TypeScript is provided on:
https://github.com/keptn/keptn-service-template
• Use the payload of the CloudEvents to implement the
functionality you want
3. Provide a service description and the channel
subscriptions in a manifest file
4. Deploy your service in the keptn namespace
•
•
•
•
•
•
jenkins-
service
github-
service
custom-
service
slack-
service
spinnake
r-service
neotys-
service

Confidential 31
Want to learn more?
• Learn how to use keptn: https://keptn.sh/docs/
• Clone, fork, star, contribute to keptn: https://github.com/keptn/
• Develop your own keptn-services: https://github.com/keptn/keptn-service-template
• Join our community (slack invite, meetings): https://github.com/keptn/community
• Follow us @keptnproject
• Fill out our survey: https://dynatrace.ai/acsurvey

Building Autonomous Operations for
Kubernetes with keptn

Confidential 33
dynatrace.com

Confidential 34
Autonomous Cloud Control Plane
event-
broker
authenticator
controlcli
keptn-channel
event-
broker-ext
deployment
evaluation
test
problem
Services
jenkins-
service
github-
service
service
now-
service
cli
configuration
new-artefact
keptn: Phase 1 – Creating a Project

Confidential 35
Autonomous Cloud Control Plane
event-
broker
authenticator
controlcli
keptn-channel
event-
broker-ext
deployment
evaluation
test
problem
Services
jenkins-
service
github-
service
service
now-
service
cli
configuration
new-artefact
keptn: Phase 2 – Onboarding a Service

Confidential 36
Autonomous Cloud Control Plane
event-
broker
authenticator
controlcli
keptn-channel
event-
broker-ext
configuration
deployment
evaluation
test
problem
new-artefact
Services
jenkins-
service
github-
service
service
now-
service
keptn: Phase 3 – Pushing an artifact and deploy to dev