With everything going on in DevOps, I think we can safely say that building pipelines is the way to deploy your applications to production. But knowing what you deploy to production and whether it is actually okay needs more data, like security checks, performance checks, and budget checks. We’ve come up with a process for that, which we call Continuous Verification “A process of querying external systems and using information from the response to make decisions to improve the development and deployment process.” In this session, we’ll look at extending an existing CI/CD pipeline with checks for security, performance, and cost to make a decision on whether we want to deploy our app or not.

1. Confidential │ ©2020 VMware, Inc.
Data Driven
Decisions in DevOps
Add More To Your Pipelines
Leon Stigter
Product Manager @VMware
July 2nd, 2020

2. 2Confidential │ ©2020 VMware, Inc.
Technology Is
Changing Rapidly
@retgits | @MyDevSecOps

3. 3Confidential │ ©2020 VMware, Inc.
Changing Technology
Means Changing
Infrastructure
@retgits | @MyDevSecOps

4. 4Confidential │ ©2020 VMware, Inc.
Changing
Infrastructure
Changing Means
Rethink How You
Deploy…
@retgits | @MyDevSecOps

5. Confidential │ ©2020 VMware, Inc. 5
2020 stack
Gone are the “simple” days of centralized stacks
Application Architectures Are Changing
2000’s stack
App
Database
Webserver
Environment
App
Environment
Cache
Database
Logs
@retgits | @MyDevSecOps

6. 6Confidential │ ©2020 VMware, Inc.
“Amazon.com does
136,000 fully
automated
deployments per
day”
Ken Exner, Director AWS Dev Resources
Source: https://techbeacon.com/devops/going-big-devops-how-scale-continuous-delivery-success
@retgits | @MyDevSecOps

7. Confidential │ ©2020 VMware, Inc. 7
Applications Are The New Unit of Measure For Ops
AWS does 136,000 fully automated
deployments every day, but…
Ø Less than 50% of organizations has CI/CD
pipelines
Ø Less than 6% has fully automated CI/CD
pipelines
Deployments
@retgits | @MyDevSecOps

8. 8Confidential │ ©2020 VMware, Inc.
“I really want to spend
my day sitting in front
of screens watching
DevOps pipelines…”
- No one ever…
@retgits | @MyDevSecOps

9. Confidential │ ©2020 VMware, Inc. 9
Who am I?
Ø Product Manager, turned
Developer Advocate,
turned Product Manager
again…
Ø Passionate about
Serverless, Containers,
and all things Cloud
Ø I love dadjokes,
cheesecake, and Go
@retgits
Leon Stigter, Product Manager App Toolkit
go.retgits.com/mydevsecops
@retgits | @MyDevSecOps

10. 10Confidential │ ©2020 VMware, Inc.
“Y’all only do VMs at
VMware, right?”
- Pretty much everyone, ever
@retgits | @MyDevSecOps

11. Confidential │ ©2020 VMware, Inc. 11
We deeply care about
building apps, for
example with the
Kubernetes ecosystem…
@retgits | @MyDevSecOps

12. 12Confidential │ ©2020 VMware, Inc.
But I’m Not Here To
Sell You Things…
@retgits | @MyDevSecOps

13. Confidential │ ©2020 VMware, Inc. 13
Let’s get back to DevOps
@retgits | @MyDevSecOps

14. 14Confidential │ ©2020 VMware, Inc.
“software
deployment” should
be high quality and
repeat-able as it
“leaves the factory”
@retgits | @MyDevSecOps

15. 15Confidential │ ©2020 VMware, Inc.
“Efficiency comes
through the (re)use
of common
components”
@retgits | @MyDevSecOps

16. Confidential │ ©2020 VMware, Inc. 16
CI/CD Pipelines are the vehicle for deployment
@retgits | @MyDevSecOps

17. Confidential │ ©2020 VMware, Inc. 17
Most operations checks are done post deployment
Re-deploy
@retgits | @MyDevSecOps
Cost
Security
Utilization
Performance

18. Confidential │ ©2020 VMware, Inc. 18
Continuous Verification
“A process of querying external
system(s) and using information from
the response to make decision(s) to
improve the development and
deployment process.”
@retgits | @MyDevSecOps

19. Confidential │ ©2020 VMware, Inc. 19
Increase Efficiency And Reduce Operational Overhead
Traditional Pipeline
Extending Pipeline(s)
External
Actor
External
Actor
Continuous
Verification
@retgits | @MyDevSecOps

20. 20Confidential │ ©2020 VMware, Inc.
“Automating your entire
pipeline, including cost
and governance, gives you
more time to spend on the
beach”
#BeachOps
@retgits | @MyDevSecOps

21. Confidential │ ©2020 VMware, Inc. 21
“A process of querying
external system(s) and using
information from the
response to make
decision(s) to improve the
development and
deployment process.”
ØContinuous Security (environment
validations, deployment security, etc)
ØCost and Resource Management in
CI/CD
ØImage Verification
ØAuthorization (normalized IAM/RBAC
between AWS, Azure, GCP, and on
prem)
ØApplication performance and SLA
Continuous Verification
What is it?
@retgits | @MyDevSecOps

22. Confidential │ ©2020 VMware, Inc. 22
Continuous Verification Scenario
Do we have capacity &
budget in the
project/account/etc? Do we follow best
practices and
security guidelines
for accounts and
permissions?
Look at performance
stats in test/dev
environment
Are my dependencies
secure?
@retgits | @MyDevSecOps

23. Confidential │ ©2020 VMware, Inc. 23
Answering questions, like
A)Will this deployment exceed the budget now?
B) Is there enough capacity for resources in EC2, RDS, etc. below the utilization budget?
C) What are the optimal resources (i.e. rightsized EC2 instance, RI, etc) to use for this deployment?
D) Is the deployment running within budget with the right resources and are there recommendations?
Because money is important…
Cost
@retgits | @MyDevSecOps

24. Confidential │ ©2020 VMware, Inc. 24
Clair Vulnerability scans
EC2 security group - port 22 is
publicly accessible
S3 Bucket or Blob Storage is
open to the world
Database instances (RDS / S3)
are not encrypted
User has active keys but has
not rotated them
AWS VPC Flow logs / Azure
Network Security Group Flow
Logs - “What’s happening in
my cloud network?”
AWS Cloud Trail / Azure
Activity Log - “Who, when and
what CRUD operations were
performed on which services
in the account?”
AWS Guard Duty / Azure
Advanced Threat Protection -
“That looks sketchy !!”
Avoids data exposure to
anonymous users/applications
Prevents privileged escalation
and unauthorized access to
resources
Ensures compliance with
regulations like PCI, HIPPA
Security
Safety matters to everyone
Pre Deployment Post Deployment Why In A Pipeline?

25. Confidential │ ©2020 VMware, Inc. 25
Open Source Security
Source: State of Open Source Security Report 2020, Snyk

26. Confidential │ ©2020 VMware, Inc. 26
Answers questions, like:
Does my K8s cluster have enough capacity to deploy this new app
Where are the bottlenecks in my architecture
Are the spikes in CPU usage normal
How can I visualize which path traffic takes through my architecture
Is there latency issues on my app using high volume traffic generation
Can the database scale and support the needed capacity
The User experience matters too!
Performance Checks

27. Confidential │ ©2020 VMware, Inc. 27
Let’s Go See That in
Action…
@retgits | @MyDevSecOps

28. Confidential │ ©2020 VMware, Inc. 28
One Small Disclaimer…
I’m about to use tools that I’m comfortable
with, you should use tools that you choose…
@retgits | @MyDevSecOps

29. Confidential │ ©2020 VMware, Inc. 29
Recap
Do we have capacity &
budget in the
project/account/etc?
Do we follow best
practices and
security guidelines
for accounts and
permissions?
Look at performance
stats in test/dev
environmentAre my dependencies
secure?
@retgits | @MyDevSecOps

30. Confidential │ ©2020 VMware, Inc.
Thank You
@retgits
The awesome icons in this deck are made by Freepik and Flat Icons from www.flaticon.com
retgits.com
go.retgits.com/ mydevsecops