1.
Delivery Pipelines as a
First Class Citizen
Cora Iberkleid
@ciberkleid | ciberkleid@pivotal.io
Advisory Platform Architect, Pivotal
#deliverAgile2019 | @AgileAlliance

2.
Why It Matters

3.
Production, production, production
awesome
product in
your dev env
ok product in
user’s hands
ok > awesome ??

4.
Is ok really ok?
Yes, if delivery is...
Fast
Safe
Secure
So you can iterate quickly & often,
because that’s how you get from ok
to awesome IN PRODUCTION

5.
New Measures of Success
● Release frequently
● Release independently
● Roll back independently

6.
Stand on the Shoulders of Giants
● Build on a layers of modern,
structured automation
● Employ modern coding & testing
practices
● Don’t reinvent the wheel!

7.
The Software Defined Delivery Manifesto
● Delivery is not a detail
● Time to apply our core skills to our own work

8.
Continuous Delivery Foundation (CDF)

9.
CI vs CD

10.
CI vs……………………………. CD
CI Pipeline
release branch
CI Pipeline
master branch
Artifact
Repo
CD Pipeline
master branch
test
stage
prod
Source
Code
Repo

11.
An Example of CI vs CD Tooling
Stateless: Rather than sharing state, every task
runs in its own container, controlling its own
dependencies.
Ephemeral: Clean and consistent builds.
Minimizes test infrastructure to maintain and gives
you flexibility to scale with the cloud.
Continuous integration: Track versions of
external artifacts used for CI (e.g., Git, S3, docker
image). Build components expressed as code with
simple modeling of modular components.
Flexible workflow automation: Extensible for
compliance or other customized tasks in a
pipeline.
Stateful: Maintains awareness of the topography
(multiple clouds, PCF Foundations, etc.) and
keeps record of all pipeline activities for
compliance and audit.
Application inventory: Maintains inventory of
deployed applications, supporting activities like
rollback, canary analysis and identifying security
vulnerabilities.
Sophisticated deployment scenarios: Canary
deploys, red/black or blue/green deploys,
progressive deployments (e.g., by time zone),
multi-cloud/PCF Foundation deployments.

12.
CI Best Practices
● Simple
● Consistent
● Early
● Follow cloud native / 12-factor principles
○ Boundaries to facilitate independent release cycles
○ Repository should contain everything necessary for build
○ Config management
○ Separate build, release, run stages
■ Containerization is plumbing - move it to release
● API-Driven Development
○ Use APIs to design / define component interactions
○ Limit the exposed surface area of your application
○ A good API decouples the “what” from the “how”
● Check for back compatibility
● Use circuit breakers

13.
CI Best Practices
● Simple
● Consistent
● Early
● Leverage modern frameworks
● Convention over configuration
● Establish basic standards for testing organization
○ Naming conventions for test class packages
○ Naming conventions for maven/gradle profiles

14.
CI Best Practices
● Simple
● Consistent
● Early
● Shift as much testing left into CI as possible
● Use contract-based stubs for API testing
● Check for breaking API changes
● Check for breaking database schema changes

15.
CD Best Practices
● Simple
● Consistent
● Easy
● Leverage the highest level of abstraction possible
for the workload runtime
○ Apps with sufficient cloud maturity can be containerized
with an off-the-shelf process (e.g. Buildpacks)
○ Containerizing is plumbing, not value-add
● Cloud Foundry is a great example of a high
abstraction level runtime target
○ OSS, multi-cloud
○ Includes Buildpacks for containerization
○ Full stack automation between IaaS and app
○ Simple API-based service provisioning, too!

16.
CD Best Practices
● Simple
● Consistent
● Easy
● Leverage purpose-built, cloud-oriented CD tooling
○ Avoid using a CI tool for CD
■ CI for getting started, team-centric CI/CD
■ Expensive (CI runs jobs individually)
■ Lack org-centric views and capabilities
● Spinnaker is a great example of a modern,
purpose-built CD tool for the cloud(s)
○ OSS, multi-cloud
○ Multi-platform integrations
○ Application Inventory
○ Efficient resource consumption

17.
CD Best Practices
● Simple
● Consistent
● Easy
● With good CI practices and the right choice of CD
tooling, sophisticated techniques can be easy
○ Blue/green, canary, and progressive deployment strategies
○ Rollbacks, canary analysis
○ Compliance, auditing, and security vulnerability
identification

18.
Spinnaker Integrations
Cloud Providers
App Engine
Amazon Web Services
Azure
Cloud Foundry
DC/OS
Google Compute Engine
Kubernetes
Openstack
Oracle
CI Systems
Jenkins
Travis CI
Wercker
Concourse (coming soon)
Artifact Support
Docker
Google Cloud Storage
GitHub
HTTP
S3
Monitoring
Datadog
Prometheus
Stackdriver
Atlas
SignalFx
(10+ more on
roadmap)Notifications
Email
HipChat
Slack
SMS via Twilio

19.
Application-centric Control Plane

20.
CI for APIs

21.
Step 1: Agree on API Contract
Consumer Producer
API v1

22.
Step 1: Codify the Contract
Consumer Producer
API v1
pull request
Contract.make {
description("""
should return a fortune string
""")
request {
method GET()
url "/"
}
response {
status 200
body "foo fortune"
}
}
Contract.make {
description("""
should return a fortune string
""")
request {
method GET()
url "/"
}
response {
status 200
body "foo fortune"
}
}

23.
Tests Can* Be Auto-generated
* e.g. Spring Cloud Contract

24.
Why is That Beneficial?
● Producer can easily ensure it complies with the contract
● Efficient
● Repeatable
● Reliable

25.
But Wait… There’s More
● Producer can easily check for API back-compatibility
○ Use an older contract to generate tests – run them against your new code
○ If the tests pass, you won’t break older clients
○ WIN: producer can release to production independently of consumer release schedule
● Even better: do this for N older contracts
○ Guarantee API back-compatibility for a range of previous contract versions
● Sounds cool, how easy is it?

26.
A Stub Jar is also Auto-Generated. Just publish it.
Producer uses the contract in the stub to
re-generate tests and runs them against
the new code base…
.... for N* number of older contracts
* where N is a small, manageable number
What’s inside?
contract (groovy)
stub (json)

27.
It’s Also Good for the Consumer
Consumer can use the stub to mock out
the producer…
.... for N* number of older stubs
* where N is a small, manageable number
What’s inside?
contract (groovy)
stub (json)

28.
Benefits to the Consumer
● The mock is more reliable
● The mock is available early
● Consumer can easily ensure it complies with the contract
● Consumer can validate cross-compatibility using older stubs
● Consumer will still work if producer needs to roll back
● WIN: consumer can release to production independently of producer release schedule

29.
The Heart of the API Testing (local versions)
######## API CHECK: PRODUCER
./mvnw clean verify -Papicompatibility -Dproduction.version=blue -Dcontracts.mode=LOCAL
######## API CHECK: CONSUMER
./mvnw clean test -Dstubrunner.ids=com.example:producer-app:blue -Dstubrunner.stubs-mode=LOCAL

30.
CI for DB Schemas

31.
Database Schema Versioning & Migration
Application
Database
v1
v1
● Frameworks such as Flyway and
Liquibase handle version schema
tracking and migrations
● Write tests to validate code against
the data store

32.
Database Schema Versioning & Migration
Application
Database
v1
v2
● Test new code against new db schema
● Check out older app code and test that
against the new db schema
.... for N* number of older app versions

33.
No More Database Rollback Nightmares
● Ensure the app can roll back in production without having to roll back the
database
● Enable blue/green deployments to production as app versions share a
database
● Can do this in CI using an in-memory database
● Stash away the new migration scripts
● Check out the old code
● Test the old code using the new db migration files
● WIN: breaking database schema changes can be caught earlier in the process

34.
The Heart of the DB Schema Testing
######## DB CHECK
cd ../app-blue;
./mvnw clean test -Dspring.flyway.locations=filesystem:../app-green/src/main/resources/db/migration

35.
CD Deployment Strategies

36.
Advanced Deployment Strategies
● Blue/green
● Automated canary analysis
● Rollbacks
Verifying back compatibility enables you to take
advantage of these advanced deployment strategies

37.
Metrics-driven Canary Analysis for Prod Deployment
NetFlix Tech Blog: https://medium.com/netflix-techblog/automated-canary-analysis-at-netflix-with-kayenta-3260bc7acc69
CF Summit Talk & Demo: https://www.youtube.com/watch?v=9C8m7n_sG38&feature=youtu.be
Ensures your applications meet
your SLAs

38.
Yet Another Layer of Abstraction

39.
Code Your Pipeline - Emergent Options
● Spin CLI
● Spinnaker Managed Pipeline Templates
● Canal (in development)
○ Kotlin and Java DSL for generating Spinnaker pipelines
○ Can manage both pipelines and templates
○ Potential for integration with other CI/CD tools in the future
○ Ability to validate pipelines structure programmatically
● Atomist
● Cloud Pipelines (as a reference for CI in particular)

40.
In Summary…
● Keep your eye on the prize
○ Production is the place to be
○ Frequent releases
○ Independent release and rollback
● Keep it simple
○ Choose tools and frameworks that will help you
○ Avoid unnecessary complexity
● CI and CD are substantially different
○ Shift left into CI - cheaper, earlier, and under developer control
○ A purpose-built CD tool has a lot built in and is more efficient

41.
Speaking of
Giants
A Word of Gratitude
For their guidance, help, and support
(in descending order of consecutive
consonants in surname):
Marcin Grzejszczak
Jon Schneider
Olga Kundzich
Clayton McCoy
Richard Seroter

42.
THANK YOU!
Demo guide:
https://github.com/ciberkleid/cna-demo-setup
Slides: https://www.slideshare.net/ciberkleid/delivery-pipelines-as-a-first-class-citizen-deliveragile2019