Companion slide from Velocity NY 2-day training. Labs: https://github.com/mesosphere/training/blob/master/velocity-training-09-2016/

1. © 2016 Mesosphere, Inc. All Rights Reserved. 1
CI/CD WITH DOCKER,
DC/OS, AND JENKINS
( Velocity NY 2016 Edition )

2. © 2016 Mesosphere, Inc. All Rights Reserved. 2
KARL WHO?
Where?
- Mesosphere
- Pivotal
What?
- DC/OS
- Kubernetes (on Mesos)
- CloudFoundry
- BOSH
- github.com/karlkfi
- twitter.com/karlkfi
- linkedin.com/in/karlkfi
- karl.isenberg.us

3. © 2016 Mesosphere, Inc. All Rights Reserved. 3
WHO ARE YOU?

4. © 2016 Mesosphere, Inc. All Rights Reserved. 4
INTEGRATION
Build/Test
Manual Review/Merge
Build/Test
Test all the pieces
together
DELIVERY
Integration
Test Release/Deployment
Manual Acceptance
Prod Release/Deployment
Integration branch is always
deployable
DEPLOYMENT
Integration
Validation
Release/Deployment
Integration branch is
validated and deployed
automatically
CONTINUOUS EVERYTHING

5. © 2016 Mesosphere, Inc. All Rights Reserved. 5
SOURCE
Branched > Developed > Compiled > Tested > Reviewed > Merged > Tagged
PACKAGES
Built > Tested > Versioned > Released > Promoted > Installed/Vendored
SERVICES
Deployed > Tested > Accepted/Validated > Promoted
RELEASE vs. DEPLOY

6. © 2016 Mesosphere, Inc. All Rights Reserved. 6
PAIN
- Expensive manual merges
- Independent refactoring
- Incompatible dependencies
- Works on my machine
- Different environments
- Who owns integration?
- Mismatched Interfaces
- Large upfront planning &
communication
WHY CONTINUOUS INTEGRATION?
RELIEF
- Small change sets
- Feature branches
- Package extraction
- Interface extraction
- Provisionable environment
- Centralized testing
- Automatic build/test triggers
- Configuration as code
- Integration package

7. © 2016 Mesosphere, Inc. All Rights Reserved. 7
PAIN
- Manual, undocumented release
process
- Siloed, tribal knowledge
- Bus factor of one
- Poor process communication
- Infrequent end-to-end testing
- Bugs caught right before/after
release/deployment
- Slow onboarding
- Manual regression testing
WHY CONTINUOUS DELIVERY?
RELIEF
- Programmatic dependency management
- Automated package management
- Automated release building
- Automated deployment
- Automated promotion
- Secrets management
- Unit/integration/acceptance tests
- Test/UAT/Acceptance environment

8. © 2016 Mesosphere, Inc. All Rights Reserved. 8
PAIN
- Unhappy customers
- Infrequent/late validation
- Many/large simultaneous
changes
- Making decisions without data
- Unhappy developers
- Process limiting innovation
- Infrequent feedback
- Long development cycles
- Large upfront planning
WHY CONTINUOUS DEPLOYMENT?
RELIEF
- Automated front-end tests
- Automated end-to-end tests
- Programmatic definition of acceptance
- Acceptance driven development
- Milestone releases, not release
milestones
- Release train
- Decouple feature promises from release
numbers
- Automatic semantic versioning

9. © 2016 Mesosphere, Inc. All Rights Reserved. 9
TOOLS

10. © 2016 Mesosphere, Inc. All Rights Reserved. 10
DOCKER
( Container Runtime, Image Format, and Repository )

11. © 2016 Mesosphere, Inc. All Rights Reserved. 11
DOCKER CONTAINER
isolated process tree
isolated filesystem
optional volumes
optional network routing
optional resource constraints

12. © 2016 Mesosphere, Inc. All Rights Reserved. 12
RUNTIME
The docker-engine runs on linux machines and manages containers and container
images.
IMAGE
A container image is a set of file system layers that can be shipped around as a unit
and is used as a template for creating containers.
REPOSITORY
A docker image repository is a service that manages and store docker images and
image layers.
CONTAINER ...

13. © 2016 Mesosphere, Inc. All Rights Reserved. 13

14. © 2016 Mesosphere, Inc. All Rights Reserved. 14
BUILD ONCE, RUN ANYWHERE
- Portable runtime environment
- Language independent
- Dependency vendoring
- Packaging format
- Dependency isolation
- More developer control
- Portable across operating systems
- Fast development cycles
HAPPY DEVELOPERS & OPERATORS
CONFIGURE ONCE, RUN ANYTHING
- Predictability
- Repeatable deployment
- Consistent, reliable results
- Low overhead, low cost
- Fast, efficient deployment
- Image Layer caching
- Developer awareness of operational
concerns
- Separation of concerns

15. © 2016 Mesosphere, Inc. All Rights Reserved. 15
CONTAINER IMAGE LIFECYCLE
Image
Image
Image
Layer
Machine 1 Machine 2Image Repository
Shared Layer
Layer
Layer
Push
Image
Pull
Dockerfile
Build
Container
Run

16. © 2016 Mesosphere, Inc. All Rights Reserved. 16
DOCKER LAB
1 - 2

17. © 2016 Mesosphere, Inc. All Rights Reserved. 17
DC/OS
( Container Orchestration )

18. © 2016 Mesosphere, Inc. All Rights Reserved. 18
IN THE BEGINNING...
Sombrero Galaxy by ESA/Hubble - CC BY 4.0

19. © 2016 Mesosphere, Inc. All Rights Reserved.
Hardware
Operating System
Application
19
COMPUTERS

20. © 2016 Mesosphere, Inc. All Rights Reserved.
Hardware
Operating System
Web Application
20
INTERNET
- Remote Users!

21. © 2016 Mesosphere, Inc. All Rights Reserved.
Web App
Hardware
Operating
System
21
DISTRIBUTION
- Horizontal Scale
- Fault Tolerance
- Availability
- Load Balancing
Operating
System
Operating
System
Hardware Hardware
Web App Web App

22. © 2016 Mesosphere, Inc. All Rights Reserved.
Service Service Service
Web App Web App Web App
Hardware
Operating
System
22
SERVICE-
ORIENTED
ARCHITECTURE
- Separation of
concerns
- Optimization of
bottlenecks
- Smaller teams
- API Contracts
- Data replication
- Complicated
provisioning
- Dependency
management
Operating
System
Operating
System
Hardware Hardware

23. © 2016 Mesosphere, Inc. All Rights Reserved.
Service Service Service
Web App Web App Web App
Machine
Operating
System
23
HARDWARE
VIRTUALIZATION
- Fast provisioning
- Isolation
- Portability
- Utilization
- Configuration
Management
- Virtual Networking
- Credential
management
Operating
System
Operating
System
Infrastructure
Machine Machine

24. © 2016 Mesosphere, Inc. All Rights Reserved.
Operating
System
Operating
System
Operating
System
ServiceApp ServiceServiceAppApp
24
MICROSERVICES
- Polyglot
- Single Responsibility
- Smaller Teams
- Utilization
- Machine
types/groups
- Dependency hell
Machine
Infrastructure
Machine Machine
ServiceService ServiceServiceServiceService

25. © 2016 Mesosphere, Inc. All Rights Reserved.
ServiceApp ServiceServiceAppApp
OS
25
CONTAINERS
- Rapid deployment
- Dependency
vendoring
- Container image
repositories
- Spreadsheet
scheduling
OS OS
Machine
Infrastructure
Machine Machine
Container Runtime Container Runtime Container Runtime
ServiceService ServiceServiceServiceService

26. © 2016 Mesosphere, Inc. All Rights Reserved.
Service
Container Runtime Container Runtime Container Runtime
OS
26
CONTAINER
ORCHESTRATION
OS OS
Container Orchestration
Machine
Infrastructure
Machine Machine
App ServiceServiceAppApp

27. © 2016 Mesosphere, Inc. All Rights Reserved. 27
CONTAINER ORCA-WHAT-NOW?
Scheduling, Coordination, & Management
of
Containerized Processes
as
Higher Order Abstractions
(e.g., Jobs, Services, Applications, Pods, etc.)

28. © 2016 Mesosphere, Inc. All Rights Reserved.
Orchestration
28
Machine Infrastructure
Web Apps & Services
Scheduling
Resource Management
Container Runtime
Machine & OS
Service Management
CONTAINER
ORCHESTRATION
Machine & OS Machine & OS
Container Runtime Container Runtime
Expanded

29. © 2016 Mesosphere, Inc. All Rights Reserved. 29
SCHEDULING
- Placement
- Replication/Scaling
- Resurrection
- Rescheduling
- Rolling Deployment
- Upgrades
- Downgrades
- Collocation
RESOURCE
MANAGEMENT
- Memory
- CPU
- GPU
- Volumes
- Ports
- IPs
- Images/Artifacts
SERVICE
MANAGEMENT
- Labels
- Groups/Namespaces
- Dependencies
- Load Balancing
- Readiness Checking
FUNCTIONAL ORCHESTRATION CAPABILITIES

30. © 2016 Mesosphere, Inc. All Rights Reserved. 30
DC/OS LAB
1 - 2

31. © 2016 Mesosphere, Inc. All Rights Reserved. 31
DC/OS
( The Datacenter Operating System )

32. © 2016 Mesosphere, Inc. All Rights Reserved. 32
SCALABILITY
Performance, Responsiveness, Efficiency
AVAILABILITY
Fault Tolerance, Robustness, Reliability, Resilience,
Disaster Recovery
FLEXIBILITY
Format Support, Interoperability, Extensibility,
Container Runtimes
USABILITY
Familiarity, Maintainability, Compatibility,
Debuggability
PORTABILITY
Host OS, Cloud, Bare-Metal, Hybrid
SECURITY
Auditability, Secrets Management, Encryption,
Isolation
NON-FUNCTIONAL PLATFORM QUALITIES

33. © 2016 Mesosphere, Inc. All Rights Reserved. 33
PLATFORM
A base of technologies on which other technologies or processes are built.
PLATFORM AS A SERVICE
A managed, multitenant platform.
CLOUD PLATFORM
A hosted platform as a service on shared infrastructure.
PLATFORM SPECTRUM
Rainbow by Susanne Nilsson - CC BY-SA 2.0

34. © 2016 Mesosphere, Inc. All Rights Reserved.
INFRASTRUCTURE PLATFORM
e.g., OpenStack, VMware vSphere
CONTAINER PLATFORM
e.g., Kubernetes, DC/OS, Rancher
APPLICATION PLATFORM
e.g., Cloud Foundry, Red Hat OpenShift, Deis
34
PLATFORMS
Oil Platform by Berardo62 - CC BY-SA 2.0

35. © 2016 Mesosphere, Inc. All Rights Reserved.
CLOUD INFRASTRUCTURE PLATFORM
e.g., Amazon Elastic Compute Cloud, Microsoft Azure,
Google Compute Engine
CLOUD CONTAINER PLATFORM
e.g., Google Container Engine, Azure Container Services,
Amazon EC2 Container Service
CLOUD APPLICATION PLATFORM
e.g., Heroku, Google App Engine, Pivotal Web Services,
IBM Bluemix
35
CLOUD PLATFORMS

36. © 2016 Mesosphere, Inc. All Rights Reserved.
APPLICATION DEVELOPMENT LIFECYCLE
Source Repo, Continuous Integration, Artifact Repo
CONTAINER ORCHESTRATION
Scheduling, Resource Management, Service
Management
DAY 2 OPERATIONS
Debugging, Maintenance
36
CONTAINER OPERATIONS

37. © 2016 Mesosphere, Inc. All Rights Reserved. 37
APPLICATION DEVELOPMENT LIFECYCLE
CONTAINER OPERATIONS
Code
Build, Test,
Package
Publish Deploy

38. © 2016 Mesosphere, Inc. All Rights Reserved.
Applications & Services
38
APPLICATION DEVELOPMENT LIFECYCLE
CONTAINER OPERATIONS
Source Control
Continuous
Integration
Artifact
Repository
Orchestration

39. © 2016 Mesosphere, Inc. All Rights Reserved. 39
APPLICATION DEVELOPMENT LIFECYCLE
CONTAINER OPERATIONS
Applications & Services
Source Control
Continuous
Integration
Artifact
Repository
Orchestration

40. © 2016 Mesosphere, Inc. All Rights Reserved. 40
DEBUGGING
- Logging & Metrics
- Emitting
- Aggregation
- Search
- Reporting
- Alerting
- Dashboards
- Shell Access
MAINTENANCE
- Package Management
- Upgrading
- Cluster Resizing
- Application Autoscaling
- Capacity Planning
- Software Defined Networking
- Backups
- Disaster Recovery
DAY 2 OPERATIONS
CONTAINER OPERATIONS

41. © 2016 Mesosphere, Inc. All Rights Reserved.
System Space
Orchestration
User Space
Source Control
Continuous
Integration
Artifact
Repository
41
CONTAINER OPERATIONS
Load Balancer
Package
Manager
Security
Networking
Logging &
Metrics
Storage

42. © 2016 Mesosphere, Inc. All Rights Reserved.
Kernel
42
OPERATING
SYSTEM
System Space
Hardware
User Space

43. © 2016 Mesosphere, Inc. All Rights Reserved. 43
DISTRIBUTED
OPERATING
SYSTEM
Hardware Hardware Hardware
User Space
Circa 1992
Microkernel
System Space
Microkernel Microkernel

44. © 2016 Mesosphere, Inc. All Rights Reserved. 44
Machine Infrastructure
User Services
Orchestration
Container Runtime
System Services
DISTRIBUTED
OPERATING
SYSTEM
Container Runtime Container Runtime
Revised
OS OS OS
Machine Machine Machine
DistributedOS

45. © 2016 Mesosphere, Inc. All Rights Reserved. 45
DC/OS
- Resource Management
- Universal Container Runtime
- Container Orchestration
- Pluggable Schedulers
- Jobs & Services
- Distributed System Services
- Virtual Networking
- Package Management
- Services Catalog & Application Ecosystem
- Day 2 Operational Focus
ENTERPRISE DC/OS
- E2E Encryption
- Identity & Access Management
- Secrets Management
- Support

46. © 2015 Mesosphere, Inc. All Rights Reserved. 46
HYPERSCALE OPERATIONS
ULTIMATE INFRASTRUCTURE FLEXIBILITY FOR MODERN APPLICATIONS
• Many “snowflakes”
• Management nightmare
• Lengthy cycles to deploy code
• Low utilization
Mesosphere DCOS
MESOSPHERE DCOS APPROACH
Container
Apps
(All) Big Data Analytics
(All)
PaaS
(All)
Stateful Service
(All)
• High performance and resource isolation
• Easy scalability and multi-tenancy
• Fault tolerant and highly available
• Highly efficient with highest utilization
Proven in
mission-critical
deployments
Deploys on-prem or in cloud
TRADITIONAL APPROACH
PaaS
1
Container
App
1
Big Data
Analytics 1
Big Data
Analytics 2
PaaS
2
Container
App
2
Stateful
Service
1
Stateful
Service
2
Complete workload
portability

47. USE, REPRODUCTION, OR DISCLOSURE IS SUBJECT TO RESTRICTIONS SET FORTH IN CONTRACT NUMBER 2014-14031000011 AND SUB CONTRACT NO. MESOSPHERE 01.
VPC
DC/OS ARCHITECTURE
DC/OS CLI
DC/OS UI
DC/OS Universe
Auto-ScalingGroup&SecurityGroup
ELB
ELB
LDAP
www
AGENT (1..n)
47
Backend Services
Spartan
Minuteman
Mesos DNS
Mesos Agent
3DT
Admin Router
Docker
User Tasks
Cassandra
Spark
Kafka
HDFS
Containers...
Apps/Tasks
Jenkins
Auto-ScalingGroup&SecurityGroup
MASTER (1..9)
Admin Router
Backend Services
Spartan
Minuteman
Mesos DNS
Resolv-Conf
Exhibitor
Zookeeper
DC/OS Signal
CA
Vault
Marathon
Mesos Master
History
3DT
Networking API
Cosmos
Bouncer
Frontend Services
Metronome

48. © 2016 Mesosphere, Inc. All Rights Reserved. 48
DC/OS LAB
3 - 5

49. © 2016 Mesosphere, Inc. All Rights Reserved. 49
JENKINS
( Open Source Automation Server )

50. © 2016 Mesosphere, Inc. All Rights Reserved. 50
JENKINS LAB
1 - 5

51. © 2016 Mesosphere, Inc. All Rights Reserved. 51
EXIT SURVEY