After spending her entire career as a software developer, with nary a moment doing operations, Cornelia Davis found herself working on an application platform that serves operations as much as development. In order to better understand that world, she spent one month on the team that runs that platform in production. The experience brought lessons in organizational design, the value of pair-ops (in addition to pair programming) and test-driven development, the importance of addressing continuous integration as a first class concern, and how separating infrastructure ops from application ops serves the business and their customers better. In this session Cornelia will share the “prod incidents” that brought these teachings; the audience will gain an appreciation not only for what, but why the lessons are so important.

1. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved.
My Great Awakening:
Top “Ah-ha” Moments As Former
Dev Manager
Devops Enterprise Summit
October 2015
1
Cornelia Davis ● @cdavisafc ● cdavis@pivotal.io
CTO, Transformation Practice

2. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 2
When I first started working on PaaS…
“For us, PaaS is a set of services aimed at developers that helps them
develop and test apps without having to worry about the underlying
infrastructure. Developers don't want to have to worry about provisioning the
servers, storage and backup associated with developing and launching an app.”
… but then I learned
[1] http://www.javaworld.com/article/2078801/java-app-dev/forrester--paas-makes-developers-happy.html
[2] http://www.infoworld.com/d/cloud-computing/paas-primer-what-platform-service-and-why-does-it-matter-212696

3. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 3
• One month on the ops team
• Production platform deployment
• 24 x 7 x 365
Lessons!!

4. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 4
My very first morning…

5. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 5
App App
App App
Log
Log

6. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 6
App App
App App
Log
Log
/dev/null
App App
App App

7. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 7
App App
App App
Log
Log
App App
App App
Log
Log

8. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 8
App App Log
Cloud Ops
App Team

9. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 9
App App Log
Cloud Ops App Team
!!OK

10. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 10
• Ops team has a backlog and a
Product Manager
• (just as app team)
• Ops team has a “prod pair”
• App team has a “prod pair”
• A very sensible way of
managing deployments

11. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 11
More About Deployments…

12. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 12
1. Do deployments during working
hours
HOW???
2. Immutable Infrastructure
NO SNOWFLAKES!!!
3. Single deployable artifact
BUILDS DONE BEFORE DEPLOY
Runtime: Dev ! MastServices: Dev ! Mast
Svc: Mast ! CR; RT: Mast ! CR
Dijon Tabasco
A1
Prod
Services: Dev; Runtime: CR Services: CR; Runtime: Dev
Services: Mast; Runtime: Mast
Services: Rel; Runtime: Rel
Svc: CR ! Rel; RT: CR ! Rel
Shared
Package
Cache

13. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 13
1. Do deployments during working
hours
HOW???
2. Immutable Infrastructure
NO SNOWFLAKES!!!
3. Single deployable artifact
BUILDS DONE BEFORE DEPLOY
4. Declarative deployment specification
EVENTUAL CONSISTENCY!
Amend
(Re)deploy
(Re)configure
(Re)connect
Monitor
Current
State Desired
State

14. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 14
More About
Single Deployable Artifact…

15. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 15

16. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 16

17. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 17
Env
Vars
Cust
DB
mock
Env
Vars
Cust
DB
Env
Vars
Cust
DB

18. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 18
Let’s Come Back to
Ops and Dev Responsibilities…

19. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 19
App App
App App
Log
Log

20. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 20
App App
App App

21. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 21
A A
A A

22. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 22
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
A A
...
Started updating job runner_z1 >
runner_z1/91. Done (00:01:06)
Started updating job runner_z1 >
runner_z1/92. Done (00:01:06)
Started updating job runner_z1 >
runner_z1/93. Done (00:01:06)
Started updating job runner_z1 >
runner_z1/94. Done (00:01:06)
Started updating job runner_z1 >
runner_z1/95. Done (00:01:06)
Started updating job runner_z1 >
runner_z1/96. Failed
!!

23. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved.
On the… BOSH Agent port: Directory Server port:
… healthy DEA 35560 34567
… unhealthy DEA 34567 Port binding failed
23
Ephemeral Port Range: 32768 to 61000
Developer should not have used port 34567
Dynamically
Assigned
Statically
Assigned

24. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 24
Your Application
Code
Platform
Virtualized
Infrastructure
Platform Operations
Application Development
Application Operations
Creates deployable artifact
• Config Prod space
• Deploys application to Prod
• Monitors application
• Scales application
(capacity)
• Deploys new app version
with zero downtime.
• Deploys platform
• Makes standard runtimes and
services available
• Monitors platform
• Scales platform
(ensuring sufficient capacity)
• Upgrades platform with zero-
downtime

25. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 25
The Takeaways

26. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 26
• Treat Ops as any other product
• backlog, prod pair
• Workday deployments
• Single deployable artifact
• Separate application operations
from platform operations
• App team = App Dev + App Ops

27. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 27
• Revelations From the Field – Life in the Operations Team
(the one about the loggregator scaling)
http://blog.pivotal.io/cloud-foundry-pivotal/p-o-v/revelations-from-the-
field-life-in-the-operations-team
• Is Continuous Delivery a First Class Concern of Your Platform?
(the one about dashboards)
http://blog.pivotal.io/cloud-foundry-pivotal/products/is-continuous-
delivery-a-first-class-concern-of-your-platform
• Cloud Foundry Ops: Ephemeral Ports and the Value of a Platform
as a Service
(the one about, well, ephemeral ports)
http://blog.pivotal.io/cloud-foundry-pivotal/p-o-v/cloud-foundry-
ops-ephemeral-ports-and-the-value-of-a-platform-as-a-service

28. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved. 28
Here’s what I need help with
https://www.washingtonpost.com/news/innovations/wp/2014/02/14/silicon-valleys-gender-imbalance-in-one-chart/
Changing the distribution will take effort from all of us!

29. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2015 Pivotal. All rights reserved.
Thank you!
Devops Enterprise Summit
October 2015
29
Cornelia Davis ● @cdavisafc ● cdavis@pivotal.io
CTO, Transformation Practice