With growing demand for containers in the enterprise, build pipelines are a bottleneck to success. Traditional workflows can't release application candidates quickly enough to fulfill demand. With over 400 development teams across many different business units, Pearson had to move away from massive installs of traditional build pipeline tools and rethink the entire concept. In this talk we'll demonstrate how we have built in security compliance, performance testing, quality assurance, abstracted away complexity, reduced overhead, aim to recover 10% of developers time and turned build tools into cattle. This represents the story to date of an in-flight engineering project to modernise the digital estate of a global enterprise organisation and how scale of the operation is leading us to challenge many established beliefs. Attendees will walk away with everything from workflows to code which they can use to get started in their own endeavors. Sched Link:

1. 1
ITNW (IF THIS, NOW WHAT)
Orchestrating an Enterprise

2. 2
We cannot solve our problems
with the same thinking we used
when we created them.
Albert Einstein
Education is what remains
after one has forgotten
everything he learned
in school.
Albert Einstein

3. 3
Business Justification
Requirements
Build Pipelines
ThirdPartyResources
Ingress Controller SSL Integration
ChatOps

4. 4
Principal Systems Architect
Technical Lead on Project Bitesize
Pearson's Enterprise Platform-as-a-Service based on
Kubernetes
Michael Ward
www.devoperandi.com
@devoperandi

5. 5
Always Learning

6. 6
Always Learning
Pearson
Denver
Malaysia
Sydney
London
Singapore
Beijing
40,000 employees
171 years in business
70 countries

7. 7
Pearson is no small potatoes:
36,000 servers
2,000
applications
400+ Dev
Teams
Up to 30 Micro-
services per
application
1:1 build
server to Dev
team

8. 8
OpenSource @pearson
KnowledgeSharing of Information
Education
Sharing of Code
OpenSource
Ingress ControllerPython Client Kubernetes Pack

9. 9
ChatOps Integration

10. 10
DEMO?

11. 11
Business Justification

12. 12
Business
Developers
Security
Quality &
Performance
Project Justification
Security
Standardization,
Compliance and
Visibility
Business
Cost
Developers
Management, Time,
Ease of use
QA & Perf
Ease of Use, Agility,
Minimal disruption

13. Business
Developers
Security
Quality &
Performance
Business
Cost
Build pipeline takes around 3 months for two Devs.
Costing on average $50,000 just in development time PER
DEV TEAM. (QA, Security, Performance not included)
400 Dev Teams x $50,000 = $20 Million
Project Justification

14. Business
Developers
Security
Quality &
Performance
Business
Recurring Cost
10-20% of a Devs time on maintenance, changes, upkeep,
troubleshooting
At 10%
400 X $10,000 = 2 Million/yr
Project Justification

15. 15
Just because I can
"Quotes from Developers"

16. 16
"Well we spent 6 months solid building a good
starting point"
Pearson Developer

17. 17
"we never upgrade Jenkins once stable cause
we can't get time for it"
Pearson Developer

18. Stakeholder Requirements
Reduce Migration Costs Compliance
"pre-baked security"
Standardization
"same build process across whole Pearson estate"
Visibility
Performance Testing
"automated perf testing"
Ease of Use
"if you don't get this I can't help you"
< 5% Time required to Manage Quality Testing
18

19. Our Requirements
Geographically Distributed
"Abstract Deployment but allow it anywhere"
Cattle
"Jenkins as Cattle"
Fast
"< 1hr"
Automated Upgrades
Self Configuring Scalability
"Horizontal"
Feedback Loops
"Clear communication across teams"
19

20. 20
High Level Product Design

21. 21
Build Pipelines

22. Key Concept:
22
1 dev team to many builds
Dev Team
Namespace Build Servers
console-dev
console-stage
console-dev
console-prod
console-stage
console-prod
console-prod
console-stage
console-prod
Oregon
Ireland
Private DC

23. 23
Standard ProcessStandard Process
Base_Builds
App_Builds

24. 24
Standard Process
Base_Builds
App_Builds
Standard Process

25. 25
Standard Process
Base_Builds
App_Builds
Standard Process

26. 26
Standard Process
Base_Builds
App_Builds
Standard Process
base image
point to build component
runtime command
All based on project name

27. 27
Standard Process
Base_Builds
App_Builds
Standard Process
component: code segment
another piece of code for build
dependencies required for build
repo and branch location
build commands for component

28. 28
Standard Process
Base_Builds
App_Builds
Standard Process
for CI/CD
Kubernetes Upgrade methon
Configures Service and Ingress
Tests to run in Build Process

29. Requirements
Geographically Distributed Cattle
Fast Automated Upgrades
Self Configuring Scalability
Feedback Loops Reduce Migration Costs
Standardization Compliance
Performance Testing Visibility
< 5% Time Management required Quality Testing
Ease of Use
29

30. 30
Base Builds
Standard Process
Base_Builds
App_Builds

31. 31
Standard Process
Base_Builds
App_Builds

32. 32
Standard Process
Base_Builds
App_Builds
Base Build
Base level of Security
Standard across the Enterprise
Ensures Compliance

33. 33
App Builds
Standard Process
Base_Builds
App_Builds

34. 34
Standard Process
Base_Builds
App_Builds

35. Requirements
Geographically Distributed Cattle
Fast Automated Upgrades
Self Configuring
Scalability
Feedback Loops
Reduce Migration Costs
Standardization Compliance
Performance Testing Visibility
< 5% Time Management required Quality Testing
Ease of Use 35

36. 36
What else?

37. 37
Manage External Resources with Kubernetes

38. 38
DEMO?

39. 39
HTTPS and automated integration
with Vault for SSL Certs

40. 40
StackStorm Integration - Michael Ward
https://github.com/StackStorm/
st2contrib/tree/master/packs/
kubernetes
Vault SSL Integration - Martin Devlin
https://github.com/devlinmr/contrib
Jenkins Build Pipelines - Simas Cepaitis
OpenSource Coming Soon
ChatOps - Simas Cepaitis & Michael Ward
OpenSource Coming Soon

41. 41
ETCD restore
to alternate clusters

42. 42

43. 43
Meet the Team
Director PM Engineer
Chris Jackson Mark Stocker Simas Cepaitis
Engineer
Bill JorgensenJohn Shirley Martin Devlin
Engineer Engineer
Peter Idah
Engineer

44. 44
Questions

45. 45
Standard Process
Base_Builds
App_Builds
Standard Process

46. 46
PaaS as an ecosystem
Build
API Management
Event Driven
Automation
Runtime
Monitoring
Secrets
Orchestration
App Config

47. Key Concept:
47
1 dev team to many namespaces
Dev Team
appeng-dev
appeng-stage
appeng-prod
appeng-stage
appeng-prod
appeng-prod
Namespace
Kubernetes
ClusterProject
console-core
console-ui
console-app