Journey from on prem to the cloud with kubernetes

1. April 20, 2021
Bob DeRosa
Senior Devops Architect
Journey From On Prem to
the Cloud with Kubernetes

2. 1
© 2021
|
Agenda
• About Broadridge
• The Project
• Why we made the switch
• Preparation
• Process
• What we should have done differently
• Results
• What’s Next
• References
• Questions

3. 2
© 2021
|
About Broadridge
• Global Fintech company with over $4.5 billion in revenues
• Handle millions of trades a day involving trillions of dollars
• Support communications that reach 75% of North American households
• Manage shareholder voting in 120 countries.
• Hosted nearly 2000 virtual shareholder
meetings in 2020
• 10,000+ total employees, thousands of
technical associates
• www.broadridge.com
Broadridge is the leading provider of investor communications

4. 3
© 2021
|
The Project
• Migrate our DevOps tools from on-prem into AWS
• Large CloudBees CI (Jenkins) installation with tens of thousands of jobs that run
regularly and thousands of deployments a day.
• Monolithic design with around 10 masters and 1000 agents (VMs).
• One team’s run-away job could
affect other teams.
• Adding resources took weeks.
• A lot of time spent on
maintenance and patching
• Mostly VMWare some AWS
some Solaris located all around
the world

5. 4
© 2021
|
Why We Made the Switch
Why Kubernetes?
• CloudBees CI optimized for K8s
• Build agents work very well as pods. (Flexibility in
scaling)
• Ability to have more customized images for
development teams.
• Easier to patch and rollback.
• Reduced infrastructure
Why the cloud?
• Corporate direction to move applications
to the cloud
• Better Scalability
• Better Reliability
• Infrastructure as a Service

6. 5
© 2021
|
Preparation
• If you don’t need to use Kubernetes, then don’t
• Not the right environment for a lift and shift
• Apply past lessons learned to the new design
• Fast, cheap and good, pick two
• Choose good!
Take advantage of move to re-architect
FAST CHEAP
GOOD
EXPENSIVE SLOW
NOT
POSSIBLE
BAD

7. 6
© 2021
|
Preparation
Rockets are hard; so is Kubernetes!
• Make sure you have the expertise to do this
• Hire folks with K8s experience
• Set expectations that this is going to take
time
• Build a Minimal Viable Product (MVP)
• Start small, work with a few trusted
customers to get it right and iterate

8. 7
© 2021
|
Process
Document, review, and test architecture
• Do a quick POC to test assumptions
• Use native services whenever possible
• Plan for Disaster Recovery
• Shared responsibility model
(code and data are our responsibility)
• Cloud is not magic
• Ensure proper security, monitoring
and backups
• Aqua for security
• Datadog for monitoring
• Kasten K10 for backup
Photo credit:
unsplash

9. 8
© 2021
|
Process
• Storing code in Version Control
• Part of CI/CD pipeline
• Auditable
• Repeatable
• Push problems to the left
Infrastructure/Configuration as Code
The Noun Project
icons

10. 9
© 2021
|
Process
• Expertise of vendors
• Capture best practices as code
• Terraform modules for EKS
• Helm Charts for tools and ingress
• Take advantage of 3rd
party Helm Charts
Don’t reinvent the wheel, just improve it
Photo credit: langevo.blogspot.com/

11. 10
© 2021
|
What We Should Have Done Differently
Hire talent faster Less features
in MVP
Fail faster sooner

12. 11
© 2021
|
Results
•Reusable EKS Terraform Modules
•Reusable Helm Charts
•Wrap 3rd
party Charts
•Developed Automation to synchronize 3rd
party Helm Charts
•Leveraged this work to quickly deploy CloudBees CD

13. 12
© 2021
|
What’s Next
• Migrating internal customers to new platform
• Scaling
• 3rd
party images
• Automating testing
• Other groups adopting code
• People who gained skills with Infrastructure as
code now helping on other projects.
• Catalyst to rethink how our DevOps pipelines
work.
Photo credit:
unsplash

14. 13
© 2021
|
References
Helm
Terraform
EKS
Cloudbees
Kasten
Aqua Security
Datadog

15. 14
© 2021
|
Questions?