Transforming the Monolith at 20M tph
•Transferir como PPTX, PDF•
1 gostou•853 visualizações
SpringOne Platform 2016 Speaker: Nick Beenham; Senior Principal Engineer, Comcast Within the Enterprise Services Team at Comcast, we have a platform that handles x million transactions a day. Over the course of a year we transformed from large monolithic deployments with cycle times of anything from 90 days to a series of lean and agile teams with cycle times measured in hours. We achieved this through cultural and technological transformations. Our culture changed from one of individual siloed development to cross-functional teams embracing Agile methods, our technology changed from heavy application server platforms on bare metal to PaaS using Cloud Foundry, from disjointed and manual builds to continuous delivery. This is our story.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (20)
Semelhante a Transforming the Monolith at 20M tph
Semelhante a Transforming the Monolith at 20M tph (20)
Mais de VMware Tanzu
Mais de VMware Tanzu (20)
Último
Último (20)
Transforming the Monolith at 20M tph
- 1. Transforming the Monolith at 20M tph August 3, 2016
- 2. Who am I? 2
- 3. What is the Enterprise Service Platform? 3 250,000,000 transactions per day Hosted on 500+ Servers Oracle, JavaEE, and Weblogic
- 4. The problem that faced us 4 Large legacy codebase Manual and people based processes Physical Infrastructure Comingled data No impact to the velocity to other teams
- 6. How we worked then 6 Dev QA Ops
- 7. Reorganized teams to be self contained 7 Team Dev QA Ops
- 8. Handling friction Presentation title (optional)8 I don’t have to… I don’t want to… You can’t make me.
- 9. We avoided running into… 9
- 10. And we started to … Standardize for Scale 10
- 11. Decomposing services into Microservices 11 Service A Service B Clients ESPRouter
- 12. De-Mingling of data 12 Service A Service B Data Access
- 13. Changes in the technology stack 13 Platform Weblogic Cloudfoundry Data Oracle Couchbase Routing IP based Consumer
- 14. Changes in the delivery stack 14 Continuous Delivery Jenkins GoCD Build Maven Gradle Deployments Script CD
- 15. CF Extensions 15 ZDD CLI Plugin: Zero Downtime deployments Open source pending Zero Downtime – Gradual rollover of applications Canary Deployments - Deployment and then promotion of new versions. https://github.com/comcast
- 16. CF Extensions Recycle Plugin: Recycles application instances 16 Scaler Service: Allows the sending of messages to scale applications Open source pending https://github.com/comcast
- 17. On the horizon 17 Reducing the impact of external processes Exploring the use of Concourse Onboarding more teams and services to CF Piloting Chat Ops with Slack
- 18. Building the runway “Always be wary of any helpful item that weighs less than its operating manual.” ― Terry Pratchett, Jingo 18
- 20. Questions? 20
- 22. Concourse Components Rally Toolkit: Recycles application instances https://github.com/Comcast/rally-rest-toolkit 22 Rally Build Resource: Resource for pushing build info to Rally https://github.com/Comcast/rally-build-resource
- 23. Current Opportunities Presentation title (optional)23 Change Management • Manual process. • Creates a bottleneck in the delivery process. • Requires external approvals and input. Continuous Deployment • Continuous deployment to production causes worry. • Maturity level is not consistent across organization. • Integration of stateful and data systems
Notas do Editor
- Who am I Intentionally left title off slide, what is my title? This talk is about that title Look for ways to innovate and change the way we work. Mostly self-directed
- business critical transactions oracle all the way down
- Legacy codebase is over 8 years old in places Most processes involve people and tickets – conway’s law Hosted on physical servers meaning no elastic scale Data is written to by multiple sources Not allowed to negatively impact the delivery schedules of other teams, so we had to look like soa, talk like soa, and act like soa, but be microservices
- Culturally we followed conway’s law and our architecture followed the same pattern processes were a representation of our org structure culture had to drive technical change
- When we started the year the team was fragmented between dev, qa, and ops builds were passed between teams and tasks executed on them, long feedback loops. Unit testing was limited, most testing was further right in the pipeline - help from pivotal Each stage was independent and passed responsibility down a chain.
- Shared responsibility Instant communication Fast feedback between team members Co-location within the same pod
- Part of the cultural change was dealing with the resistance, usually fell into the 3 categories
- we've all seen and experienced this no longer happens due to shared goals and success as a team not as an individual or role
- Move from culture to technology Started with less prescriptive methods and allowed teams to innovate, allowed the better methods to rise and become de facto standards Creating reusable methods and technologies Created centralized systems for authentication and authorization Created libraries to encapsulate standardized patterns Created logging framework Created esp-connector Output was ESP-Boot Common gradle tasks were extracted and placed in plugins Yeoman generator to create base service
- Clients expected to call existing service Implemented method level router in Go Based on soap method call request is routed to correct end-point
- De-mingle that data by extracting specific schemas and where access is required by more than one service place an access service in the way Moved from oracle to couchbase
- GoCD pipeline templates - Why - Issues with jenkins, detecting change, git plugin, ssh keys, user management. Pipelines were rudimentary - Standardized templates within GoCD that are applied to all pipelines. Gradle scripts and plugins Extraction of common tasks Sharing of plugins Deployments Fully automated complex deployments – canary –test – scaleover Deployment manifests separate from the codebase allow for deployment into any environment Cloud config allows for environment based configurations
- ZDD Plugin – zero downtime deployments via scaleover and canary
- Recycle Plugin: Recycles application instances Use Case: There are situations such as property changes where a restart is required, restarting all at once can cause outages ZDD Plugin – zero downtime deployments via scaleover and canary
- Going forward the purpose is to identify and prepare solutions to appearing problems Do not over-engineer solutions, allow for flexibility and extensibility. Example is concourse replacing gocd Rigid templates Multiple instances of GoCD Versioning of tools and software packages Over reliance of data in GoCD Continue to iterate and refine processes and bottlenecks - Evaluate new technology and methods. - Organize and implement a pipeline for technology adoption. - Identify and develop technology to reduce friction.
- Who am I - Snr Principal Engineer at Comcast working within the application and platform services group Been at comcast for 10 years I’m a builder Look for ways to innovate and change the way we work. Mostly self-directed