O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Surviving the Script-apocalypse

122 visualizações

Publicada em

Did you know the average age of the top four banks is 175 years old? And those four banks are the end result of merging or acquiring 35 separate companies over the last 40 years? This means lots of different cultures, tech stacks, apps, processes, and controls – and SCRIPTS!!

Join Chris Nowak, Principal at FlowStates, and Avantika Mathur, ElectricFlow Product Manager, as they share hard-won lessons and emerging patterns in DevOps and release orchestration that can help any regulated organizations climb out of scripting hell. You’ll hear:

1) Why it’s important to prioritize delivery improvement across all your teams

2) How the theory of constraints is limiting your success

3) And how simple things like version control for artifacts can help improve success rates!

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Surviving the Script-apocalypse

  1. 1. © Electric Cloud | electric-cloud.com | @electriccloud Surviving the “Script-Apocalypse”Surviving the “Script Apocalypse” Why it’s important to prioritize delivery improvement across all your teams
  2. 2. © Electric Cloud | electric-cloud.com | @electriccloud Speakers Chris Nowak Founder FlowStates @DevOpsFlow Avantika Mathur Product Manager Electric Cloud @avantika_ec / @electriccloud - #ecwebinar
  3. 3. © Electric Cloud | electric-cloud.com | @electriccloud The “Script-ocalypse” • How many people are devoted to maintaining your scripts? • What happens if the script owner leaves? • Are you afraid to make changes to your automation? • Do you have visibility into what is was executed and current status? • How long does it take to run a deployment? • How do you prepare for an audit?
  4. 4. © Electric Cloud | electric-cloud.com | @electriccloud The Script Reality • Spaghetti scripts • Piece-meal automation connected by more scripts • Maintenance cost/time is unknown • Not a scalable solution • No auditability and visibility • No reusability • Knowledge retention and sharing
  5. 5. © Electric Cloud | electric-cloud.com | @electriccloud • NetEnt • 2-4 hour maintenance window for upgrades • Scale by adding people • ~80% of time spend on non-value add • Business Payment Solutions • Could not keep up with business growth • 5000 deployments take 1 month – 4-10 people / 8hrs a day • Limited SSH connections • Limited visibility of disparate deployment scripts Example
  6. 6. © Electric Cloud | electric-cloud.com | @electriccloud
  7. 7. © Electric Cloud | electric-cloud.com | @electriccloud • How many people are devoted to maintaining the scripts? • What happens if the script owner leaves? • Are you afraid to make changes to your automation? • Do you have visibility into what is was executed and current status? • How do you prepare for an audit?
  8. 8. © Electric Cloud | electric-cloud.com | @electriccloud Where You Want to Be • Increased Productivity • Time is spent developing value and innovating • Improved process robustness and scalability • Reduced Risk • System level visibility across the processes
  9. 9. © Electric Cloud | electric-cloud.com | @electriccloud
  10. 10. © Electric Cloud | electric-cloud.com | @electriccloud Don’t Just Automate – Orchestrate the Entire Solution • Having scripts does not mean you are fully automated • Every action needs to be automated and driven by your pipeline • Automate approvals and handoffs • Are you fully automated? “High performers automate significantly more of their configuration management, testing, deployments and change approval processes than other teams” – 2017 State of Devops Report Iterative process – doesn’t happen in a day DEV/CI QA Deploy Release Operate
  11. 11. © Electric Cloud | electric-cloud.com | @electriccloud Orchestrating Your Pipeline Automated or Manual Approval Gates • Architect your pipeline with the right team • An Automated pipeline is critical to success • Treat your pipeline like a product
  12. 12. © Electric Cloud | electric-cloud.com | @electriccloud Best Practices
  13. 13. © Electric Cloud | electric-cloud.com | @electriccloud Approaching the Problem Climbing out of the Script Hole 1. Create a Value Stream Map 2. Organize Your Scripts 3. Score and Prioritize Scripts 4. Re-Factor Iterative agile process
  14. 14. © Electric Cloud | electric-cloud.com | @electriccloud DevTeam Pre-Picture: Build / Deploy Value Stream Analysis – prod DML File Red boxes / lines are manual ticket and email processes DevOpsPlatform Services(DPS) Release Mgmt(RM) Build Engineerin g(BE) (Work initiates via no standard method: email, tickets, etc.) DML changes made to .prod file Create / Update Name / Value pair Check-in to SCM Email chains for dev approvals Create Build Eng Ticket1 (BET). BE opens DPS Jira ticket2 to inform of migration need Close BET Request BE fails BET ticket (sysgen mail) then does warm hand-off to Dev 1. Apply SCM label to new file version 2. Extract & Stage file Validation Pass? Run Validation (Jenkins) BET hits work queueYes No Begin Deploy Process DPS Approvals are gathered Alert Dev & RM Goes to RM. RM confirms build #’s w/ Dev and puts in Release Plan (Excel). Then notifies DPS. RM approves BET request to start DML Build / Validate / Deploy Process Release for testing. Testing OK? END Yes No Prep Files: 1. Get .prod from staging 2. Get .base from local Filesystem 3. Concatenate files (.base + .prod) Validate (custom prog) Deploy Pass? Deploys DML (cust. Prog generates DML and loads) Corresponding Ear needs to do env progression in deploy / release tool Validation Pass? DPS confirms validation Fail Program flags deploy as FAIL DPS Triage (others as needed) Yes No No Yes START
  15. 15. © Electric Cloud | electric-cloud.com | @electriccloud DevTeam Process Collapse: Build / Deploy Value Stream Analysis – prod DML File Red boxes / lines are manual ticket and email processes DevOpsPlatform Services(DPS) Release Mgmt(RM) Build Engineerin g(BE) (Work initiates via no standard method: email, tickets, etc.) DML changes made to .prod file Create / Update Name / Value pair Check-in to SCM Email chains for dev approvals Create Build Eng Ticket1 (BET). BE opens DPS Jira ticket2 to inform of migration need Close BET Request BE fails BET ticket (sysgen mail) then does warm hand-off to Dev 1. Apply SCM label to new file version 2. Extract & Stage file Validation Pass? Run Validation (Jenkins) BET hits work queueYes No Begin Deploy Process DPS Approvals are gathered Alert Dev & RM Goes to RM. RM confirms build #’s w/ Dev and puts in Release Plan (Excel). Then notifies DPS. RM approves BET request to start DML Build / Validate / Deploy Process Release for testing. Testing OK? END Yes No Prep Files: 1. Get .prod from staging 2. Get .base from local Filesystem 3. Concatenate files (.base + .prod) Validate (custom prog) Deploy Pass? Deploys DML (cust. Prog generates DML and loads) Corresponding Ear needs to do env progression in deploy / release tool Validation Pass? DPS confirms validation Fail Program flags deploy as FAIL DPS Triage (others as needed) Yes No No Yes START Validation Pass? Run Validation (Jenkins) AutomatedEliminated
  16. 16. © Electric Cloud | electric-cloud.com | @electriccloud DevTeam Current State: Build / Deploy Value Stream Analysis – prod DML File Red boxes / lines are manual ticket and email processes DevOpsPlatform Services(DPS) Release Mgmt (RM) Build Engineeri ng(BE) (Work initiates via DPS mail or Standard Request System DML changes made to .prod file Create / Update Name / Value pair Check-in to SCM ALERT Release for testing. Testing OK? END Yes No Validation Fail: 1. Workflow Fails 2. Automail to Dev & RM w/error output START AutomatedEliminated Triggers build Singular Build Workflow: 1. Automatic Build on check-in 2. Run Validation (CustProg steps) Validation Pass: 1. Apply SCM Labels 2. Upload to Artifact Repo 3. Success mail to Dev, RM, DPS ALERT Pass: 1. Generate DML 2. Load to DML Target 3. Success mail to Dev, RM, DPS 4. Update Release Orchestrator Fail: 1. Workflow Fails 2. Automail to Dev & RM w/ error output Singular Deploy Workflow: 1. Use .base & .prod files already in artifact repo 2. Run Validation if needed DPS Triage (others as needed) RM Decides when to execute Deploy Begin Deploy Process
  17. 17. © Electric Cloud | electric-cloud.com | @electriccloud Results 75% Faster In All Environments Event Legacy Process DevOps Team Handoffs 5-11 1 Email / MOC / Coordination 2-5 0-1 (other than automated) Service Requests filled (tickets) 2-4 0 Process Steps 28 Manual or isolated auto. 5 Manual, 12 Automated Tools / SW used 5 Different Vendors 1 Vendor, Suite of 4 Tools Primary Support Team Count 4 (BE, DPS, WAS, BL) 2 (DPS, WAS)
  18. 18. © Electric Cloud | electric-cloud.com | @electriccloud Create a Value Stream Map • Bring everyone involved in one room • Draw out entire process • What is the actual process for each step (copy/paste script, manual steps) • What tools are you using for each step? • How are you handling approvals? • Include detailed sequencing and timelines à Identify Bottlenecks
  19. 19. © Electric Cloud | electric-cloud.com | @electriccloud Organize Your Scripts • Take stock of all scripts across your Value Stream Map • Bucket scripts into categories • Configuration • Deployment • integrations with external tools • Approvals • Test automation • vm provisioning • Monitoring • Continuous Integration • Etc…
  20. 20. © Electric Cloud | electric-cloud.com | @electriccloud Score and Prioritize • For each bucket, score each script • Script Complexity (Legacy scripts, no expertise) • Repeated tasks (VM provisioning, test automation) • Where and how many times it is used (how much time is spent on this task?) • Is there an easier way? • Identify Redundancy, dependencies and bottlenecks • For those items that take a long time, but too complex to optimize individually, consider adding parallelization where possible to improve efficiency
  21. 21. © Electric Cloud | electric-cloud.com | @electriccloud Re-Factor • Object-Oriented • Think about objects/building blocks that are reusable • Configuration Management • VM provisioning • Access control • Notifications/approvals • Test Automation – process for test is the same, • Monitoring • Design for reusability • Abstract common functions that can be called from anywhere • Parameterize • Maintain one definition - update once and get benefit everywhere
  22. 22. © Electric Cloud | electric-cloud.com | @electriccloud Start Orchestrating • Start with just wrapping existing scripts and orchestrating • In time start to chip away and re-factor based on priorities and scores • Leverage new technologies
  23. 23. © Electric Cloud | electric-cloud.com | @electriccloud Guiding Principles Separate application definition and processes (configuration/Deployment) from the environment where it is being deployed for reusability and consistency • Repeatability and consistency between deployments • Portability between environments
  24. 24. © Electric Cloud | electric-cloud.com | @electriccloud Look for Solutions that Solve Problems For You • Don’t script and maintain what you don’t have to • Configuration management • Automated rollbacks • Rolling Deployments • Security gates and approvals • Integrations with your toolchain • Artifact management
  25. 25. © Electric Cloud | electric-cloud.com | @electriccloud Treat Your Pipeline Like a Product • Establish a central repository for maintenance and common access • Integrate CI and testing for your pipeline • Think about collaboration across the team with multiple developers working on different parts of the pipeline • Provide centralized visibility for users and stakeholders to see the pipeline definition as well as status and metrics at any given point in time • As much automation as possible • Don’t forget to automate the handoffs between stages with automated approvals/gates and even notifications for manual approval
  26. 26. © Electric Cloud | electric-cloud.com | @electriccloud Considerations
  27. 27. © Electric Cloud | electric-cloud.com | @electriccloud Future Proof • Don’t get locked in to certain tools and technologies • Estimate and plan for long term scale • Shared visibility • Reusability • Performance • Measure and validate scalability with metrics and benchmarks
  28. 28. © Electric Cloud | electric-cloud.com | @electriccloud Self-Service • Promote self service • Resource provisioning • Configuration • Test Automation • Release Pipelines • … Vetted Automation • Goal is to allow any user to enter the right parameters to generate and run their own release pipeline • Eliminate drift and ensure compliance and security • Enable teams to use vetted and standardized processes across the organization
  29. 29. © Electric Cloud | electric-cloud.com | @electriccloud Measure and Improve • Consider how to automatically pull and correlate data from all of the tools, environments, processes • Select the right KPIs • Cycle time, lead time, failure rates, MTTR, automated vs manual processes • Determine your baseline and measure your progress • Provide Centralized Visibility • Enables easy auditability • Log for each stage become your audit log
  30. 30. © Electric Cloud | electric-cloud.com | @electriccloud Surviving the Script-ocalypse • It isn’t easy! • Take time to untangle and understand your process • Orchestrate your entire pipeline – and treat it like a product • It’s an iterative process
  31. 31. © Electric Cloud | electric-cloud.com | @electriccloud Surviving the “Script-Apocalypse” Questions?

×