O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Jose Luis Soria - Codemotion 2014 - Designing a release pipeline

1.475 visualizações

Publicada em

Slides from my presentation about release pipelines at Codemotion (Madrid - November 2014)

Publicada em: Software

Jose Luis Soria - Codemotion 2014 - Designing a release pipeline

  1. 1. MADRID · NOV 21-22 · 2014 Designing a Release Pipeline Jose Luis Soria Continuous Improvement Manager at Ria Financial @jlsoriat
  2. 2. MADRID · NOV 21-22 · 2014 What is a Release Pipeline?
  3. 3. MADRID · NOV 21-22 · 2014 What is a Release Pipeline? Automated manifestation of your delivery process. Feedback mechanism. Detection of unfit release candidates. Pull system. Useful for CD, or any other delivery model.
  4. 4. MADRID · NOV 21-22 · 2014 Pipeline design considerations Emergent design. No BDUF. Start early. Start simple and evolve with the system. Begin with the most valuable assets. Address the bottlenecks.
  5. 5. MADRID · NOV 21-22 · 2014 #1 Define Components
  6. 6. MADRID · NOV 21-22 · 2014
  7. 7. MADRID · NOV 21-22 · 2014 What is a (software) component? (*) A set of artifacts (binaries, dynamic code, configuration files, other supporting files) that can be deployed and verified together without affecting other areas of the application. (*) in a release pipeline
  8. 8. MADRID · NOV 21-22 · 2014 Choosing components Deploy and test the smallest independent entity. Rely on the architecture: Logical / physical. Layers / tenants. See the whole.
  9. 9. MADRID · NOV 21-22 · 2014 What do we need to know? For each component: Meaningful name. Description. Priority / order (when to address it?) Source (most likely, version control.) Target (where it gets deployed.) Pre-requisites. Dependencies. Configuration tokens.
  10. 10. MADRID · NOV 21-22 · 2014 Configuration tokens 1.Make a list of environment-dependent information. 2.Tokenize it in the configuration. 3.Gather the values for all the environments.
  11. 11. MADRID · NOV 21-22 · 2014 Component sheet
  12. 12. MADRID · NOV 21-22 · 2014 Activity: defining components Our application consists on: A web site built on top this technology stack: MVC framework. Client-side logic (HTML5, JavaScript.) Entity model mapped to the DB using a ORM. Data model residing in a DB.
  13. 13. MADRID · NOV 21-22 · 2014 Activity: defining components Our application consists on: A web site built on top this technology stack: MVC framework. Client-side logic (HTML5, JavaScript.) Entity model mapped to the DB using a ORM. Data model residing in a DB. Key business logic resides on a web services layer. We also maintain a mobile client for two platforms.
  14. 14. MADRID · NOV 21-22 · 2014 Activity: defining components Our application consists on: A web site built on top this technology stack: MVC framework. Client-side logic (HTML5, JavaScript.) Entity model mapped to the DB using a ORM. Data model residing in a DB. Key business logic resides on a web services layer. We also maintain a mobile client for two platforms. For some operations we make calls to third-party web services.
  15. 15. MADRID · NOV 21-22 · 2014 Example: component sheet
  16. 16. MADRID · NOV 21-22 · 2014 Implementation notes It should be possible to independently build, deploy and test anything defined as a component. You should decide how dependencies will be made available: Source Control. Artifact repositories. Deployed (third-party) artifacts. Etc.
  17. 17. MADRID · NOV 21-22 · 2014 Tooling Version control systems Git, Mercurial, SVN, TFS… Artifact repositories Artifactory, Nexus, Frog, NuGet…
  18. 18. MADRID · NOV 21-22 · 2014 #2 Identify Sub-pipelines
  19. 19. MADRID · NOV 21-22 · 2014 Single pipeline A single pipeline servicing all the components and teams. May be able to detect which component has changed and operate only on that one.
  20. 20. MADRID · NOV 21-22 · 2014 Onepipeline per component Each component has its own pipeline. Different pipelines may have different designs. Individual pipelines may fan-in to a system pipeline. More flexible but more complex.
  21. 21. MADRID · NOV 21-22 · 2014 Onepipeline per team Each team has its own pipeline. Different pipelines may have different designs. Individual pipelines may fan-in to an integration pipeline.
  22. 22. MADRID · NOV 21-22 · 2014 Mixedapproach Different teams building different components. Keep it simple!
  23. 23. MADRID · NOV 21-22 · 2014 Implementationnotes It is easier if you use a tool that allows to define sub-pipelines, fan-in, fan-out, etc.
  24. 24. MADRID · NOV 21-22 · 2014 #3 Define Stages& Orchestration
  25. 25. MADRID · NOV 21-22 · 2014
  26. 26. MADRID · NOV 21-22 · 2014 Whatisa stage? A set of steps or activities that are performed on a release candidate. It lets any release candidate advance towards production, or discards it. When a release candidate passes through a stage, our confidence on it is increased. It is a source for feedback. Frequently taken for environments (but they’re not the same)
  27. 27. MADRID · NOV 21-22 · 2014 Whatisorchestration? It is the way we arrange the stages so release candidates flow through them, in their way to production.
  28. 28. MADRID · NOV 21-22 · 2014 Tipsforstages& orchestration(I) Feedback is the key. Arrange stages and orchestration based on the feedback we need. Stages are filters. The orchestration should be arranged to stop the pipeline if a stage fails. Stages can contain both manual and automated steps.
  29. 29. MADRID · NOV 21-22 · 2014 Tipsforstages& orchestration(II) Stages can be manually or automatically triggered (think about approvals.) Automate as much as possible. Including the approvals.
  30. 30. MADRID · NOV 21-22 · 2014 Tipsforstages& orchestration(III) Grow your pipeline wide, not long http://bit.ly/1jsNGP5 Build only once. Use environment-agnostic binaries. Version everything.
  31. 31. MADRID · NOV 21-22 · 2014 Whatdo weneedto know? For each stage: ∘Meaningful name. ∘Clear goal. ∘Does it need a manual approval to be triggered? ∘Does it need a manual verification when it has finished? ∘Sources. ∘Flow (orchestration.)
  32. 32. MADRID · NOV 21-22 · 2014 Pipeline-levelorchestration(examples) CommitManual testingReleaseMinimum pipelineFully automatedPartially automated, or manualLegend:
  33. 33. MADRID · NOV 21-22 · 2014 Pipeline-level orchestration (examples) Commit Acceptance testing Release Exploratory testing Capacity testing Security testing User Acceptance testing Complex pipeline Fully automated Partially automated, or manual Legend:
  34. 34. MADRID · NOV 21-22 · 2014 Stage-levelorchestration(example) Code is builtOnce and Only Once: In the first stage. Subsequent stagesare run in parallel if configured that way. Pipeline gets triggered: When a developer does a check-in, or manuallyA new instance of the pipeline is createdGet next stage, relate it to the pipeline instance, prepare parameters, notify for monitoringAutomatically- triggered stage? Trigger stageWait for the user to trigger the stageGather stage results and notify them for monitoringSucceded? Stop the pipeline instanceAny stages left? NOYESNONOYESYES
  35. 35. MADRID · NOV 21-22 · 2014 Aboutsources Version control. Artifactrepositories. Environmentlibraries.
  36. 36. MADRID · NOV 21-22 · 2014 Whichstagesdo I need? Think about the kind of feedback you need. Think about what should stop a release candidate to get to production. Create a Value Stream Map.
  37. 37. MADRID · NOV 21-22 · 2014 Valuestreammap(example) AssessmentApprovalPlanningCapacity testsReleaseAcceptance testsCodeSpecification2 days1 dayValue- added timeWaittimeDevelopment: cycle time ~ ? 3 days3 days? ?? ? 4 days1 day2 days2 weeks?? Delivery: lead time ~ ? Exploratory tests? UAT? ?
  38. 38. MADRID · NOV 21-22 · 2014 Prevalentstages: theCommitstage Eliminate early release candidates that are unfit for production. Close to (or the same as) a CI build. Quick validations: build, unit testing, static analysis, etc. Packaging. For Continuous Delivery, it runs on each commit (no branches –feature toggles.) For other models, decide when it gets triggered (for example, on each merge to trunk.)
  39. 39. MADRID · NOV 21-22 · 2014 Prevalentstages: theCommitstage http://bit.ly/1jsSkwA
  40. 40. MADRID · NOV 21-22 · 2014 Prevalent stages: the Automated Acceptance Test stage http://bit.ly/1jsSkwA
  41. 41. MADRID · NOV 21-22 · 2014 Prevalent stages: the Manual Test stage http://bit.ly/1jsSkwA
  42. 42. MADRID · NOV 21-22 · 2014 Prevalent stages: non-Functional Testing stages http://bit.ly/1jsSkwA
  43. 43. MADRID · NOV 21-22 · 2014 Activity: defining stages & orchestration “We have a basic suite of automated acceptance tests that we plan to grow along with the system.” “The team does (manual) functional testing.”
  44. 44. MADRID · NOV 21-22 · 2014 Activity: defining stages & orchestration “We have a basic suite of automated acceptance tests that we plan to grow along with the system.” “The team does (manual) functional testing.” “We need to support 2,000 concurrent users.”
  45. 45. MADRID · NOV 21-22 · 2014 Implementationnotes Choose a tool that allows to easily model and visualize the flow. Choose a tool that supports what you need for orchestration: ∘Approvals. ∘Validations. ∘Parallelization. ∘Alerts. ∘Etc.
  46. 46. MADRID · NOV 21-22 · 2014 #4 Define Environments
  47. 47. MADRID · NOV 21-22 · 2014 Whatisanenvironment? A set of servers, devices or any other resources we need in order to run and validate a release candidate in its way to production.
  48. 48. MADRID · NOV 21-22 · 2014 Tipsfordefiningenvironments Prepare for deployment automation. Lock down environments. Restrict access. Different stages could target the same environment if needed. Prepare for auto-provision. Make environments disposable. Don’t turn them into bottlenecks. Environments may not be tied to stages. It should be easy to point any stage to any environment.
  49. 49. MADRID · NOV 21-22 · 2014 Activity: defining environments “We have a basic suite of automated acceptance tests that we plan to grow along with the system.” “The team does (manual) functional testing.” “We need to support 2,000 concurrent users.”
  50. 50. MADRID · NOV 21-22 · 2014 Implementationnotes Use virtualization. Use cloud-based environments. Use tools for managing templates, configuration, auto-provision, etc.
  51. 51. MADRID · NOV 21-22 · 2014 Tooling Phisicalmachines Virtualization tools VmWare, VirtualBox, Hyper-V, etc. Containers Docker Cloud Amazon, Google, Azure… Environment definition Vagrant, PowerShell DSC…
  52. 52. MADRID · NOV 21-22 · 2014 #5 DefineSteps
  53. 53. MADRID · NOV 21-22 · 2014 Whatisa step? Any activity that is done in the context of a stage, that allows us to get feedback and prove the fitness of the release candidate. Examples: ∘Deploy a component. ∘Run automated tests. ∘Run manual tests. ∘Update metrics. ∘Alert the user of some event. ∘Etc.
  54. 54. MADRID · NOV 21-22 · 2014 Tipsfordefiningsteps(I) Consider: The goal of the stage. The kind of feedback you need. Sources. Targets (environments.) Build and package only in the Commit stage.
  55. 55. MADRID · NOV 21-22 · 2014 Tipsfordefiningsteps(II) Consider: Most times, deployment is present, but not always. (Automated) Smoke Testing should follow any deployment. Think about both automated and manual steps.
  56. 56. MADRID · NOV 21-22 · 2014 Activity: defining steps “We want to filter out anything producing static analysis warnings.” “We want to try exploratory testing.” “We may use the same environment for load testing and security testing.”
  57. 57. MADRID · NOV 21-22 · 2014 #6 Define Automation & Tooling
  58. 58. MADRID · NOV 21-22 · 2014 Tipsforstepautomation(I) Automate everything. Automate everywhere (for all the environments.) Preference for automation: Fully automated steps. Manually triggered automatic steps. Manual steps.
  59. 59. MADRID · NOV 21-22 · 2014 Tipsforstepautomation(II) Build only once. Version everything. This includes the automations. Have environment lockdown in mind.
  60. 60. MADRID · NOV 21-22 · 2014 Deploymentautomationconsiderations Deploy the same way to every environment. The target environment should be a (implicit) parameter for the automations. Set up (tool-agnostic) one-click deployments. Treat configuration tokens as parameters for the automations. Prepare for rollbacks.
  61. 61. MADRID · NOV 21-22 · 2014 Databasedeploymentconsiderations Databasedeploymentisnotthesameas databasedevelopment. Decide aboutthedeploymentstrategy: Schema& Data compare. Delta scripts (betterforContinuousDelivery.) ORM tools(schemaupdate, migrations, etc.)
  62. 62. MADRID · NOV 21-22 · 2014 Test automationconsiderations Q2 tests are not necessarily run through the UI. Smoke tests may be run through the UI. Frequently, non-functional testing can be automated. Leave environments and data in a known state. A few things can’t be automated (UAT & Q3 testing.)
  63. 63. MADRID · NOV 21-22 · 2014 Whatdo weneedto know For each step to be automated: Automation tool or technology. Execution model. Parameters (at least you’ll have the configuration tokens.) Source / target.
  64. 64. MADRID · NOV 21-22 · 2014 Aboutexecutionmodels Native OS tool. Agent. Remote execution.
  65. 65. MADRID · NOV 21-22 · 2014 Activity: defining automation & tooling “Production environment is in an isolated network” “Operations people won’t allow us to install anything there”
  66. 66. MADRID · NOV 21-22 · 2014 Tooling Environment provision Puppet, Chef, Ansible Virtualization tools (VM templates, etc.) App deployment Scripting (Unix shell, PowerShell, etc.) Puppet, Chef, Ansible, VS Release Management DB Deployment RedGate, DBDeploy, etc. Testing Testing frameworks UI automation frameworks Non-functional testing tools
  67. 67. MADRID · NOV 21-22 · 2014 #7 Define Execution model, monitoring & metrics
  68. 68. MADRID · NOV 21-22 · 2014 Continuousdeliveryflowmodel Pipeline instances are created on each commit. Any commit is a release candidate. One-piece continuous flow model. There is no way back. Any error makes the release candidate to be discarded. Fixes are treated as new release candidates. They are run through the entire pipeline from the beginning.
  69. 69. MADRID · NOV 21-22 · 2014 Otherflowmodels Pipeline instances are created as needed. A release candidate might comprise several commits. Decide on the batch size. Larger batches may be cheaper but limit feedback. Errors might be fixed in the context of the stage where they arise.
  70. 70. MADRID · NOV 21-22 · 2014 Monitoringthepipeline Transparency. Rely on a proper tool. Set up alerts for key events. Use a Cumulative Flow Diagram (CFD.) Gather metrics and act on them.
  71. 71. MADRID · NOV 21-22 · 2014 Primarymetrics Cycle time. Mean Time Between Failures (MTBF.) Mean Time To Recover (MTTR.)
  72. 72. MADRID · NOV 21-22 · 2014 Secondarymetrics Test coverage. Duplication of code. Coupling. Compilation warnings. Code churn. Build frequency Etc.
  73. 73. MADRID · NOV 21-22 · 2014 Tooling Outof thebox metricsin theorchestrationtool. Ad-hoc reporting. Sonar, FxCop, Ndepend, FindBugs…
  74. 74. MADRID · NOV 21-22 · 2014 #8 Plan forfutureenhancements
  75. 75. MADRID · NOV 21-22 · 2014 Examples–DevOps culture Improvebranchingmodelifneeded. Debuggingoptimization, symbol servers. Canaryreleases. Blue/greendeployments. A/B Testing.
  76. 76. MADRID · NOV 21-22 · 2014 Examples–DevOps culture Preventiveprofiling. Telemetry, analytics, ApplicationPerformance Monitoring(APM.) Proactiveresiliencyenablement(Simianarmyhttp://nflx.it/SPeTGj)
  77. 77. MADRID · NOV 21-22 · 2014 Tooling Applicationprofilers APM New Relic App Insights Raygun CompuwareAPM
  78. 78. MADRID · NOV 21-22 · 2014Artifact and metadata repositoriesBinaries repositoryTFS Build Drops folderReporting system (receives data from all stages) TFS Reporting – tracks Cycle Time, MTTR & MTBF Custom TFS Reporting on Work ItemsCommit stage TFS Build Definition, customized from Default templateCodeGet dependencies (package restore) BuildCommit testsContinuous integrationCode analysisBasic functional tests (manual) Version artifactsName the pipeline instance (set the build number) Update the artifact repositoryUpdate the Symbol ServerAcceptance test stage TFS Build Definition, customized from Lab template. MTM for Test Case managementChange configuration (automated) Deployment (automated) MSBuild for WPPowershell for AzureAcceptance tests (automated) MSTest, CodedUI (?) Release stage TFS Build Definition, customized from Lab templateBranch or merge to Release branch (for archiving) Deployment (automated) MSBuild for WPPowershell for AzureSmoke testing (automated) CodedUI (?) Error detection and recovery (resiliency enablement) SCOM-TFS IntegrationIntellitrace in ProductionPreemptive AnalyticsCapacity test stage TFS Build Definition, customized from Lab templateDeployment (automated) MSBuild for WPPowershell for AzureSmoke testing (automated) CodedUI (?) Performance & load testing (automated) Visual Studio perf & load testsEnvironments (automatic provision, locked down so only automated deployments are allowed) Using Lab Manager SCVMM EnvironmentsDevelopment (isolated) Local deploymentto WP and Azure emulatorsTestProductionIteration 6 Pipeline: further improvementsManualtriggerAutomatictriggerVersion Control (code and configuration) TFS Version ControlMain branch (w/ feature toggles) Release branch (for archiving) StagingDeployment & testingtriggered by Test AgentPipeline monitoringCustom solutionUAT stageUAT (manual) Acceptance criteria from TFS requirementsTFS Feedback toolManualtriggerDeployment triggeredby Test AgentUsing deploymentfrom Capacity test stageDeployment & testingtriggered by Test AgentSecurity test stageSecurity tests (manual) WACAAny additional tools? AutomatictriggerUsing deploymentfrom Capacity test stageExploratory test stage TFS Build Definition, customized from Lab templateDeployment (automated) MSBuild for WPPowershell for AzureExploratory testing (manual) MTMManualtriggerExploratoryTestDeployment & testingtriggered by Test AgentArtifact repositoryNuGet ServerSymbol ServerTFS Symbol Server
  79. 79. MADRID · NOV 21-22 · 2014 #9 Inspect & Adapt
  80. 80. MADRID · NOV 21-22 · 2014 Continuouslyimprovethepipeline Component or architectural changes. New skills in the team. New resources, tools, environments. Reserve time, and make the team accountable for improvement.
  81. 81. MADRID · NOV 21-22 · 2014 Summary 1. Define components. 2. Identifysub-pipelines. 3. Define Stages& Orchestration. 4. Define Environments. 5. Define Steps. 6. Define Automation& Tooling. 7. Define executionmodel, monitoringand metrics. 8. Plan forfutureenhancements. 9. Inspect& adapt.
  82. 82. MADRID · NOV 21-22 · 2014 Thanks! Jose Luis Soria ContinuousImprovementManager at Ria Financial jlsoriat@gmail.com-@jlsoriat http://geeks.ms/blogs/jlsoria Slides: http://www.slideshare.net/jlsoria http://aka.ms/releasepipeline

×