O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

DevOps@Morpho for ParisDevOps - 2nd of December 2014

10.553 visualizações

Publicada em

A short timelined description of DevOps at Morpho made for ParisDevOps meetup of 2nd of December 2014

Publicada em: Tecnologia
  • Login to see the comments

DevOps@Morpho for ParisDevOps - 2nd of December 2014

  1. 1. 0 / Morpho DevOps approach time line Jean-Charles JOREL (jean-charles.jorel@morpho.com) May. 2015 DevOps Infrastructure Service Line
  2. 2. 1 / DevOps approach brief  Agenda  A word about Me  Time travel from 2010 to 2014
  3. 3. 2 / About me…  Jean-Charles JOREL (jean-charles.jorel@morpho.com)  DevOps Service Line Manager  Leading a Team of 18 people dedicated to DevOps deployment & associated operations  Safran Morpho Expert  Promote Morpho Technical Excellence outside of the corporation  Areas of Expertise: DevOps…, Cloud Techs, Network protocols & SDN, Innovation process, Linux hacking…  Help to bring new Tech Trends inside Morpho
  4. 4. The DevOps@Morpho time line (Presentation made for the ParisDevops meetup of 2nd of December 2014)
  5. 5. 4 / Mid-2010 : Root of everything…  Our Products were using ‘old school’ technologies mainly based on C++ and Home made frameworks  Our Products were specialized on a small set of OS and HW Morpho DevOps Service Line 2010 2013 2014 2015 Morpho creates the Morpho Core Platform initiative & associated Team MCP Let’s migrate to support of wide choice of OS (Linux’es + Windows) and Open Frameworks with 
  6. 6. 5 / Mid-2012 : Houston, we have a problem…  So many Operating Systems, so many App Servers, so many components, that could be seen as ‘constantly moving parts’  But also, so many subjects where Experts are not in the MCP Team… Morpho DevOps Service Line 2010 2013 2014 2015 Morpho Core Platform has successes but also difficulties… (inability to test the full scope at the right pace, team seen by others as not legitimate on many subjects) Time for a choice: 1. Stop or reduce the scope, 2. Change the way we are working to increase drastically Productivity Let’s Try Option 2. !
  7. 7. 6 / Mid-2012 : First « DevOps » word usage but…  Once you have a « Software forge », teams may want to move further by doing deeper Integration tests  This quickly leads to issues regarding « software forge » IT environment not matching the target deployment platform, overload the forge resources, possible contamination of forge environment.  That’s one of the reason why SW teams always required access to real test benches where they can do integration  “DevOps” is a term describing the seamless link of the 2 needs:  SW forge to build and do unit testing  Relevant integration test environments where to deploy what we just compiled SVN SVN Too Early!!
  8. 8. 7 / Mid to End-2012: Not focusing on DevOps but Shaping a PoC about Test Env Automation Got a High Profile sponsor for a LEAN GreenBelt Made >40 meetings to explain to all involved teams (Dev & Ops + IT Dept) expected benefits  Try to focus on highly visible gains linked to Test Bench management 15 involved people (PoC Builders & early Users)… Morpho DevOps Service Line
  9. 9. 8 / -IaaS manager -VM repository -Block Storage (SAN) Virtual IaaS network #nVirtual IaaS network #1 2012/Proposed PoC setup Server 2012 VM + ActiveDirectory + OpenAM + NTP CentOS VM + PuppetMaster + DNS + DHCP CentOS VM + Supervision COTS+ PostFix CentOS VM + Puppet AgentCentOS VM + Puppet AgentCentOS VM + Puppet Agent REHL6 VM + Oracle RAC RHEL6.3 VM + Oracle RAC Node Dashboard access Physical nodes Server 2012 VM + ActiveDirectory + OpenAM + NTP CentOS VM + PuppetMaster + DNS + DHCP CentOS VM + Supervision COTS+ PostFix CentOS VM + Puppet AgentCentOS VM + Puppet AgentRHEL 6.3 VM + Puppet Agent REHL6 VM + Oracle RACCentOS VM + PostGreSQL HA Nexus + YUM Windows 7 VM + Puppet Agent Deployment configuration
  10. 10. 9 / End-2012 to March 2013 : A successful PoC! Proved that we can automate Test Env Deployment with the OpenStack IaaS!  Perfect IP Address Plan segregation confirmed so allowing our Must-have Datacenter cloning use-case! Let’s apply the concept to Morpho DevOps Service Line 2010 2013 2014 2015 MCP
  11. 11. 10 / DevOps Practices & Platform 2013: “DevOps” word only known as a way to help MCP + = DBA, On-Site installers, Site support, Network engineers, … Ex: Java developper, integrators, Testers, … MCP  We ask Ops to deliver pre-packaged IT to the MCP Team (and so Dev teams)  Oracle Database, Operating Systems, Supervision tools, Security hardening
  12. 12. 11 / March 2013 to Oct-2013: Building the first « DevOps » tool chain A small set of MCP Team working on it BUT with external contributions from other teams, From 10 to 100 users involved (Only volunteers) First benefits:  Sharp increase of Test coverage of MCP  First packaged Oracle deliverables from DBA  Quick-wins of OpenStack usage (snapshots) by users Morpho DevOps Service Line 2010 2013 2014 2015 MCP
  13. 13. 12 / Mid-2013 : First DevOps platform concept Extended scope MCP Programs DevOpsplatform Product & Suites …/…Components (Simplified platform view)
  14. 14. 13 / Nov-2013 to March-2014: Viral usage of « DevOps » tool chain 150 users involved in France, Ireland, Germany and Morocco  All from Dev (almost none from Ops side… not yet…) Creation of the DevOps Service Line on 1st of January!  DevOps word is officially used as a Practice!! …but most people still do not know exactly what it is…  Morpho DevOps Service Line 2010 2013 2014 2015
  15. 15. 14 / Morpho DevOps Infrastructure, Technologies, Process Supervision & SLA OS, Virtualization & Networks Database Management Installation & Lifecycle Suite or Product or MSU Suite or Product or MSU Suite or Product or MSU Suite or Product Agile Application Platform MCP Common Binary Storage Solutions Solutions SolutionsPrograms & Solutions Agile DevOps Our Devs… Our Ops…
  16. 16. 15 / Morpho DevOps Service Line 4 teams for Full Product to Program Lifecycle support Stress Test services & Test Benches operations Software Configuration Management Artifact Lifecycle Management Standardized IT Deliverables MCP Collaborative Software Methods for developers & integrators
  17. 17. 16 / DevOps Service Line teams Software Method definitions SCM & ALM Operations IT pre-packaging for Tools & Product / Solutions Stress Test Mgmt for Products & Solutions HW & OS Technology scouting SCM/ALM Op team ~5 people Software Collaborative Method team ~3 people Test Bench Operations (OpenStack for Dev/Int/Test) MCP-IT team ~5 people Stress Test & Test Bench team ~5 people
  18. 18. 17 / March-2014 to June-2014: Hard days for the DevOps Service Line DevOps platform is near collapse... Still more and more users but unhappy with the SLA  New Oracle RAC use-cases had catastrophic impacts on the Platform  Lots of unexpected Support toward users rediscovering some IT aspects  Lots of OpenStack Grizzly issues due to heavy Orchestration use-cases Emergency investments performed!!  Local storage with SSD for OpenStack Compute Nodes solved most of issues  Strong automation effort of supervision and administration tasks to govern DevOps platform resource usage in near real-time Morpho DevOps Service Line 2010 2012 2013 2014
  19. 19. 18 / July-2014 to now: First big rewards! DevOps platform under heavy usage and integrated by most of Dev teams  Very interesting early DevOps use-cases shown (see next slide)  People thinking Automation and Resilience  Best-Practices for DevOps Platform Resource Governance accepted by users  Under process to make most our Ops also use the Platform…  Ok for new Customer Programs, Difficulties with Ops supporting our legacy (process to move back and forth out/in DevOps platform not yet mature…)  Envisioning future with more and more DevOps agility (OpenStack over OpenStack and OpenStack over other clouds…) Morpho DevOps Service Line 2010 2012 2013 2014
  20. 20. 19 / DEVOPS@MORPHO: FIRST EARLY SUCCESS STORIES  Our products stressing continuously their integration with Oracle RAC  Permanent Performance & Resilience test campaign  One Oracle RAC failover simulated every ten minutes Automated through OpenStack API directly by the Dev team. Not possible cheaply before OpenStack IaaS!  One critical random bug discovered in JBoss 7.2 leading to data loss “1 out of roughly 100,000 requests at an Oracle RAC fail over event every 10 minutes would translate into 1 out of 15 fail over events assuming a throughput of 1 million requests in 24 hours (which was roughly the throughput of the test system). Assuming that the customer has a bad database which has an Oracle RAC fail over event every month, a data loss will statistically occur every 15 months of processing” Bug discovered in Development phase instead of Production!  Fixed before Product official release!  DevOps approach provided the ability to forecast impacts of a rare random issue by compressing time
  21. 21. 20 / DevOps vs non-DevOps: How to identify them based on their Test Resource consumption ?  DevOps oriented projects are only possible if you have a Test & Integration Infrastructure providing a near unlimited amount of very cheap IT resources Amount of Test & Integration resources used End of project End of project DevOps project non-DevOps project Powered by
  22. 22. 21 / Annex CONFIDENTIEL / DATE / DIRECTION
  23. 23. 22 / 0 200 400 600 800 1000 1200 0 5000 10000 15000 20000 25000NumberofVMscreatedpermonth janv.-13 févr.-13 mars-13 avr.-13 mai-13 juin-13 juil.-13 août-13 sept.-13 oct.-13 nov.-13 déc.-13 janv.-14 févr.-14 mars-14 avr.-14 mai-14 juin-14 OpenStack Instances created 28 463 618 318 274 676 658 1095 1045 1762 4689 6436 6140 9890 10359 13562 14536 21849 OpenStack Instances deleted 27 413 572 287 234 619 634 1049 979 1715 4641 6442 6005 9814 10252 13409 14453 21678 OpenStack Active instances 4 14 49 76 124 183 224 294 238 401 474 349 626 694 783 617 966 1100 VM resources KPI for the IAAS (Infrastructure-as-a-service) One VM created every 2 minutes!
  24. 24. 23 / 75,0% 80,0% 85,0% 90,0% 95,0% 100,0% 105,0% 0 200 400 600 800 1000 1200 NumberofVMS Ratios janv.-13 févr.-13 mars- 13 avr.-13 mai-13 juin-13 juil.-13 août-13 sept.-13 oct.-13 nov.-13 déc.-13 janv.-14 févr.-14 mars- 14 avr.-14 mai-14 juin-14 OpenStack Active instances 4 14 49 76 124 183 224 294 238 401 474 349 626 694 783 617 966 1100 Turnover ratio (global) 85,7% 97,1% 95,6% 94,7% 92,7% 92,3% 92,6% 92,9% 95,4% 94,2% 95,9% 98,1% 97,4% 98,0% 98,2% 98,9% 98,7% 98,8% Turnover ratio (month) 96,4% 89,2% 92,6% 90,3% 85,4% 91,6% 96,4% 95,8% 93,7% 97,3% 99,0% 100,1% 97,8% 99,2% 99,0% 98,9% 99,4% 99,2% VM Turnover ratios KPI for the IAAS More than 1000 Equivalent Full Time VMs but 99,2% lived less than one month…
  25. 25. 24 / 0,0% 20,0% 40,0% 60,0% 80,0% 100,0% 120,0% 0 500 1000 1500 2000 2500 3000 3500 4000 4500 Capacity RAM (GBytes) Active RAM (Gbytes) RAM usage ratio KPI for the IAAS [RAM selling] New servers added! (Stress Test / Performance HW zones) Test benches are bigger and bigger to manage more complex use-cases (Various Suite and Stress Test activities)
  26. 26. 25 / 0,0% 20,0% 40,0% 60,0% 80,0% 100,0% 120,0% 140,0% 160,0% 180,0% 200,0% 0 100000 200000 300000 400000 500000 600000 700000 800000 vCPU Capacity Hours VCPU Hours vCPU usage ratio KPI for the IaaS [vCPU selling] Increase of vCPU hour selling. New servers added especially for Stress Test activities

×