O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Angelique henry performance non regression

276 visualizações

Publicada em

Jenkins world Nice 2018 - Présentation about performance tests with Gatling, Jenkins and Maven in CI process. Lessons learns and real project feedback.

Publicada em: Tecnologia
  • DOWNLOAD FULL MOVIE, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. MOVIE 4K,FHD,HD,480P here { https://tinyurl.com/yybdfxwh }
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD FULL MOVIE, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. MOVIE 4K,FHD,HD,480P here { https://tinyurl.com/yybdfxwh }
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD FULL MOVIE, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. MOVIE 4K,FHD,HD,480P here { https://tinyurl.com/yybdfxwh }
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Angelique henry performance non regression

  1. 1. Adding performance non- regression in your CI pipeline with Jenkins, Docker and Gatling Angélique HENRY@AngliqueHenry
  2. 2. © 2018 All Rights Reserved. 2| Angélique HENRY | @AngliqueHenry Angélique HENRY « Make women visible in tech » https://www.duchess-france.org/ https://mixteen.github.io/ « Code with kids » Production expert Automation enthusiast Site Reliability Engineer Convince by DevOps culture « Build to Run applications »
  3. 3. © 2018 All Rights Reserved. 3| Angélique HENRY | @AngliqueHenry Inclusion begins now [About conferences] « I’m trying to make them more accessible to people with disabilities … I gonna read all the slides and any text, describe any images content and this is part of the presentation so you probably won’t even notice. This not only helps people that are visually impaired but someone who’s dyslexic, has trouble reading, people who don’t speak English as their first language, if you forgot your glasses at home, are to far from the screen. … It’s very important to make our content as accessible as possible to as many people possible. » Code Is Not Neutral: the Ethics of Programming Clarissa Peterson
  4. 4. © 2018 All Rights Reserved. 4| Angélique HENRY | @AngliqueHenry In this talk “Adding performance non-regression in your CI pipeline “ • Why performance non regression in CI? • How ? What do we need ? •Basic concepts •Tools •Where in the development process • Lessons learned from project experiences
  5. 5. WHY ? Where the story beginsAdding performance non-regression in your CI pipeline
  6. 6. © 2018 All Rights Reserved. 6| Angélique HENRY | @AngliqueHenry Back to 2015 Worked on a public website with high traffic: • Used 24h/24h 7/7 • 600 000 users / hour (up to 2000 / min) Before the first production opening: • Performances tests run with success !
  7. 7. © 2018 All Rights Reserved. 7| Angélique HENRY | @AngliqueHenry Back to 2015 After one year of development with a team of 6 developers: • Production issues every weeks ! • Future rush days will completely break everything ! It took 9 months with 2 experts to go back to stability
  8. 8. © 2018 All Rights Reserved. 8| Angélique HENRY | @AngliqueHenry Back to 2015 What could we have done better ? • we should have detect it long before production ! But how ?? With non regression tests of course: • Unit tests ? Did not detect, we needed the full stack to be deployed to reproduce issues • Integration tests ? It would have not detect, was working fine with one user The only regression was in performance
  9. 9. © 2018 All Rights Reserved. 9| Angélique HENRY | @AngliqueHenry Back to 2015, and the future One goal: detect performance regression earlier and easily Continuous full performance tests ??? No of course not BUT Launch small load tests as often as possible Detect performance drop with new developments
  10. 10. How ? What do we need ?Adding performance non-regression in your CI pipeline
  11. 11. © 2018 All Rights Reserved. 11| Angélique HENRY | @AngliqueHenry A disposable playground Applications are complex: - Several services to dozens of micro-services - Front and Back servers - Database, cache management, messaging, etc. Tests should break application when needed, anytime Tests should be re-run at will We need an environment close to production, not shared with client or end- users that we can break anytime.
  12. 12. © 2018 All Rights Reserved. 12| Angélique HENRY | @AngliqueHenry Some users, some data and some isolation Performance: - Should validate concurrency - Should validate processing times - Should validate your application, not the application of others We need several users concurrently doing stories with simulated or mocked external services.
  13. 13. © 2018 All Rights Reserved. 13| Angélique HENRY | @AngliqueHenry Effortless for developers Developers should just « git push » code And be notified when new code brings performance issues
  14. 14. © 2018 All Rights Reserved. 14| Angélique HENRY | @AngliqueHenry Continuous delivery Automatic deployment of application in some environment, without any human action. I will call it « CD »
  15. 15. © 2018 All Rights Reserved. 15| Angélique HENRY | @AngliqueHenry The main idea Deliver application Init data Start services CD Performance test Validations Success Fail Alert Integration tests Do nothing
  16. 16. How ? ToolsAdding performance non-regression in your CI pipeline
  17. 17. © 2018 All Rights Reserved. 17| Angélique HENRY | @AngliqueHenry Docker : run services - Easy to deploy a full application with docker-compose - Can be deploy nearly anywhere - Used by the development team - Used in production for this project
  18. 18. © 2018 All Rights Reserved. 18| Angélique HENRY | @AngliqueHenry Mattermost: notifications and alerts - Used by development team to communicate - API easy to use (just a Curl) - People can subscribe / unsubscribe to notifications
  19. 19. © 2018 All Rights Reserved. 19| Angélique HENRY | @AngliqueHenry - Open-source tool - Easy to launch e.g: mvn gatling:test - Default reports have nice presentation, can be shared with your client - Not that easy to write (scala) but includes nice functions for performance tests - E.g: random CSV feeders, ramp-up injection of users, scenario - Good documentation and cheat-sheet - Jenkins plugin Gatling: performance tests
  20. 20. © 2018 All Rights Reserved. 20| Angélique HENRY | @AngliqueHenry Jenkins: automation tool - Open source, easy to install - Well known by the development team - A lot of features for automation - Schedule, watching git, job dependencies - A lot of plugins, including gatling - Failure notification build-in - With the usage of pipelines, automation script is versioned in git
  21. 21. © 2018 All Rights Reserved. 21| Angélique HENRY | @AngliqueHenry Gatling report
  22. 22. © 2018 All Rights Reserved. 22| Angélique HENRY | @AngliqueHenry Gatling plugin evolution
  23. 23. How ? In the development processAdding performance non-regression in your CI pipeline
  24. 24. © 2018 All Rights Reserved. 24| Angélique HENRY | @AngliqueHenry Remember the main idea Deliver application Init data Start services CD Performance test Validations Success Fail Alert Integration tests Do nothing
  25. 25. © 2018 All Rights Reserved. 25| Angélique HENRY | @AngliqueHenry The sooner, the better On feature branch ? develop Create branch ‘my-nice-feature’ [commit] I did some job. [commit] review corrections. Merge branch ‘my-nice-feature’ Launch CD and performance tests Question ! Given: - A team of 6 developers - 1 integration env. for CD (one branch at a time) - CD: 10 min for the setup, 20 min of tests When - All developers push in quite the same time Then How long does the last developer has to wait to see the result ?
  26. 26. © 2018 All Rights Reserved. 26| Angélique HENRY | @AngliqueHenry Why only one CD environment then ? • Automation save project quality but have costs: • Time to set up • Platforms to install • Time to maintain • Time to learn new tools to go deeper in automation • KISS (Keep It Simple and Stupid) compliant • Good to start with small things the first time • Good to have « something that works » very quickly
  27. 27. © 2018 All Rights Reserved. 27| Angélique HENRY | @AngliqueHenry The sooner, the better On the branch were features are merged ? develop Create branch ‘my-nice-feature’ [commit] I did some job. [commit] review corrections. Merge branch ‘my-nice-feature’ Launch CD and performance tests Only one CD environment needed
  28. 28. © 2018 All Rights Reserved. 28| Angélique HENRY | @AngliqueHenry Our CD environment • In Wordline we have a sandbox on-premise cloud: to try stuffs • We took one CentOS VM, where we installed: • Jenkins • Docker and docker-compose • Git • On every commit on main branch “develop” • Build application packages from latest version • Build docker images with packages • Run application parts • Check that all services are up • Launch Gatling with maven
  29. 29. © 2018 All Rights Reserved. 29| Angélique HENRY | @AngliqueHenry Usage of Jenkins pipelines • We split into 2 pipelines • One to install • One to launch Gatling tests • After install, the Gatling pipeline is launch automatically • Why did we split ? • On the last 3 days of sprints, a lot of merge on develop branch • The application was restarted very often, several time per hour • We wanted to create a stable demo environment for internal tests • We re-used the install pipeline on a separate VM to deliver once a day
  30. 30. © 2018 All Rights Reserved. 30| Angélique HENRY | @AngliqueHenry First pipeline: install and start last version
  31. 31. © 2018 All Rights Reserved. 31| Angélique HENRY | @AngliqueHenry Second pipeline: launch gatling tests
  32. 32. © 2018 All Rights Reserved. 32| Angélique HENRY | @AngliqueHenry Mattermost notifications
  33. 33. Lessons learns Adding performance non-regression in your CI pipeline
  34. 34. © 2018 All Rights Reserved. 34| Angélique HENRY | @AngliqueHenry What is the state today ? • We set up this continuous performance testing with 2 projects: • One build at the end of 2017, with a production on January of 2018 ▸3 months build ▸6 developers ▸6 « micro-services » (java) ▸2 front services (ember js) • One still in construction in 2018, with a first opening at the end of January of 2019 ▸1 year of build ▸8 developers ▸8 « micro-services » so far.. (java) ▸2 front services (ember js)
  35. 35. Lessons learns Adding performance non-regression in your CI pipeline
  36. 36. © 2018 All Rights Reserved. 36| Angélique HENRY | @AngliqueHenry The sooner, the better CD must be set up before the development begins: - It should be an habit for developers - It should be part of the initial training When rush of sprints came in, only habits have a chance to survive !
  37. 37. © 2018 All Rights Reserved. 37| Angélique HENRY | @AngliqueHenry Learning curve Gatling is not that easy to write (scala) Jenkinsfile for Jenkins pipelines, it’s a new syntax to learn Docker and Dockerfile require system and network skills Beware of « DevOps tools » competence SPOF (Single Point Of Failure) Be sure to have at least 2 people mastering CI and CD workflow and tools. Train as much people as possible Concentrate efforts on high critical parts of the application
  38. 38. © 2018 All Rights Reserved. 38| Angélique HENRY | @AngliqueHenry CD is a small production Deploying database, application servers, Jenkins and launch Gatling on single VM: - Needs RAM and CPU We had to upgrade VM to « stronger » On larger application we will have “to go to cloud” to have everything working together without capacity issues
  39. 39. © 2018 All Rights Reserved. 39| Angélique HENRY | @AngliqueHenry CD is a production for the dev team CD must be available all the time CD evolve with the application architecture You need an “acceptance” CD to tests development on: - Jenkinsfile - Dockerfile, docker-compose - New services There is some maintenance to do on this new platform, time to time: - full, reboot, jenkins upgrade, …
  40. 40. © 2018 All Rights Reserved. 40| Angélique HENRY | @AngliqueHenry Not a silver bullet - You will detect a lot of issues earlier - It’s a way to train your team to write load test - It can be used as functional tests - It can be used as internal demo (for Product owners, projects leaders) BUT - It’s still not a production - It does not replace real load tests
  41. 41. Lessons learns Nexts stepsAdding performance non-regression in your CI pipeline
  42. 42. © 2018 All Rights Reserved. 42| Angélique HENRY | @AngliqueHenry It’s only the beginning - For now we only check HTTP 200 OK status - We have to set maximum response time per scenario / request - We do not monitor RAM, CPU or I/O consumption - We were not able to detect load balancing issues, having only one instance of services BUT - Dev team write quite easily Gatling tests: adoption is done - Dev maintains in full autonomy pipelines and CD env: habits are present - We have part of scenario for real load tests: nearly ready - Gatling scenario are based on stories examples: small functional tests (based on REST API) - Integration issues raises on the same day the feature is merge on develop => The code causing the issue is still in the developer head, and corrections are made by people without asking them to do
  43. 43. © 2018 All Rights Reserved. 43| Angélique HENRY | @AngliqueHenry (?) Wonder for future (?) - Will we use some Gatling scenario to validate delivery on acceptance ? - Will we use some « read-only » scenario to validate production, nightly for example ? - Will we use APM (Application Performance Management) to monitor performance tests ? - Dynatrace, App dynamic, pin point, elastic APM, … - Will we move forward and launch performance tests on every feature branches ?
  44. 44. Conclusion Adding performance non-regression in your CI pipeline
  45. 45. © 2018 All Rights Reserved. 45| Angélique HENRY | @AngliqueHenry
  46. 46. Thank you ! Image crédit: https://pixabay.com

×