This document discusses how to fit performance testing into DevOps environments. It recommends adopting best practices like shifting performance testing left to earlier stages, conducting continuous performance validation as part of continuous delivery, automating what can be automated while accelerating other tests, and ensuring collaboration between performance engineers and DevOps teams. The presentation provides an example of how a performance testing tool can integrate into a DevOps toolchain at different stages like build, deploy, test and release. It emphasizes the importance of performance testing for software quality in fast-paced DevOps environments.
4. The context of performance testing in DevOps environments
Performance Testing in DevOps Best Practices
Demo – How to fit a Performance Testing Tool into a DevOps Toolchain
Agenda
5. The context of performance testing in
DevOps environments
6. Optimized software delivery is a competitive advantage
Software delivery is accelerating dramatically to sustain the competitive advantage
Organizations adopt Agile and DevOps as a solution
Software Delivery Pace is Accelerating Dramatically
7. High Performing DevOps Enables Better Software Quality
Source: 2016 State of DevOps Report
8. • Performance testing is time consuming and still highly manual
• Modern applications are complex: risk of failure is high
• Risk assessment is required
• Provisioning testing environments can be challenging
• More test = more data. How do you act on it?
But performance testing in DevOps is challenging
9. • Testing maturity is “a key differentiator”
Yet Performance Testing is Essential to DevOps
“DevOps – It’s all about continuous testing!”
by Marc Hornbeek, Principal Consultant - DevOps at Trace3
(Source: devops.com)
• Performance is a key determinant in the user
experience
• Performance bottlenecks impact
• Business and brand
• Employee’s productivity
11. 1. Shift left
2. Continuous performance validation for Continuous Delivery
3. What can be automated vs what can be accelerated
4. Collaboration to support the Devops culture
5. Performance Engineers become key players
Best Practices
12. How automated are your performance testing processes?
• Fully automated
• Mostly automated with Continuous Integration
• Mostly manual
Polling Question #1
13. Why Shift Left?
• Manages the frequency of releases and cost of testing maintenance
• Identify performance issues earlier
How?
• Test APIs, microservices & components
1) Shift Left: Test Early
14. 2) Continuous performance validation for Continuous Delivery
Early Risk Analysis
1
Understand the application
Evaluate the performance risk
Write performance SLAs in user stories
Automation
2 Start Testing Early Components
Faster more frequent performance testing
Validation
3 System Wide Testing
Test against your (business) SLAs
Monitor
4 Detect problems in production
Collaborate
15. 3) What can be automated vs what can be accelerated
Unit component
Component
Production
activity
Limit
Development
Environment
QA
Environment
Soak
Connection
Mobile
Performance
Or
User
experience
Failover Cloud
Preproduction
Environment
Production
Environnement
Robustness of the code Architecture stability End to end experience
Load in
production
Test can (should) be
automated
Test can be accelerated
Complexityofperformancetest
Stage in the SDLC
Single
User
17. • Use of collaborative tools to distribute test results to the entire Devops team
• Enable real time monitoring during tests
• Customize test data views
4) Collaboration To Support The DevOps Culture
18. Performance engineering is a cultural transformation
Performance Engineers skill set needs to evolve
• Methodology proficiency
• New performance testing practices (More frequent component/API testing,
Monitoring, Feedback loop)
• Integration skills
• Automation skills
• Dev expertise
5 ) Performance Engineers: Supporters of The DevOps Team
19. Who manages performance testing within your Devops organisation?
• Performance engineers in a separate test team
• Decentralized performance engineer from the test team in the Devops team
• Each devops team levarages its own performance testing skills
Polling Question #2
21. receive task
integration
test
Typical Integration in the Devops Toolchain
Build
reques
t
Deploy Release
DE
V
QA
UA
T
PROD
Test
Prepare
Build
Compile
Packagin
g
Integration Functional Performance Security
23. Build
reques
t
Deploy ReleaseTest
Stage 2: Deploy
DE
V
QA
UA
T
Deployment scripts
Deployment tools
Chef
3 environments where build may
be deployed depending on
where you are in the
development process
24. Build
reques
t
Deploy ReleaseTest
Stage 3: Test
receive task
integration
test
DEV QA UAT
Integration Functional Performance Security
Test Pass or fail
Jira: collaboration
25. Continuous Integration
• Specific Plugin
• API or command line
Other Test Tools
• Reuse functional test assets
• Service Virtualization
Deployment Tools
• Test Infrastructure
• SLA Pass/Fail JUnit
Integration Strategy
26. Stage 4 : Release and Operate
PROD
Build
reques
t
Deploy ReleaseTest
27. Demo – How to fit a Performance Testing
Tool into a DevOps Toolchain
28. Next Steps
Contact Us
Visit Neotys blog www.neotys.com/blog/
Download NeoLoad Free Edition
go.neotys.com/NeoLoad-Free-Edition-Download.html
Questions?
www.neotys.com/solutions/devops-performance-testing-solution
Notas do Editor
Hi everyone, thank you for joining this webinar where we will show you how to fit performance testing in DevOps.
I am Sylvain Fambon VP Marketing at Neotys, and I will be presenting we Kirk who is senior perf engineer at Neotys
For those who do not know Neotys, we a load testing tool company, we develop NeoLoad a load and performance testing tool designed to answer the requirements of continuous performance testing from dev/api testing up to the system wide testing. And we also provide NeoSense a performance monitoring tool for apps in production.
And we see many customers moving to agile, and Continuous Integration/deployment, and then Devops. So
With digital transformation and business being supported by software, Optimizing the software delivery value chain has become the new way to better business outcomes
The most-advanced digital businesses prove to be more successful than their competitor who are less mature on digital transformation.
To sustain the competitive advantage, organizations need to accelerate the time to market of their digital services => Software delivery pace is accelerating dramatically
Organizations Adopt Agile and DevOps as an Answer, to release new features more frequently
We learn from the latest State of DevOps Report that
High-performing organizations are decisively outperforming their lower-performing peers in terms of throughput and software quality. High performers see 24 times faster recovery times and three times lower change failure rates.
Performance testing is complex, time consuming, and requires human action
Modern applications embedded complex technology: risk of failure is high
Enhanced user experience urge new technologies adoption
Integration of different systems: Website, supply chain/ERP
Third party services
Performance varies with mobile or web users
It is not possible to test everything under load: how to determine performance test coverage?
Provisioning testing environments may be challenging
Testing more frequently increases the volume of test data which may become difficult to analyze
DevOps does not reduce the need for testing, which is still an integral part of product development and delivery, and warrants quality.
Testing maturity is “a key differentiator of best practices in DevOps”.
As Diego La Guidice, Principal Analyst at Forrester, says, “The term everyone is using is DevOps, but I think it should really be DevTestOps. Testing is really crucial for achieving quality & speed”
Multiple feedback loops along the way ensure that software gets delivered to users quickly, at a high level of quality.
Test
Automated Test Execution with Continous Integration Server During Nightly Builds
Track the Evolution of Performance Test Results build after build
Deployment
For Test environment
Continuous Deployment automation based on performance test result
Automate Component/Microservices/API testing
Automate performance testing with CI integration
Run performance tests during nightly builds
Test 3rd party service / Service virtualization
Low design/ low maintenance
Use Collaborative Tools To Distribute Test Results to the entire Devops team, so they have an accurate, always up-to-date insight into the performance of the app and the risk of the features in
the delivery pipeline
Enable real time monitoring during tests to speed up reaction time / re run test when they fail
Customize test data views to fit with each stakeholder requirements (Dev, QA, Ops, Business) to analyze test results and efficiently identify bottlenecks with a growing amount of test data
Performance engineering is a cultural transformation that builds in performance throughout an organization: performance engineers are key players in a devops team
Their skill set also need to evolve
Explain than the test team have to adapt their way to work to this new methodology
New skills
DEV skills may be required
Process autoamtion
More test require a better understanding and monitoring of the test tools accuracy
Component testing
New method, test earlier needs to understand how microservcies are architectured
Expertise
Ponctual help in the cycle to be sure the process is always relevent and working with accuracy
Role is changing !
Deployment environments
Dev = Dev environment
QA = QA/test environment
UAT = User Acceptance Testing environment / Pre-prod
Start Q&A session
(While waiting for questions, read the next steps)