The document outlines seven best practices for accelerating continuous testing (CT): 1) make CT comprehensive and focus on results, 2) test fast and don't stop the testing pipeline for failures, 3) use powerful, relevant tests, 4) shift testing left to earlier phases, 5) foster a CT-ready culture and infrastructure, 6) integrate CT-ready tools across the pipeline, and 7) accelerate test execution with analytics. It provides examples of how following these practices improved release cycles and defect rates for a case study client.

1. Seven Best Practices for
Accelerating Continuous Testing
Marc Hornbeek
Senior Solutions Architect

2. 2Spirent Communications
“There is no continuous delivery
without testing”
Dev Ops

3. 3Spirent Communications
Fast CT is required to race
through the many test steps in
the pipeline from Pre-Flight
through to Delivery

4. 4Spirent Communications
CT infrastructure must
be stable and accurate
because test failures at
any point in the pipeline
cause delays or roll-
backs

5. 5Spirent Communications
Consequences of
poor CT practices
80% failure rate when
using current tools
IDC Fortune 1000 survey,
December 2014

6. 6Spirent Communications
Accelerated Continuous Testing
Best Practices Principles
Comprehensive
&
Results
Test fast
&
Don’t Stop
Powerful
Relevant

7. 7Spirent Communications
CT best practices are across the pipeline
“Shift Left”

8. 8Spirent Communications

9. 9Spirent Communications
Focus on customer relevant defects
not quantity of defects
“Quality is everyone’s
Responsibility.”
Dr. W. Edwards Deming

10. 10Spirent Communications
Pre-Flight Team-workDesign for CT
CT End-to-End Culture

11. 11Spirent Communications
CT Process design
Fault tolerant
Obfuscation
Intelligent dashboards
CT-Ready Infrastructure

12. 12Spirent Communications
CT-Ready Tools
 Restful APIs
 Cache and pipeline
 Virtual and physical
 Program agnostic
 Large scale

13. 13Spirent Communications
CT Tools Integration
 Pre-Flight
 CI Tests
 Integration
 Regression
 System
 Deployment

14. 14Spirent Communications
Orchestrate Test Topologies
 Physical and Virtual
 Utilization
measurement
 Topologies
 Test Automation

15. 15Spirent Communications
Accelerate Test Execution
 Powerful servers
 Test design
 Pipeline
 Scale horizontally
 Thresholds
 Dynamic test schedules

16. 16Spirent Communications
Fast Relevant CT Analytics
 Continuous Monitoring (CM)
 Relevance
 Dashboards, Alerts, logs,
thresholds
 Aggregation
 Snapshots and trends
 Telescope for diagnostics

17. 17Spirent Communications
Case Study
 360 developers, six sites
 36 MLOC+, 100+ features
 86 build targets
 complex interdependencies
 200+ topologies
 12K test cases
 80+ deployment packages
(physical and virtual)
1,280,000,000
test possibilities

18. 18Spirent Communications
The primary contributing
factor was CT and CM !
CT Case Results
Metric
Major release
(#months)
Minor release
(#weeks)
# Features
Defects
Integrations / day
Tests / day
Automated tests
After
3
2
150
10
100
10
85%
Before
6
4
113
1260
0.5
0.3
5%

19. 19Spirent Communications
Best practices toolBest Practices Assessment Tool
simplifies the assessment process.

20. 20Spirent Communications
Summary
 7 CT best practices emphasize
speed, don’t stop,
comprehensive and relevant.
 Benchmark against Best
Practices Blueprint and
DevOps Assessment.
 Don't fall into the “do-CT-
yourself / NIH” trap!
“There is no continuous delivery
without testing”

21. 21Spirent Communications
www.spirent.com/clear
1) CT Blueprint white papers
2) CT eBook
3) CT Assessment Tool
Take-aways
http://www.spirent.com/go/AMR

22. 22Spirent Communications
How can Spirent help?:
Test tools
CT Orchestration
Lab Management
Professional Services

23. 23Spirent Communications
© Spirent Communications, Inc. All of the company names and/or brand names and/or product names and/or logos referred to in this document, in particular the name
“Spirent” and its logo device, are either registered trademarks or trademarks pending registration in accordance with relevant national laws. All rights reserved.
spirent.com
Thank you
Seven Best Practices for Accelerating Continuous Testing
Marc Hornbeek
Senior Solutions Architect
Marc.Hornbeek@spirent.com
Twitter: mhexcalibur