1.
Clint Sprauve
Malcolm Isaacs
The Evolution of Test
Automation for Dev/Ops

2.
Quality driving value through DevOps
The “why”
--How DevOps presents a great opportunity
for quality professionals to expand their
organizational impact
The “what”
--Specific new processes and skills quality
professionals bring to DevOps and Shift Left
models
The “how”
-- How quality professionals and teams are
adopting DevOps practices and capabilities

3.
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 3
Quality matters to DevOps
Why?

4.
Evolution of Functional Testing
Agile and DevOps
challenge testing processes
Current challenges…
Open Source and
Continuous Integration Tools
User Experience changes
the definition of quality
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4
93%
of IT leaders use Agile
for software development
projects
$60Bannual savings by
businesses using
open source
Global population estimated
to be connected by 2019*
autonomous things
estimated to be attached to
the Internet by 2020*
57%
30B

5.
Database
activity
Security
validation
Inventory
query
Database
activity
3rd party
ship service
Logistical
services
Acc’t
info
3rd party
database
3rd party
reporting
Security
validation
3rd party
payment services
Account
authorization
Shopping app
(web or mobile)
Purchase
receivedAdd to
cart
Shipping
detail
Delivery
tracking
Customer
login
Checkout
A Retailer Real-Life Scenario
A Modern, Complex Architecture
Geographies of
users?
Flows?
User types?
(mobile, web, other)
Services availability
and cost?
Single vs.
Multiple?
Type of network
connections?
Environment
availability?
Is it secure?
Does it
function?
Functional
Development
USE CASE:
Performance
DevOps
Security
Mobile
QUIT SCENARIO

6.
DevOps is a transformation
It’s a sustained effort to achieve speed with Quality
• With DevOps there is no one true authority
• It’s not a product, spec, job title
• There is no one “right way”
• But one thing is certain- Quality matters
The role and importance of Quality in
today’s AD teams is moving from finding
defects to protecting the corporate image
World Quality Report– 2015-16
People
Process Technology

7.
The good news– Quality is central
Dispel the myths that Quality goes away with DevOps
Dev Ops
Quality

8.
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 8
Quality in the age of DevOps
What

9.
Quality Within DevOps
Defining The What
1. Starts with Culture Change
2. Quality Integrated Throughout
3. Test Constantly
4. Relentlessly Automate
5. Create Visibility

10.
Starts with Culture Change
Quality within DevOps
1. Starts with Culture Change. Teams unite around making the org as a whole faster,
more competitive.
“Coming together is a beginning;
keeping together is progress;
working together is success”
Henry
Ford

11.
Quality Integrated Throughout
Quality within DevOps
2. Quality Integrated Throughout. Move everything to happen earlier. Testing happens
earlier, get comfortable with IDEs, leverage development assets, but add your testing
acumen.

12.
Test Constantly
Quality within DevOps
3. Test Constantly. If you’ve followed point 2 above, you’re testing earlier. You’re also going
to test much more often. And consider the Quality Trifecta—ensure Functionality,
Performance, Security.

13.
Relentlessly Automate
Quality within DevOps
4. Relentlessly Automate. Automated testing shortens time to market and lowers risk of
human error in testing, while increasing the cycles in same amount of time, and increasing
coverage. Automate testing, provisioning and virtualize. Then tie it to build life and
automated execution integration.

14.
Create Visibility
Quality within DevOps
5. Create Visibility. Seamless communication between people, tools and processes.
Removing constraints that slow you down and connecting tools through a framework.

15.
Framework
Quality within DevOps
AutomateOrchestrate Transform
Application
Delivery
Management
IT Operations
Management
Application testing for
desktop, web and
mobile
Infrastructure, domain
and compliance
automation
From silos to integrated
lifecycle management
Processes to improve
agility
Customer experience by
simulating real world
scenarios
Customer experience by
delivering products and
services
DevOps
Shared Dev & Ops
methodology across
Dev/Test/Stage/Production

16.
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 16
In Practice
How

17.
How we have delivered at HPE Software
Quality within DevOps

18.
Starts with Culture Change
Quality within DevOps
• Why we thought to change = Productivity
• Development environment based on shared infrastructure
• Unstable
• Can be easily damaged
• Common for everyone
• Operational cost is very high
• Unexpected behavior
• Work from anywhere
• Latency is killing your connections

19.
Quality Integrated Throughout
Quality within DevOps
Plan Define Develop Build Test Deploy Release Operate
AGM AGM AGM
Sonar
OO
SM/SAW
Puppet
UCMDB
BSMFortify
Jenkins
Wiki
OpenGrok
GIT/Stash
IntelliJ Maven
ALM OO
Puppet
SM/SAW
Nexus
LeanFT
LR SiS
CSA
Nexus Nexus
Maven Site
Vagrant
Sahara
3RDPartiesProcessHP
CSA
Docker
JUnit
JBehave
Jasmine
Selenium
SharePoint
NV and SV

20.
Organization Example - Evolve the Team Structure
Change based on product complexity and team skill set
Dev/Test Organization Additional Responsibilities
QA & DEV – one team Support Tickets
Automation from day 1 Forums & social networsk
Manual Testing is Essential Build CI/CD infrastructure
No more deliveries to QA Train internal teams on dev/ops
practices
 Structure and responsibilities will vary from team to team, project to project
 Do not implement a “cookie cutter” approach to quality in Dev/Ops

21.
Test Constantly
Quality within DevOps
Tests run on each push
13,000 automated functional tests run daily
Service Anywhere 3.3 Service Anywhere 3.4
336 application tests 624 application tests
40 platform tests 70 platform tests
Manual coverage 100% sanity coverage
Can run only on CI deployment Runs on real deployment too

22.
Relentlessly Automate
Quality within DevOps
• End2End automated flow
for continuous testing
• Jenkins acts as the
orchestrator
• Flexible & customizable
to accommodate specific
customer workflows
Build
Deploy
Test
Jenkins
HP CodarALM
Test
case1
Test case2
Test case
3
.
.
.
.
Test case
n

23.
Example – Dev/Ops Automation Test Coverage Distribution
TEST TYPE Client Backend Protocols/Regions
Coverage 60% 80% 100%
# of Active
Tests
133 306 All Protocols, All
regions

24.
Create Visibility
Quality within DevOps
• Accurately Predict if a Code Commit Will Break a
Build
• Accurately Predict and Plan for Defect
Introduction

25.
Key Lessons Learned
Case Study Results
• DevOps is possible. Huge potential for speed and quality
• The culture changes when you incorporate DevOps
• Discipline – DevOps requires focus and dedication
• Automation and tools are important, but not everything
• Communication, collaboration and trust

26.
More info
– Trials and free editions of tools you can use
– Trials
–NV Trials: http://www8.hp.com/us/en/software-solutions/network-virtualization/try-now.html
–LeanFT: http://saas.hp.com/signup/try/leanFT
– Community Editions
–LR 12.50 with NV: http://saas.hp.com/signup/try/LoadRunner
– Freemium(s)
–NV Freemium: http://www.hp.com/go/NVFreemium

27.
Thank you
27