This document discusses continuous testing in the context of DevOps. It defines continuous testing as including automated testing, managing production and non-production environments, application monitoring, and evaluating business objectives. Continuous testing relies on DevOps activities like automating builds, infrastructure, deployments, and monitoring. It advocates for smaller, more frequent deliveries through practices like test automation, infrastructure as code, and treating testing as integral to the software delivery process. The conclusion emphasizes automation, configuration management, and obtaining frequent feedback to enable continuous testing.
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
AfterTest Madrid March 2016 - DevOps and Testing Introduction
1. - Technical Architect @Planday
- Lead consultant & trainer at @
LeanSoftwareServices
- Based in London & Madrid
Peter Marshall
MAIL
info@leansoftwareservices.com
TWITTER
@petemar5hall
LINKEDIN
https://www.linkedin.com/in/petedmarshall
3. Testing software in high
frequency delivery
environments
DevOps &
Testing
Understanding the
Foundations
@petemar5hall
4. Gain insight into how devops and organisational change are
required for continuous testing
Understand how to leverage devops & continuous testing practices to
gain more time testing
Objectives
6. “A survey by IBM and uTest showed
that almost 40% of testers spent
between 50% and 100% on non
testing activities per week.”
7. Sources: Applied Software Measurement: Global Analysis of
Productivity and Quality by Capers Jones, & IBM The Future of Testing
Most forms of testing find fewer than 30% of all bugs
almost 40% of testers have seen an increase in the frequency of release
Productivity and quality are directly coupled: projects with
high quality have high productivity and vice versa
11. the test and QA
headache is
compounded
further by the need
to release faster
and more often
Amazon is on record as making changes
to production every 11.6 seconds on
average in May of 2011.
Facebook releases to production twice a
day.
Many Google services see releases
multiple times a week, and almost
everything in Google is developed on
mainline.
Etsy push unto 70 changes a day to its
monolithic software!
13. Continuous Testing
…not just about test
automation! (ignore
wikipedia)
Actually includes, but not limited to:
Automated testing
Automated management of production and non
production environments
Automated application feedback through
monitoring
Automated evaluation of business objectives
Evaluating product concepts
Testing
engaging in XP practices
organisational change
14. The DevOps movement is built
around a group of people who
believe that the application of a
combination of appropriate
technology and attitude can
revolutionize the world of software
development and delivery.
Stephen Nelson-Smith - 2010
its about:
Automating building, testing, and
deployment
Bringing consistency to the way we
use and create technical assets
Providing teams with the tools
required to achieve technical
excellence
Continuous
testing is
dependent on
DevOps
Activities
15. Build Agent & Devel
The continuous test flow
Pre commit tests
Code review
PO/UX review
Branch / Merge
Build
Unit tests
Contract tests
Component tests
Coverage
Static analysis
Component
performance
End to end
component tests
NFR tests
Exploratory Test
performance testing
Story sign off /
Demos
NFRs
Analytics
Monitoring
Production tests
canary or blue /
green release
CI Environments
Developer
Local
Pre-Production Production
Virtualised Full stack environments
Commit Auto-Deploy Promote Release
Build Pipeline:
There can be multiple pipelines, usually per system, should execute in minutes
Commit:
The commit can include changes to the infrastructure, database, or application code.
Outputs:
Software, infrastructure and configuration
Environment manifests
Release notes
18. vs.
script & test your
configuration!
say NO to hand crafted configuration files!
remove configuration issues by scripting your configuration.
19. automate & test
the infrastructure
Remove configuration defects by having a repeatable
mechanism for creating environments.
Over 70% of testers are delayed
on a regular basis by environment
issues
20. automate & test
deployments
say NO to sneaker net
say NO to manually to copying files
say NO to manually pulling and pushing servers in and out of load balancers
deploying software is one of the
most mission critical aspects of
what we do, let a machine do it!
21. vs.automate the testing
continuous testing does away
with the concept of
regression testing.
..an effective software delivery process
requires no regression testing stages. It
becomes integral to everything we do..
write tests for:
Configuration
Functionality
Deployments
Infrastructure
22. monitoring &
Alerting
One of the greatest and
overlooked tools in the arsenal of
the tester is monitoring.
Realtime access to
application logs
Realtime access to
error logs
Realtime access to
infrastructure logs
Testing also
occurs in
production.
23. “A good, easily accessible
event & error monitoring
system is essential for anyone
releasing software. Build an
early warning system into your
environment!”
24. “Outputting more logs and
events is as good as having a
houseful of automated
checks!”
25. If your system is not producing any or enough
data, ask your developers to change that
Log technical and business events
Log successes as well as failures
Use tools to capture logs that allow
visualisation