More Related Content Similar to Software Quality as a Competitive Differentiator (20) More from DevOps.com (20) Software Quality as a Competitive Differentiator 1. © 2018 VERACODE INC.1
Software Quality as a Competitive
Differentiator
Maria Loughlin, VP Engineering
@marialoughlin
2. © 2018 VERACODE INC.2
On This Webinar
1.
Quality and
Business Success
2.
DevOps
Promises & Gaps
3.
Building a
Quality System
3. © 2018 VERACODE INC.3
Poll: Who’s Attending This Webinar?
• Quality Professional (Tester, QA Eng, SWET, Architect)
• Developer / DevOps / Operations Engineer
• Product Manager / UX Designer
• Engineering Manager / Executive
• Other
4. © 2018 VERACODE INC.4
Revenue and Net Promoter Score
SOURCE: The Net Promoter System. Bain &
Company, Inc.
“On average, an industry’s Net
Promoter leader outgrew its
competitors by a factor greater
than two times.”
5. © 2018 VERACODE INC.5
High Quality, High Trust
SOURCE: https://cxi.today/2018-cx-trends/analytics-infographic-5-trends-
shaping-cx-in-2018/
6. © 2018 VERACODE INC.6
Quality
Productivity
Predictability
Employee
Happiness
Innovation
7. © 2018 VERACODE INC.7
Maria Loughlin
VP Engineering, Veracode
• Two decades of software engineering
leadership
• Waterfall to Agile to DevOps
• Monolith to MicroServices
• Manage development and operations for the
FedRAMP instance of Veracode’s Application
Security products.
• Deep expertise in Secure SDLC and DevSecOps.
10. © 2018 VERACODE INC.10
DevOps Promise: Continuous Testing
Image: https://www.parasoft.com/solutions/continuous-testing
11. © 2018 VERACODE INC.11
DevOps Promise: Comprehensive Testing
Unit
Component
Integration
E2E
UI
Shift
Right
Shift
Left
Automation throughout the
stack
Automation throughout the
lifecycle
12. © 2018 VERACODE INC.12
Unit
Compo
nent
Integrat
ion
End to
End
DevOps Reality: Inconsistent Testing
• Quality investment often driven
by delivery team, independent
of overall strategy
• All sorts of tests with almost
equal priorities
13. © 2018 VERACODE INC.13
State of Software Quality
SOURCE: GitLab 2018 Global Developer Report, https://about.gitlab.com/developer-survey/2018/ /
40% sacrifice quality to
meet a deadline
14. © 2018 VERACODE INC.14
SOURCE: GitLab 2018 Global Developer Report, https://about.gitlab.com/developer-survey/2018/ /
Testing
causes
delays
15. © 2018 VERACODE INC.15
The Change Failure
Rate for high
performers is
5 times lower
than for low
performers
SOURCE: Puppet 2017 State of DevOps Report,
https://puppet.com/resources/whitepaper/2017-state-of-devops-report
16. © 2018 VERACODE INC.16
The Mean Time to
Repair (MTTR) for
high performers is
96 times faster
than for low
performers
SOURCE: Puppet 2017 State of DevOps Report,
https://puppet.com/resources/whitepaper/2017-state-of-devops-report
17. © 2018 VERACODE INC.17
85% of
applications are
vulnerable35.9%
33.5%
85.1%
84.9%
First Scan
Latest Scan
High or
Very High
Severity
Any
Severity
Percent of Applications with Findings
Source: Veracode SOSS Volume 9
SOURCE: Veracode SOSS Volume 9, https://www.veracode.com/state-of-software-security-report
State of Software Security
18. © 2018 VERACODE INC.18
The percent of
applications
passing OWASP
Top 10 Policy
on first scan is
consistent over
time
23% 77%
13% 87%
32.3% 67.7%
38.6% Passed 61.4% Did Not Pass
30.2% 69.8%
2010
2013
2015
2016
2017
Percentage of Applications Passing OWASP on First Scan
Source: Veracode SOSS Volume 9
SOURCE: Veracode SOSS Volume 9
19. © 2018 VERACODE INC.19
What’s The Challenge?
1. Reinvested quality process
2. Unfocused quality efforts
3. Relentless pressure to deliver
4. Complexity of software –
more than ever before
20. © 2018 VERACODE INC.20
Challenge 1: Who’s Responsible for Quality?
Dev Product
Tester
Designer Mgr
Dev Product
Tester
Designer Mgr
Monitoring
Analytics
SupportInfra-
structure
Dev Product
Tester
Designer Mgr
Waterfall Agile DevOps
21. © 2018 VERACODE INC.21
Challenge 2: Unfocused Quality Efforts
Quality can be subjective
and contextual.
23. © 2018 VERACODE INC.23
Challenge 4: Software Is Increasingly Complex
Today’s software is
• Distributed
• Embedded in complex systems
• Autonomously learning and
evolving
• Deployed to untrusted
environments
25. © 2018 VERACODE INC.25 © 2018 VERACODE INC.
Part 3:
Creating a
Quality System
26. © 2018 VERACODE INC.26
Creating a Quality System
Specify
CI/CD across organization with recommended tools
Drive
Quality
Strategy
Quality
Process
Test
Automation
27. © 2018 VERACODE INC.27
Strategy: Quality Goals
`
Strategy Process Automation
Is Your Customer
Getting the Value
They Expect?
• Functional
• Great user experience
• Consistent, reliable
• High performing
Will Your Team
Remain Productive?
• Maintainable
• Scalable
• Secure
28. © 2018 VERACODE INC.28
Strategy: When and Where?
• Pre-production
– Test functionality, stability, security, customer satisfaction, compliance
• Production
– Test functionality, performance, resilience, stability
– Experiment to test new ideas
`
Strategy Process Automation
29. © 2018 VERACODE INC.29
Strategy: Who?
Unit
Component
Integration
E2E
UI Delivery team owns the tests
• Maturity of organization impacts
exact staffing
– Lower layers always developer
• Quality mindset always present
– QA architect and ever-present
voice of customer
`
Strategy Process Automation
30. © 2018 VERACODE INC.30
Process: Investment by Phase
• Inspect and adapt process
• Continuous production feedback
• Customer data
• Upper layers of test pyramid
• Quality dashboards
• Security testing
WALK
CRAWL
RUN
• Deployment automation, CI/CD
• Unit tests and mocking code
• Test infrastructure
`
Strategy Process Automation
31. © 2018 VERACODE INC.31
Process: Definition of Done
`
Strategy Process Automation
Test investment
32. © 2018 VERACODE INC.32
Process: Metrics
Internal View
• Test coverage
• Reopened issues
Customer View
• Escaped defects
• MTTR
• Service interruption
`
Strategy Process Automation
33. © 2018 VERACODE INC.33
Automate Everything
1. DevOps Infrastructure
`
Strategy Process Automation
Unit
Component
Integration
E2E
UI
2. Tests
34. © 2018 VERACODE INC.34
SOURCE: Atlassian Marketplace for DevOps Apps, https://marketplace.atlassian.com/categories/devops
35. © 2018 VERACODE INC.35
Automation: Infrastructure
`
Strategy Process Automation
Infrastructure Consideration
CI / CD Pipeline Reliable, repeatable
Example: Jenkins
Test Environments Easy to create and scale. Monitor for cost
Test Frameworks Can be integrated with build pipeline, e.g. GitLab
Or separate tool, e.g. Robot / TestNG
Quality Metrics Transparent, consistent
Example: SonarCube, Bug tracker with analytics
36. © 2018 VERACODE INC.36
Automation: The Test Stack
Test Layer Consideration Example Tools
UI Match your UI language Protractor for Single Page Apps,
or Selenium, Cyprus, Jest
E2E Include performance testing Selenium, Protractor, Cyprus,
Jest
Integration Focus on interactions
between microservices and
external services
API: Rest Assured, Postman
Component Include performance testing Mockito for mocking framework
Unit Match your language
primitive
Junit, PyUnit
UI Unit tests: Karma, Jasmine
`
Strategy Process Automation
37. © 2018 VERACODE INC.37 © 2018 VERACODE INC.
Driving Quality
Across The
Organization
38. © 2018 VERACODE INC.38
Creating a Quality Culture
Break the
Silos
Support the
Team
Learn
Continuously
39. © 2018 VERACODE INC.39
Guilds: Experts Support Each Other
• Identify your leaders and
practice experts
• Hold regular ‘birds of a
feather’ meetings
• Share learnings, trends and
best practices constantly
• Encourage & reward
participation
41. © 2018 VERACODE INC.41
What About Security Quality?
Specify
Security
Strategy
Security
Process
Security
Automation
42. © 2018 VERACODE INC.42
Strategy: Security Policy
Policy defines and supports your tolerance for risk.
• Requirements for remediation of vulnerable code
and components
• Standards for software licence usage
• Recommended libraries, frameworks, embedded
components
43. © 2018 VERACODE INC.43
Process: Security Maturity Model (SAMPLE)
Activity Base Beginner Intermediate Advanced Expert
Training
Secure Design
Security Code
Review
Security
Testing
Third Party
Activity Base Beginner Intermediate Advanced Expert
Training
Secure Design Security is not a
design
consideration
Security reqts are
generally defined
after
development has
started or
completed
Threat modeling
before major
components or
features
Security reqts are
defined before
major componen
ts or features
Threat modeling is
incorporated into
the story process
Security reqts are
defined as story
Acceptance
Criteria on
relevant stories
Security
Acceptance
Criteria defined
for all relevant
stories
Security Code
Review
Security
Testing
Third Party
44. © 2018 VERACODE INC.44
Security Automation
The best app security is
invisible to developers
45. © 2018 VERACODE INC.45
Recap: On This Webinar
1.
Quality and
Business
Success
2.
DevOps:
Promises &
Gaps
3.
Building a
Quality System
“Quality is never an accident; it is always the result of high
intention, sincere effort, intelligent direction and skillful execution”
– William A. Foster