From Requirements to high quality deliverables - Visure Solutions & Wind River

1. Software Test Optimization:
Reducing the cost of testing, using a
requirements-driven, change-aware process
An Integrated Solution from Wind River and Visure Solutions
Fernando Valera, Alliances Manager– Visure
Ido Sarig, VP Marketing – Wind River

2. Why do we fail?

3. Our View of the Market: Market Pains 1
Trend 1: Growing Complexity of Software and Product Variants
50
45
•Key issues:
•Relative growth of software, increase 40
of 33% compared to mechanics and
35
electronics
Millions lines of code
•Complexity of product lines and 30 2006
large-scale systems 2007
•Examples: 25 2008
2009
•Aircraft carrier, 1,000,000 20 2010
requirements 2011
•Hospital information system: 90,000 15
2012
requirements 10
•Train control system: 14,000 contract
requirements 5
•Modern airliner: 6,500,000 LOC 0
2006 2007 2008 2009 2010 2011 2012
•Ford Motor Company’s Prediction of Growth of Software Content in Automobile embedded software. Calendar years
•CPDA, The Requirements Engineering Process - Panel Discussion of Critical Issues in Requirements Management
•Nationale Roadmap Embedded Systems, ZVEI, www.zvei.org, 2010
•Requirements Engineering: A Roadmap, Bashar Nuseibeh & Steve Easterbrook, 2000

4. Our View of the Market: Market Pains 2
Trend 2: Missing Connection to Market Needs and Business Requirements
Key issues:
•< ~52% of requirements go into product release
•1% to 3% per month of requirements become obsolete
•RDM tools do not seamlessly connect upstream and downstream
activities
•Lack of communication and visualization of RDM features for
scoping & descoping incl. decision support
•Nationale Roadmap Embedded Systems, ZVEI, www.zvei.org, 2010
•Systems Engineering Vision 2020
•Information Technology for European Advancement (ITEA) 2009 Roadmap, http://www.itea2.org/itea2_roadmap_3/step/2, p.18, 3rd Edition
•M. Bone, R. Cloutier: The Current State of Model-Based Systems Engineering:
Results from the OMG SysML Request for Information 2009

5. Our View of the Market: Market Pains 3
Trend 3: Requirements Quality as an Essential Driver for Software Product Quality
Key issues:
~ 60% of all software defects traceable back to poor RDM practices
Changing requirements: Lack of effective change process and tool
enforcement
Absent formal RDM approach, > 90% of companies use still use
natural language approach
No defined RDM process
80% of organizations do not use RDM tool at all
Connecting adjoint development activities:
Testing
Design
Product risk management
Project management
•Institute for Experimental Software Engineering (IESE), Fraunhofer Gesellschaft, Survey on Root Causes of Software Defects in Medical
Devices, 2007; http://www.iese.fraunhofer.de/Images/ studie_denger_070507ple_v3 _e_tcm27-7391.pdf
•Information Technology for European Advancement (ITEA) 2009 Roadmap, http://www.itea2.org/itea2_roadmap_3/step/2, p.18, 3rd Edition
•Chaos Chronicles v 3.0, Standish Group International, 2003, http://www.standishgroup.com/

6. Our View of the Market: Market Pains 4
Trend 4: Compliance with standards and agile approach
Software is becoming more safety–critical,
and therefore, more regulated:
New standards:
 IEC 62304
 ISO 26262
 IEC 61508
 DO-178B/C
 DO-254
 CMMI
 SPICE
Shifting to different development model
 V-model and Waterfall to a more agile approach

7. Visure’s Differentiated Approach
Process Organization with mature Requirements
Engineering PROCESSES are proven to
consistently produce better quality products on
time, and within budget

8. Visure’s Differentiated Approach
Collaboration A COLLABORATIVE requirements platform has
proven to have a direct impact in the project
success bridging the gap between business and
IT, communicating distributed teams, integrating
clear information between the teams, and
establishing workflows

9. Visure’s Differentiated Approach
Product quality QUALITY needs to be introduced straight from
the beginning and at every stage of the
development, from writing good requirements,
keeping changes under control, to reusing high
quality assets

10. Visure Requirements
VISURE REQUIREMENTS INTEGRATES
PROCESS, QUALITY & COLLABORATION IN A SINGLE PLATFORM

11. Visure Requirements
A COMPREHENSIVE PLATFORM FOR VERIFICATION & VALIDATION OF
PRODUCT AND EMBEDDED SOFTWARE DEVELOPMENT
Unit test Cases

12. Visure Requirements: Collaboration
VISURE REQUIREMENTS ALLOWS USERS TO KEEP USING THEIR OWN
TOOLS, WHILE KEEPING EVERYTHING CENTRALIZED

13. The Testing Process
Development Costs
Requirements
Maintenance 15%
10%
Design
10%
Test
40%
Development
25%

14. Full Lifecycle Traceability
Quality Management Needs Strong Linkages to Requirements, Tests and Code
User Design
Code Test Cases
Requirements Specification
Bi-directional traceability
• Traceability maintained throughout the whole ALM, including WindRiver’s solution
• Identify un-tested requirements and code
• Measure requirements, code and test coverage
• Identify feature creep
Understand the impact of change
• Identify test suites impacted by requirements changes
• Identify what tests to run based on code changes
Traceability is the key to compliance
14

15. Requirements change? Automate where it matters most!
 Executing all the tests on each change is unrealistic… expensive and inefficient!
 Alternatively…
1. Highest Return: Execute tests that contribute to high coverage of the requirements, then any others…
2. Quickly Lower Risk: Execute tests for the most critical requirements, then any others…
3. Practically Safe: Execute tests for all the critical requirements, then any others…
 Especially since often ~20% of the test cases are covering ~80% of the business value
No. of Requirements High Requirements coverage
 Most critical Requirements
All critical Requirements

Low contribution

Test Suites
15

16. The Value of Wind River Test Management
Build 6 Build 7
TC1
TC3
TC2
Compare builds’ binaries to identify Only a subset of test cases are testing the
changed, new and deleted functions actual changes in the code
 Identifies the real  Works on unmodified,
impact of change fully optimized binaries
 Maps tests to the code  Does not require access
they exercise to source code
 Enables easy fault-  Facilitates collaboration
HIGH RISK: 85% of functions were not
injection tested amongst distributed
team members
16

17. Build Automation:
Requirement Management: IBM Buildforge, Jenkins, Electric Cloud
Excel, Visure Requirements
Central repository to store
manual and automated test cases
Dashboard Virtual lab manager
Flexible and views
Reports generator Easy and clear interpretation
of individual test case results
Powerful
Framework
Easily integrated with existing
build and test infrastructure
Defect Tracking:
JIRA, Rational, ClearQuest
Test results
parsed for each
test case and
each step within STDOUT for each test case step
test case captured individually

18. Wind River
Test Management
Identifies high risk code areas
Coverage
 After running all test cases
what areas of the code were
NOT tested HIGH RISK: 85% of functions were not tested
 Map test cases to code
 Increase preparedness for
certification process
Performance
 Measure actual functions
response time
 See impact of change on
performance
…all on production code.
HIGH RISK: Performance degradation between builds

19. Wind River Build 6 Build 7
Test Management
Manages the impact of change
Build Differences
 Compare builds to know the
impact of change Compare builds’ binaries to identify
changed, new and deleted functions
 Gain visibility to new code
introduced to the build
 Identify what tests to run to TC1
verify only the changes
TC3
TC2
Only a subset of test cases are testing the
actual changes in the code
…all on binary code.

20. Wind River
Test Management
Optimizes test execution
 The test suite generator
utilizes existing and
generated test suites and
enables test suite
optimization for criteria
such as coverage, fixed
defect validation, and
requirements traceability.
What tests will generate maximum coverage if I have only 5
hours to run my tests?

21. Wind River
Test Management
Tests edge conditions and error
handlers
Fault Injection
 Validate fault handlers.
 Automate fault injection.
 Do away with complex test
scenario setups.
 Fully validate exception
conditions fast.
 Simulate data input/output.
 Increase coverage, save time,
and improve quality.
…all on production code.

22. Summary:
Full lifecycle quality management for embedded SW
Requirements Management Quality Management
Visure Requirements Wind River Test Management
Visure Requirements Wind River Test Management
• Dynamic runtime instrumentation, data capture
• Integrated requirements traceability and test
and analytics
coverage
• Change-driven, optimized automated test suite
• Semantic Analysis
generation
• Centralized information repository
• Open execution engine
• Collaboration platform
• Host-based and on-board execution
• Requirements dashboard and reporting
• Virtual lab management

23. Questions?
Contact Details:
Fernando Valera fvalera@visuresolutions.com
Ido Sarig ido.sarig@windriver.com