Mais conteúdo relacionado Semelhante a Kerim Cakmak, Moshe Cohen -- Continuous Verification and Validation (20) Mais de Anatoly Levenchuk (20) Kerim Cakmak, Moshe Cohen -- Continuous Verification and Validation 1. Applying Continuous Verification and Validation
to achieve the right Quality in Systems delivery
Business
Risk
Technical
Risk
Kerim Cakmak, PhD
IBM Rational Tiger Team CEE
Moshe S. Cohen
QM Market Manager and Evangelist
© Copyright IBM Corporation 2013
2. Software and Systems Engineering | Rational
The V is dead! Long live the new V with continuous Verification
and Validation across the full V!
Abstract:
While we know how to reduce technical risks in Systems delivery through Verification activities,
our business risk is on a steep rise. This is largely due to the fact that "Validation" is performed
at the end of the development lifecycle, rather than throughout, and it is done against customers
requirements rather than end-users needs. In this session we will discuss some new best
practices around Verification and Validation that were observed through recent successful and
even more unsuccessful projects, and show you how to continuously verify and validate your
designs throughout the whole lifecycle and not only at the tail end of it. Not only will this improve
the quality of your products and systems, but it will also allow you to converge earlier on the right
requirements, compressing the time to end users feedback, and improve your time to market
predictability. While it doesn't change the V model, it is a new V.
2
© 2012 IBM Corporation
3. Software and Systems Engineering | Rational
Bottom Line is…
While delivering Quality is already a major challenge, the definition of Quality is
changing…
– Low tolerance for bad quality (real and perceived)
– Expected short cycles with continuous improvements
To deliver Quality, we need a holistic approach to Quality
– Continuously verify and validate our designs, throughout the whole lifecycle.
Business results:
– Deliver the right system
• Compressing time to customer’s feedback
– Better time to market predictability
• Earlier convergence on the right Requirements
– Lower cost of Quality
• Defect* Avoidance, Lower defect density, Higher defect removal efficiency
* Defect, as defined in ISO 9000 (2008): The non-fulfillment of intended usage requirements.
The term Defect will be used here in the general sense, regardless whether it is a product
defect, design defect, manufacturing defect, an error, a fault or a failure.
3
© 2012 IBM Corporation
5. Software and Systems Engineering | Rational
In the News… Business Implications of a Failed missile test
Could result in major financial consequences, unless Raytheon succeeds in
demonstrating that it is not its fault.
5
© 2012 IBM Corporation
6. Software and Systems Engineering | Rational
Business Implications of Recalls
Not only recalls are very expensive, but they also drive corporate behaviors.
Sony recalls: “Sony-related recalls are following one another, and that may ruin
the company’s brand image,” said Keita Wakabayashi, an analyst at Mito
Securities Co. with a “neutral plus” rating on the stock. Bloomberg, Oct 2011.
Toyota Prius gas pedal recall, 2010: ~$3B (CNNMoney.com)
– Repair cost (quality related expenses):
$1.12B
– Legal cost (class-action lawsuit):
$1.1B (WSJ, Dec 27, 2012)
– Image cost (sales losses):
$770M-$880M
– “Toyota said Thursday that a software problem is causing problems with the brakes”…
CNNMoney, Chris Isidore, senior writer, Feb 4, 2010.
GM presses suppliers for future recall costs (Automotive News, Aug 5, 2013)
– GM has adopted a new purchasing contract that would allow it to recover the cost of safety
recalls from its suppliers.
– The new GM contract has open-ended implications, stating that the supplier's components
"will not, at any time (including after expiration or termination of this contract), pose an
unreasonable risk to consumer or vehicle safety."
6
© 2012 IBM Corporation
7. Software and Systems Engineering | Rational
So… What is Quality?
Exceeding end users needs, even when they keep changing.
No single definition, most definition are very detailed and
verbose… but the common theme is most definitions is:
Yesterday: Meeting the Spec.
Today: Meeting or exceeding customers and endusers needs and expectations
– “End users needs” ≠ “Initial Requirements”
“The test of innovation lies not in its novelty, its
scientific content or its cleverness.
It lies in success in the marketplace.”
Peter Drucker.
(Business Management Philosopher)
7
© 2012 IBM Corporation
9. Software and Systems Engineering | Rational
Continuous Verification & Validation Process
End users needs, at least initially, are imprecise and incomplete – This is Normal!
Initial Elicitation
of end-users needs
Starts with early elicitation of end-users needs
Helping the customer/end user to flash out their needs (Requirements Elicitation)
– Imagine you want to build a new home, do you know the exact size of the house?
Notice that this results in Initial understanding of the needs, still partial and incomplete
Continuously
Verify…
Continuously
Validate…
Continuously verify, throughout
Development, that you are building the
system right.
•
•
As the builder checking with the architect.
As the city inspections, making sure the house is
being built in compliance with regulations.
Continuously validate with the
customer, as the design is getting more
concrete, that you are building the right
system.
•
As an architect and/or the builder validating
assumptions they have made as your house
is being built.
Continuously improving understanding of
customer/end-users needs
Delivery
© 2012 IBM Corporation
10. Software and Systems Engineering | Rational
Different perspectives
What all the stakeholders need
What the customer needs
What the customer thinks they need
What the customer says they want
Given this
How do we
get to this?
© 2012 IBM Corporation
11. Software and Systems Engineering | Rational
Verification vs. Validation Examples
While Verification is “Internal”, Validation is “External”
Verification: The evaluation of whether or not a product, service or system complies
with a regulation requirement, specification, or imposed condition. It is often an internal
process. (IEEE)
• Code should never run thru a divide by zero as it may cause a crash or unexpected results.
• Did you implement the design pattern properly?
• Does this block diagram represent the best architecture for the system?
Validation: The assurance that a product, service, or system meets the needs of the
customer and other identified stakeholders. It often involves acceptance and suitability
with external customers. (IEEE)
• On a cycling computer… The customer asked for certain data fields to be displayed… but
wouldn't the end user want to customize them?
• Is this {…} the right sequence of voice commands for making a phone call while driving
your car?
• What's the right behavior of the wifi enabled home dialysis machine when the connection is
intermittent?
11
© 2012 IBM Corporation
12. Software and Systems Engineering | Rational
Verification vs. Validation
While Verification is “internal”, Validation is “External”
Verification: Building the system right
– Testing against requirements
• “Internal” requirements
– Utilizes Test, Simulations, Analysis,
Inspections and Demonstrations
Customers
Needs
– Assuring robust design and implementation
Requirements
Analysis
System
Testing
– Compliance with industry regulations
– We “know” how to do that… often done well!
Validation: Building the right system
Systems Analysis
Systems Design
Integration
Testing
– Validating against your customers needs
• As expected, imprecise and incomplete
• Often negotiated with several teams
• Often negotiated with people who may not be
the end users
– Do we know how to do that? Do we do it?
12
Detailed Design
&
Implementation
Note that the V does not represent a
development process but rather a flow
of information as we decompose and
recompose a system.
© 2012 IBM Corporation
13. Software and Systems Engineering | Rational
Verification vs. Validation
While Verification detects defects, Validation is about avoiding them.
Left
side of the V
Verification
• Early defect
removal
Right
side of the V
• Expensive
defect
removal
Customers
Needs
System
Testing
Requirements
Analysis
Validation
13
• Defect
Avoidance
• Early
detection of
potentially
business or
mission
failures
• Early
Acceptance
tests
• Final
Acceptance
test
Systems Analysis
Systems Design
Integration
Testing
Detailed Design
&
Implementation
© 2012 IBM Corporation
14. Software and Systems Engineering | Rational
Verification vs. Validation
While Verification reduces Technical risk, Validation reduces Business or
Mission risk
Verification: Reduces Technical risks
– It is a Testing process…
Business
Risk
– We “know” how to do that… often done well!
Validation: Reduces Business risks
Technical
Risk
– Focuses on closing the loop with the end-user as
often as possible
• Can’t just blame the Requirements being wrong…
• “Meeting the spec” is not a guarantee for business
or mission success…
– It is NOT a Testing process…
but rather an on-going Discovery process
• Important to “developers”
• Critical to End Users – Helping users figuring out their own Requirements!
– Do we know how to do that? Do we do it?
14
© 2012 IBM Corporation
15. Software and Systems Engineering | Rational
Continuous Verification and Validation (V&V)
Compressing time to customers feedback!
Very
Common:
Customers
Requirements
Late Validation (Re-Evaluate)
Long/Costly
Process
System
Testing
Requirements
Analysis
System
Testing
Requirements
Analysis
Systems Analysis
Systems Design
Integration
Testing
Systems Analysis
Systems Design
Requirements
Analysis
Integration
Testing
System
Testing
Systems Analysis
Systems Design
Integration
Testing
Detailed Design
&
Implementation
Detailed Design
&
Implementation
Detailed Design
&
Implementation
Example:
AA and United approached Boeing for a new flight entertainment system
Job has been sub’ed to Sony Trans Com., who designed & delivered a state-of-the-art system.
End result:
– Premium paying passengers had trouble operating the system
– Flight attendants had trouble supporting the passengers
End-End result: Flight attendants shutting down the system before take off, claiming malfunction.
© 2012 IBM Corporation
16. Software and Systems Engineering | Rational
Continuous Verification and Validation (V&V)
Compressing time to customers feedback!
Late Validation (Re-Evaluate)
Very
Common:
Long/Costly
Process
Customers
Requirements
System
Testing
Requirements
Analysis
System
Testing
Requirements
Analysis
Systems Analysis
Systems Design
Integration
Testing
Systems Analysis
Systems Design
Integration
Testing
Requirements
Analysis
System
Testing
Systems Analysis
Systems Design
Integration
Testing
Detailed Design
&
Implementation
Detailed Design
&
Implementation
Detailed Design
&
Implementation
Continuous V&V
per Feature:
Use
Scenarios?
Validation
Operational
Scenarios?
Validation
Prototypes?
Validation
HIL/SIL?
Validation
Final?
Validation
Customers
Needs
Verification
16
Verification
Verification
Verification
Verification
© 2012 IBM Corporation
18. Software and Systems Engineering | Rational
Collaboration through Reviews
Testing for early feedback from Internal and External stakeholders
Review process where peers and other stakeholders collaborate in order to identify potential
errors and deviations from the spec.
Objectives include:
– Detect wrong understanding, bad assumptions and defects close to when introduced (Functional review)
– Identify potential improvements and/or risks (Functional review)
– Verify compliance with regulations and conformance to standards (Quality review)
Audience:
– External Reviews: With customers and/or end-users,
often focused on Validation
– Internal Reviews: With internal stakeholders (peers, management),
often focused on Verification
Formal inspections (with defined roles &
process) to ad-hoc team reviews and walkthroughs.
FDA Example: On Going Reviews / Late Validation
20
© 2012 IBM Corporation
19. Software and Systems Engineering | Rational
Reviews are collaboration points
Include the voice of the customer for on-going validation
Applies to the full development life cycle and to all artifacts
Reviews are short, and per feature (best practice: focus on end user feature)
Examples:
Review
Internal/External
Requirements elicitation
Identifying Use
Scenarios
External Review
What are the main
functions/activities?
Requirements Review
Defining Operational
Scenarios
External Review
How does the system
operate?
Architecture Design
Review
Is this the “best”
architecture?
Internal Review
Design documents
Initial Prototype Review
Early feedback based
on feature execution
External review
Low Fidelity prototype,
early feedback.
Advanced Prototype
Review
Production code in
HIL/SIL environment
External Review
Hi Fidelity prototype,
still early feedback.
Test Plan Review
Agreement across all
stakeholders
Internal and/or External
Review
Sunny & Rainy days
scenarios
Test Execution Review
21
Objective
Discuss Quality status
and next actions
Internal and/or External
Review
Focused on high level
operational scenarios
© 2012 IBM Corporation
20. Software and Systems Engineering | Rational
Reviews Impact
Reviews are ~2x more efficient in removing defects than Testing
(Not including Defect Avoidance)
Reviews
Testing
Source: Capers Jones, 2012
22
© 2012 IBM Corporation
21. Software and Systems Engineering | Rational
Reviews Impact
Reviews are ~2x more efficient in removing defects than Testing
(Not including Defect Avoidance)
Reviews
Testing
23
© 2012 IBM Corporation
22. Software and Systems Engineering | Rational
End Users Operational Scenarios
Use them as top level test cases to continuously drive all V&V activities
Operational Scenarios
– Quasi formal but yet intuitive
– Can be extended with sketches and
Interaction Diagrams
Operational Scenarios are a Testable version
of the requirements!
– Can be converted into System level test cases
System level test cases:
– Should be Validated (reviewed and approved)
with the customer and/or end-user
– To be used as a Golden Reference model
throughout design and implementation for
Verification purposes.
Although diagrams shown here are SysML diagrams,
concept applies to any system.
24
© 2012 IBM Corporation
23. Software and Systems Engineering | Rational
To leverage System Level Test Cases, you need…
Use Operational scenarios as Tests throughout the whole development Lifecycle
Validate your understanding of the needs by reviewing the expected scenarios with the
stakeholders and getting them approved.
A Modeling environment that support Use Case analysis
A Quality Management process that support the conversion of Operational scenarios into
test cases
– Test cases for either Manual Testing or Automated Testing
– May require internal review and approval
Traceability: Needs Requirements Operational Scenarios Test Cases
Operational
Scenarios
System
Level
Test
Cases
Real
Prototype
25
Virtual
© 2012 IBM Corporation
24. Software and Systems Engineering | Rational
Example User Scenario
Boat
loaded
Boat lifted
Boat on car
Ready to
sail
Boat
unloaded
sequential
Boat
rigged
parallel
To have
sailed
and
survived
Mast rigged
Center-plate
rigged
Rudder rigged
Gybed
Subgoals
Sailed
normally
Tacked
alternate
Overal
l
goal
Sailed
periodic
Returned
home
Cruised
Maneuvered
alternate
Ashore
Recover
from
Capsize
Righted
exception
Coast
guard
contacted
© 2012 IBM Corporation
25. Software and Systems Engineering | Rational
Requirements from Scenarios
Goal hierarchy
Boat
loaded
Boat lifted
traceability
Two people shall be able
to lift the boat onto the
roof of the average
saloon car.
Boat on car
Ready to
sail
Boat
unloaded
sequential
Boat
rigged
parallel
To have
sailed
and
survived
Stakeholder
requirements
Mast rigged
Center-plate
rigged
The sailor shall be able to
perform a gybe.
Rudder rigged
Gybed
Subgoals
Sailed
normally
Tacked
alternate
Cruised
Maneuvered
Sailed
Overall
goal
periodic
Returned
home
alternate
Ashore
Recover
from
Capsize
exception
Righted
Coast
guard
contacted
The sailor shall be able to
contact the coastguard
when the boat is
capsized.
© 2012 IBM Corporation
26. Software and Systems Engineering | Rational
Virtual Models
Enabling early Verification and Validation… before building anything real.
Virtual models that can be used for early analysis and trade studies (functionality, behavior,
architecture, structure, performance, reliability, safety, etc.)
– Models abstract mechanics, electronics and/or software entities, operating independently or integrated
– Different domains:
• Mechanical/Control models (such as Mathworks, NI)
• Graphical languages (such as SysML)
• Textual language (such as SystemC for HW design)
Can be used for
– The system being designed AND its TestBench (Plant models)
Use System level test cases to drive analysis
– Execution, Simulation or Prototyping
Various configurations driving on going system analysis:
• Eg. 1: Virtual sensors driving virtual HW+SW models
• Eg. 2: Mechanical prototype driving virtual HW+SW models
• Eg 3.: Mechanical prototype driving virtual HW+production SW
28
© 2012 IBM Corporation
27. Software and Systems Engineering | Rational
Modeling has low potential of introducing defects
Models are best in removing defects
Models are Executable
– You can ask Questions and get consistent
answers!
Models can be Analyzed
Model Based:
Low Defect Potential,
Best Removal Efficiency
– Tradeoff studies
– Performance
– Behaviors
– Etc.
Leverage correct-by-construction Synthesis
capabilities to guide the design and
implementation process
Source: Capers Jones, 2012
– Eg. 3D printers, C/C++ for SW, SystemC for HW
– Enabling Co-Execution for V&V purposes
29
© 2012 IBM Corporation
28. Software and Systems Engineering | Rational
Effective and Efficient Testing
Single source of truth for all Quality related artifacts and processes
Central Quality Management Hub, acting as a single source of truth, enabling:
– Collaborative Test Planning
• Development and approval for Verification and Validation test plans
– As well as specific plans as needed, such as Safety.
• Traceability back to Requirements (why we need which test?)
– Managing test execution (manual and automated) according to Test Plan
• Who is running which test, when, why, pre-conditions, test lab/equipment reservation etc.
– Integrated Defect Management
• What went wrong, how to reproduce, defect resolution, reporting, etc.
– Traceability across the full lifecycle
• User Needs Requirements Test cases Test Results Defects Resolutions
Other key capabilities
– Collaboration around Rainy days scenarios
– Test assets ReUsability
– Reconciliation process between Requirements and Test Plans/Test Cases
30
– Actionable Analytics to drive process improvements
© 2012 IBM Corporation
29. Software and Systems Engineering | Rational
Overall Impact on Testing Effectiveness
Collaborative Test Planning, Traceability, Test Execution and Analytics
Impact of Quality Management on Process Efficiency
100%
85%
90%
76%
80%
70%
58%
60%
85% 87%
Detect 3,
release 7
75%
60%
Detect 3,
release 2
50%
40%
32%
30%
30%
20%
15%
10%
0%
1
QM Impact:
20%
2
3
40%
40%
CMMI Levels
4
5
40%
10%
W/O QM
W QM
W/O QM: Percentage of defects detected during Requirements review, design
reviews, unit testing and Functional testing – Current practice
W QM: Percentage of defects detected thru Requirements review, design
reviews, unit testing and Functional testing – Improved practice
31
Source: Analysis of over 850 GBS projects and Industry data
© 2012 IBM Corporation
31. Software and Systems Engineering | Rational
Key Take-Aways
Verification in Inbound; Validation is Outbound. Both needed.
Verification is a Testing process. Validation is a Discovery process, helping the customer
refine their understanding of their own needs.
Compressing time to feedback allows for change… and therefore reduces surprises.
Reduce business risk with Early and Frequent feedback from all stakeholders
– Use Scenarios and Operational Scenarios
– Validation Test Plan
– Low Fidelity and Hi Fidelity prototypes
Operational scenarios = Testable Requirements = Golden reference
– Use them for your Testing throughout the whole development process and not only at the end!
Question:
Unit Testing and Integration Testing are both very important.
Which is more important?
Which one would you do first?
34
© 2012 IBM Corporation
32. Software and Systems Engineering | Rational
Continuous V&V – Reducing Technical and Business Risks
Quality can be delivered by compressing the time to customer feedback with
continuous Verification and Validation.
Both Technical and Business risks can be lowered
Business Implications:
– Lower cost of Quality
Business
Risk
Technical
Risk
• Defect Avoidance, Lower defect density, higher defect removal efficiency
– Deliver the right system
• Compressing time to customer feedback
– Better time to market predictability
• Earlier convergence on the right Requirements
35
© 2012 IBM Corporation
34. Software and Systems Engineering | Rational
www.ibm.com/software/rational
© Copyright IBM Corporation 2013. All rights reserved.
The information contained in these materials is provided for
informational purposes only, and is provided AS IS without
warranty of any kind, express or implied. IBM shall not be
responsible for any damages arising out of the use of, or
otherwise related to, these materials. Nothing contained in
these materials is intended to, nor shall have the effect
of, creating any warranties or representations from IBM or its
suppliers or licensors, or altering the terms and conditions of
the applicable license agreement governing the use of IBM
software. References in these materials to IBM
products, programs, or services do not imply that they will be
available in all countries in which IBM operates. Product
release dates and/or capabilities referenced in these
materials may change at any time at IBM’s sole discretion
based on market opportunities or other factors, and are not
intended to be a commitment to future product or feature
availability in any way. IBM, the IBM logo, Rational, and other
IBM products and services are trademarks of the International
Business Machines Corporation, in the United States, other
countries or both. Other company, product, or service names
may be trademarks or service marks of others.
37
© 2012 IBM Corporation
Notas do Editor Reality is that the definition of Quality is changing… low tolerance for inadequate quality, and even Tesla is now delivering updates over the air.To deliver quality, we need to take a holistic approach to Quality, where we continuously verify and validate our designs, throughout the whole development life cycle, and not just at the right side of the V, or even worse – at the top right side of the V.I will show you how to do it, so that you will be more predictable in your delivery schedule, you’ll deliver the right system and at a lower cost of quality. I must here that when I talk about Defect, I mean it in the ISO 9000 sense, which is non-fulfillment of intended usage requirements. It is being used in the general sense, covering product defect, design defect, manufacturing defect, an error, a fault or a failure. Let’s look at the agenda.By now, you already know that you need to take a holistic approach to Quality. We will take a quick look at some recent examples of inadequate quality, talk about Cont V&V… then key capabilities that are needed to deploy it successfully, and then we will take a closer look at how the Rational Systems and Software Eng solution supports Cont V&V.Let’s look at some examples. Bad quality can turn into penalties. In the government determines that the test failed due to quality issues at Raytheon, that could result in major financial consequences to Raytheon. Wall street is watching.Reminds me of a meeting with another contractor, a missile test failed and they were supposed to pay heavy penalties. Like here.The contractor was able to demonstrate full traceability from Requirements to Test plans to Test Cases, very detailed. Therefore they could analyze all their tests, back to Requirements, only to find that the DoD executed a test that was outside the tolerances that were specified in the Requirements. And indeed – the Test failed – as it should have!In this case, Traceability, and being able to demonstrate it, saved this contractor a lot of money, including not being at risk of loosing future projects awards.We never expected this to be a value prop, but to this contractor, this was a major one! Bad quality hurts your stock valuation. Like in the serial Sony recalls. The Prius gas pedal costs Toyota $3B, only a third was for repair cost, 2/3rds were business related.And we start to see different corporate behaviors… GM is trying to pass on cost of recalls to their suppliers. Remains to be seen how suppliers will respond… So, what’s a good quality?While there are many definitions to Quality, the common aspects to most of them is meeting or exceeding customers and end users needs and expectations. Look at the mp3 players… all met the specs… non is around today….And end users keep changing! Let’s look at the agenda.By now, you already know that you need to take a holistic approach to Quality. We will take a quick look at some recent examples of inadequate quality, talk about Cont V&V… then key capabilities that are needed to deploy it successfully, and then we will take a closer look at how the Rational Systems and Software Eng solution supports Cont V&V.Let’s look at some examples. Process quality assurance (automotive spice or CMMI) Processes are written according to some process framework. Does our product development lifecycle complies with these processes Cycling computer. Desinged for cyclers. Customer: Garmin, focused specifically on distance, pace, time. What about people doing cycling for loosing weight? Calories? Types of Verification: Analysis,inspection, review, test, demo, COC (Certification of Conformance, Compliance) Testing code that will eventually reside in a black box without the actual hardware is called Software-in-the-Loop (SIL). Installing software in an actual device and testing that device in a virtual environment is called Hardware-in-the-Loop (HIL). If human interaction is desired or required, the test also includes a Man-in-the-Loop (MIL) component. Let’s look at the agenda.By now, you already know that you need to take a holistic approach to Quality. We will take a quick look at some recent examples of inadequate quality, talk about Cont V&V… then key capabilities that are needed to deploy it successfully, and then we will take a closer look at how the Rational Systems and Software Eng solution supports Cont V&V.Let’s look at some examples. Food and Drug Administration Let’s look at the agenda.By now, you already know that you need to take a holistic approach to Quality. We will take a quick look at some recent examples of inadequate quality, talk about Cont V&V… then key capabilities that are needed to deploy it successfully, and then we will take a closer look at how the Rational Systems and Software Eng solution supports Cont V&V.Let’s look at some examples. End users needs, at least initially, are never precise or complete, therefore we need to help the end user to flash them out. Like if you were to build a new home, would you know up front the exact size of the home and how many rooms you need? No… but the architect will help you figure this out. Same here.As you go down the V, you will continuously do both Verification and Validation. Verification is where you verify that you are building the system right. Like the builder checking with the architect, or the city doing inspections that you build to code.But you will also do continuous Validation…. Not wait to the end, like the builder and/or the architect coming back to you to check with you as they make more decisions on your dream home.