Presentation of the Model Based Testing process, enhanced with real automotive OEM and T1 customers feedback.

1. Results of Model-Based Testing
deployments performed for the Automotive
Embedded Systems Validation
MaTeLo
Boost your test efficiency

2. AGENDA
 ALL4TEC MBT History
 Automotive Market
 Automotive Electronic Challenge
 Automotive Electronic Testing
 ALL4TEC Customer Slide
 Why MBT in Automotive
 Usage Model Design
 Test Case Generation
 Test Execution
 Test Campaign Analysis
 Results
MaTeLo
Boost your test efficiency

3. MaTeLo History
 10 years of MBT
 Starting in Automotive with Magneti Marelli
 MaTeLo as “Markov Test Logic”
1
•Model with
INI File
2
•Model with
PowerPoint
3
•Test Campaign
Analysis
4
•Test Automation
•NI TestStand
•EXAM
•Proprietary
•Requirements
•IBM Doors
•XML, CSV
•Test Management
•HP QC
•Test Oracles
•Simulink
•Scilab
•Python
5
• Eclipse

4. ALL4TEC MBT Experience in Automotive
 Instruments Clusters
 Multimedia Unit
 Seat Belt ECU
 Display
 Body Control Unit
 Engine Control Unit
 Chassis ECU
 Climate Control
 Stop & Start
 Electrical Lateral Door
 ….
intelligent-vehicles.

5. Automotive Electronic
N Lines of Codes
Ni.com
Software Engineering
Economics
10 M
40 %
*1000
80 M

6. The Simple Automotive World
Standard
Tools
Safety
Product Lines
Platform
Car Variant
Complexity
Process
CMMI Spice
AutoSar
HIS
Vector
ETAS
Mathworks
dSPACE
NI
Menthor
OSEK
MISRA
UDS
ASAM
ISO26262
Multi OEM
ECU
ESP
VW, AUDI, PORSCHE,
SKODA, SEAT,
GM, CHEVROLEY, OPEL,
PEUGEOT, CITROEN
A Heavy Truck
is almost unique
EMS
NAVIGATION
1-10 Networks
20-110 ECUs
x0 000 signals x km of wire
x00 000
Requirements

7. ECU 1
An Integration Market !
ECU A
ECU 1
ECU 4
ECU 2
ECU 3
ECU C
ECU B
OEM 1 OEM 2
T1 A
COMPETITION
Parts
Suppliers
Vehicle
Manufacturers

8. OEM/T1 Integration V Cycle
Parts
Suppliers
System
Definition
Architecture
Design
ECU
Requirements
System
Design
System
Validation
ECU
Verification
System
Verification
Product
Validation
Software
Development
INTEGRATION
Vehicle
Manufacturers

9. Test Level ?
Parts
Suppliers
System
Definition
Architecture
Design
ECU
Requirements
Architecture
Design
System
Validation
ECU
Verification
System
Verification
Product
Validation
Software
Development
System
MIL
Feature
SIL
Feature
MIL
Feature
PIL
ECU
HIL
ECU
HIL
System
HIL
Field
Vehicle
Manufacturers

10. Test Object ?
Parts
Suppliers
System
Definition
Architecture
Design
ECU
Requirements
Architecture
Design
System
Validation
ECU
Verification
System
Verification
Product
Validation
Software
Development
System
MIL
Feature
SIL
Feature
MIL
Feature
PIL
ECU
HIL
ECU
HIL
System
HIL
Field
Multi Domain
Model
Software
Model
Software
Embedded
Code
ECU
Electronic
System
Vehicule
Vehicle
Manufacturers

11. The Testing Spectrum
Embedded SoftwareControl Logic
Software Level Testing
Control Procedure Level Testing
Network Level Testing
Diagnostic Level Testing

12. Automobile Functional Integration HIL
ATZ Magazine

13. Journée Utilisateurs MaTeLo 2012
> 100 peoples

14. Why MBT (1)

15. Why MBT (2)

16. MBT in the V Cycle

17. MBT in the Agile Cycle (Continuous Integration)

18. MATELO USAGE MODEL
(4) CONSISTENCY
TESTS <-> REQUIREMENTS
BASED ON USE CASES
(2) ALL STRATEGIC
USAGES SCENARIO
(3) FORMAL
REQUIREMENT
TEST SPECIFICATION
Enabler for
 Global Test Strategy
 Graphical Design/Review
 Automatic Generation
 Model Checking
 Impact Analysis
(1) BIG PICTURE

19. MATELO USAGE MODELS
BEGINNING
END
STATE
TRANSITION
MACRO CHAIN
 Setup “Transition” as “Test Step”
 Mapping of Requirements
 Configuration for Test Automation

20. Usage Model Design

21. REQUIREMENTS MANAGEMENT
MaTeLo Requirements Library MaTeLo Usage Model
Model &
Trace
(N,M)
IBM Rational Doors
Import
Requirement Update
New Baseline
Update
Quick
Model
Update
Requirements
Change
Analysis
Model
Impact
Highlighting

22. Stimulations
Inputs Stimulation
Equivalence Classes
Timing
Verifications Points
Expected Ouputs Results
Timing
MODEL TRANSITION = TEST STEP
Requirements
Test Automation
Select_Gear(5)
Accelerate(100%)
Check_Speed(185)
Computed Test Oracle
Outputs = f(Inputs)
Derived Test Oracle

23. MATELO PROFILE
 Every Alternative can be qualified by Weight
 Test Profiles
 Operational Profiles
347
0-100-150-200-500
87 (+/- 25)
Data distribution Path frequency

24. MATELO PROFILE
0-500 0-500 0-500

25. MATELO PROFILE
Profile 1
87 (+/- 25)

26. MATELO PROFILE
0-100-150-200-500 87 (+/- 25)
347
Profile 2

27. Automotive Product Lines
TFT
KM/H M/H
DOT MATRIX
Display
Country
Navigation
OEM
Required
Feature Products Line
Constraint
Variability Point
Product

28. MaTeLo Product Line Manager
TCG
TCG
TCG
TCG
MG
Product Usage Model
Derived from PL Model
Test Cases Generation
By Product Usage Model

29. THEORETICAL MaTeLo WORKFLOW
Automation
Test Framework
Test Scheduler
Simulator
System
Under
Test
Test
Execution
Log
Usage
Model
Test
Cases
Functional
Requirements
Atomic
Test
Action
Test
Cases
Verdicts
Confidence
& Coverage
Reports
Test
Scripts

30. Automatic Test Framework Setup

31. Test Automation Framework integration

32. MaTeLo Eco-System
TCL Proprietary Script
EXAM
TestStand - VeriStand
Java
CAPL
Python
VB
HTML
ATX
MaTeLo

33. Select the Test Cases

34. 4 Generation Strategies
depending on your Test Strategy
4 TESTS STRATEGIES
Most probable
approach
Start
Close
FREQUENCY
FOCUS
Start
Close
Risk based
approach
Custom
Test profile
CRITICALITY, COMPLEXITY
UPDATE FOCUS
Arcscoverage
approach
Close
Start
REQUIREMENTS
COVERAGE
Usage
Test profile
Usage
approach
Start
Close
OPERATIONAL
COVERAGE

35. New Test
Cases
Generation
TEST CAMPAIGN PROCESS
Test
Cases
System
Requirements
Usage
Model
Test Execution
Platform
System
Under
Test
Adaptor
Test
Scripts
Requirements
Traceability
Model
Coverage
Reliability
Mean Time
To Failure
Intensity
of Test
Test
Cases
Test
Cases
System
Requirements
System
Requirements
Usage
Model
Usage
Model
Test Execution
Platform
Test Execution
Platform
System
Under
Test
System
Under
Test
Adaptor
Test
Scripts
AdaptorAdaptor
Test
Scripts
Test
Scripts
Requirements
Traceability
Model
Coverage
Requirements
Traceability
Model
Coverage
Reliability
Mean Time
To Failure
Intensity
of Test
Reliability
Mean Time
To Failure
Intensity
of Test
Test
Cases
System
Requirements
Usage
Model
Test Execution
Platform
System
Under
Test
Adaptor
Test
Scripts
Requirements
Traceability
Model
Coverage
Reliability
Mean Time
To Failure
Intensity
of Test
Test
Cases
Test
Cases
System
Requirements
System
Requirements
Usage
Model
Usage
Model
Test Execution
Platform
Test Execution
Platform
System
Under
Test
System
Under
Test
Adaptor
Test
Scripts
AdaptorAdaptor
Test
Scripts
Test
Scripts
Requirements
Traceability
Model
Coverage
Requirements
Traceability
Model
Coverage
Reliability
Mean Time
To Failure
Intensity
of Test
Reliability
Mean Time
To Failure
Intensity
of Test
Test
Cases
System
Requirements
Usage
Model
TestExecution
Platform
System
Under
Test
Adaptor
Test
Scripts
Requirements
Traceability
Model
Coverage
Reliability
MeanTime
ToFailure
Intensity
ofTest
Test
Cases
Test
Cases
System
Requirements
System
Requirements
Usage
Model
Usage
Model
TestExecution
Platform
TestExecution
Platform
System
Under
Test
System
Under
Test
Adaptor
Test
Scripts
AdaptorAdaptor
Test
Scripts
Test
Scripts
Requirements
Traceability
Model
Coverage
Requirements
Traceability
Model
Coverage
Reliability
MeanTime
ToFailure
Intensity
ofTest
Reliability
MeanTime
ToFailure
Intensity
ofTest
Stopping Quality Criteria
Coverage
Reliability
Confidence Indicators
Test
Cases
System
Requirements
Usage
Model
TestExecution
Platform
System
Under
Test
Adaptor
Test
Scripts
Requirements
Traceability
Model
Coverage
Reliability
MeanTime
ToFailure
Intensity
ofTest
Test
Cases
Test
Cases
System
Requirements
System
Requirements
Usage
Model
Usage
Model
TestExecution
Platform
TestExecution
Platform
System
Under
Test
System
Under
Test
Adaptor
Test
Scripts
AdaptorAdaptor
Test
Scripts
Test
Scripts
Requirements
Traceability
Model
Coverage
Requirements
Traceability
Model
Coverage
Reliability
MeanTime
ToFailure
Intensity
ofTest
Reliability
MeanTime
ToFailure
Intensity
ofTest
Bugs
New
Package
Test
Cases
System
Requirements
Usage
Model
TestExecution
Platform
System
Under
Test
Adaptor
Test
Scripts
Requirements
Traceability
Model
Coverage
Reliability
MeanTime
ToFailure
Intensity
ofTest
Test
Cases
Test
Cases
System
Requirements
System
Requirements
Usage
Model
Usage
Model
TestExecution
Platform
TestExecution
Platform
System
Under
Test
System
Under
Test
Adaptor
Test
Scripts
AdaptorAdaptor
Test
Scripts
Test
Scripts
Requirements
Traceability
Model
Coverage
Requirements
Traceability
Model
Coverage
Reliability
MeanTime
ToFailure
Intensity
ofTest
Reliability
MeanTime
ToFailure
Intensity
ofTest
New version
Test
Cases
System
Requirements
Usage
Model
TestExecution
Platform
System
Under
Test
Adaptor
Test
Scripts
Requirements
Traceability
Model
Coverage
Reliability
MeanTime
ToFailure
Intensity
ofTest
Test
Cases
Test
Cases
System
Requirements
System
Requirements
Usage
Model
Usage
Model
TestExecution
Platform
TestExecution
Platform
System
Under
Test
System
Under
Test
Adaptor
Test
Scripts
AdaptorAdaptor
Test
Scripts
Test
Scripts
Requirements
Traceability
Model
Coverage
Requirements
Traceability
Model
Coverage
Reliability
MeanTime
ToFailure
Intensity
ofTest
Reliability
MeanTime
ToFailure
Intensity
ofTest
Requirements
Coverage
Functional Coverage
Operational
Coverage
Stopping criteria

36. Test Execution

37. Test Campaign Analysis

38. Results with a Powertrain Parts Supplier

39. Results with a Car Manufacturer

40. Results with a Body Supplier

41. Results with a Car Manufacturer

42. QUESTION ?
MaTeLo
Boost your test efficiency