Confidence in software tools rests on the effectiveness of tool verification – essentially, asking the right questions. To determine the right questions for WCET tools, the full presentation includes our WCET tool test effectiveness framework and explains how it influences our tool testing.

1. Timing Tool Test
Effectiveness for WCET
Analysis Tools
Avionics Europe 2013
Zoë Stephenson
zoe.stephenson@rapitasystems.com

2. Confidence in software tools rests on the
effectiveness of tool verification –
essentially, asking the right questions.
To determine the right questions for WCET
tools, this presentation includes our WCET
tool test effectiveness framework and
explains how it influences our tool testing.

3. Overview
The presentation is split into five distinct
sections:
Motivation – WCET analysis in context
Obtaining confidence in tools
How timing tools are used
How we evaluated our own testing efforts
What the evaluation means for RapiTime

4. MOTIVATION

5. Confidence
How do we show that we have
effective tests of a timing tool?

6. Context: general SW development
The V model: requirement, software, test result
evaluate
Timing Timing
Requirement Evidence
improve
develop test
Software

7. Context: timing requirements
This is where we introduce a timing analysis tool and
method to check that the software meets those
requirements
evaluate
Timing Timing
Requirement Evidence
improve
develop test
Timing Analysis
Software Tool + Method

8. Context: timing requirements
Now we run tests to ensure the tool is operating correctly
(according to its TOR)
evaluate
Tool
Operational Tool Verification
Requirements Results
improve
develop
test

9. Context: tool test effectiveness
Now we review whether the test suite for the tool has
sufficiently exercised the tool
evaluate
Criteria for Test Test
Effectiveness Effectiveness
improve
develop review

10. How RVS aids testing
RVS
RapiTime
collect data on-target during execution
transmit data to host computer
combine with program static analysis: beyond end-to-end tests
report test coverage and potential untested worst-case behaviours
direct tool user to define more comprehensive tests

11. CONFIDENCE

12. DO-178B says…
The objective of the tool qualification process is to ensure that
the tool provides confidence at least equivalent to that of the
process(es) eliminated, reduced or automated.

13. Qualification context
The background and what we are concerned with
for effectiveness of the test of RapiTime (circled in
green)
Tool User
Tool Use Cases
Tool Qualification Tool Accomplishment
Plan Summary
Tool Development Tool Testing
Tool Vendor

14. Tool testing: effective
A tool test is effective if it can distinguish a tool that meets
the requirements from a tool that does not
correct tool tool testing accept
incorrect tool tool testing reject

15. Tool testing: representative
If a test is representative, you can infer real-world
correctness from correctness during the test
tool tool testing accept
tool real world accept

16. TOOL USAGE

17. Timing tool usage
Adding detail to the model of the tool and its usage
highlights the factors that are considered in the
assessment
Tool Verification
Timing Analysis Tool + Method Results
user
test
target
Tool Test / Analysis Suite
tool
test test input
procedures programs

18. ASSESSMENT

19. Assessment approach
Usage Model Undesired Outcome
custom SHARD /
HAZOP process
adjust testing Causes and
Mitigations
requirements
derivation
∆ Test plans Conditions of Use

20. Guideword selection
Applicability of general guidewords for test effectiveness:
Guideword Application
No Test not present / not done
More Over-constrained analysis, cases missed
Less Shallow test, cases missed
Part of Incomplete test, not whole programs
As well as N/A
Reverse N/A
Error Test claims tool works, but it does not
(and similarly for other artefacts and flows)

21. Top-level analysis
Provide both procedures and review criteria for test selection and
customisation
Test procedure review criteria:

from “less”
 Depth
 Generalisability from “more”
 Completeness
General tool derived requirements: from “part of”
 If main tool calls further tools, propagate back error return code

22. Timing tool analysis
structural diversity threads, tasks,
schedules…
subprograms:
direct, indirect…
blocks, entries,
exits…
selection, loop,
nesting…
…with expected
expressions, execution times
decisions…
compound
statement, exits…

23. Timing tool analysis
Testing must be applicable across hardware features that
lead to variations in execution time at different scales
execution time
diversity
bus interaction denormalized
branch prediction
with peripherals numbers
representing the
deployed system

24. RapiTime analysis
Creating software with a known execution time
Independent time Other calibrated
source delay
Ensuring that on-target measurement is
representative
Target measurement Target measurement Target measurement
library testing for multicore for ARINC 653
Helping the user to manage the execution time
analysis
Traceability by Workflow to validate
configuration ID annotations

25. RAPITIME IMPACT

26. Strengthened confidence
RapiTime works on large programs, peculiar
code structures, a variety of OS styles
RapiTime works with a large range of data
collection and extraction mechanisms
RapiTime provides comprehensive traceability
mechanisms for observed measurements and
computed execution times

27. Improving tool offering
New integration possibilities for multicore
and time-partitioned systems
More comprehensive assessment advice
for different target hardware and
measurement capabilities
More flexible verification kits for on-site
tool qualification

28. New features in RVS 3.1
Graphical report comparisons help to
show where a test effective in the lab falls
short on site
Wider range of path highlighting facilities
show WCET path deviations at a glance
Commandline data export to CSV, XML
and text formats help to trace between
tool assessment and individual tests

29. Summary
Motivation - what do we want to test?
Confidence - how do I assess tools?
Tool usage - how do I use timing tools?
Evaluation - How do we evaluate our efforts in
testing RapiTime?
Impact - how has the evaluation affected RapiTime?