The document discusses the theoretical foundations of source coverage analysis from execution traces. It outlines the original needs to provide accurate source coverage information and meet certification standards. Several challenges were encountered, including mapping statements and conditions from traces to source code given compiler optimizations. Formal modeling and model checking helped define source coverage metrics precisely and prove relationships between different coverage criteria. The results allowed developing open source tools that provide certified source coverage analysis for high integrity software development.

1. Theoretical foundation of source
coverage analysis from execution traces!
Thomas Quinot!

2. Summary!
• Original Needs & Goals!
• Challenges along the way!
• Main Results!

3. Original Needs!
• Structural Coverage Analysis is required by
certification standards:!
• Open source Coverage Tools exist but are not usable in a HI context!
• Proprietary Tools exist but do not support all versions of Ada!
• Complete the GNAT Pro Toolset for the High Integrity
Market!
• Better support for the rapidly evolving versions of
Ada (83 … 95 … 2005 … 2012 …)!

4. Original Goals!
• Provide an High Quality Open Source alternative to existing
proprietary tools!
• Provide Support for Agile/Lean Development!
• In particular: Continuous Integration/Certification!
• Open-DO initiative!
• Find the best compromise between Source and Object Coverage!

5. The Couverture Project (2008-2010)!
• One of the first FUI projects from the GTLL at System@atic!
• 4 partners (AdaCore, Openwide, Telecom PT, Paris 6)!
• Effort of 160 man-month (2,23 M€) over 2 years
• 45% Financed by the city of Paris, IdF region, DGE !
This project gave us the capability to meet
the unexpected challenges we were facing.

6. Object Coverage vs Source Coverage!
• Big debate in the Certification Community!
• Which one is the most Accurate / Appropriate ?!
• Which one is the most efficient ?!
Source
Object
- Statement/Decision are source concepts
- on final code (no instrumentation)
- usually works by instrumenting the code
- on final hardware
- can be done on fast native platforms
- not language specific
- requires double testing strategy
- more precise

7. Object Coverage vs Source Coverage!
• Object coverage metrics:!
• Instruction Coverage!
• Object Branch Coverage (OBC)!
• Source coverage metrics:!
• Statement Coverage!
• Decision Coverage (DC)!
• Modified Condition/Decision Coverage (MC/DC)
Independent influence of each condition within a decision!

8. Challenge 1!
!It is difficult to provide accurate source coverage info
from execution traces:!
! !- no trace of “statement” / “condition” / “decision” at
! binary level!
! !- optimization can change significantly the control flow

9. Accurate Source Coverage Info!
Sources
Sources
Sources
Sources GNAT Pro
Executable
Exec
decorated Exec
decorated traces
Exec
sources
decorated traces
sources GNATcoverage
traces
sources

10. Accurate Source Coverage Info!
Not sufficient to locate
precise statements, decisions,
or conditions boundaries
Sources
Sources
Sources
Sources GNAT Pro
Executable
Debug info
Exec
decorated Exec
decorated traces
Exec
sources
decorated traces
sources GNATcoverage
traces
sources

11. Accurate Source Coverage Info!
Source Coverage
Information
(Static analysis)
-fpreserve-control-flow
Executable
Sources
Sources Debug info
Sources Enhanced
Sources
GNAT Pro
SCOs
Exec
decorated Exec
decorated traces
Exec
sources
decorated traces
sources GNATcoverage
traces
sources

12. Challenge 2!
OBC does not imply MC/DC!
We need better theoretical foundations !

13. Initial ideas!
• General Belief at beginning of project :!
 Object Coverage => Statement Coverage!
 Object Branch Coverage => Decision Coverage!
 Object Branch Coverage => MC/DC (when using short
circuit operators)!
• But a FAA study arrived after the beginning showing
unexplainable differences between OBC and MC/DC
DOT/FAA/AR-07/17!

14. Elementary counter-example!
function P (A, B, C : Boolean) return Boolean is
begin Conditions
if ( A and then B ) or else C then
return True;
end if;
end P; Decision
OBC
MC / DC
A
B
C
if statement
A
B
C
if statement
T
T
F
T
T
T
x
T
A
F
T
F
F
T
F
T
T
C
B
F
T
T
T
F
x
F
F
T
F
F
F
3 tests are sufficient
At least n+1 tests
n = number of conditions

15. Counter-measures!
• Definition of a formal model to express
coverage metrics based on BDD (Binary
Decision Diagram)!
• Express OBC and MC/DC in this model!
Use
• Find counter-examples! Open Source
Model Checker
• Find precise perimeter where the Alloy
equivalence can be proven!
• Formally prove this result!

16. Evaluation of short circuit boolean expressions!
• Evaluating a short circuit decision is a traversal
of its Reduced Ordered Binary Decision Diagram!
• Each ROBDD node is a test for a condition!
• Evaluate conditions left to right!
• Do not evaluate RHS if LHS is sufficient!
• A condition vector denotes a path trough the ROBDD!
A
T
F
( A and then B ) or else C
F
B C
T
T
F
T
T
F

17. More counter-examples (Alloy)!
A
function P (A, B, C : Boolean) return Boolean is
begin T
F
if ( A and then B ) or else C then
return True; B
F
end if;
end P; T
C
T
F
T
F
BDD
C0
T
F
function P (C0, C1, C2, C3, C4 … : Boolean) return Boolean is C1
begin C2
F
if ((((…(C0 and then C1) or else C2) and then C3) or else C4 … T
T
F
then
return True; C3
end if; F
C4
end P; T
F
T

18. Pathological case!
(((C0 AND THEN C1) OR ELSE C2) AND THEN C3) OR ELSE C4...
• N conditions
• MC/DC requires at least N + 1 tests
• OBC can be achieved in 3 tests!
C0
T
C1
F
T
F
C2
T
C3
F
T
F
C4
T
F
T
T
F
3 tests sufficient instead of N+1, for any N

19. What does that mean?!
• For a given test campaign!
• OBC (BDDBC) are local properties of each BDD
node: stateless (union of all paths are covering
the BDD)!
• MC/DC is a property of trajectories taken through
the ROBDD: stateful (all paths through the BDD
are taken)!
• In general MC/DC requires complete history of each
conditional branch instruction (each BDD node)!
• Are there specific cases where we can do better?!

20. Equivalence can be proven when !
• There are no diamonds in the BDD (nodes that can
be reached through multiple paths)!
• How does this translate in “User Terms” ?!
• No easy formulation… the best we found is!
• Transform Boolean expression in “Negative
Normal Form”!
• No “and then” in left operand of a “or else”!
• No “or else” in left operand of a “and then”!

21. Proof sketch!
• In the no-diamond case, each path covers a distinct
terminal edge
⇒ all terminal edges covered implies all paths
covered, MC/DC is achieved!
• If thereʼs a diamond, we construct a covering path
set that fails to show independent influence of one
condition (all paths through that condition have the
same outcome)!
A
T
F
B
F
T
C
T
F
T
F

22. Main Results!
• Emulation is key to Agile cross development!
• GNATcoverage takes advantage of the theoretical
results to:!
• Implement properly MC/DC in the complex case!
• Optimize the simple case by using OBC!
• Definition of specific compilation artefacts (SCOs)
and of a certification-friendly code generation mode
in GCC (-fpreserve-control-flow)!
• Creation of “open source” qualification material as
part of Open-DO!

23. Conclusion!
• The Couverture project allowed us to concentrate on
solving properly the unexpected challenges!
• Existing Open-Source technologies have played a
key role:!
• Qemu is the base of GNATemulator!
• Alloy helped a lot for the mathematical proofs!
• As a result, new industrial-ready Open Source tools
are now available for the HI developersʼ community !