Using Tiers of Assurance Evidence to Reduce the Tears! Adopting the “Wheel of Qualification” for an Alternative Software Safety Assurance Approach

15 November 2019
© Crown copyright 2019 Dstl

Using Tiers of Assurance Evidence to
Reduce the Tears!
- Adopting the “Wheel of Qualification” for an
Alternative Safety Assurance Approach
Dr Mark Hadley CEng MIET & Mike Standish MSc CEng MBCS MINCOSE
Dstl, Software and Systems Dependability Team
DSTL/PUB118156. © Crown copyright (2019), Dstl. This material is licensed under the terms of the Open Government Licence except
where otherwise stated. To view this licence, visit https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3 or write
to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email: psi@nationalarchives.gsi.gov.uk.
15 November 2019

Disclaimer
This presentation is an overview of UK MOD sponsored research and is
released for informational purposes only. The contents of this presentation
should not be interpreted as representing the views of the UK MOD, nor
should it be assumed that they reflect any current or future UK MOD policy.
The information contained in this presentation cannot supersede any
statutory or contractual requirements or liabilities and is offered without
prejudice or commitment.
.
15 November 2019

Open Government Licence (OGL)
• Content published under the OGL; see the title slide. OGL includes, but
is not limited to, the following…
• You are free to:
– copy, publish, distribute and transmit the Information;
– adapt the Information;
– exploit the Information commercially and non-commercially for example, by
combining it with other Information, or by including it in your own product or
application.
• You must (where you do any of the above):
– acknowledge the source of the Information in your product or application by
including or linking to any attribution statement specified by the Information
Provider(s) and, where possible, provide a link to the licence.
.
15 November 2019

What’s To Come
.
Problem
Space
What a
Proposed
Solution
Offers
Overview of
the “Wheel of
Qualification”
Generating
the
Evidence
Caveats to
the
Approach
Putting
the
Approach
into
Practice
Next Steps
15 November 2019

The Problem Space
.
15 November 2019

Software Safety Assurance (MOD)
Design and Airworthiness Requirements for Service Aircraft
DS 00-970, Part 13, Issue 11, 1.7 Safety Related Programmable Elements
Reqts for Safety of
Programmable
Elements (PE) in
Defence Systems
DS 00-55, Issue 4
MOD - Ministry of Defence
DS - Defence Standard
Reqt - Requirement
ARP - Aerospace Recommended Practice
CEH - Complex Electronic Hardware
Safety Assessment
process as per ARPs
4754A / 4761
Safety Assessment
report as per DS 00-56,
Issue 5
Cyber Security via
RTCA DO-326A and
RTCA DO-356
Safety Related
Software via DO-178C
(and supplements)
Safety Related CEH via
DO-254
acceptable
Additional
supporting
standards as
required (e.g.
DO-200B
Aeronautical Data)
15 November 2019
Legacy systems can be legitimately qualified to prior standards/guidelines

You Can’t Always Get What You Want
• There is a defined system for making judgements on qualitative and quantitative
process evidence to determine if a software level has been achieved (e.g. DO-
178C).
• However, not all scenarios allow for qualitative and quantitative process
evidence to be judged by those that need to. This can be due to a multitude of
reasons:
• Systems of interest can be, but not be limited to: Military-Off-The-Shelf (MOTS),
Commercial -Off-The-Shelf (COTS), Foreign Military Sales (FMS) equipment, or novel
(i.e. not subject to previous certification).
• Military systems may not be developed to standards/processes which are recognised by
extant regulatory requirements.
• Legacy systems may not have the availability of artefacts to be reviewed (but they may
have in-service data!).
• Limited access to information due to Intellectual Property Rights (IPR) and/or
International Traffic in Arms Regulations (ITAR) (e.g. low-level firmware design).
.
15 November 2019

The Focus of the Problem
• “Brownfield” developments which have pre-existing systems with (or
without) prior qualification.
• Qualification of the overall platform from a software and CEH
perspective. Focus is not to solely qualify an individual Line
Replaceable Unit (LRU) but a collection of relevant LRUs (although
there is a need to assess individual LRUs!).
• Mitigation of shortfalls in the evidence against extant regulatory articles.
However, there is a need to have a valid approach in its own right.
• There is no “cookbook” for the differing assurance activities. Multiple
levels of diversity is the strongest form of evidence and any qualification
approach should embrace this concept.
CEH - Complex Electronic Hardware
15 November 2019

A Proposed “Solution”
.
15 November 2019

Principles
• All evidence is of relevance although there will be different weightings.
• Current approaches use quantitative and qualitative process artefacts to
judge confidence to a defined software level. A revised approach is to
use this evidence and to adopt a wider set of evidence.
• Need to take a platform and system level perspective. The approach
should adopt the wider system elements (e.g. integration testing, flight
trials, life-cycle reviews).
• Acknowledging the evidence of systems to create dialogue. An
approach should create an open dialogue regarding the availability of
evidence and any relative strengths.
.
15 November 2019

Moving Away From “Labels”
• Revised approach moves away from a “tick-box” compliancy process to
a more engineering and independent assurance driven approach.
Approach still includes the cornerstones of the regulatory articles (e.g.
acceptable means of compliance such as DO-178C).
• Not an objective based approach but is based upon Subject Matter
Expert (SME) judgement.
• Evidence can take many forms in the revised approach – this can
create a complicated solution space.
• The quantity and format of the evidence can be difficult to comprehend
by stakeholders as an objective-by-objective approach is not adopted.
.
15 November 2019

A Visualisation to Tell a Story
• Need to inform stakeholders to allow the data to be made sense of.
Being able to visualise data and tell stories with it is key to turning it into
information that can be used to drive better decision making.
• Any data visualisation needs to facilitate understanding*:
– Perceiving. What does it show?
– Interpreting. What does it mean?
– Comprehension. What does it mean to me?
• We’ve developed a stakeholder communication model to assist with
getting across a more complicated assurance scenario which isn’t
confined to a pure process-based approach.
* Kirk, A. 2016. Data Visualisation: A Handbook for Data Driven Design. SAGE.
.
15 November 2019

.
15 November 2019
.
15 November 2019
Wheel of
Qualification
(WoQ)

.
15 November 2019
WoQ:
Systems of
Interest

.
15 November 2019
WoQ:
Tier 3 –
Traditional
Process
Evidence

.
15 November 2019
WoQ:
Tier 2 –
System
Integrator
Evidence

.
15 November 2019
WoQ:
Tier 1 –
Government
Acceptance
Evidence

.
15 November 2019
1
2
4
5
3
6
WoQ:
Linking the
Systems to
the Evidence

Generating the Evidence
• Evidence to support the WoQ is developed via the review/analysis of existing
life-cycle artefacts, development of in-service arguments, and dedicated one-to-
one workshops to gather tier evidence. Key concept is to work with what already
exists!
• A number of outputs underpin the WoQ to ensure that the qualification strategy
can be formally articulated to stakeholders:
– Report detailing the justification and the argument for adopting the WoQ strategy.
– Reports detailing the specific evidence for each system of interest.
– Reports outlining the evidence and confidence gained from tiers 1 and 2 of the WoQ.
– Report detailing any mitigations/shortfalls.
• …all of the above reports are currently in development as the qualification
strategies proceed.
WoQ – Wheel of Qualification
15 November 2019

What the Visualisation Offers
• Visualisation helps to hide the complexity of the assurance activities to
assist the engagement with multiple stakeholders who have an interest
in the qualification approach.
• Simplifying the complicated solution space provides the ability to
holistically view: the evidence, the evidence sources, the systems, and
the relationships.
• Stakeholders can make informed decisions on the evidence to be
gathered/generated and to interpret the consequences of any shortfalls.
.
15 November 2019

Caution!
.
15 November 2019

Proceed With Caution
• The approach needs to be underpinned by artefacts to support the qualification
strategy. However, the visualisation acts as a “dashboard” to provide direct links
to the artefacts and allows progress and confidence to be readily reviewed.
• A revised approach does require a level of understanding and competency to
develop/interpret the evidence.
• The approach is very much SME based and is focussed on capturing
judgements. There is a reliance on Suitably Qualified and Experienced
Personnel (SQEP).
• The approach is novel and therefore there is a risk to its acceptance.
• The WoQ and the strategy is not an easier route to qualification than traditional
approaches.
SME – Subject Matter Expert
WoQ – Wheel of Qualification15 November 2019

Strategy In Action
.
15 November 2019

Putting it into Practice (1)
• The revised qualification approach is the cornerstone of a current
airborne platform qualification strategy which has had engagement with
a range of relevant stakeholders.
• Current qualification approach adopted for a platform with a range of
software and CEH levels (A-E).
• Approach is to be implemented within a number of future qualification
strategies. In discussion with further DE&S Delivery Teams (DTs) to
mitigate identified shortfalls in any evidence.
CEH – Complex Electronic Hardware
15 November 2019

Putting it into Practice (2)
• Engagement with UK MOD Military Aviation Authority (MAA).
• Strategy has removed the need to conduct full level C process
assessments due to the confidence gained via the oversight activities
(i.e. tiers 1 & 2).
• Includes process conformance (in keeping with extant standards) but
also takes into account the properties of the system.
• Takes a process and engineering approach.
.
15 November 2019

Next Steps
.
15 November 2019

Engagement and Refinement
• Plan to present a fuller explanation of the strategy with case studies
within a number of future conference papers (target is 2020).
• Plan further refinement based upon lessons learnt from the current
projects and to improve the underlying meta-model of the WoQ.
• Implementation of the “dashboard” to provide direct access to the
underpinning evidence.
– Implemented via tooltips displayed to provide explanatory information and
hyperlinks (to file directories, SharePoint, web pages etc.)
– IP protection is ongoing.
• Produce guidance for end users to adopt the WoQ. This will allow users
to refine the segments and tiers to reflect specific projects of interest.
15 November 2019
WoQ – Wheel of Qualification
IP – Intellectual Property

To Wrap Up
.
15 November 2019

Summary (1)
• With existing approaches not all scenarios allow for evidence to be judged
by those that need to.
• Revised approach adopts a wide set of evidence which builds on the
principles of using diverse evidence. Approach articulates the actual
evidence to support the systems and creates open dialogue. Leads to an
informed qualification assurance argument.
• Moves away from a “tick-box” compliancy process to a more engineering
and independent assurance driven approach.
.15 November 2019

Summary (2)
• Evidence can take many forms in the revised approach – this can create a
complicated solution space. The visualisation provides the ability to
holistically view: the evidence, the evidence sources, the systems, and the
relationships.
• A number of outputs underpin the WoQ to ensure that the qualification
strategy can be formally articulated to stakeholders. Approach part of an
existing strategy.
• Approach and is not a panacea and there are a number of caveats to its
adoption.
WoQ – Wheel of Qualification15 November 2019

Any Questions?
15 November 2019

Using Tiers of Assurance Evidence to Reduce the Tears! Adopting the “Wheel of Qualification” for an Alternative Software Safety Assurance Approach

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Using Tiers of Assurance Evidence to Reduce the Tears! Adopting the “Wheel of Qualification” for an Alternative Software Safety Assurance Approach

Semelhante a Using Tiers of Assurance Evidence to Reduce the Tears! Adopting the “Wheel of Qualification” for an Alternative Software Safety Assurance Approach (20)

Mais de AdaCore

Mais de AdaCore (20)

Último

Último (20)

Using Tiers of Assurance Evidence to Reduce the Tears! Adopting the “Wheel of Qualification” for an Alternative Software Safety Assurance Approach