In today’s cost conscious marketplace, solution providers gain advantage over competitors when they deliver measurable benefits to customers and partners. Systems of even small scope often involve distributed hardware/software elements with varying execution parameters. Testing organizations often deal with a complex set of testing scenarios, increased risk for regression defects, and competing demands on limited system resources for a continuous comprehensive test program. Learn how designing a testable system architecture addresses these challenges. David Campbell offers practical guidance on the process to make testability a key discriminator from the earliest phases of product definition and design. Learn approaches that consistently deliver for high achieving organizations, and how these approaches impact schedule and architecture performance. Gain insight on how to select and customize techniques that are appropriate for your organization’s size, culture, and market.

1. T8
Test Techniques
5/8/2014 11:15:00 AM
Designing for Testability:
Differentiator in a Competitive
Market
Presented by:
David Campbell
MITRE Corporation
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. David Campbell
MITRE Corporation
David Campbell has twenty-seven years of technical management and software development
experience focusing on high performance and real-time systems. Much of that time was spent
with Silicon Graphics Computer Systems and Kasenna, Inc. working on joint programs with
customers and partners from a wide variety of markets and development cultures. Today Dave
works for the MITRE Corporation directing a lab that focuses on research and evaluation of time
critical systems.

3. 4/26/2014
1
| 1 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Designing For Testability
Dave Campbell
MITRE
Approved for Public Release; Distribution Unlimited. 14-0356
| 2 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Biography
Dave Campbell
– MITRE Corporation
Time Critical Systems
dcampbell@mitre.org
25 years of software development and technical
management
– Silicon Graphics Inc.
High Performance Servers and Workstations
Operating and Real Time Systems
Entertainment, Media, High Performance Solutions
– Kasenna Inc.
IPTV and Video Delivery Platforms

4. 4/26/2014
2
| 3 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Continuing Problem
Software Systems are often key contributors to
program/project execution risk
- Schedule Delays
- Cost Overruns
- Reliability
- Security and Information Assurance
| 4 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Observation
When analyzing all the software systems that I
have observed over 27 years a trend becomes
clear:
Development organizations that take the
long view and invest in testing
infrastructure consistently outperform all
others in cost, schedule and quality
execution

5. 4/26/2014
3
| 5 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Design for Testability (DFT)
Earliest stages of design must include test team
representation
– Requirements, Scheduling, Funding
No design is approved without a complete picture of
how each subsystem is tested independently or
integrated into system
Need to have long term view
– Initial hit to schedule and EVM (Earned Value Metrics)
Agile, Continuous, Waterfall – all methodologies can
support approach
Steps to Enforce Design Testability will Follow
| 6 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
What to Avoid
Evolving development into a process where
any comprehensive integration testing
requires all software and hardware system
components to be effective
Full system configuration dependency for
testing slows engineering productivity
–Lose agility
–Lose effectiveness of test coverage
–Forces big bang integration testing late in
development
What is your experience?

6. 4/26/2014
4
| 7 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Example: Radar Processing
Classic example, involves distributed
hardware and software system components
–System of systems
Time Critical and batch processing
requirements
Service level agreement requirements
System that will be in operation for over 25
years with technology refreshes over lifespan
| 8 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Radar System Components
Web services takes requests/presents results
Other components manage radar and data it generates
Antenna
Controller
Radar Signal
Processing
Operations
and
Control
Web
Services
Repository
Clients
Does the design require all these
components to be available to
effectively integration test?

7. 4/26/2014
5
| 9 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Designing for Testability, Step 1
Define Time Critical Core of System
Identify all timing critical areas of the system
– Understand constraints
– Concurrency and potential race conditions
Ensure implementation is self monitoring
– Reports violations
– Effective under loading and scaling
All ongoing testing must verify the time critical
processing is not compromised as system is
integrated
| 10 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Designing for Testability, Step 2
Ensure System Can Be Characterized
System must be able to be diagnosed
– Avoid debug and production builds
– Logging system must not alter system timing
– Distributed system traceability and time stamping
– Keep diagnostic capability native
Design to greater than capacity and service load
limits
– Define denial of service behavior
– Data drops or loss handling

8. 4/26/2014
6
| 11 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Designing for Testability, Step 3
Define Component Simulators
Create high fidelity software simulations of key
system components
–Formalizes interfaces and messaging
–Resources for every developer, no sharing
–Validates the design of component
–Much more than stubbing
Simulated components allow testing of related
components independent of hardware availability
–Allow for introduction of error states
–Can be used throughout lifecycle
| 12 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Designing for Testability, Step 4
Data Injection and Taps
Identify areas in system where key data items
can be injected or extracted without performance
impact
–Identify APIs and necessary state setup
Create low impact tools to perform function,
insure the tools meet required performance
–Load generators, IA probes
Injectors and taps allow for post processing
analysis and inserting data generated from
hardware prototypes

9. 4/26/2014
7
| 13 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Designing for Testability, Step 5
Apply Continuous Integration Testing
Integration testing never stops
– Avoid waiting on testing anything
– Capacity and service loading early and often
– Avoid point testing only of a capability
– Apply automation and testing frameworks
Use simulators, injectors and any available hardware
to construct test configurations
– Not uncommon for system to be used for hardware
debugging during day and running integration tests with
simulated hardware at night
Complex distributed systems of systems may have more end to
end dependencies than designers can predict, mitigate with a
testable design
| 14 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Tale of Two Engineering Teams
Two teams design and implement the same Video on
Demand (VOD) system for competitive contract award
– Requirements bound implementation choices
Team A designs for testability and implements many
of the necessary steps
Team B does not implement any testability
Team B initially performs better but as development
advances they become more dependent on limited
hardware configurations
Team A emerges with a product that has a much lower cost
of ownership

10. 4/26/2014
8
| 15 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Key Factors Impacting Decision
Cost advantage emerges for Team A
– Shorter implementation time to add new capabilities
– Short reintegration time to fix and test defects
– Simulation components can be utilized for training and
provisioning of deployed system
Flexibility advantage emerges for Team A
– VOD was a new technology and many deployment
trials were needed with different configurations
– Increased ability to work with more partners, investors
DFT concepts eventually become part of the Open Cable
Architecture Specification
| 16 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Sample Steps Review
Define Time Critical Core of System
– Ensure this never breaks
Ensure System can be Characterized
– No performance impact, one implementation
High Fidelity Simulation Components
– Refine design, Provide flexibility, Key artifact
Data Injectors and Extractors
– Design for key points of system (i.e. IA boundaries)
Continuous Testing Approach
– Avoid the “big bang” and “one and done” scenarios
– Test hard stuff first, capacity and service loading
Use approaches that work within your project and culture

11. 4/26/2014
9
| 17 |
© 2014 The MITRE Corporation. All rights reserved. For internal MITRE use
Conclusion
There is additional initial cost to a testable
design
Testable design needs to be defined in
specification and requirements, contract
deliverable
During development execution, need to resist
temptation to pull development resources
and funding away from testability efforts
High Performing Engineering Organizations Design for
Testability