In the tradition of James Whittaker’s book series How to Break … Software, Jon Hagar applies the testing “attack” concept to the domain of embedded software systems. Jon defines the sub-domain of embedded software and examines the issues of product failure caused by defects in that software. Next, Jon shares a set of attacks against embedded software based on common modes of failure that testers can direct against their own software. For specific attacks, Jon explains when and how to conduct the attack, as well as why the attack works to find bugs. In addition to learning these testing skills, practice the attacks on a device—a robot that Jon will bring to the tutorial—containing embedded software. Specific attack methods considered include data issues, computation and control structures, hardware-software interfaces, and communications.
Exploring the Future Potential of AI-Enabled Smartphone Processors
How to Break Software: Embedded Edition
1. MI
AM Tutorial
4/29/13 8:30AM
How to Break Software: Embedded
Edition
Presented by:
Jon Hagar
Independent Test Consultant
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
2. Jon Hagar
Jon Hagar is a systems-software engineer and tester consultant supporting software product integrity and
verification and validation, with a specialization in embedded and mobile software systems. For more than
thirty years Jon has worked in software engineering, particularly testing, supporting projects including
control system (avionics and auto), spacecraft, mobile-smart devices, IT, and attack testing of smart
phones. Jon has built and managed embedded test labs with test automation; publishes and speaks
regularly with more than fifty presentations and papers; and authored a new book on mobile/embedded
software, scheduled for publication in 2013, and parts of three other books.
3. 4/11/2013
How to Break Software:
The Embedded Edition
Jon Hagar
embedded@ecentral.com
Jon.d.hagar@gmail.com
Jon Hagar Copy right 2010
How to Attack Embedded Software
1
Some Thoughts
• Are you in the right class?
– Definitions and introductions will help here
• Why do we test and what do we need to test?
– Risks
– Information
– Users
– Attacks
Jon Hagar Copyright 2010
How to Attack Embedded Software
2
1
4. 4/11/2013
This is a Workshop Tutorial
• A bit of a talking head (with charts)
– Based on my book
• Attendees should be prepared to:
– Do some reading & thinking
– Use the reference material
– Talk & ask questions
– Share (lessons learned and retrospectives)
• LET’S PLAY TEST… and try out some new
things
Jon Hagar Copyright 2010
How to Attack Embedded Software
3
Agenda
•
•
•
•
•
•
Definitions and introductions
Risk based concepts
Exploratory approaches
Attacking the scenario(s)
Attacking the hardware-software interface
Wrap up and references
• Exercises and testing
Jon Hagar Copyright 2010
How to Attack Embedded Software
4
2
5. 4/11/2013
Definitions
Embedded Software Systems . . .
•
Interact with unique hardware/systems to solve specialized problems in
the “real world”
– IT software runs with largely “generic” hardware
– Users are barely aware the device uses or has software
•
Usually have significant hardware interface issues and concerns
– Initialization, noise, power up/down, timers, sensors, etc.
Often are resource constrained
– RAM, ROM, stack, power, speed, time, etc.
Typically has a restricted or no Human User/Computer Interface (HCI) but
is evolving rapidly
Often no way (or only a risky way) to update and/or change the software
Involves risks, hazards, safety, and/or some specialized domain knowledge
and logic/algorithms usually controlling hardware
•
•
•
•
Jon Hagar Copyright 2010
How to Attack Embedded Software
5
Close Cousins:
Mobile, Smart, and Handheld
•
•
•
•
As the names imply, these are devices—small, held in the hand, often
connected to communication networks, including
– Cell and smart phones – apps (not covered today)
– Tablets
– Medical devices
Typically have:
– Many of the problems of classic “embedded” systems
– The power of PCs/IT
– More user interface (UI) than classic embedded systems
– Fast updates
Are getting more power, memory, and features (software, e.g., apps)
The “hot” area of computers/software
– Testing rules are “evolving”
Jon Hagar Copyright 2010
How to Attack Embedded Software
6
3
6. 4/11/2013
What do these look like?
Examples
– Avionics systems: planes, cars, rockets, military,…..
– Telecom: switch, routers, phones, cell devices,….
– Transportation: traffic control, railroad, trucking, ….
– Industrial control: lighting, machines, HVAC, nuclear/power,…
– Medical: pacemaker, dispensers, …….
– Home and office systems: control, entertainment (TV box), …
– And the list goes on
• Now starting to include PDA’s and other items that “blur” the
lines
Jon Hagar Copyright 2010
How to Attack Embedded Software
7
Fundamental Software Capabilities
Dr. James Whittaker lists four capabilities:
• Software accepts inputs from its environment
• Software produces output and transmits it to its
environment
• Software stores data internally in one or more data
structures
• Software performs computations using input or stored
data
Embedded devices can be refined with
1. Function in/with Time
2. Use/control of unique hardware, OR
3. Refinement of items 1 and 2
Jon Hagar Copyright 2010
How to Attack Embedded Software
8
4
7. 4/11/2013
Embedded: Knowing the Bug (error)
•
Handheld/Embedded software has
similar defects to other software
• Requirements & Design
• Logic & Math
• Control Flow
• Data
• Initialization & Mode changes
• Interfaces
• Security
• Gaming
• etc. . .
Jon Hagar Copyright 2010
But adds context defects/issues
• Software and hardware
development cycles done in
parallel, where aspects of the
hardware may be unknown to the
software development effort
• Hardware problems which are
often fixed with software late in
the project
• Small amounts of dense complex
functions often in the control
theory or safety/hazard domains
• (a big one) Very tight real-time
performance issues (often in millior micro-second ranges)
How to Attack Embedded Software
9
The “World” of Mobile-Smart/Embedded Software
Response-Outputs
Stimulus-Inputs
Expected
Unexpected
Wanted
Hardware
Unwanted
Software
•
•
Inputs and outputs involve hardware, software, and humans
Time dependent
– NOTE: most software has “time” (performance) issues but here things are
often “hard real time”
– Embedded and real-time “time” may be a requirement
Jon Hagar Copyright 2010
How to Attack Embedded Software
10
5
8. 4/11/2013
Exercise: Why do we test?
• Handheld Mobile/Embedded Software
Jon Hagar Copyright 2010
How to Attack Embedded Software
11
YAM* Lifecycle Embedded
(*yet another model)
• Software - Many builds, iterations and increments
– Test “circles” around schedule milestones
start
Lab drop
end
Build 1
start
Eng drop lab drop
end
Build 2
…………
…Prototype………
………Prototype n…..
But what about the hardware lifecycle?
Jon Hagar Copyright 2010
How to Attack Embedded Software
12
6
9. 4/11/2013
Example High Level Embedded Lifecycle
System Creation
Hardware Build
Hardware Build
Hardware Build
Hw
Issue
Software Build
Software Build
Software Build
Software Build
Software Build
Results: Software is “late”
Jon Hagar Copyright 2010
How to Attack Embedded Software
13
My Assumptions…
• This is not a “general” class on systems, software,
and/or testing and I assume the following
knowledge:
–
–
–
–
–
–
–
Test plans and planning
Requirements testing
Test labs and building labs
Standards you operate under (yes, there are many)
Tools you use
Testing experience (a software system)
Embedded design for testability is an accepted practice
• That you want something more . . .
Jon Hagar Copyright 2010
How to Attack Embedded Software
14
7
10. 4/11/2013
If What I Assume is False
(when you get home)
•
•
•
•
•
•
Reference list is available to do some reading
Other full classes are available
You are reading books
You will ask questions
Looking to have an epiphany
You are ready to learn
Keep in mind that I do not have all the answers
Jon Hagar Copyright 2010
How to Attack Embedded Software
15
Section 1: Testing Preliminary
Jon Hagar Copy right 2010
How to Attack Embedded Software
16
8
11. 4/11/2013
Exercise: Test the Embedded Game
•
•
•
•
Break into teams
Define a test
Define some rules: No destructive testing please
List of requirements
– This is a handheld game
– You think of something (say spinach) and it figures out what you are thinking by proposing
20 questions to you
– Questions begin with animal, vegetable, mineral and go from there
– Game has non-standard input keys, display screen, and embedded software
– Game knows things and will figure out what you are thinking of
• Now . . . build a test for this device
Jon Hagar Copyright 2010
How to Attack Embedded Software
17
What do you mean you cannot test?
• What is wrong?
• What do you need to do testing?
• Is this not the world many testers live
in?
• We should start simple in testing, but
maybe this is not simple enough?
Jon Hagar Copyright 2010
How to Attack Embedded Software
18
9
12. 4/11/2013
So, Let’s Back up a Little
• Let me give you some attack support concepts
& techniques (in case you don’t know these)
• You can apply these if you are a staff tester or
a “crowd source” contractor
• This is a simulation, but in the real world,
often you will just be given the software or a
device to test ---- You CAN test…….
Jon Hagar Copyright 2010
How to Attack Embedded Software
19
Risk and Exploratory-Attack Testing
• You cannot test everything
• Risk(s) based testing helps bound the test
scope problem
• Testing is about providing information and
understanding
• Exploration gets you started with whatever
you have (or don’t have)
Jon Hagar Copyright 2010
How to Attack Embedded Software
20
10
13. 4/11/2013
Risk-Based Testing
• Address, mitigate, attack and retire product risks
• Do you remember what a risk is?
– Potential problem - Consequence and effect
– Occurrence – likelihood or chance of happening
– Impact – what happens
• Do this from the beginning (proposal) to the end
(retirement) of the product (Hw-Sw) lifecycle
• Risks should feed the Attacks (more on that later)
Jon Hagar Copyright 2010
How to Attack Embedded Software
21
Sample Product Risks Testers
Should Consider
?
Jon Hagar Copyright 2010
•
•
•
•
•
•
•
•
•
•
•
•
•
Safety
Security
Hazard
Business impacts
Control (loss of)
Computation
Functional elements
Non-functional
Data
Regulation (s) and legal factors
Output noise
Environment and input factors
System factors – complexity, interfaces,
human/non-human
How to Attack Embedded Software
22
11
14. 4/11/2013
How to Use Risk Analysis in Testing
• Goal oriented testing (where to focus)
• Priority of attack and scenario
– Never enough time to test everything
– Can define the “un attacked” (risks)
• Minimization of risks by focusing on the
scary/critical first
• Provide information back to the team sooner
Jon Hagar Copyright 2010
How to Attack Embedded Software
23
Risk-Based Testing Process (simple)
• Identify the product
• Find product supporting information
• Identify risks associated with the product
• Risk priority (what you will test first?)
• The resulting risks by priority define the attacks
Jon Hagar Copyright 2010
How to Attack Embedded Software
24
12
15. 4/11/2013
Risk Analysis Throughout the Test Process
• Many testers just think “requirements” in embedded, but…
• Always be “thinking” risks, since it can drive and control your
testing
• Do this by team brainstorming (make lists)
• Tests and analysis provide learning/data points/information
– Errors in an area of code?
– Hardware that doesn’t work?
– Piece of code from a vendor is more complex?
– Operations the system will/can do?
• Particularly off nominal and unusual (where bugs hide)
Jon Hagar Copyright 2010
How to Attack Embedded Software
25
Exercise: Redeaux
• Back into your teams
• Conduct a risk exercise for your device
Risk Statement ( If x, then y happens)
Jon Hagar Copyright 2010
How to Attack Embedded Software
Priority
26
13
16. 4/11/2013
Risks Should Define Exploration
For mobile-embedded,
exploratory testing can be important
Jon Hagar Copy right 2010
How to Attack Embedded Software
27
Exploratory –Attack Testing
• What is it?
–
–
–
–
Scientific “methods”
Engineering understanding
May call it something else, but most of us do it
Attacks “target” specific bugs using test techniques
• How and when to apply?
–
–
–
–
Jon Hagar Copyright 2010
As early in a lifecycle as possible (with prototypes, models, etc.)
When you want to “learn” and test at the same time
When being a little “informal” is OK
All the time?
How to Attack Embedded Software
28
14
17. 4/11/2013
Exploratory–Attack Testing Definition
Bach/Kaner:
“Exploratory testing is simultaneous learning,
test design and test execution.”
• Exploratory testing has rules and concepts
• Underlying it is a “model” of human understanding of
software and knowing how it fails
• NOT AD HOC: Ad hoc has all too often been associated with
sloppiness, carelessness, no documentation, non-repeatable,
and so froth—but may have a place at times too
Jon Hagar Copyright 2010
How to Attack Embedded Software
29
In Embedded
• Exploratory testing is situational - Use it when…
•
•
•
•
•
•
•
•
•
•
Jon Hagar Copyright 2010
Rapid feedback
Learning
Upfront rapid learning
Attacking
Risk
Independent assessment
Target a defect
Prototyping
Need info
Test beyond the requirements
How to Attack Embedded Software
30
15
18. 4/11/2013
General Concept of Exploratory
Many authors define it as:
1.
2.
3.
4.
Time/Schedule (limited)
The Tester (your team)
A Testing Mission (also called “Charter”)
Results
• Usually in the form of opened Defects
• Sometimes an annotated Mission statement
and opened Defects list
• Maybe a “report”
• Retrospective (more on that in a minute)
Jon Hagar Copyright 2010
How to Attack Embedded Software
31
Exploratory Critical Components
Test Design
Critical Thinking
Diverse Ideas
Rich Resources
Careful Observation
Jon Hagar Copyright 2010
How to Attack Embedded Software
32
16
19. 4/11/2013
Process for This Class (one of many)
• Have an outline (top level plan and/or risk list)
• Create a flip chart, notecard, state model, or some
representation of each test task
– No “heavy” weight documentation of the “test case”
– See Exploratory Charter (test objective)
• Have a Target concept - charter (Risk, Attack, Bug, Learning, …)
• Have a schedule/time box (hours — not more than 1-2 days)
• Do the test
– Design test
– Execute test
– Learn about the product: change the risk list, modify/add
tests, and so on
• Repeat the process as needed
Jon Hagar Copyright 2010
How to Attack Embedded Software
33
Exploratory Test Card (Charter)
Name of Test:
•
Who is testing (test team)
What to Test:
– Risk (s):
Success Criteria:
1.
– Attack
2.
–
3.
Other (requirements, …..)
•
Support items needed:
•
•
Role (User you play during the test):
Actions:
– 1.
– 2.
– 3.
Others Steps
Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)
Document all of these.
Jon Hagar Copyright 2010
How to Attack Embedded Software
34
17
20. 4/11/2013
Exercise: So Let’s Go Back to
the Embedded Game Device to Do Testing
• Test of the Game App
• Use the risks (chart 25) for the device to define test objectives
• Apply Exploratory-Attack
– Do a Charter
• Learn – do one cycle of exploration
Jon Hagar Copyright 2010
How to Attack Embedded Software
35
Group Flip Chart
Feedback - Retrospective
• What did you accomplish?
– Find any bugs? If so, how many?
• What did you think of?
• What would you do differently?
Jon Hagar Copyright 2010
How to Attack Embedded Software
36
18
21. 4/11/2013
Section 2:
So we have Risk Analysis &
We have done a first Exploratory Test
• Basic and addressed in many books
• What’s next?
• Lets get to a real embedded device
Jon Hagar Copy right 2010
How to Attack Embedded Software
37
What is an Attack
• Attacking your software–In part, the process of attempting
to demonstrate that a system (hardware, software, and
operations) does not meet requirements, functional and
non-functional objectives
– Embedded/handheld software testing must include “the
system” (hardware, software, operations, users, etc.)
• Attacks go after common modes of failure and bugs to
demonstrate that “does not meet” exists
• We go after our enemy with many approaches
–
–
–
–
–
Jon Hagar Copyright 2010
Tools
Levels
Attacks
Techniques
Etc.
How to Attack Embedded Software
38
19
22. 4/11/2013
An Attack Is…
• Based on a common mode of failure seen over and over
– Maybe seen as a negative, when it is really a positive
– Goes after the “bugs” that may be in the software
– Based on or using classic test techniques and test concepts
• Lee Copeland’s book on test design
• Many other good books
• A Pattern (more than a process) which must be modified
for the context at hand to do the testing
• Testers learn these in a domain after years and form a
mental model (most good testers attack)
Jon Hagar Copyright 2010
How to Attack Embedded Software
39
Kinds of Attacks
• Whittaker offers a good starting point for software
attacks in general that can be applied to embedded:
– User Interface Attacks
– Data and Computation
– File System Interface
– Software/OS Interface
• Whittaker’s “How to Break Software” lists 23 attacks
• “Software Test Attacks to Break Mobile and
Embedded Devices” lists 32 attacks and 8 sub attacks
Jon Hagar Copyright 2010
How to Attack Embedded Software
40
20
23. 4/11/2013
Embedded Attack Classification
•
•
•
•
•
•
•
Developer Attacks (unit/code testing)
Control System Attacks
Hardware-Software Attacks
Mobile and Embedded Software Domain Attacks
Time Attacks (Performance)
Human User Interface Attacks
Smart and/or Mobile Phone Functional App Attacks
• Mobile/Embedded Security Attacks
• Generic Attacks
– Functional, mind mapping, and combinatorial tests
Jon Hagar Copyright 2010
How to Attack Embedded Software
41
The Software You Test
• Do you know how it fails?
• Do you test for success or failure?
– Both?
• Will this workshop give you all the answers and
all possible attacks?
– No, but you can start asking questions and thinking
Jon Hagar Copyright 2010
How to Attack Embedded Software
42
21
24. 4/11/2013
Introducing the Robots
•
•
•
•
Requirements – in the class hand out for each robot grouping
Rules (this exercise takes some thinking and reading)
– NO destructive testing (Please BE CAREFUL with the robots)
– There are bugs to be found (record and report them)
– Each group defines an attack and gets “time” on the devices (but time in our
environment is limited—just as it is in the real world)
Environment
– This room, but we have some “test tools”
– This is software testing, but within the hardware it is embedded in
This will be a simple testing process, but use the attack concepts and report
experiences back in the debrief
1. Define risks based on hardware, software, requirements, and bugs (risk list)
2. Conduct each attack session using a charter
3. Define a test attack using provided attack pattern (handout)
4. Will do a debrief after each test session
5. Group will rotate different robot configurations
Jon Hagar Copyright 2010
How to Attack Embedded Software
43
Additions Considerations
•
•
•
•
We will try to get at least 2 different attacks
Use the concepts we used on games
Ask questions (of each other & me)
No destructive testing and I am the “tester”
(you must tell me what to do)
• Use the tools at hand
• But first we need to think about embedded
users – next exercise
Jon Hagar Copyright 2010
How to Attack Embedded Software
44
22
25. 4/11/2013
More Test Time
• We are going to do another attack on a different embedded
device
• Each group will be tasked with one of the two attacks
• Each group should complete a charter and see the “test
master” for access to the device
• You’ll have a robot with software loaded
• Practice identifying: Risks, Users, Exploration, and Attacks
• Follow the “suggested patterns” of the attacks
• We’ll go until no more time left
Jon Hagar Copyright 2010
How to Attack Embedded Software
45
Understanding Users for
Embedded Attacks
• Let’s list some of my Game App users
because
• Users play into risks, attacks, bugs, what to
look for
•
Jon Hagar Copyright 2010
You should be able to do the same for the robots (or any
software you test)
How to Attack Embedded Software
46
23
26. 4/11/2013
So you have a few basics:
Risk thinking
Exploration
Software’s users
Attack patterns are provided next
Jon Hagar Copy right 2010
How to Attack Embedded Software
47
Attack Group 1
Stories, Tours, and Scenarios
• Call them what you will
• There are subtle differences depending on
whose material you have read
• They are how the system gets used end-to-end
• They combine use, users, information,
techniques, tools, and (maybe) attacks
Jon Hagar Copyright 2010
How to Attack Embedded Software
48
24
27. 4/11/2013
Apply This Attack When…
• Time interacts with the software, events, inputs, and
outputs
• Checklist of things to look for & consider (possible
bugs)
–
–
–
–
–
–
–
–
–
–
–
Order problems
Too Long
Too Fast
Not at Right Time mark or point
Late
Late or early
Early
Deadlocked caused by a race condition(hard to find)
Extra input or output events
Missing events
Wrong input/output within events
Jon Hagar Copyright 2010
How to Attack Embedded Software
49
Attack Factors
• What - Look for things not in the right order
• Who – Test team
• Where – Lab and/or field testing where hardware
and software interact
– Tools may be important here
Jon Hagar Copyright 2010
How to Attack Embedded Software
50
25
28. 4/11/2013
How?
•
•
•
•
•
•
•
Understand what the system does or is supposed do
– a sequence of events or functions
– Look in: concepts of operations, user guides, use cases, models, & any other
information that will detail functions and usage over time
– From these, organize a sequence or set of sequences
First attack case: Focus on a typical situation based on requirements and/or use cases
Second attack case: Consider the off–normal , non–failure modes
– Look for the failure modes and effects—does the software recover well?
– Review and understand system errors and failure history from the field.
Build up histories of attacks based on outputs and log files
– Warning: log files can contain large amounts of detailed data and this can also
adversely affect the performance (especially timing) of the software
Conduct risk analysis as the effort progresses
Final Attack cases: Build Extreme cases
– such as “Soap Opera” Tests
Warning: Watch becoming “script” bound
Jon Hagar Copyright 2010
How to Attack Embedded Software
51
Attack Group 2
Refer to second attack handout
Here we are attacking the hardware-to-software
interface
Jon Hagar Copyright 2010
How to Attack Embedded Software
52
26
29. 4/11/2013
Attack: Analog-Digital Hw-Sw Interfaces
• When - The software is “controlling” the unique hardware
• What – Look at the interface, hardware (as a user), and what
the software is controlling
• Who – Test team (independent)
• Where – Lab where the hardware and software are both
present
• Bugs to look for (next page)
Jon Hagar Copyright 2010
How to Attack Embedded Software
53
Taxonomy: A2D and D2A Bug Possibilities
Type
A2D
A2D
A2D
D2A
D2A
D2A
Jon Hagar Copyright 2010
Situation
Impact
A2D representation information is lost Software
because measurement is not precise computation is
based on incorrect
data
A2D information is contaminated with Software
noise
computation use
noise when it
should not
A2D information is calculated
Computation has
correctly
unknown error
D2A conversion losses “least
significant bits” (LSB) in conversions,
but bits are, in fact, important
because computer word sizes are too
small
D2A information does not account for
noise of the real world
D2A information is calculated
correctly because of internal factors
Output to analog
device is wrong
Software
computation does
not include a factor
for noise
Computation has
unknown error
How to Attack Embedded Software
Notes
Number of bits used to store the analog
converted data is not large enough or
sampling rate to get bits is not correct.
The noise term may not be known,
accounted for, or misrepresented.
Sources of error can come from:
calibrations used on variables, variables
lacking initialization, or calculations are not
done with enough accuracy (single versus
double floating point
Number of bits stored from the digital
world to the analog world do not have
enough precision, so analog data is
incorrect.
The analog values are not correct given the
noise of the real world (output data may be
lost in the noise).
Sources of error can come from:
calibrations used on variables, variables
lacking initialization, or calculations are not
done with enough accuracy (single versus
double floating point
54
27
30. 4/11/2013
How?
•
•
•
•
•
•
•
•
•
•
•
•
•
Upfront data gathering and analysis are important beginnings – what do we know
(or can ask about)
Identify input devices
Identify output devices
Define the input disturbances (unexpected system inputs)
Define possible output disturbances (unexpected system outputs)
Determine what is or is not possible in the test environment
Conduct a risk analysis (see likely bugs table)
Identify the users of the device and software
- Testers should be aware that embedded systems have resource constraints in
memory, CPU usage, and time
Use the above information to define an exploratory chart attack
Go run that attack
Learn
Design
Repeat (until time out)
Jon Hagar Copyright 2010
How to Attack Embedded Software
55
Questions to Ask with This Attack
•
•
•
•
•
•
If the hardware is a prototype (not like what will be in the field), will that
impact testing or test results?
If a simulation is used, what bugs might be missed because actual
hardware or software is not used?
If the test inputs and environment are not representative of the real world
both in terms of expected and unexpected values, what risks will be
acceptable?
If the hardware is not understood, will testing be weak?
If the major sources of “noise” are not defined, will the system be
susceptible to impacts from unexpected inputs or outputs?
All of these questions will involve test tradeoffs, accepted risk, and
compromise
– 40% of this kind of attack should be “normal” situations
– Start normal and move to off normal and stress cases
Jon Hagar Copyright 2010
How to Attack Embedded Software
56
28
31. 4/11/2013
Group Flip Chart
Feedback – Retrospective Session
• What did you accomplish?
– Bugs?
– Tests?
• What things did you think of?
– Wish I had a ???? I need more time???
• What favors or opposes an attack?
• What would you do differently next cycle?
Jon Hagar Copyright 2010
How to Attack Embedded Software
57
Final Thoughts
Jon Hagar Copy right 2010
How to Attack Embedded Software
58
29
32. 4/11/2013
Wrap Up
• This tutorial only covered some basic
introduction (key attacks) and sampling
– There are many more
• Understanding your local context and error
patterns is important (one size does NOT fit all)
• Attacks are patterns…you still must THINK
• These attacks target Embedded and Mobile
Jon Hagar Copyright 2010
How to Attack Embedded Software
59
More Attacks (from my book and others)
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Attack 1: Static Code Analysis
Attack 2: Finding White–Box Data Computation Bugs
Attack 3: White–Box Structural Logic Flow Coverage
Attack 4: Finding Hardware–System Unhandled Uses in
Software
Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs
Attack 6: Long Duration Control Attack Runs
Attack 7: Breaking Software Logic and/or Control Laws
Attack 8: Forcing the Unusual Bug Cases
Attack 9 Breaking Software with Hardware and System
Operations
9.1 Sub–Attack: Breaking Battery Power
Attack 10: Finding Bugs in Hardware–Software
Communications
Attack 11: Breaking Software Error Recovery
Attack 12: Interface and Integration Testing
12.1 Sub–Attack: Configuration Integration Evaluation
Attack 13: Finding Problems in Software–System Fault
Tolerance
Attack 14: Breaking Digital Software Communications
Attack 15: Finding Bugs in the Data
Attack 16: Bugs in System–Software Computation
Attack 17: Using Simulation and Stimulation to Drive
Software Attacks
Jon Hagar Copy right 2010
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Attack 18: Bugs in Timing Interrupts and Priority Inversion
Attack 19: Finding Time Related Bugs
Attack 20: Time Related Scenarios, Stories and Tours
Attack 21: Performance Testing Introduction
Attack 22: Finding Supporting (User) Documentation Problems
Sub–Attack 22.1: Confirming Install–ability
Attack 23: Finding Missing or Wrong Alarms
Attack 24: Finding Bugs in Help Files
Attack 25: Finding Bugs in Apps
Attack 26: Testing Mobile and Embedded Games
Attack 27: Attacking App–Cloud Dependencies
Attack 28 Penetration Attack Test
Attack 28.1 Penetration Sub–Attacks: Authentication —
Password Attack
Attack 28.2 Sub–Attack Fuzz Test
Attack 29: Information Theft—Stealing Device Data
Attack 29.1 Sub Attack –Identity Social Engineering
Attack 30: Spoofing Attacks
Attack 30.1 Location and/or User Profile Spoof Sub–Attack
Attack 30.2 GPS Spoof Sub–Attack
Attack 31: Attacking Viruses on the Run in Factories or PLCs
Attack 32: Using Combinatorial Tests
Attack 33: Attacking Functional Bugs
How to Attack Embedded Software
60
30
33. 4/11/2013
Summary: Thank You (ideas used from)
•
•
•
•
•
•
James Whittaker (attacks)
Elisabeth Hendrickson (simulations)
Lee Copeland (techniques)
Brian Merrick (testing)
James Bach (exploratory & tours)
Cem Kaner (test thinking)
• Many teachers
• Generations past and future
• Books, references, etc.
Jon Hagar Copyright 2010
How to Attack Embedded Software
61
Book List (my favorites)
• “Software Test Attacks to Break Mobile and Embedded Devices”
– Jon Hagar, to be published in 2013
• “How to Break Software” James Whittaker, 2003
– And his other “How To Break…” books
• “Testing Embedded Software” Broeckman and Notenboom, 2003
• “A Practitioner’s Guide to Software Test Design” Copeland, 2004
• “A Practitioner’s Handbook for Real-Time Analysis” Klein et. al.,
1993
• “Computer Related Risks”, Neumann, 1995
• “Safeware: System Safety and Computers”, Leveson, 1995
• Honorable mentions:
–
–
–
–
–
–
Jon Hagar Copyright 2010
“Embedded System and Software Validation” Roychoudhury, 2009
“Systems Testing with an Attitude” Petschenik 2005
“Software System Testing and Quality Assurance” Beizer, 1987
“Testing Computer Software” Kaner et. al., 1988
“Systematic Software Testing” Craig & Jaskiel, 2001
“Managing the Testing Process” Black, 2002
How to Attack Embedded Software
62
31
34. 4/11/2013
More Resources
• www.stickyminds.com – Collection of test info
• www.embedded.com – info on attacks
• Association of Software Testing
– BBST Classes
http://www.testingeducation.org/BBST/
• Your favorite search engine
Jon Hagar Copyright 2010
How to Attack Embedded Software
63
Definitions (for this class)
•
•
•
•
•
•
•
•
•
•
•
•
Taxonomy - the practice and science of classification.
Test – the act of conducting experiments on something to determine the quality
and provide information
Test case – One set of inputs, environmental set up, and results (expected and un)
Attack – to set up, forcefully, and attempt to “damage” the system or software,
using tools, methods, and techniques
Bug (error) – Results that depart from the expected (from requirements, design,
standards, user, etc.)
Lifecycle – From beginning-to-end, the steps, stages, and activities to create (birthto-death)
Procedure – a particular way of accomplishing tests, usually written (one or more
test cases)
Tour – a journey to find information (tests) with a focus/direction (story)
Scenario – a sequence of events with a test plot or story
Script – see procedure, but normally uses automation
Users – someone/something that interacts with the system/software (can be
human or machine, or?)
Quality – Value to someone and that they will pay for
Jon Hagar Copyright 2010
How to Attack Embedded Software
64
32
35. 4/11/2013
Exploratory Test Card (Charter)
Name of Test:
•
Who is testing (test team)
What to Test:
– Risk (s):
Success Criteria:
1.
– Attack
2.
–
3.
Other (requirements, …..)
•
Support items needed:
•
•
Role (User you play during the test):
Actions:
– 1.
– 2.
– 3.
Others Steps
Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)
Jon Hagar Copyright 2010
How to Attack Embedded Software
65
Exploratory Test Card (Charter)
Name of Test:
•
Who is testing (test team)
What to Test:
– Risk (s):
Success Criteria:
1.
– Attack
2.
–
3.
Other (requirements, …..)
•
Support items needed:
•
•
Role (User you play during the test):
Actions:
– 1.
– 2.
– 3.
Others Steps
Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)
Jon Hagar Copyright 2010
How to Attack Embedded Software
66
33
36. 4/11/2013
Exploratory Test Card (Charter)
Name of Test:
•
Who is testing (test team)
What to Test:
– Risk (s):
Success Criteria:
1.
– Attack
2.
–
3.
Other (requirements, …..)
•
Support items needed:
•
•
Role (User you play during the test):
Actions:
– 1.
– 2.
– 3.
Others Steps
Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)
Jon Hagar Copyright 2010
How to Attack Embedded Software
67
Exploratory Test Card (Charter)
Name of Test:
•
Who is testing (test team)
What to Test:
– Risk (s):
Success Criteria:
1.
– Attack
2.
–
3.
Other (requirements, …..)
•
Support items needed:
•
•
Role (User you play during the test):
Actions:
– 1.
– 2.
– 3.
Others Steps
Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)
Jon Hagar Copyright 2010
How to Attack Embedded Software
68
34
37. 4/11/2013
Exploratory Test Card (Charter)
Name of Test:
•
Who is testing (test team)
What to Test:
– Risk (s):
Success Criteria:
1.
– Attack
2.
–
3.
Other (requirements, …..)
•
Support items needed:
•
•
Role (User you play during the test):
Actions:
– 1.
– 2.
– 3.
Others Steps
Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)
Jon Hagar Copyright 2010
How to Attack Embedded Software
69
35