It proposes a 3-phase framework: 1) Model valid business processes by monitoring normal user behavior. 2) Manipulate workflows by modifying states and transactions. 3) Analyze results to detect deviations from expected behavior, indicating potential logic defects. The goal is to overcome challenges of testing application logic, which is hard to define, domain-specific, and lacks consistent patterns. A demo is provided as a proof of concept for how such a framework could work. Contributions to further the research are welcomed.
Defying Logic - Business Logic Testing with Automation
1. Defying Logic
Theory, Design, and Implementation
of Complex Systems for
Testing Application Logic
Rafal Los & Prajakta Jagdale â HP Application Security
4. Define: Application Logic
Just what is âapplication logicâ?
âDesign components of a program,
including the sequencing of steps
prescriptive for how to execute intended
business processes in a piece of
softwareâ
5. Logic Primer
A business process, ordering
Log In Select Add Verify
Restaurant Items Order
Log Out Loyalty Transaction Add
(quit) Points Success Payment
6. Logic Primer
A broken business process?
Log In Select Add Verify
Restaurant Items Order
Log Out Transaction Loyalty Add
(quit) Success Points Payment
7. Logic Primer
Can we manipulate intended process?
Log In Select Add Verify
Restaurant Items Order
Log Out Transaction Loyalty Add
(quit) Success Points Payment
8. Logic Primer
In this simple example:
Manipulate a business process
Early loyalty points without paying
â Points = fraud (free stuff!)
â Free meals = financial lo$$ to vendor
Business process manipulated = theft,
fraud, financial loss
9. Define: Logic Defect
A defect that exposes the component
business processes (or execution flows)
to manipulation from the attacker
perspective to achieve unintended and
undesirable consequences from the
design perspective; without disrupting
the general function or continuity of the
application.
10. New Classifications
How is this different than existing
classifications?
â OWASP Top 10
â WASC Threat Classification v2
â MITRE CWE Top 25 Best âfitâ
A fundamentally different way of
looking at software vulnerabilities.
11. Building Onto CWE
â MITRE CWE Top 25
âClassification of business-logic weaknesses is worth deeper
investigation by researchers. While business logic rules are
domain-specific, there may be some domain-independent
patterns to them. We are likely to find new wrinkles to old
problems, and maybe some new problems that will be easier to
find once we know what to call them.â âSteven Christey (MITRE)
12. âTaxonomyâ
2 Types of logic-based defects
âą Privilege manipulation
âą Transaction control manipulation
13. Privilege Manipulation
Flaw based on broken or incomplete
authentication/authorization mechanism
âą Horizontal/vertical privilege escalation
âą Access unauthorized content
âą Perform privileged system functions
15. Transaction Control
Manipulation
Flaw based on broken business process
continuity allowing for manipulation of
intended business process/flow
âą Circumvent system limitations
âą Tamper with application flow
âą Manipulate business resources
19. Manipulating âLogicâ
The server should control the logic of the
application.
In this case, the server should know 30 is
not a valid number âŠright?!
âŠâŠâŠâŠâŠâŠâŠâŠâŠ..Nope.
20. To Be Clear
The key point to remember:
This research seeks to better enable the
tester through automation.
We are addressing âdesigned processesâ
We are not addressing âback-end
business processâ (non-visible)
21. Starting a Wave
FEW previously given talks or papers on
logic vulnerabilities
â Ideas ï talks ï papers ï experiements?
â A more tangible/usable effort is needed
NO existing R&D effort by a vendor in this
space as far as we could tell â
proprietary or open source
22. Lots of TalkingâŠ
âSo why arenât there any readily available,
even semi-mature tools or frameworks
for testing application logic?â
23. Act 2 â The Turn
APPLICATION LOGIC VS.
AUTOMATION
24. Logic vs. Automation
Application Logic Automation
ïŒ hard to define! o pattern-dependent
ïŒ domain-specific o programmatic
ïŒ Industry o scale is in
ïŒ Business process repeatability
ïŒ not pattern-based o no concept of
ïŒ not easy to âRegExâ process
25. Challenges
For humans For technology
âą understand the âą map the application
application processes
â Business processes â simple, flexible maps
â âApplication flowâ âą identify control logic
âą drive automation â critical parameters
â work with technology âą differentiate test
âą document & repeat success or failure
26. Application Logic
Application âlogicâ is trickyâŠ
â found on the server side
â found in the client âcacheâ (offline use?)
â no consistent patterns to match/test
âą Remember the AT&T/iPad email hack?
â logic implies human thought required
27. Simple Logic
Simplest logic block exampleâŠ
If <expression> Then
do something
ElseIf <expression> Then
do something
Else
do something else
End If
28. Challenges
Bridging a vast disconnect
Client (browser) App Server
DOM
IF (var1 = A) {
var1=A do IsUser; }
var2=B ELSEIF (var1 = B)
var3=1 {
var4=@b do IsAdmin; }
var5= ENDIF
29. Challenges
How to overcome random âfuzzingâ
âą identify var1 as a critical parameter
â Human or automation-driven
âą manipulate var1
â human or technology driven
â âfuzzingâ or data-set driven
Technology enables scalability & repeatability
30. Future State ï Fantasy
âą Point ânâ shoot application logic testing
â This was never a good strategy anyway*
âą Dynamic testing tool âlearnsâ business
processes, logic flow and thinks
â Iâve seen this movie before âŠâSkyNet?â
âą Security continues to operate
independent of business
31. Future State ï Reality
âą Logic testing enabled through
automation
â Base logic mapping on QA methodology
â Continue to evolve combined functional,
exploratory & security testing
âą Technology allows repeatable testing
throughout application after initial setup
32. This Has Been Coming
âą Initially I talked about mapping
application execution flowâŠ
âą Dynamic testing technology matured
âą Static testing technology matured
âą New âhybridâ technology gives even
greater insight into application function
33. Act 3 â The Prestige
BUILDING TEST
FRAMEWORKS
34. Overview
Test framework will constitute 3 phases
I. Capturing the assumed application
behavior
II. Manipulation of the workflows to evoke
(un)expected behavior patterns
III. Analyzing the results of the workflow
manipulations to detect non-conformance
with defined business processes
35. 3 Step Process
âą Model the business process
â create valid workflows
âą Manipulate application workflow
â âfuzzing logicâ
âą Analyze the results
â Were we successful in evoking a non-
standard response?
36. Modeling the Business
Processes
âą Purpose: Discover business processes that
define the application workflow
âą Challenge: Application workflow
specifications are rarely documented and
are often unknown to the testers
âą Solution: Passively monitor & record normal
user behavior through the application
37. Modeling the Business
Processes
Proposed Approach
Devise a mechanism to:
â Capture permissible user actions
â Store the state of the application before
and after the invocation of each user event
â Record expected transitions between
application states
38. Manipulating the Application
Workflow
âą Purpose: Induce deviations in the application
behavior that do not conform with the
assumed business rules
âą Challenge: Meaningfully manipulating
applicationâs behavior with minimal
knowledge about its purpose & context
âą Solution: Apply pre-defined modifications to
the state identifiers and state transactions
recorded in phase I
39. Manipulating the Application
Workflow
Proposed Approach
Discover logic defects by:
â Modifying the state of the application
before passing it as an input to a
transaction
â Fuzz the sequence of user actions
captured during the phase I to detect bugs
in transaction control
40. Analyzing the Results
âą Purpose: Determine the success of
workflow manipulation
âą Challenge: Measuring the deviation in
application behavior with minimal
understanding of application context
âą Solution: Apply comparison metrics to
infer deviations in the application
workflows
41. Analyzing the Results
Proposed approach
â Measure the impact of workflow manipulations on
the application state
âą compare state identifiers (e.g. system variables, page
DOM) in the tainted workflow with the original
â Apply pre-defined set of rules to detect violations
of the business rules governing the application
flow
âą E.g. An acceptable application end state can be reached
despite inaccuracies in one or more intermediate states
of the workflow