Slides for the UK London BCS Sigist March Keynote

1. Push Your Functional
Testing Further
Alan Richardson
EvilTester.com
@EvilTester
Technology
and
Security
into

2. @EvilTester 2
Part the first, wherein we
describe Functional Testing in
terms of Systems and Models,
and expand A model of testing
to include Technical Testing.

3. @EvilTester 3
Functional Testing
● Testers learn how to
test systems
– Requirements
– 'What' a system
'should' do

4. @EvilTester 4
Systems
● System under Development
● System of Development

5. @EvilTester 5
System Under Development
● System under Development
– Requirements
– Architecture
– Environments
– etc.

6. @EvilTester 6
System Of Development
● Methodological Context
● Social Constructs
● Model Different Systems of Testing
– Systems of Feedback
– Systems of Learning
– Systems of Questioning
– ...

7. @EvilTester 7
Requirement Example – RestMud

8. @EvilTester 8
A Model of Testing
● Modelling,
● Observation,
● Intent,
● Reflection,
● Manipulation

9. @EvilTester 9
We can push our functional testing
further
● “What is it supposed to
do?”
– vs “What does it do?”
● Comparison to other
models
● Is it viable?
● Precondition analysis
● Presupposition analysis

10. @EvilTester 10
We can push our functional testing
further
● Explore 'How' the system does what it does
● Understand the technology used to build the
system
– Identify technology risks
– Identify risks at different levels of the stack
– Work at different levels of the stack

11. @EvilTester 11
A Model of Technical Testing
● Modelling,
● Observation,
● Interrogation,
● Reflection (includes intent),
● Manipulation

12. @EvilTester 12
Part the second, wherein two
technical models are provided
with a discussion of possible
technical testing approaches.

13. @EvilTester 13
Example – a Java App
● HouseOfTest.se
– /2016/02/testers-contest-crappy-little-datagenerator/

14. @EvilTester 14
Observation Example – a Java App
● Double click run – see GUI
● tail -f DataGeneration.txt
● Text Editor
– Line endings
– Refresh
● No exceptions shown

15. @EvilTester 15
Observation Example – a Java App
● java -jar crappy_little_datagenerator_v_1.0.jar
– Now can see stdout written to command line,
exceptions, errors etc.

16. @EvilTester 16
Interrogation Example – a Java App
● Decompile e.g. http://jd.benow.ca/
● Load .jar as a library into a Project

17. @EvilTester 17
Interrogation Example – a Java App

18. @EvilTester 18
Observe - Multiple Entry Points

19. @EvilTester 19
Modelling Example – a Java App
GUI
CLI
DataGeneration

20. @EvilTester 20
Manipulation Example – a Java App

21. @EvilTester 21
Manipulate Example – a Java App

22. @EvilTester 22
Risks
● Does this test approach add risk?
– Because testing at a lower level in the stack?
– Because working against a non-deployed version?
– Because it is not how the user would run it?

23. @EvilTester 23
Risk
● Mitigating risk provides one reason for testing
● No test approach mitigates all risk
● Multiple test approaches required

24. @EvilTester 24
Modelling – a Web App
HTML GUI
Web App
Browser Web Server
App Server

25. @EvilTester 25
Observe & Interrogate – a Web App
HTML
& DOM
Web App
Web Server
App Server
HTTP
via
Proxies
View Source
Dev Tools
HTTP
via
Dev Tools

26. @EvilTester 26
Manipulate – a Web App
HTML
& DOM
Web App
Web Server
App Server
HTTP
via
Proxies
Dev Tools

27. @EvilTester 27
Modelling – this Web App
Web App
Web Server
App Server
Player GUI
Admin
GUI
Rest API

28. @EvilTester 28
Part the third, wherein the
overlap between technical
testing and security testing are
explored.

29. @EvilTester 29
Deeper Testing

30. @EvilTester 30
Technical Testing
● We will find defects and issues we would
otherwise miss
– Observing, Interrogating, Manipulating lower
● Some defects normally associated with security
testing
– I am not a security tester
– Technically Informed Modelling

31. @EvilTester 31
Security Overlap Examples
● Code reviews find hard-coded security issues
● Form field inspection exposed emails,
executable shell code, file names for survey
answers
● HTML Commented URLs to 'secret' parts of the
application
● User HTTP json contains more info than
displayed as HTML

32. @EvilTester 32
Security Overlap Examples
Because...
● Code reviews find hard-coded security issues
● Form field inspection exposed emails,
executable shell code, file names for survey
answers
● HTML Commented URLs to 'secret' parts of the
application
● User HTTP json contains more info than
displayed as HTML
Observed More Deeply
Interrogated More Deeply
Interrogated More Deeply
Observed, Interrogated, Manipulated
More Deeply
Because Modelled More Deeply

33. @EvilTester 33
Any Methodology : Any Tester
● Methodology context does not dictate
– 'Process' context might dictate
– 'Social' context might dictate
● Any Tester can do this
– Limited by technology knowledge
– Limited by technical skill
– Limited by choice

34. @EvilTester 34
Part the fourth, wherein the
steps to increase technical
ability are made lay afore the
public.

35. @EvilTester 35
I am fairly Technical
● Books
● SeleniumSimplified.com
● EvilTester.com
● JavaForTesters.com
● Online Training Courses
● Consultancy, work hands on
with teams

36. @EvilTester 36
I grew up with computers...
http://www.retrogamer.net/profiles/hardware/zx-spectrum-hardware-profile/

37. @EvilTester 37
I grew up reading computer books...
http://www.usborne.com/catalogue/feature-page/computer-and-coding-books.aspx

38. @EvilTester 38
...but not with the internet
http://www.worldofspectrum.org/hardware/feat24.html

39. @EvilTester 39
I had to learn how to test the web

40. @EvilTester 40
How to learn to test the web
● Model What You Know
– HTML? HTTP? Browsers?
● Increase your ability to Observe at the GUI
– View Source
– Inspect Element - Dev Tools
– You will see things you don't understand (add to
your model & research)

41. @EvilTester 41
How to learn to test the web
● Increase your ability to Manipulate at the GUI
– Inspect Element - Dev Tools
– Amend DOM prior to submitting a form
– Inspect and manipulate URLs
● Cookies, Local Storage
– Inspect
– Figure out how to manipulate (plugins required?)

42. @EvilTester 42
Basic Web Challenges
● View Source and inspect Element of:
– Your favourite web sites
● How do they do 'that'?
● Any free 'pdf' report that requires 'email'
– find the download without adding your email
● Newspapers - 'you have read too many articles
today' – how can they tell? Manipulate to
bypass?

43. @EvilTester 43
How to learn to test the web
● Observe HTTP Traffic in Browser
– Network tab in Dev Tools
● Observe HTTP Traffic outside Browser
– HTTP Proxy
– Fiddler, Charles, BurpSuite, Owasp ZAP
– Interrogate and Manipulate Traffic with a Proxy

44. @EvilTester 44
How to learn to test the web
● Learn features in the browser
– View Source, Users, Dev Tools
– How can the feature help you test?
● Learn features in the proxies
– Replay Message, Fuzzers, Auto Responders
– How can the feature help you test?

45. @EvilTester 45
Technology Basics
● Model the Technology
– Where are the gaps in your understanding?
– These gaps are risks to your testing.
● How can I observe X?
● How can I interrogate X?
● How can I manipulate X?
● Repeat

46. @EvilTester 46
Application Basics
● Model the application
● What is it actually doing?
– Not just what is it supposed to do
● How does it do X?
– Observe, Interrogate, Manipulate

47. @EvilTester 47
Going Further

48. @EvilTester 48
Pushing Functional Testing Further
Go Even Further:
● Explore and automate systems below the GUI
● GUI as API
● Quickly enter more combinations of input than
would otherwise be feasible
● Fuzzers, Setup Test Data
● Test at an API level without specialist tools
● Proxy message creation
Even

49. @EvilTester 49
Do the work!
● Research
● Experiment
● Learn
● Take Small Steps

50. @EvilTester 50
Do The Work!
● Alan Richardson
– @EvilTester
– www.EvilTester.com
– www.JavaForTester.com
– www.SeleniumSimplified.com
– www.CompendiumDev.co.uk