This document discusses fuzz testing techniques. It describes generating random data and inputs to test systems in unexpected ways. This allows discovering bugs like hangs, crashes, and performance issues. Examples mentioned include testing a math library by generating random expressions, testing a text messaging system by simulating many parallel messages, and testing a tree parsing algorithm with huge randomly generated directory trees. Fuzz testing uncovered bugs in areas like regular expressions, logging libraries causing bottlenecks, and parsing hangs.

1. Fuzz Testing
Atul S. Khot (atul.khot@gmail.com)
VodQA ThoughtWorks Pune - 2013

2. Random behavior aka Insanity

Testing the “drink maker”

lemon juice + milk + tea leaves + (black?) salt

Rather a fuzzy drink ;-)


We human beings are somewhat “conditioned” - computers
aren't
And that is good!!!

3. Of talking gibberish

Try throwing senseless data at your system

And see what is uncovered



Hangs/infinite loops/exceptions/Deadlocks/race conditions whatever ;-)
Better let the computer go insane (it is all raring to go...)
And no call to recall your initial C days... Pointers going
haywire? Etc...

4. Is tommath right?


How do I test tommath gets its arithmetic right?
Generate random numbers – next generate artihmetic
expressions (*,/,+,-)

Run the expressions throught tommath

Run the expressions through gnu bc


Compare – 30 million – different expressions – over 4 days You get a fair good idea
All gory details in my Linux For You article

5. Uncovering performance
bottlenecks





A campaign manager – customer needs to send a text sms to
16 million cell numbers
Cannot test – as one run would cost $35000/Decouple (very handy techique) – instead of sending to real
webservice – send it to a mock
Shell scripts run in parallel – you can spawn many thousand
parallel processes easily...
Each process is a simple socket client – sending a mobile
number – and the message

6. The surprise is revealed

Our algorithms were right

No big deadlocks

For this huge run – profiler indicated log4j as the culprit

Log4j's writing to a log file – was a bottleneck


Solution - use an Async appender – Events are logged
asynchronously
Nobody thought log4j as a possible suspect ;-)

7. Ideas galore




Needed to test a complex tree manipulation algorithm written
in TCL
I coded the algorithm – to test I needed very big trees
Directories – Perl slicing and dicing – C++ boost library (open
source) – Files correspond to leaves in the tree
Directories are essentially random trees –

8. Bugs surface...

Revealed a bug - we needed to make some regex greedier

Was a corner case

Hard to see how we could have come upon it with manual
testing

A TCL expert from Norway carefully reviewed

Okayed – big moment ;-)

9. Platypus – (http://platypus.pz.org/)

It is just (?) simplified Latex

Elaborate parser

Fuzz unleashed

Produced a hang

Deemed low priority –

Will eventually get addressed

10. Platypus – (http://platypus.pz.org/)

It is just (?) simplified Latex

Elaborate parser

Fuzz unleashed

Produced a hang

Deemed low priority –

Will eventually get addressed