1.
Scalable and Effective Fuzzing
of Google Chrome Browser
Positive Hack Days VI
Moscow, 17-18 May 2016
Max Moroz, Google

2.
● Google Chrome Security team, Bugs--
● BalalaikaCr3w, LC↯BC
● CTF, BugBounty, etc
Bio
2

3.
1. ClusterFuzz overview
2. Memory Debugging Tools
3. LibFuzzer
4. Bugs statistics
5. Vulnerability Reward Program
6. Get some $$$ with ClusterFuzz
Agenda
3

4.
A software testing technique, often
automated or semi-automated,
that involves passing invalid,
unexpected or random input to a
program and monitor result for
crashes, failed assertions, races,
leaks, etc.
Definition: Fuzzing
4

5.
The Big 3s in Fuzzing
5
Fuzzers Memory Tools Scale
Know
WHERE
to
reach
Know
WHEN
you
reached
Know
HOW
to
reach
FASTER

6.
6
ClusterFuzz overview

7.
Suppliers of Chrome vulns
7

8.
ClusterFuzz
8
BackendFrontend
Bots
Builder Bots
1. Trunk (Release (-
O2+-O1), Debug)
2. Stable, Beta
Sync Bots using
GlusterFS
1. Fuzzer Data
bundles
2. Build Cache
3. Code Coverage
Local Storage
1. Tests
2. Fuzzers
3. Some Fuzzer
Data Bundles
4. Builds
Google Cloud
Storage
Blobstore
1. Custom Binaries
2. Crash testcases
3. User uploaded testcases
4. Fuzzers
5. Small data bundles
Task Pull
Queue
High replication datastore
All the metadata - Jobs, Crash
List, Crash Stats, Fuzzers, Bot
tasks, Bot heartbeats, etc
Tasks
1. Fuzz
2. Minimize
3. Impact
4. Regression
5. Fixed
6. Sheriff
7. Coverage
ClusterFuzz
UI
Appengine Google Compute Engine and Chrome Lab

9.
1. Automated crash detection, analysis and management
2. Fully reproducible and minimized testcases
3. Real-time regression and fixed testing
ClusterFuzz: Goals
9

10.
ClusterFuzz: Goals
10

11.
ClusterFuzz: Life of a Crash
11
Find Actionable, Reproducible and Minimized Tests
Verify
Test Deduplication, Regresse, Production ImpactAnalyze
Fixed, Merged, Released
Prevent Regressions

12.
12
Goal 1:
Crash automation & management

13.
Fuzzer Types
13
Generation Based
Generate from scratch with
no prior state
Example
https://bugs.webkit.org/show_bug.cgi?id=60831
<script>
document.body = document.createElement(‘iframe’);
</script>

14.
Fuzzer Types
14
Mutation Based
Mutate existing state
based on some rules
Example
crbug.com/552046
--- orig.pdf
+++ crash.pdf
@@ -57,7 +57,7 @@
/DecodeParms [null 8 0 R]
/Type /XObject
/Width 1760
-/Filter [/FlateDecode /DCTDecode]
+/Filter [/JBIG2Decode /DCTDecode]
/Height 1248
/Length 2277

15.
Fuzzer Types
15
Evolutionary
Generation or mutation
based or both, in-process
with code coverage
feedback
Example
crbug.com/575205
SELECT'xef(xfb;
DSx1aLEETABLxfeES'REGEXP';0ttC LE|A*
(xc8*.+!*)*h*00x0b$T''&'

16.
● Setup build {trunk}, run application with test
● Choose params
● Gestures, tool settings, timeout multiplier, window
location+size, etc.
● Figure out resource dependencies
● Test for reproducibility
● Test for duplicates
● Store crash, coverage, stats, etc
Infrastructure
16

17.
● Disable inline frames (-inlining in llvm-symbolizer)
● Crash_type - e.g. “Heap-buffer-overflow READ 2”
● Crash_state - top 3 frames
● +namespaces, -line_numbers
● E.g.
● WebCore::SVGDocumentExtensions::removeAnimationElement
● WebCore::SVGSMILElement::removedFrom
● WebCore::ContainerNode::removeChild
● security_flag
Testcase Duplication Check
17

18.
Code Coverage: Aggregate View Sample
18

19.
Scale, Scale and Scale (in cores)
19
12k
5k
Linux
4k
Android x86
200+
Android devices
3k
Windows
20
Mac
{wip}
iOS

20.
20
Goal 2:
Fully reproducible and minimized
testcases

21.
● Same bot configuration as crash
● Multi-threaded minimization based on Delta Debugging
● Custom minimizers for some file types
● + Gesture minimization
● + Resource minimization
● + Tool settings correction (redzone {asan}, history size
{tsan}, origins {msan}, etc)
Fully reproducible & minimized tests
21

22.
● Tokenize the input
● Generate hypotheses that certain groups of tokens are not
required for the crash
● Test hypotheses by running the test with the tokens from the
hypothesis removed
● If it crashes, removing them was fine
● If not, try breaking it into smaller groups
How does the minimizer work?
22

23.
● Stack trace with default redzone (128)
==9485== ERROR: AddressSanitizer heap-use-after-free on address
0x7f8f653ff11e at
pc 0x7f8f849fbb10 bp 0x7f8f5514a0a0 sp 0x7f8f5514a098
READ of size 2 at 0x7f8f653ff11e thread T14
#0 0x7f8f849fbb10 in WTF::charactersToIntStrict(unsigned short const*,
unsigned long, bool*, int)
#1 0x7f8f8589d863 in WebCore::InlineTextBox::isLineBreak() const
#2 0x7f8f858a771d in WebCore::InlineTextBox::containsCaretOffset(int)
● Actual stack trace with bigger redzone (1024 / 2048)
==14334== ERROR: AddressSanitizer heap-buffer-overflow on address
0x7f7e42b9b81c
at pc 0x7f7e8f79a6ca bp 0x7f7e3cc30040 sp 0x7f7e3cc30038
READ of size 2 at 0x7f7e42b9b81c thread T15
#0 0x7f7e8f79a6ca in WebCore::InlineTextBox::isLineBreak() const
#1 0x7f7e8f7abddd in WebCore::InlineTextBox::containsCaretOffset(int)
const
#2 0x7f7e8e06b19d in WebCore::Position::inRenderedText() const
Tools Settings Correction: ASAN example
crbug.com/118662
23

24.
24
Goal 3:
Real-time regression and fixed testing

25.
● Use LKGR builds archived on Google Cloud
● Account for bad builds / startup crashes
● Use a LOOK_BEHIND_WINDOW
● If previous step failed, then use binary bisect
● Use FindIt to find culprit changelist
Real-time regression and fixed testing
25

26.
● Manual owner triage is usually …
● Slow
● Inaccurate
● Complex
● From factors such as refactorings, size of regression range, etc
FindIt: Culprit CL Finder
26

27.
1. Parse the stacktrace
FindIt: How it works
27

28.
2. Parse the changelog(s) in the regression range
FindIt: How it works
28

29.
3. Generate a list of suspected CLs, and sort / filter the results
FindIt: How it works
29

30.
4. Show blame information if no results are available
FindIt: How it works
30

31.
FindIt: How it works
31

32.
32
ClusterFuzz: Sample

33.
Testcase Report
33

34.
FindIt: Culprit CL Finder
34

35.
Filed bug: crbug.com/430925
35

36.
Patched + Verified: < 1 day
36

37.
Or not patched...
37

38.
38
Memory Debugging Tools

39.
Memory Debugging Tools
39
Tool OS Android Chrome OS Linux OS X Windows
ASan + + + + +
MSan +
UBSan +
TSan +
LSan +
CFI +
SyzyASAN +
DrFuzz +
DrMemory +

40.
● AddressSanitizer (aka ASan)
● Detects use-after-free, buffer overflows (heap, stack, globals),
stack-use-after-return, container-overflow
● Cpu: 2x, memory 1.5x-3x
● ThreadSanitizer (aka TSan)
● Detects data races, esp on use-after-frees. Object vtpr
● Cpu: 4x-10x, memory: 5x-8x
Memory Debugging Tools
40

41.
● MemorySanitizer (aka MSan)
● Detects uninitialized memory reads
● Cpu: 3x, memory: 2x
● Special mode: origins
● UndefinedBehaviorSanitizer (aka UBSan)
● Detects several classes of bugs (19?), esp on type confusion,
signed-integer-overflow, undefined shift, etc.
● Cpu: 10-50%
● Memory: ~1x (no allocator, no shadow)
Memory Debugging Tools
41

42.
● Others
● SyzyASAN
● DrFuzz
● DrMemory
Memory Debugging Tools
42

43.
● Container-overflow (ASan):
#include <vector>
#include <assert.h>
typedef long T;
int main() {
std::vector<T> v;
v.push_back(0);
v.push_back(1);
v.push_back(2);
assert(v.capacity() >= 4);
assert(v.size() == 3);
T *p = &v[0];
// Here the memory is accessed inside a heap-allocated buffer
// but outside of the region `[v.begin(), v.end())`.
return p[3]; // OOPS.
}
Memory Debugging Tools: Examples
43

44.
44
LibFuzzer

45.
● Fuzzing very wide scope
● Cannot fuzz specific function
● Hard to fuzz network protocols
● Speed of regular fuzzers (html, css, dom, etc mutators):
● From ~30K to ~2.5M testcases per week
● Regular fuzzers are great, but not enough...
Problems of regular fuzzing of browsers
45

46.
● More granulated fuzzing
● Faster fuzzing
● Smarter fuzzing
● Easier fuzzer writing
Goals
46

47.
● In-process guided fuzz testing
● Very effective at function / protocol level
● It’s easy not hard to write a “target function”
● Can be checked along with unit-tests
● Maintained by Googlers :)
LibFuzzer
47

48.
libFuzzer example [code]
48
#include <stdint.h>
#include "third_party/icu/source/i18n/unicode/regex.h"
// Entry point for LibFuzzer.
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
UParseError pe = { 0 };
UErrorCode status = U_ZERO_ERROR;
URegularExpression* re = uregex_open(reinterpret_cast<const UChar*>(data),
static_cast<int>(size) / sizeof(UChar),
0, &pe, &status);
if (re)
uregex_close(re);
return 0;
}

49.
libFuzzer example [crash]
49
==19494==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x4d306d in icu_54::RegexCompile::doParseActions(int)
third_party/icu/source/i18n/regexcmp.cpp:1702:13
#1 0x4a1f3c in icu_54::RegexCompile::compile(UText*, UParseError&,
UErrorCode&) third_party/icu/source/i18n/regexcmp.cpp:228:13
#2 0x5fc98e in icu_54::RegexPattern::compile(UText*, unsigned int,
UParseError&, UErrorCode&) third_party/icu/source/i18n/repattrn.cpp:357:5
#3 0x601be7 in uregex_open_54 third_party/icu/source/i18n/uregex.cpp:155:20
#4 0x49dc3f in LLVMFuzzerTestOneInput
testing/libfuzzer/fuzzers/icu_uregex_open_fuzzer.cc:13:28
Uninitialized value was created by an allocation of compiler in the stack frame
of function _ZN6icu_5412RegexPattern7compileEP5UTextjR11UParseErrorR10UErrorCode
#0 0x5fc220 in icu_54::RegexPattern::compile(UText*, unsigned int,
UParseError&, UErrorCode&) third_party/icu/source/i18n/repattrn.cpp:325

50.
● Dictionaries
● Seed corpus
● Custom mutators
● Coverage visualisation
Ways to improve a libFuzzer
50

51.
● dictionary and custom options for ICU uregex_open() fuzzer:
○ coverage ↑ 25+%
● dictionary for libxml fuzzer:
○ 5 new security bugs
● dictionary and seed corpus for libpng fuzzer:
○ coverage ↑↑↑ 70+%
Dictionaries are awesome!
51

52.
52

53.
53

54.
54

55.
55

56.
● Timeout (DoS)
● Heap-buffer-overflow
● Use-of-uninitialized-value
● Heap-buffer-overflow
● Heap-buffer-overflow
sqlite3 fuzzer with dictionary at ClusterFuzz
56

57.
● ASan, MSan, UBSan builds
● Corpus Synchronization with Google Cloud Storage
● Support for dictionaries and custom runtime options
● Automated exploitability scoring for crashes
● Statistics dashboard
LibFuzzer & ClusterFuzz
57

58.
● ~85 target functions for Chromium
● 88 bugs filed
○ 46 security bugs
● 700 GCE bots running target functions at ClusterFuzz 24/7
● ~310 billion testcases daily
● ~8,000 corpus synchronization jobs daily
● From ~200k to ~800k corpus files
● Best coverage: 100.00% of funcs, 89.32% of edges
Numbers
58

59.
● 200+ bugs in third party, Google, other software
● http://llvm.org/docs/LibFuzzer.html#trophies:
Trophies of LibFuzzer outside of Chrome
59

60.
● Slow inputs and timeouts
● Out-of-memory with valid inputs
● You have to improve / refresh some fuzzers
● Stuck at easy crashes
Problems
60

61.
Example: 10 lines fuzzer
61
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
{
auto var = xxxxxxxxxxxxxx(data, size, NULL, 0);
if (var) {
xxxxxxxxxx_Free(var);
}
return 0;
}

62.
Example: 10 lines fuzzer ⇨ 10+ bugs
62
● Memory leak
● 7 × Heap-buffer-overflow
● Timeout (DoS)
● Heap-based buffer-underreads
● Heap-based buffer overread
● Undesired side effects

63.
Self-fuzzing!
63
WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x56b2594 in fuzzer::Fuzzer::AlarmCallback()
third_party/llvm/lib/Fuzzer/FuzzerLoop.cpp:138:7
<...>
#9 0x56abbb7 in fuzzer::FileToVector() third_party/llvm/lib/Fuzzer/FuzzerIO.cpp:
65:3
<...>
#14 0x56a08a8 in main third_party/llvm/lib/Fuzzer/FuzzerMain.cpp:25:10
Uninitialized value was created by an allocation of 'F.i' in the stack frame of
function '_ZN6fuzzer12FuzzerDriverEiPPcPFiPKhmE'
#0 0x56a1040 in fuzzer::FuzzerDriver()
third_party/llvm/lib/Fuzzer/FuzzerDriver.cpp:395

64.
64

65.
● The same target function
● Shared corpus:
● Need to be pruned or minimized on regular basis
● Different mutations:
● When one fuzzer got stuck, another one may help
● PoC is ready:
● https://github.com/llvm-mirror/llvm/blob/master/lib/Fuzzer/afl/afl_driver.cpp
● Work in progress
Multiple fuzzing engines approach
65

66.
66
return 0;

67.
67
Bugs statistics

68.
Bugs statistics
68

69.
69
Bugs statistics

70.
70
Bugs statistics

71.
71
Vulnerability Reward Program

72.
● Reporter submits a bug on crbug.com
● Bug gets fixed
● Reward panel determines reward based on table and precedent
● Reporter gets cash money
Chromium VRP: high level process
72

73.
Reward amounts
73 https://www.google.com/about/appsecurity/chrome-rewards/index.html

74.
Recent updates
74 https://www.google.com/about/appsecurity/chrome-rewards/index.html
● Max reward: $50,000 ⇨ $100,000:
● compromise a Chromebook or Chromebox with device persistence in guest
mode (i.e. guest to guest persistence with interim reboot, delivered via a web page)
● Download Protection bypass (i.e. SafeBrowsing) *:
● Baseline: $0 - $500
● High-quality report: $1,000
* Landing a blacklisted test binary on disk where a typical user could execute it, on Mac
or Windows. The file type on disk must lead to non-sandboxed code execution after
minimal user interaction with the file.

75.
Chromium VRP stats
75
● Over $500,000 paid out last year
● More than $2,500,000 in rewards since 2010
● Median payment went from $2,000 to $3,000
● Top 0x05 reporters take ~70% of the earnings

76.
76
Trusted Researcher Program

77.
ClusterFuzz’s first days...
77

78.
ClusterFuzz now
78

79.
● Invitation-only
● Skilled fuzzer developers
● Fuzz at Google scale
● Get 100% of the reward for bugs + $500 bonus:
● Bugs shouldn’t be found by our* fuzzers within 48 hours
● *our - written by Googlers
● Target functions for LibFuzzer are also in scope!
How to get an invite?
● Submit at least one bug found by one of your fuzzers
● mailto:mmoroz@chromium.org
Trusted Researcher program
79
https://www.google.com/about/appsecurity/chrome-rewards/index.html
http://www.chromium.org/Home/chromium-security/bugs/developing-fuzzers-for-clusterfuzz

80.
● More than $120,000 paid out for 2014-2015
● Rewards in range: $500 - $4,500
Stats
80

81.
Thank you
Contacts
● email: mmoroz@chromium.org
● twitter: @dor3s
● telegram: @dor1s
Q & A
81