4. curl CI
100+ builds and test “rounds” per commit
Tests code style, indenting etc
Thousands of tests per build
Builds and tests on tens of platforms
20-25 hours of CI per commit
@bagder
@bagder
5. Writing test cases should be easy
A curl test case is a single file in a human readable
well-documented format
• conditions and features needed to run
• what command (line) to run
• what the test wants returned from a server
• how the protocol exchange should look like
• stdout and stderr contents
• expected error code
• … and more
@bagder
6. Custom test servers
We avoid using “real” servers for testing
Test servers are as dumb as possible
Controlled from the test case what it should send and expect
Allows “crazy” behaviors and send/receive “anything”
Makes the test servers smaller and simpler
Helps with test suite portability
TLS is done by stunnel-fronting
Servers run on random port numbers
@bagder
7. Tools of the trade
Valgrind
Clang sanitizers
Clang tidy
“torture tests”
Scan-build
Lgtm
Lift
CodeQL
Monocle AI
Deepcode AI
Coverity
Zuul CI
Appveyor
Cirrus CI
Circle CI
Github Actions
Azure Pipelines
Buildbots
OSS-Fuzz
CI-fuzz
@bagder
@bagder
8. Torture tests – error injection
Build with a debug option
Use wrapper functions for fallible
functions
Each wrapper function can
optionally return error
The complete individual test case
is first run once
count fallible function invokes
rerun the test case that number of
times and for each iteration make
next fallible function fail
Verify nothing crashed and no
memory leaked
Repeat for all tests
@bagder
9. Source code policy
Fix all warnings (eye roll)
No defects left
Use the strictest and most picky options
As many tests as possible
Fix security issues as soon as possible
@bagder
@bagder
10. A million build combos, 86 OSes and 22 CPUs
Testing all combinations is simply not possible
Test the common setups
Test on as many platforms as possible
Test on several different CPU architectures
“white spots” in test coverage handled by review
Users keep finding untested areas and build combinations
@bagder
12. Code audit pending
via OSTIF: Open Source Technology Improvement Fund
sponsored by OpenSSF
performed by Trail of Bits
during September 2022
@bagder
19. But does it work?
10 billion installations is no proof
★ Maybe decreasing number of CVEs introduced
★ Decreasing number of OSS-Fuzz reports over time
★ manageable(?) number of C mistakes
★ Increasing bounty rewards
@bagder
24. License
This presentation and its contents are
licensed under the Creative Commons
Attribution 4.0 license:
http://creativecommons.org/licenses/by/4.0/
@bagder