Presentation from curl up 2019. The state of the curl project. In English.

2. The state of curl 2019

3. The curl project 2019

4. Stats
Mostly from 2010 or later
Due to data availability
Represents “the modern curl project”

5. Number of lines of “product code”
2010-02-09
2010-06-16
2010-10-12
2011-02-17
2011-04-22
2011-09-13
2011-11-17
2012-03-22
2012-07-27
2012-11-20
2013-04-12
2013-08-11
2013-12-16
2014-03-26
2014-07-16
2014-11-05
2015-02-25
2015-04-28
2015-08-11
2015-12-01
2016-02-08
2016-05-17
2016-07-21
2016-09-07
2016-11-02
2016-12-22
2017-02-24
2017-06-14
2017-08-13
2017-10-23
2018-01-23
2018-05-15
2018-09-04
2018-12-12
0
20000
40000
60000
80000
100000
120000
140000
160000
180000

6. Is a 160K a lot or a little?
A dozen TLS backends
Two SSH backends
Three name resolver backends
Feature packed; 221 command line options and 267 setopt() options
More portable than most
More compliant than most
More feature-packed than most
25% comments

7. C!
Efficient and portable!
Some security problems could be avoided using something
else
Lots of “reach” would then also be avoided
Mitigation: readable code, reviews, tests, fuzzing, static code
analyzing

8. Coverity on curl – fixed defects

9. Coverity on curl – defects over time

10. OSS-Fuzz reports over time
2017-06
2017-07
2017-08
2017-09
2017-10
2017-11
2017-12
2018-01
2018-02
2018-03
2018-04
2018-05
2018-06
2018-07
2018-08
2018-09
2018-10
2018-11
2018-12
2019-01
2019-02
2019-03
0
2
4
6
8
10
12
14
16

11. Test cases over time
2010-02-09
2010-06-16
2010-10-12
2011-02-17
2011-04-22
2011-09-13
2011-11-17
2012-03-22
2012-07-27
2012-11-20
2013-04-12
2013-08-11
2013-12-16
2014-03-26
2014-07-16
2014-11-05
2015-02-25
2015-04-28
2015-08-11
2015-12-01
2016-02-08
2016-05-17
2016-07-21
2016-09-07
2016-11-02
2016-12-22
2017-02-24
2017-06-14
2017-08-13
2017-10-23
2018-01-23
2018-05-15
2018-09-04
2018-12-12
0
200
400
600
800
1000
1200
1400

12. Source vs tests over time
2010-02-09
2010-08-11
2011-02-17
2011-06-23
2011-11-17
2012-05-24
2012-11-20
2013-06-22
2013-12-16
2014-05-20
2014-11-05
2015-04-22
2015-08-11
2016-01-27
2016-05-17
2016-08-03
2016-11-02
2017-02-22
2017-06-14
2017-10-04
2018-01-23
2018-07-11
2018-12-12
0
20000
40000
60000
80000
100000
120000
140000
160000
-100
100
300
500
700
900
1100
1300
Test cases
Lines of code
Linesofcode
Numberoftestcases

13. Source lines per test file since 20107.20.0
7.20.1
7.21.0
7.21.1
7.21.2
7.21.3
7.21.4
7.21.5
7.21.6
7.21.7
7.22.0
7.23.0
7.23.1
7.24.0
7.25.0
7.26.0
7.27.0
7.28.0
7.28.1
7.29.0
7.30.0
7.31.0
7.32.0
7.33.0
7.34.0
7.35.0
7.36.0
7.37.0
7.37.1
7.38.0
7.39.0
7.40.0
7.41.0
7.42.0
7.42.1
7.43.0
7.44.0
7.45.0
7.46.0
7.47.0
7.47.1
7.48.0
7.49.0
7.49.1
7.50.0
7.50.1
7.50.2
7.50.3
7.51.0
7.52.0
7.52.1
7.53.0
7.53.1
7.54.0
7.54.1
7.55.0
7.55.1
7.56.0
7.56.1
7.57.0
7.58.0
7.59.0
7.60.0
7.61.0
7.61.1
7.62.0
7.63.0
7.64.0
7.64.1
120
130
140
150
160
170
180
The y-axis is not zero-based!

14. Test coverage
Good to know, hard to measure
72 - 78% on coveralls.io
For a single TLS – SSH – resolver – config setup!
Some tests too slow for coverage runs in the cloud (torture)
Some code paths still hard to test with existing test suite

15. Daniel’s share of curl commits
2010-01-13
2010-08-10
2011-04-16
2011-11-01
2012-06-19
2013-02-06
2013-07-09
2013-12-22
2014-05-04
2014-10-29
2015-02-19
2015-08-20
2016-03-29
2016-10-24
2017-04-29
2017-09-15
2018-05-02
2018-11-23
0
10
20
30
40
50
60
70
80

16. Commits per release since 2010
7.20.0
7.20.1
7.21.0
7.21.1
7.21.2
7.21.3
7.21.4
7.21.5
7.21.6
7.21.7
7.22.0
7.23.0
7.23.1
7.24.0
7.25.0
7.26.0
7.27.0
7.28.0
7.28.1
7.29.0
7.30.0
7.31.0
7.32.0
7.33.0
7.34.0
7.35.0
7.36.0
7.37.0
7.37.1
7.38.0
7.39.0
7.40.0
7.41.0
7.42.0
7.43.0
7.44.0
7.45.0
7.46.0
7.47.0
7.47.1
7.48.0
7.49.0
7.49.1
7.50.0
7.50.1
7.50.2
7.50.3
7.51.0
7.52.0
7.52.1
7.53.0
7.53.1
7.54.0
7.54.1
7.55.0
7.55.1
7.56.0
7.56.1
7.57.0
7.58.0
7.59.0
7.60.0
7.61.0
7.61.1
7.62.0
7.63.0
7.64.0
7.64.1
0
50
100
150
200
250
300
350
400
450
500

17. Commits per year
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
0
200
400
600
800
1000
1200
1400
1600
1800

18. Commit authors in curl since 20102010-01
2010-03
2010-05
2010-07
2010-09
2010-11
2011-01
2011-03
2011-05
2011-07
2011-09
2011-11
2012-01
2012-03
2012-05
2012-07
2012-09
2012-11
2013-01
2013-03
2013-05
2013-07
2013-09
2013-11
2014-01
2014-03
2014-05
2014-07
2014-09
2014-11
2015-01
2015-03
2015-05
2015-07
2015-09
2015-11
2016-01
2016-03
2016-05
2016-07
2016-09
2016-11
2017-01
2017-03
2017-05
2017-07
2017-09
2017-11
2018-01
2018-03
2018-05
2018-07
2018-09
2018-11
2019-01
2019-03
0
2
4
6
8
10
12
14
16
18
20
22
24
26
28
0
100
200
300
400
500
600
700
First Commit Authors Total count
Date
Authorspermonth
Authorsoveralltime

19. Authors per month, excluding first-timers
2010-01
2010-04
2010-07
2010-10
2011-01
2011-04
2011-07
2011-10
2012-01
2012-04
2012-07
2012-10
2013-01
2013-04
2013-07
2013-10
2014-01
2014-04
2014-07
2014-10
2015-01
2015-04
2015-07
2015-10
2016-01
2016-04
2016-07
2016-10
2017-01
2017-04
2017-07
2017-10
2018-01
2018-04
2018-07
2018-10
2019-01
0
2
4
6
8
10
12
14
16
18
20

20. Top-10 commit author share since forever
Marc Hoersken
Kamil Dudka
Patrick Monnerat
Jay Satiro
Gisle Vanem
Guenter Knauf
Dan Fandrich
Steve Holme
Yang Tse
(The rest)
Daniel Stenberg
0 10 20 30 40 50 60

21. Top-10 commit author share since 2017
Kamil Dudka
Viktor Szakats
Johannes Schindelin
Michael Kaufmann
Daniel Gustafsson
Dan Fandrich
Patrick Monnerat
Jay Satiro
Marcel Raad
(The rest)
Daniel Stenberg
0 10 20 30 40 50 60

22. Days between curl releases since 2010
Average: 50 Median: 56
7.20.0
7.20.1
7.21.0
7.21.1
7.21.2
7.21.3
7.21.4
7.21.5
7.21.6
7.21.7
7.22.0
7.23.0
7.23.1
7.24.0
7.25.0
7.26.0
7.27.0
7.28.0
7.28.1
7.29.0
7.30.0
7.31.0
7.32.0
7.33.0
7.34.0
7.35.0
7.36.0
7.37.0
7.37.1
7.38.0
7.39.0
7.40.0
7.41.0
7.42.0
7.43.0
7.44.0
7.45.0
7.46.0
7.47.0
7.47.1
7.48.0
7.49.0
7.49.1
7.50.0
7.50.1
7.50.2
7.50.3
7.51.0
7.52.0
7.52.1
7.53.0
7.53.1
7.54.0
7.54.1
7.55.0
7.55.1
7.56.0
7.56.1
7.57.0
7.58.0
7.59.0
7.60.0
7.61.0
7.61.1
7.62.0
7.63.0
7.64.0
7.64.1
0
10
20
30
40
50
60
70
80
90
Max: 83 Min: 2

23. Bug-fixes per release since 20107.20.0
7.20.1
7.21.0
7.21.1
7.21.2
7.21.3
7.21.4
7.21.5
7.21.6
7.21.7
7.22.0
7.23.0
7.23.1
7.24.0
7.25.0
7.26.0
7.27.0
7.28.0
7.28.1
7.29.0
7.30.0
7.31.0
7.32.0
7.33.0
7.34.0
7.35.0
7.36.0
7.37.0
7.37.1
7.38.0
7.39.0
7.40.0
7.41.0
7.42.0
7.42.1
7.43.0
7.44.0
7.45.0
7.46.0
7.47.0
7.47.1
7.48.0
7.49.0
7.49.1
7.50.0
7.50.1
7.50.2
7.50.3
7.51.0
7.52.0
7.52.1
7.53.0
7.53.1
7.54.0
7.54.1
7.55.0
7.55.1
7.56.0
7.56.1
7.57.0
7.58.0
7.59.0
7.60.0
7.61.0
7.61.1
7.62.0
7.63.0
7.64.0
7.64.1
0
20
40
60
80
100
120
140

24. Bug-fixes per day since 2010February92010
April142010
June162010
August112010
October132010
December152010
February172011
April172011
April222011
June232011
September132011
November152011
November172011
January242012
March222012
May242012
July272012
October102012
November202012
February62013
April122013
June222013
August122013
October142013
December172013
January292014
March262014
May212014
July162014
September102014
November52014
January82015
February252015
April222015
April292015
June172015
August122015
October72015
December22015
January272016
February82016
March232016
May182016
May302016
July212016
August32016
September72016
September142016
November22016
December212016
December232016
February222017
February242017
April192017
June142017
August92017
August142017
October42017
October232017
November292017
January242018
March142018
May162018
July112018
September52018
October312018
December122018
February62019
March272019
0
1
2
3
4
5
6

25. Vulnerability reports since 2010
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
0
5
10
15
20
25

26. Lessons from past vulnerabilities?
Integer overflows are tricky things – different architectures make them
more so
Most flaws linger in the code a long time until detected
Fuzzing is king
Fixing the flaws is usually straight-forward
Bug bounties can help

27. Top-20 changed source files since 2010
lib/url.c
lib/vtls/openssl.c
lib/imap.c
lib/http2.c
lib/smtp.c
lib/multi.c
lib/pop3.c
include/curl/curl.h
src/tool_getparam.c
lib/transfer.c
src/tool_operate.c
lib/http.c
lib/connect.c
lib/urldata.h
lib/ssh.c
include/curl/curlver.h
lib/ftp.c
lib/curl_sasl.c
lib/vtls/darwinssl.c
lib/vtls/nss.c
0 50 100 150 200 250

28. Annual user survey
What is used, what is ignored
What is good, what is bad
What should be added, what should be removed
How are we doing

29. How good is the project to handle
2014 2015 2016 2017 2018
3
3.2
3.4
3.6
3.8
4
4.2
4.4
4.6
4.8
5
security
credit
patches
bug reports
information
newcomers
minorities
(According to the annual user survey)

30. curl’s top-5 areas according to users
the libcurl API
the support of many protocols
documentation
its availability and functionality on many platforms
the quality of the products, curl/libcurl
0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00%
2017 2018

31. curl’s worst-5 areas according to users
project web site and infrastructure
welcoming to new users and contributors
the libcurl API
its build environment/setup
documentation
0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00%
2017 2018

32. User survey 2019
Around May time frame
Very much interested in feedback on where to take it and
what to ask for
Received 670 responses 2018
https://daniel.haxx.se/blog/2018/06/12/curl-survey-2018-analysis/

33. Web site traffic 2019
Fastly makes our lives easier
1.5 million requests/day (from 1.8)
41.6 TB the last 12 months
Fast web site, close to most users
No logs, no tracking, very little stats

34. [curl] 34,550 times
[libcurl] 2,510 times

35. Google trends, worldwide search
Wget rsync curl
Includes wget and rsync only to provide references with similar projects

36. CII Best Practices
https://bestpractices.coreinfrastructure.org/en/projects/63
100% passing
96% Silver
26% Gold
“SHOULD have a legal mechanism
where all developers of non-trivial
amounts of project software assert
that they are legally authorized to
make these contributions”

37. Everyone uses curl 2019
Apps: Youtube, Instagram, Skype, Spotify, ...
OS: iOS, macOS, Windows, Linux, ChromeOS, AOSP, ...
Cars: Mercedes, BMW, Toyota, Nissan, Volkswagen, …
Game consoles: PS4, Nintendo Switch, ...
Games: Fortnite, Red Dead Redemption 2, Spider Man, …
Estimate: 6 billion installationsEstimate: 6 billion installations

38. Done the last 12 months

39. Defaults (1/4)
multiplexing enabled by default
defaults to "2TLS"
leave secure cookies alone
high resolution timestamps on Windows
headers output in bold

40. New features (2/4)
DNS-over-HTTPS support
URL parsing API
curl_easy_upkeep()
--resolve supports wildcard hosts
trailing headers support for chunked transfer uploads
alt-svc

41. Improvements (3/4)
%{stderr} and %{stdout} for --write-out
support for HTTP Bearer tokens
IMAP changed from "FETCH" to "UID FETCH"
MesaLink is a new TLS backend
microsecond resolution timers for seven getinfo intervals

42. New setopts (4/4)
CURLOPT_CURLU
CURLOPT_UPLOAD_BUFFERSIZE
CURLOPT_TLS13_CIPHERS and CURLOPT_PROXY_TLS13_CIPHERS
CURLOPT_DISALLOW_USERNAME_IN_URL
CURLOPT_HAPROXYPROTOCOL
CURLOPT_DNS_SHUFFLE_ADDRESSES
(the alt-svc pair)

43. Everything curl
70K words, 10K lines
332 pages (PDF version)
“95.1% complete”
https://ec.haxx.se/

44. Everything curl – printed
https://curl.haxx.se/book.html

45. Less good
Flaky tests/CI
Slow CI tests
Vulnerabilities are still reported
Still regressions, but less frequently?
Could use more people who stick around

46. FutureFuture

47. Planning
I can’t tell what “we” will do
I have some ideas about what to do next
Things change all time time
Tell us what you want!

48. Version 8
Release every 56 days
7.65.0 is next
A bump in every release gives us 35 * 56 = 1960 days until
version 7.100
I want to avoid reaching 7.100 due to confusions it’ll create
1960 days == 5 years and 4.5 months == September 2024
Evolutionary, not revolutionary?

49. libcurl work to consider
Keep up with browsers
HTTP/3 and QUIC
ESNI
Hardcode localhost
Refuse HTTP => HTTPS redirects
Option to let
CURLOPT_CUSTOMREQUEST be
overridden on redirect
HSTS
"menu config"-style build
feature selection

50. New APIs?
Config file reader

51. curl tool work to consider
Parallel transfers
Support for HTTP/2 Push
Master/slave mode
Make --retry resume
This list is identical to
last
year’s curl tool list!

52. Finally