An introduction to how the Linux kernel works: maintianers, scaling trust, and no regressions. This talk also gives tips to people who want to get involved with Linux kernel development, either through reporting bugs, reviewing code, or developing code.

1. Linux Device Drivers
Sarah Sharp
Dec 6, 2014

2. Sarah Sharp
● USB 3.0 host controller driver author
● Linux kernel developer for 9 years
● Worked in Intel's Open Source
Technology Center (OTC) for 7 years
● Driver maintainer for 5 years

3. How did I get into
open source?
● Didn't program until college
● Portland State University
● My boyfriend introduced me to
– open source, Linux, rockets

4. Portland State Aerospace Society

5. Rockets and Open Source

6. How did I get into
Linux kernel development?
● Professor Bart Massey
introduced me to
– The local Linux kernel community
– Greg Kroah-Hartman
● Greg suggested usbfs2 project
● Bart helped me get:
– College credit for kernel project
– Intel undergraduate research grant

7. How did I get a
job in open source?
● Bart encouraged me to present at
OSCON on usbfs2
● Kristin Accardi was on
OSCON paper selection
committee
● Through network
(Bart → Greg → Kristen)
I got a job

8. Sarah Sharp: Linux kernel
USB 3.0 driver developer
● Hired by Intel in July 2007
● USB 3.0 specification draft
● FPGA prototypes
● Driver merged in April 2009

9. What is it like to be a
Linux kernel developer?
● very tight-knit group, very slow turn over
● 80% of kernel developers are paid
● 20% are volunteers and students
● all work done through mailing lists
● a few IRC channels
● rarely get together in person

10. Linux kernel subsystems
● Many different parts of the kernel
● Broken into subsystems
● Each has a maintainer
● Most have a mailing list
● MAINTAINERS file
● scripts/get_maintainer.pl

11. What Do
Subsystem Maintainers Do?
● Experts in their particular field
● Usually 5-20 years of experience
● Approve changes to their area
● Often have 1-4 sub-maintainers
● This is how Linux scales trust

12. Intel
Linux Kernel Maintainers
● ACPI - Rafael Wysocki, Zhang Rui
● Bluetooth - Marcel Holtmann
● NFC - Samuel Ortiz
● USB 3.0 - Mathias Nyman
● Intel graphics - Daniel Vetter
● EFI - Matt Fleming
● x86 - H. Peter Anvin, Peter Z
● RAS, machine check - Tony Luck
● Code ellision - Andi Kleen
● NVMe - Matthew Wilcox
● IOMMU - Dave Woodhouse
● Temperature - Fenghua Yu
● DMA offload - Vinod Koul
● Wimax - Inaky Perez-Gonzalez
● Intel wifi - Johannes Berg
● Intel idle - Len Brown
● Intel ethernet - Jeff Kirsher, Jesse
Brandeburg, Carolyn Wyborny
● Intel C600 SAS, IOAT - Dan
Williams
● Serial, GMA500/600/3600, misc -
Alan Cox

13. What is Being a
Linux Kernel Maintainer Like?
● Hundreds of emails a day
● 1:Many relationship with
contributors
● Rapid development cycle
● Looking at long-term health
of the project
● Maintains code for 3 - 20 years
● Reputation is on the line for
each patch

14. “Meritocracy”
● Maintainers are vested
● Developers will come and go
● Long-standing contributors gain trust
● Easier for them to do complex things
● Harder for newcomers

15. Community Trust and
Social Engineering
● Build a trust relationship with maintainers
– Review patches
– Ask intelligent questions
– Submit small bug fixes
● Learn community norms
– Email format
– Coding style
– Testing

16. Tips for
Asking Intelligent Questions
● Always Cc the maintainer
● Do your research
● Document where you looked
● Be concise and precise
● Ask a question you know the
answer to, then one you don't

17. Tips for
Bug Reporting
● Always Cc the maintainer
● System information is crucial
– What hardware?
– What kernel, what distro
– 32-bit vs 64-bit
– lspci, lsusb, acpidump
● Reproducibility thinking
● Be responsive
● Hold onto hardware

18. Linux Kernel
Quality Assurance
● Regressions are not tolerated
● Buggy new code is fixed or reverted
● Shifting towards static analysis tools
– sparse, smatch, Coccinelle
● 0-day testing infrastructure
● kselftest

19. Getting into Linux kernel
development
● Read the code, review other people's patches
● Participate and ask questions on the mailing list
● First patch tutorial
– http://kernelnewbies.org/FirstKernelPatch
● Staging cleanup patches
– http://kernelnewbies.org/CheckpatchTips
● Linux kernel exercises
– http://eudyptula-challenge.org/

20. Leveling Up in
Linux Kernel Development
● Larger patchsets
● Use complex tools (sparse, Coccinelle)
● Find an area you're passionate about
● Look for todos, bugzilla entries
● Before starting a large task:
– Always ask the maintainer
– build trust with the maintainer

21. Tips for Code Review
● Keep a list of the feedback
● Address ALL feedback in revisions
● Track changes in your cover letter
● If you disagree with a maintainer,
back yourself up with facts
● Maintainers don't care about your deadline
● Code must be complete and clean
● Duplicate code or drivers won't be accepted

22. More Linux Kernel Resources
● Linux Device Drivers book:
http://lwn.net/Kernel/LDD3
● Linux Device Driver tutorial:
https://github.com/gregkh/kernel-tutorial
● FOSS Outreach Program for Women (OPW)
(paid open source and Linux kernel internships)
http://gnome.org/opw

23. Photo Attribution
● Tux:
http://upload.wikimedia.org/wikipedia/commons/thumb/b/b0/NewTux.svg/400px-
NewTux.svg.png
● Jamey and Sarah Sharp: https://www.flickr.com/photos/igalko/7570033604
● Sarah Sharp and PSAS photos CC BY-SA-NC Sarah Sharp
● Bart Massey: https://www.flickr.com/photos/reidab/9326868060
● OSCON sign: https://www.flickr.com/photos/kubina/912778997
● USB 3.0 host PCI card: https://www.flickr.com/photos/hades2k/9069181710
● Logitech webcam: https://www.flickr.com/photos/bshunrichs/162199777
● Greg Kroah-Hartman: https://www.flickr.com/photos/tian2992/6208491794
● David Brownell:
https://www.flickr.com/photos/13825348@N03/sets/72157608741347102/detail/

24. Photo Attribution
● Computer guts: https://www.flickr.com/photos/orijinal/4832982682
● Kernel Summit: https://www.flickr.com/photos/linuxfoundation/2987611750
● Mail:
https://www.flickr.com/photos/donebythehandsofabrokenartist/1502717818
9
● Nerd merit badge: https://www.flickr.com/photos/momentum/4203541703
● Trust: https://www.flickr.com/photos/dobs/10726756606
● Question: https://www.flickr.com/photos/marcobellucci/3534516458
● Bug: https://www.flickr.com/photos/103499652@N04/14232715868
● Quality: https://www.flickr.com/photos/dieselbug2007/414348333
● Editing: https://www.flickr.com/photos/nics_events/2349631515