Every kernel developer knows that Linux comes with plenty of precious documentation as an integral part. From coding style to how to post patches, almost everything has been documented. However, history shows that error is human nature. Sometimes developers do not well know Don’ts, but there are also cases when they make mistakes despite being aware of such rules. Why this happen is unsolved, but a documentation, so far missing, of the consequences of this misbehavior could discourage it. The presenter is project manager of TOMOYO Linux, a security enhancement feature merged in version 2.6.30. Thinking open-minded, he decided to share the errors his project made, wishing it could be a helpful warning to other projects, especially newcomers. In this presentation, it will try to explain the mistake circumstances in TOMOYO Linux project, highlighting the thoughts of project members and the community reactions. No prior kernel development experiences are necessary.

1. KERNEL DEVELOPMENT:
DRAWING LESSONS FROM
"MISTAKES"
Japan Linux Symposium 2009
October 23, 2009
Toshiharu Harada
haradats@nttdata.co.jp
NTT DATA CORPORATION

2. ABSTRACTS
Every kernel developer knows that Linux comes with plenty of precious
documentation as an integral part. From coding style to how to post
patches, almost everything has been documented. However, history
shows that error is human nature. Sometimes developers do not well
know Don’ts, but there are also cases when they make mistakes despite
being aware of such rules. Why this happen is unsolved, but a
documentation, so far missing, of the consequences of this misbehavior
could discourage it. The presenter is project manager of TOMOYO
Linux, a security enhancement feature merged in version 2.6.30.
Thinking open-minded, he decided to share the errors his project
made, wishing it could be a helpful warning to other projects, especially
newcomers. In this presentation, it will try to explain the mistake
circumstances in TOMOYO Linux project, highlighting the thoughts of
project members and the community reactions.

3. “Experience is the name everyone gives to
his mistakes” --- Oscar Wild

4. WHAT’S THIS ALL ABOUT

5. WHAT’S THIS ALL ABOUT
• Linux comes with a set of great documentation

6. WHAT’S THIS ALL ABOUT
• Linux comes with a set of great documentation
• DOs and DON’Ts are there already

7. WHAT’S THIS ALL ABOUT
• Linux comes with a set of great documentation
• DOs and DON’Ts are there already
• Yet we keep on making mistakes

8. WHAT’S THIS ALL ABOUT
• Linux comes with a set of great documentation
• DOs and DON’Ts are there already
• Yet we keep on making mistakes
• Finding a missing piece

9. WHAT IS MISSING?

10. WHAT IS MISSING?
• In my humble opinion:

11. WHAT IS MISSING?
• In my humble opinion:
• Human nature (hard to fix)

12. WHAT IS MISSING?
• In my humble opinion:
• Human nature (hard to fix)
• Most of us don’t really like readings (hard to fix)

13. WHAT IS MISSING?
• In my humble opinion:
• Human nature (hard to fix)
• Most of us don’t really like readings (hard to fix)
• Real-life examples taken from TOMOYO Linux project

14. WHO AM I?

15. WHO AM I?
• Project manager of TOMOYO Linux

16. WHO AM I?
• Project manager of TOMOYO Linux
• What is “project manager”?

17. WHO AM I?
• Project manager of TOMOYO Linux
• What is “project manager”?
• Something put in between an Open Source projects and a
Company

18. WHO AM I?
• Project manager of TOMOYO Linux
• What is “project manager”?
• Something put in between an Open Source projects and a
Company
• It’s an adventurous role (experimental stage)

19. OUCH, IS THIS “SECURE LINUX” TALK?
• No. (so please remain seated, you are safe)

20. COVERED TOPICS
• Chapter 1: Where to find DOs and DON’Ts
• Chapter2: TOMOYO Linux posting history
overview
• Chapter
3: Step by step introduction of DOs and
DON’Ts of the TOMOYO Linux

21. CHAPTER 1
Where to find DOs and DON’Ts

22. Gentle Reminder

23. Gentle Reminder
Documentation is a part of Linux
Kernel

24. Gentle Reminder
Documentation is a part of Linux
Kernel
After checking out the kernel, cd to
“Documentation”

25. Gentle Reminder
Documentation is a part of Linux
Kernel
After checking out the kernel, cd to
“Documentation”
Problem is “there are just too many
files and directories” and people prefer
coding than reading

26. How Great is It?

27. How Great is It?
$Documentation/ManagementStyle

28. How Great is It?
$Documentation/ManagementStyle
“Most people are idiots, and
being a manager means you'll
have to deal with it, and
perhaps more importantly, that
_they_ have to deal with
_you_.”

29. How Great is It?

30. How Great is It?
$Documentation/ManagementStyle

31. How Great is It?
$Documentation/ManagementStyle
“Thing will go wrong,
and people want
somebody to blame. Tag
you’re it.”

32. Okay, I’ll do so

33. Okay, I’ll do so

34. Okay, I’ll do so
Me
Blame

36. The truth is people will blame you regardless
of you are tagged or not (you can omit)
Just these two statements illustrate the
essential part of managements
Linux documentation is so
practical

37. Minimal Reading

38. Minimal Reading
Entire Scheme

39. Minimal Reading
Entire Scheme
$Documentation/HOWTO

40. Minimal Reading
Entire Scheme
$Documentation/HOWTO
Submitting Patches

41. Minimal Reading
Entire Scheme
$Documentation/HOWTO
Submitting Patches
$Documentation/SubmitChecklist

42. Minimal Reading
Entire Scheme
$Documentation/HOWTO
Submitting Patches
$Documentation/SubmitChecklist
$Documentation/CodingStyle

43. References

44. References
Note that the title is “How to Participate in
the Linux Community”

45. References
Note that the title is “How to Participate in
the Linux Community”
Making your code upstream
means your participation in the
Linux Community (Be nice!)

46. My favorite one
http://www.linuxfoundation.jp/jp_uploads/
seminar20070710/Jon-Dev-Process.pdf

47. Picked up Two pages

51. I was laughing when I saw the slides for the
first time (in 2007)
When I came to realize that it was true, I
couldn’t laugh any more ...
They kept asking me “not yet?” ;-)

52. CHAPTER 2
TOMOYO Linux by Numbers

53. Leo Tolstoy said
All Happy Families Resemble Each
Other, Each Unhappy Family Is
Unhappy in Its Own Way.

54. 2,700,000
9,230
2,700,000

55. Number of employees
2,700,000
of NTT DATA
2,700,000
CORPORATION

56. 2,700,000
3
2,700,000

57. 2,700,000
Number of project
2,700,000
members

58. 2,700,000
0.0325%
2,700,000

59. Possibilities to be
2,700,000
assigned to the
2,700,000
project

60. 2,700,000
3
2,700,000

61. TOMOYO is the 3rd
2,700,000
(and the latest) LSM
2,700,000
module merged
upstream

62. 2,700,000
0
2,700,000

63. Number of people
2,700,000
who expected
2,700,000
TOMOYO would be
merged

64. 2,700,000
15
2,700,000

65. 2,700,000
We posted patches 15
2,700,000
times

66. 2,700,000
716
2,700,000

67. 2,700,000
Merged since 716
days after the first
2,700,000
post

68. 2,700,000
162
2,700,000

69. Number of comments
2,700,000
from LKML
2,700,000
(0.2 comments/day)

70. Proposal History
http://tomoyo.sourceforge.jp/wiki-e/?JLS2009

71. CHAPTER 3
Drawing Lessons from the “Mistakes”
of TOMOYO Linux Project

72. CHAPTER 3
Drawing Lessons from the “Mistakes”
of TOMOYO Linux Project
B lame
Me

73. IN THE 1ST POSTING
I wrote:
snip
All right, that's almost everything. Please
visit the following
URL for the code and documents:
http://tomoyo.sourceforge.jp/wiki-e/
If you want to see the code first, then:
http://tomoyo.sourceforge.jp/cgi-bin/lxr/
source/security/tomoyo/?v=linux-2.6.21.3-
tomoyo-2.0

74. DON’T
Send URL
Send patches

75. DON’T
Send URL
Send patches

76. WHAT HAPPENED?
•Igot a personal message from Stephen Smalley, a
maintainer of that famous SELinux
–“If you really want feedback or to get your code
into the kernel, you need to do more than post a
URL to the code - you need to break your code
down into a number of patches and post them,
just like the AppArmor folks have been doing. “

77. SO WE RUSHED TO POSTED
PATCHES NEXT DAY
• Pavel Machek gave a comment
•“Looks whitespace-damaged to me.”

78. DON’T
Ignore the Linux standard coding style
Always apply checkpatch.pl

79. DON’T
Ignore the Linux standard coding style
Always apply checkpatch.pl

80. DO

81. DO
• Carefully read the $Document/CodingStyle

82. DO
• Carefully read the $Document/CodingStyle
• Check your code with $scripts/checkpatch.pl

83. DO
• Carefully read the $Document/CodingStyle
• Check your code with $scripts/checkpatch.pl
• Also use other $scripts/check*.pl

84. • Jiri Kosina pointed us to make patches bisectable
• “Justa trivial minor nitpick - IMHO this breaks
bisectability. It might be better to add the Kconfig/
Makefile patch at the end of the whole series, so
that bisect doesn't end up in the tree in which
Makefile references non-existing files/directories.”

85. DO
• Add the Kconfig/Makefile patch at the end of the whole
series, so that bisect doesn't end up in the tree in which
Makefile references non-existing files/directories.”

86. DO
• Add the Kconfig/Makefile patch at the end of the whole
series, so that bisect doesn't end up in the tree in which
Makefile references non-existing files/directories.”

87. IN THE 3RD PATCH
• James Morris taught us series of patches should form a
thread
•“I'dalso suggest making all of the
patches a reply to the first email, so
they can be threaded.”

88. LIKE THIS

89. DO
• Send series of patches as children of the first message so that
they can form a thread or you want people to read your
messages

90. DO
• Send series of patches as children of the first message so that
they can form a thread or you want people to read your
messages

91. • James Morris said:
• “Please use standard kernel list handling, per include/linux/list.h”
• YOSHIFUJI Hideaki also mentioned:
• You'reintroducing a custom API, which is open-coded repeatedly
throughout your module.
• All linked lists (at least, new ones) must use the standard kernel
list API.

92. DON’T
Propose new data structure
Use existing one

93. DON’T
Propose new data structure
Use existing one

94. IN THE 5TH POSTING
• James Morris suggested to CC netdev mailing list
• “Youshould send anything which touches core networking to
netdev, too, and get an ack from one of the core developers
there.”

95. DO
• Carefully choose CCs and get a review from them

96. DO
• Carefully choose CCs and get a review from them

97. IN THE 6TH POSTING
•
Tetsuo posted 30 series of messages with the subject,
“Subject: [TOMOYO #7 00/30] TOMOYO Linux 1.6.0
released”
• The problem was “TOMOYO 1.6.0” did not use LSM and
implemented different hooks

98. DON’T
Try to invent a new API
Respect a standard and follow one

99. DON’T
Try to invent a new API
Respect a standard and follow one

101. • Tetsuoand I knew that posting such patches will never be
accepted
• However, we had been stuck and we couldn’t find another
way
• Our posting was thoughtless, but we were so serious to make
our code upstream

102. IN THE 7TH POSTING
I changed my mind and wrote:
We apologize for the confusion we caused in
the last posting,
but we don't want to give up returning our
work to the mainline.
We cordially request LSM changes to pass
vfsmount parameters.

103. • Stephen Smalley kindly responded on the list
–“Don't cordially request it - submit
patches to make it happen. Or work
with others who have been
submitting such patches. “

104. DON’T
Cordially ask to merge your code ;-)
Make good patches and post them

105. DON’T
Cordially ask to merge your code ;-)
Make good patches and post them

106. IN THE 16TH POSTING
• Tetsuoposted 25series of messages with the subject,
“Subject: [TOMOYO #16 00/25] Starting TOMOYO 2.3”
• The patches included enhancements as well as garbage
collector functionality (so Tetsuo had a reason for 25)

107. • Pavel Machek commented:
•You are expected to submit
diffs in smaller steps, not here
it is, totally rewritten, take it or
leave it.

108. IT HAPPENED IN THE 7TH
• Serge E. Hallyn once gave us a same comment:
• First
let me point out that reviewing patches is
always a lot of work. What you've done here by
posting an entirely new 30-patch implementation
of tomoyo when (I hope) you're not even serious
about that is to basically tell us our time means
nothing to you...

109. CONCLUSION

111. • Our mistakes, presented in this slides, are merely the tip of the
iceberg
• We made many failures and we sometimes behaved very
badly
• Nevertheless, this Linux community reacted and even merged
our code

113. • Feedbacks were not limited by mailing list replies
• There were people who sent off-list messages and left advices
in face

114. WITH ALL MY EXPERIENCES
I CAN SAY

115. WITH ALL MY EXPERIENCES
I CAN SAY
•Linux is not just free code

116. WITH ALL MY EXPERIENCES
I CAN SAY
•Linux is not just free code
• Linux is great because people are great

117. WITH ALL MY EXPERIENCES
I CAN SAY
•Linux is not just free code
• Linux is great because people are great
• Sending your code is a conversation with people

118. WITH ALL MY EXPERIENCES
I CAN SAY
•Linux is not just free code
• Linux is great because people are great
• Sending your code is a conversation with people
• Theymight not appear friendly for the first time, but try to
speak them first

119. ACKNOWLEDGMENTS

121. Al Viro, Albert Cahalan, Alexey Dobriyan, Andi Kleen,
Andrew Morton, Bodo Eggert, Casey Schaufler, Chris
Wright, Christoph Hellwig, Crispin Cowan, Daniel
Walker, David Howells, David Lang, David P. Quigley,
Greg KH, James Morris, Jamie Lokier, Jiri Kosina,
Jonathan Corbet, Joshua Brindle, KaiGai Kohei,
Kamezawa Hiroyuki, KOSAKI Motohiro, Kyle
Moffett, Linus Torvalds, Matthew Wilcox, Miklos
Szeredi, Paul E. McKenney, Paul Moore, Pavel
Machek, Peter Dolding, Peter Zijlstra, Rik van Riel,
Serge E. Hallyn, Seth Arnorld, Shaya Potter, Stephen
Smalley, Tim Bird, Trond Myklebust, Valdis Kletnieks,
William Leibzon, YOSHIFUJI Hideaki

122. WE COULDN’T HAVE DONE IT
WITHOUT YOUR HELP
Al Viro, Albert Cahalan, Alexey Dobriyan, Andi Kleen,
Andrew Morton, Bodo Eggert, Casey Schaufler, Chris
Wright, Christoph Hellwig, Crispin Cowan, Daniel
Walker, David Howells, David Lang, David P. Quigley,
Greg KH, James Morris, Jamie Lokier, Jiri Kosina,
Jonathan Corbet, Joshua Brindle, KaiGai Kohei,
Kamezawa Hiroyuki, KOSAKI Motohiro, Kyle
Moffett, Linus Torvalds, Matthew Wilcox, Miklos
Szeredi, Paul E. McKenney, Paul Moore, Pavel
Machek, Peter Dolding, Peter Zijlstra, Rik van Riel,
Serge E. Hallyn, Seth Arnorld, Shaya Potter, Stephen
Smalley, Tim Bird, Trond Myklebust, Valdis Kletnieks,
William Leibzon, YOSHIFUJI Hideaki

123. AND OF COURSE
• We thank NTT DATA CORPORATION for their support of
the project

124. RELATED PRESENTATIONS
Realities of Mainlining
- Case of the TOMOYO Linux Project -
Toshiharu Harada
haradats@nttdata.co.jp
haradats@gmail.com Time to Glean
!#$%'$()*+,-$.)/0'1$2*3$0.4$%+0+'4
NTT DATA CORPORATION
July 9, 2008 July 25, 2008
Toshiharu Harada
haradats@nttdata.co.jp
Kentaro Takeda
Tetsuo Handa
NTT DATA CORPORATION

125. What does it mean being an
Open Source project manager
in Enterprise
enterprise edition What does it mean being an
Open Source project manager
LinuxCon 2009 (Business)
September 23, 2009 in Enterprise
Toshiharu Harada
haradats@nttdata.co.jp Open Source Edition
NTT DATA CORPORATION
LinuxCon2009 (Business)
September 23, 2009
Toshiharu Harada
haradats@gmail.com
TOMOYO Linux Project

126. TRADEMARKS
• Linuxis a registered trademark of Linus Torvalds in Japan and
other countries
• TOMOYOis a registered trademark of NTT DATA
CORPORATION in Japan