Software architecture is often overlooked: pushed by deadlines and hurry, we tend to make our applications a giant ball of mud. Here is where the unix philosophy comes to help. In this talk I will give an overview of the unix philosophy and I will explain how I have applied it to django development for the OpenWISP and NetJSON projects (a set of software modules and standard formats that can be used to deploy wireless networks and implement a public wifi service) with encouraging results. In this talk I will cover the benefits and the downside of such approach, showing example implementations that are being tested in the real world.

1. Unix Philosophy & Django projects
Applying the Unix Philosophy to
Django projects
a report from the real world

2. Who am I?
Federico Capoano (A.K.A. Nemesis)
OpenWISP core developer
Working on public wif at Cineca since 2012

4. What is OpenWISP?
OpenWISP is a set of software modules that can be
used to deploy and manage wireless networks
(public wif, backbone, mesh networks)

5. OpenWISP 1
OpenWISP 1: started in 2008-2016
Focused on public wif for Italian cities
Fast growth of features to accommodate requirements

6. Problems of OpenWISP 1
● monolithic ruby on rails apps

7. Problems of OpenWISP 1
● monolithic ruby on rails apps
● tight coupling

8. Problems of OpenWISP 1
● monolithic ruby on rails apps
● tight coupling
● duplicated logic

9. Problems of OpenWISP 1
● monolithic ruby on rails apps
● tight coupling
● duplicated logic
● policy not separated from mechanisms

10. Problems of OpenWISP 1
● monolithic ruby on rails apps
● tight coupling
● duplicated logic
● policy not separated from mechanisms
● complexity

11. Negative consequences (OpenWISP 1)
These problems caused several
very negative consequences

12. 1. "Giant ball of mud" problems
Many features in a single codebase
Hard to add new features in a clean way
Hard to maintain over time (eg: upgrade
dependencies, fx bugs)

13. 2. Inflexibility
Hard to extend without changing the core
Hard to adapt it and reuse it in different contexts
Specialized features available in the
codebase are not easily reusable

14. 3. Little or no contributions
Complexity scared away
potential external contributors
We only got low quality patches
which were never merged

15. 4. Costs and risk of death
Increased costs to keep the project alive
Many years of work risked to fade into oblivion

16. 5. Duplication of efforts
Little reusability means:
reinventing the wheel
no ecosystem of interoperable tools

17. Nikola is not happy

18. This can't be...
There must be a better way!

20. The Unix Philosophy
What is the unix philosophy?

21. 1. Approach to software development
● Bottom-up

22. 1. Approach to software development
● Bottom-up
● Pragmatic

23. 1. Approach to software development
● Bottom-up
● Pragmatic
● Grounded in experience

24. 1. Approach to software development
● Bottom-up
● Pragmatic
● Grounded in experience
● Founded in empirical skepticism

25. 2. Mature
Evolved in the Unix community since 1970

26. 3. Change tolerant
Born during the digital revolution 1970-2010
It kept pace with the dramatic
evolution of technology

27. The Unix Philosophy in short
What are the main concepts
of the Unix philosophy?

28. "Do one thing and do it well"
● Break down big problems in smaller problems

29. "Do one thing and do it well"
● Break down big problems in smaller problems
● Write simple programs that solve one problem well

30. "Do one thing and do it well"
● Break down big problems in smaller problems
● Write simple programs that solve one problem well
● Write programs to work together

31. "Do one thing and do it well"
● Break down big problems in smaller problems
● Write simple programs that solve one problem well
● Write programs to work together
● Value simplicity and maintainability over
performance and complexity

32. What advantages does it bring?
What are the main advantages
of applying the Unix Philosophy?

33. 1. Saner development
Solving one problem at time is simpler

34. 2. Maintainability
Maintaining a simple program is less expensive

35. 3. Usability
A simple program is easier to use

36. 4. Readability
A simple program is easier to read
and understand

37. 5. Agility
A simple program will be more easily
readapted to new requirements

38. 6. Reuse
Being easier to use, understand, read and maintain
it will be readapted to work in different contexts
and will attract more contributors

39. 7. Ecosystem
Many such programs form an ecosystem of
interoperable tools
(like basic lego building blocks)
and
Allow to solve complex problems faster

40. ME GUSTA

41. The Unix Philosophy in detail
The 17 rules of the Unix Philosophy
From "The art of unix programming"
by Eric Raymond

42. 1. Rule of Modularity
Write simple parts connected
by clean interfaces
source

43. 2. Rule of Clarity
Clarity is better than cleverness
source

44. 3. Rule of Composition
Design programs to be connected
to other programs
source

45. 4. Rule of Separation
Separate policy from mechanism;
separate interfaces from engines
source

46. 5. Rule of Simplicity
Design for simplicity;
add complexity only where you must
source

47. 6. Rule of Parsimony
Write a big program only when it is clear by
demonstration that nothing else will do
source

48. 7. Rule of Transparency
Design for visibility to make inspection
and debugging easier
source

49. 8. Rule of Robustness
Software is said to be robust when it performs well
under unexpected conditions which stress the
designer's assumptions, as well as under normal
conditions
source

50. 9. Rule of Representation
Fold knowledge into data so program logic
can be stupid and robust
source

51. 10. Rule of Least Surprise
In interface design, always do
the least surprising thing
source

52. 11. Rule of Silence
When a program has nothing surprising
to say, it should say nothing
source

53. 12. Rule of Repair
When you must fail, fail noisily
and as soon as possible
source

54. 13. Rule of Economy
Programmer time is expensive; conserve it
in preference to machine time
source

55. 14. Rule of Generation
Avoid hand-hacking; write programs to
write programs when you can
source

56. 15. Rule of Optimization
Prototype before polishing;
get it working before you optimize it
source

57. 16. Rule of Diversity
Distrust all claims for “one true way”
source

58. 17. Rule of Extensibility
Design for the future, because it will
be here sooner than you think
source

60. How to apply it to Django?
How to apply the Unix philosophy
to django projects?
Let's see some real world
examples from OpenWISP 2

61. 1. Rule of Modularity
● Develop main features as reusable django apps

62. 1. Rule of Modularity
● Develop main features as reusable django apps
● One django app for each group of related features

63. 1. Rule of Modularity
● Develop main features as reusable django apps
● One django app for each group of related features
● Document public API

64. 1. Rule of Modularity
● Develop main features as reusable django apps
● One django app for each group of related features
● Document its public API
● Include a license and a changelog

65. 1. Rule of Modularity
● Develop main features as reusable django apps
● One django app for each group of related features
● Document its public API
● Include a license and a changelog
● Publish your app on pypi and djangopackages.org

66. 1. Real world examples
● Confgurations of routers and VPNs:
django-netjsonconfg

67. 1. Real world examples
● Confgurations of routers and VPNs:
django-netjsonconfg
● PKI management (x509 certifcates): django-x509

68. 1. Real world examples
● Confgurations of routers and VPNs:
django-netjsonconfg
● PKI management (x509 certifcates): django-x509
● Multi-tenancy: contributed to django-organizations
reuse existing projects when possible!

69. 2. Rule of Clarity
Explicit is better than implicit
This is already a widely accepted concept in
the python world

70. 3. Rule of Composition
Combine, extend and customize django
reusable apps in your fnal django project

71. 3. Real world examples
● openwisp-users: extends django-organizations

72. 3. Real world examples
● openwisp-users: extends django-organizations
● openwisp-controller: depends on openwisp-users
and extends django-netjsonconfg and django-x509

73. 3. Real world examples
● openwisp-users: extends django-organizations
● openwisp-controller: depends on openwisp-users
and extends django-netjsonconfg and django-x509
● Final result handled by ansible-openwisp2 in an
(almost) transparent manner

75. 4. Rule of Separation
● implement mechanisms as libraries

76. 4. Rule of Separation
● implement mechanisms as libraries
● implement policy as a confguration

77. 4. Rule of Separation
● implement mechanisms as libraries
● implement policy as a confguration
● make these libraries highly confgurable

78. 4. Rule of Separation
● implement mechanisms as libraries
● implement policy as a confguration
● make these libraries highly confgurable
● make these libraries work with data

79. 4. Rule of Separation
● implement mechanisms as libraries
● implement policy as a confguration
● make these libraries highly confgurable
● make these libraries work with data
● clearly defne input and output

80. 4. Real world examples
● netjsonconfg: python library for generating
router & VPN confgurations from NetJSON
objects

81. 4. Real world examples
● netjsonconfg: python library for generating
router & VPN confgurations from NetJSON
objects
● django-netjsonconfg: web interface that uses
netjsonconfg under the hood

83. >>> from netjsonconfig import OpenWrt
>>> router = OpenWrt({
... "general": {"hostname": "HomeRouter"}
... })
>>> print(router.render())
package system
config system 'system'
option hostname 'HomeRouter'
option timezone 'UTC'
option zonename 'UTC'

84. 5. Rule of Simplicity
● start with very basic features

85. 5. Rule of Simplicity
● start with very basic features
● release your project early
(even if you feel it's incomplete)

86. 5. Rule of Simplicity
● start with very basic features
● release your project early
(even if you feel it's incomplete)
● add one feature at time as your understanding grows

87. 5. Rule of Simplicity
● start with very basic features
● release your project early
(even if you feel it's incomplete)
● add one feature at time as your understanding grows
● add complexity only when necessary

88. 5. Real world examples
as of April 2017:

89. 5. Real world examples
as of April 2017:
● 24 releases of netjsonconfg

90. 5. Real world examples
as of April 2017:
● 24 releases of netjsonconfg
● 25 releases of django-netjsonconfg

91. 5. Real world examples
as of April 2017:
● 24 releases of netjsonconfg
● 25 releases of django-netjsonconfg
● 5 releases of django-x509

92. 5. Real world examples
as of April 2017:
● 24 releases of netjsonconfg
● 25 releases of django-netjsonconfg
● 5 releases of django-x509
OpenWISP 2 still lacks some features of OpenWISP 1

93. 6. Rule of Parsimony
Prefer creating new reusable apps
when adding big features
unless doing this complicates things a lot
with no real advantages

94. 6. Real world examples
PKI management in new app: django-x509
VPN confgurations added to existing app:
django-netjsonconfg

95. 7. Rule of Transparency
Log unexpected events using
the python logging facility

96. Logging of bad requests in django-netjsonconfg
import logging
logger = logging.getLogger(__name__)
def invalid_response(request, error, status):
logger.warning(error, extra={'request':
request, 'stack': True})
return ControllerResponse(error,
status=status)

97. 7. Real world examples
Provide good default logging
Provide support for sentry
Take a look at a real example

98. 8. Rule of Robustness
Add constraints to your reusable django apps only
when necessary:
● avoid very strict validation rules
● provide confgurable settings

99. 8. Real world examples
● settings in django-netjsonconfg
● settings in django-x509

100. 9. Rule of Representation
Fold complex information in data structures
Process these data structures with algorithms
Try to make algorithms framework-agnostic
(if possible)

101. 9. Real world examples
in OpenWISP 2, confguration of routers is
implemented as a single text feld formatted as
NetJSON
(in OpenWISP 1 each available confguration had its
own database table and model,
very hard to maintain and evolve)

102. # NetJSON DeviceConfiguration example
{
"general": {"hostname": "HomeRouter"},
"interfaces": [
{
"name": "eth0",
"type": "ethernet",
"addresses": [
{
"address": "192.168.1.1",
"mask": 24,
"proto": "static",
"family": "ipv4"
}
]
}
]
}

105. 10. Rule of Least Surprise
This has become a widely accepted
concept in the IT industry

106. 11. Rule of Silence
Just don't make your apps annoying
It's not that hard :-P

107. 12. Rule of Repair
Errors should never pass silently
Unless explicitly silenced
(from the zen of python)
Let's add: fail fast, noisily and early

108. from django.core.exceptions import ImproperlyConfigured
from .settings import REGISTRATION_ENABLED, SHARED_SECRET
if REGISTRATION_ENABLED and not SHARED_SECRET:
msg = 'NETJSONCONFIG_SHARED_SECRET not set!'
raise ImproperlyConfigured(msg)
Full real world example available in
django-netjsonconfg

109. 13. Rule of Economy
When Unix was born this was a radical idea:
assembler was the norm, C was considered a higher
level language
Python is a consequence of the success of that
radical idea
Embrace this concept

110. 14. Rule of Generation
When you fnd yourself writing lots of boilerplate
code, try to use
meta-programming or code generators

111. 14. meta-programmed urlpatterns
urls.py in openwisp-controller

112. 14. Auto-generated UI from json-schema
JSON-Schema of this UI available on github

113. 15. Rule of Optimization
● write a frst early version of your django app
● do not forget to write automated tests
● refactor and polish

114. 16. Rule of Diversity
Maintain a skeptic attitude towards
dogmas and fads
Measure the effectiveness of an
approach before embracing it

115. 17. Rule of Extensibility
Provide ways to extend and customize the
behaviour of modules without the
need of changing the core code

116. 17. Extensibility: abstract models
● Provide abstract models in your
most important modules

117. 17. Extensibility: abstract models
● Provide abstract models in your
most important modules
● Store these in a python fle which does not import
concrete models
(otherwise other django apps won't be able to import them)

118. 17. Extensibility: abstract models
abstract models in django-netjsonconfg
example usage

119. 17. Extensibility: abstract models
abstract models in django-x509
example usage

120. 17. Extensibility: base admin
● Provide base admin classes in your main modules
(avoid duplication)

121. 17. Extensibility: base admin
● Provide base admin classes in your main modules
(avoid duplication)
● Store them in a python fle which does not import
concrete models
(otherwise other django-apps won't be able to import them)

122. 17. Extensibility: base admin
base admin classes django-netjsonconfg
example usage

123. 17. Extensibility: base views
● If your reusable django app has views, provide
generic views that can be extended
● Store them in a python fle which does not import
concrete models
(otherwise other django-apps won't be able to import them)

124. 17. Extensibility: base views
base views in django-netjsonconfg
example usage

125. 17. Extensibility: reusable urls
If your app provides views that can be extended,
third party apps will have to redefne their URLs
You may want to avoid this required duplication by
providing a mechanism to import urls

126. 17. Extensibility: reusable urls
reusable urls in django-netjsonconfg
example usage:
from django_netjsonconfig.utils import get_controller_urls
from . import views # customized views
# creates new url patterns hooked to customized views
urlpatterns = get_controller_urls(views)

127. 17. Extensibility: AppConfig
If your reusable django app relies on signal
connection for some features,
provide a base AppConfig class

128. 17. Extensibility: AppConfig
base AppConfig class in django-netjsonconfg
from django_netjsonconfig.apps import OpenWispAppConfig
class MyOwnApp(OpenWispAppConfig):
name = 'yourapp.config'
label = 'config'
def __setmodels__(self):
# these are your custom models
from .models import Config, VpnClient
self.config_model = Config
self.vpnclient_model = VpnClient

129. Positive results
Positive results achieved in OpenWISP

130. 1. Faster release cycle & evolution
We release new features more often
The project is evolving rapidly

131. 2. Easier maintainance
Once a bug in a specifc module is replicated
fxing it is easier compared to the work needed to
fx bugs in OpenWISP 1

132. 3. More derivative work
OpenWISP 2 was released officially
less than 1 year ago (as of April 2017)
notwithstanding that, there are already
a couple of derivative works

133. 4. Growth
The user base has been growing rapidly
Users send feedback and patches

134. 4. Growth: mailing list

135. 4. Growth: GSOC 2017
OpenWISP has also been accepted
as a mentoring organization
for the Google Summer of Code 2017

137. Disadvantages
Is not a bed of roses

139. 1. Integration issues
Combining more django apps may
result in integration issues
Changes to project confguration can result
in bugs that are not caught by unit tests

140. 1. Solution
Integration tests are important in this case
Shortcut: I was able to import tests of a base app
and repeat them in the extension app
This easy fx was good enough for my case

141. 2. Big features
When I needed to introduce a new major feature I
needed to change at least a couple of modules

142. 2. Solution
many times this will be necessary
If you have to change many modules every time,
refactor your code to be more loosely coupled

143. 3. Repository management overhead
● working with many repositories can be overwhelming

144. 3. Repository management overhead
● working with many repositories can be overwhelming
● more git tags, pypi releases, versioning, changelogs

145. 3. Repository management overhead
● working with many repositories can be overwhelming
● more git tags, pypi releases, versioning, changelogs
● more announcements for new releases

146. 3. Solutions
● import several projects in a single window of your editor

147. 3. Solutions
● import several projects in a single window of your editor
● automate versioning and changelogs
(I haven't done this yet)

148. 3. Solutions
● import several projects in a single window of your editor
● automate versioning and changelogs
(I haven't done this yet)
● do not send announcements for minor releases which
contain no real advantage for end users

149. 4. Documentation fragmentation
We have many repositories with
their own READMEs and docs
We don't have a single documentation website yet

150. 4. Solutions
I added links to specifc module docs from the website
I often have to send those links on the mailing list
But a central comprehensive documentation website
would be better (we don't have this yet)

151. 5. Bug reports
Users don't know where to send bug reports
and they write to the mailing list

152. 5. Solution
● For the moment I reply by pointing to them where
to open issues

153. 5. Solution
● For the moment I reply by pointing to them where
to open issues
● But a single issue tracker for all the repositories
would be better
(even though it may not really solve the "which module?" problem)

154. 6. Dispersion of popularity
We no longer have a single popular
repository with many github stars

155. 6. Solution
Stop caring about this
Github stars are not an accurate indicator
of the success of a project

156. Conclusions
The creators of Unix are not newcomers
Following their heritage is a good idea

157. Delve deep
Further resources on the Unix Philosophy
and related concepts:
● the art of unix programming
● the pragmatic programmer

158. Practice
Start to gradually apply these concepts in your projects
EG: extract big features to separate projects

159. NO to dogmas
● Practice empirical (scientifc) skepticism

160. NO to dogmas
● Practice empirical (scientifc) skepticism
● Test your ideas

161. NO to dogmas
● Practice empirical (scientifc) skepticism
● Test your ideas
● Measure your results

162. Read existing code
Read the code of OpenWISP
@openwisp organization on github
(or other projects like @openstack)

163. Unix Philosophy for the win!

164. Thank you
Ideas, critical feedback, suggestions?
Talk to me!
Find me also on:
● twitter (@nemesisdesign)
● github (@nemesisdesign)
● linkedin (Federico Capoano)
You can fnd the slides on slideshare