This document summarizes the steps taken by the Enlightenment project to transition from infrequent 10+ year release cycles to more agile 3 month release cycles. Technological improvements like switching to Git, adding automated builds, static analysis tools, and increasing test coverage helped improve code quality. Social changes like releasing responsibility, communication of benefits, and leading by example helped establish release-focused habits and norms within the distributed developer community. While more work remains, such as improving automated testing and the release process, significant progress was made through automation and gradually socializing developers towards the value of regular releases.

1. Bring Your Project From a 10 Years to a
3 Months Release Cycle
Stefan Schmidt

2. Agenda
Introduction
Release History
Technological Side
Social Influences
Next Steps
Summary

3. Introduction
Enlightenment is not a web project
Instead it is written in plain C
Enlightenment is not developed by a StartUp
Instead it is developed by the FOSS community and companies
such as Samsung
Enlightenment is not developed by a 5 person team in one
office
Instead it is developed by an international team distributed over
many timezones
All this might result in different understandings of agile
development :-)

4. Release History

5. Release History
The Enlightenment project started in 1997 by Carsten Haitzler
Some early releases until E 0.16
A 12 years gap from first code to release E 0.17
Building a whole new set of libraries for it but no releases for
them either for a long time
More release driven now but still feature based and not time
based schedule

6. Reasoning
Most of the active developers at that time did not see a need for
releases
It was a project driven by volunteers which used the code
straight out of CVS/SVN
Some argued for it as the fastest agile development cycle ;-)
Nobody stood up for taking care of the releases so it did not get
done

7. What Changed?
Different people joined the project which had more interested in
releases
Most Linux distributions wanted us to do releases before
including our code
Commercial parties also asked for releases
With the pressure also the understanding grow that releases
would help the project forward

8. How did we do it?
That is what the rest of the talk is about
From the technological side we improved our tools and
infrastructure
On the social side we steadily worked on giving releases a
more prominent role

9. Technological Side

10. Tools and Automation
One of the biggest roadblocks is to build trust in the code base
you are going to release
In the Enlightenment project with the two main libraries alone
having around 850.000 lines of code
You need a variation of tools that will help you to build up this
trust
SCM, automated builds and maybe continuous integration,
static analyzers just to name a few

11. SCM and Understandable Commits
As an very old project CVS was used in the beginning
Switch to SVN for a newer tool but same old practices
Large amounts of work committed as one change
Many unrelated changes at once
Switch to git
After some struggling it helped to allow people working on
different things without dumping everything into master
Improved overall commits, code as well as commit messages
People start to develop larger changes in branches

12. Automated Builds
Setup automated builds with Jenkins
Quick builds for every change pushed pushed to master
Building for x86, x86_64, x32 with gcc as well as clang and also
cross-compilation for windows with mingw
Very useful as many developer only use one system for their work
Many of the discovered problems are getting fixed instantly (see
social pressure)
In the dark times before we had this setup builds have been broken
for some configurations for months
During the nightly builds we run time intensive things like static
analyzers and other testing

13. Static Analyzer
Static analyzer can be very helpful for a plain C project
Started with scan-build, part of clang, from the LLVM project
Output quite noise and sadly a lot false positives
Also experimented with klocwork but the proprietary nature
made it hard to share results with the community
Now also using Coverity which is also proprietary but offers a
free service for FOSS projects
Especially Coverity uncovered quite some issues which have
been overlooked by code review
You still have to deal with false positives but the signal to noise
ratio is way better

14. Test Suite
We have unit testing for a lot
of components
But with 358 tests we only
cover 29.4% of our lines of
code
We are an UI toolkit so work
is also ongoing on pixel
perfect regression testing
We are still weak on the
automated testing side and
need to improve
One of the bigger worries
before every release

15. Build Complexity
By history the Enlightenment project was happy to add
configuration options
Over the last two years we tried to reduce these options to
something maintainable
Merged libs allow easier maintenance. 11 different libs have
been merged into EFL
All of these actions had direct influence on how much different
configurations we can test
The reduced set of libraries also eased the release process

16. Social Influences

17. General Social Changes
Once we had IRC and email notification about broken builds
social pressure grow to fix them
Keep people in the loop and highlight positive changes (e.g.
weekly QA newsletter)
Most people have been ok with adapting to changes but you
have to provide docs and guidance
People follow the lead but would not have done it on their own.
(e.g. backports of fixes to stable branches)

18. Towards Releases
Easier once we came over a long time without a release
Getting into the habit to do releases more often
Core application now do releases a while after the libraries
Happened naturally
Switch to time based releases. Now fine tuning the process to
identify problems
Took over as release manager (Only because I build up trust to
the code base due to all the automation)
There need to be proof that it will really work
Invest time to make a sane and realistic schedule

19. Next Steps

20. Changes to come
ABI checker to avoid breakage surprises after a release
Jenkins job that builds a magic developer branch
Allowing developers to tests their changes for many configurations
Thinking about adding arm builds to jenkins
Thinking about running automated valgrind runs to detect memory
issues (-fsanitize* could be an alternative)
Fine tuning of the release process
Think about regression test. Having them mandatory for fixes might
blow our man power.
Integrate the pixel perfect regression tester with jenkins for fast
feedback

21. Changes that might never come
Every commit runs through our jenkins testing before it goes
into master (Qt project does this)
More code review for core developers before changes hit the
repo
Hard to bring in as many developers want to avoid any delay
before the code hits the repo
Automate the release process (NEWS generation,
announcement templates, upload tarballs, etc)
A simple version bump and pushed git tag might be enough to
get it all done :-)

22. Summary – What you should take home
Automate as much as possible to keep the workload down and
the project always in a good state
Invest in all kind of tools to make your job easier
Show the benefits of more regular releases and proof that it is
possible
Keep on fine tuning the process to avoid as much problems as
possible
Expect that people will only follow if you give them the lead and
show how the process could work

23. Thank you.

24. References
http://www.enlightenment.org
http://build.enlightenment.org
http://git.enlightenment.org
Imagery sources
http://www.linuxtag.org/2014/
http://www.enlightenment.org/
https://openclipart.org/detail/192629/gear-tools-by-ben-192629
http://git-scm.com/downloads/logos
http://jenkins-ci.org/
http://www.coverity.com/
https://openclipart.org/detail/26498/present-blue-pack-by-minduka