Apache CloudStack (ACS) is one of the best-of-breed in the current IaaS landscape. However, despite its high functional quality, its technical quality (the one you observe when you look "under the hood") is bellow industry average. Miguel Ferreira was initiated in ACS development in the early days of 2014 and presents, in this talk, his perspective, as a newcomer, on how it felt to him like he was working on a skyscraper in the early days of the XX century, without safety-net, nor rope. Together with his former colleague Dennis Bijlsma (SIG), they dived into the code of ACS to find evidence to support a change in the way ACS is maintained, that combines the best of modern software engineering practices and cutting-edge innovation. Miguel proposes that the ACS community take charge of keeping to the highest standards in software technical quality, while continuing to foster creativity.

1. CCCE 2014
WORKING ON A SKYSCRAPER IN THE EARLY
YEARS OF THE XX CENTURY
CCCE – 2014 – Budapest
Talk: Miguel Ferreira (SBP)
CloudStack evaluation: Dennis Bijlsma (SIG)

2. CCCE 2014
THIS TALK
It’s about how I’ve experienced CloudStack development

3. CCCE 2014
WHO AM I
Portuguese
Mission Critical Engineer at Schuberg Philis
Interests:
• Software maintainability
• Metrics-driven engineering
• Testing

4. Studied systems and software engineering

5. Computer science research

6. Learned to see code as mathematical structures

7. Focus on building the right software

8. Real world software was not so mathematical

9. Software is a mess of building blocks

10. /legacy

11. Focus on building software right

12. Tremendous mental shift!

13. Code as building blocks

14. Complexity
Size
Duplication
Coupling

15. Categorize & Compare

17. CloudStack

18. CCCE 2014
WHAT SOFTWARE TAUGHT ME
Maintainable software is made of tiny blocks of code
Simple blocks, with meaningful names
That blend in together to create a consistent picture
Each piece is independently testable

19. CCCE 2014
MAINTAINABILITY IS THE ENABLER
Maintainability enables all other software quality characteristics
Security, Reliability, Performance, Portability, …

20. CCCE 2014
MAINTAINABLE SOFTWARE
Changes often
Changes fast
by Arthur John Picton, source http://www.flickr.com/, licensed under CC BY 2.0

21. CCCE 2014
UNIT-TESTS ENABLE MAINTAINABILITY
The single most effective feedback loop
Without it the risk of continuously grooming
code is prohibitive
System
Integration
Unit

22. Wasn’t this about CloudStack?

23. Healthcare domain

25. Mission critical cloud (with CloudStack)

26. CCCE 2014
CLOUDSTACK
What kind of software system is it?
“CloudStack is an open source software
platform that pools computing resources to build
public, private, and hybrid Infrastructure as a
Service (IaaS) clouds.”

27. CCCE 2014
IN OTHER WORDS
CloudStack coordinates the work of multiple computing systems in order to offer computing
resources in a seemingly integrated fashion
What kind of systems are we talking about?
• Hypervisors
• Storage devices
• Network devices
Pretty complex stuff

28. CCCE 2014
MY FIRST REACTION
Excitement to work on such an important piece of software
It must be really good, given the adoption

29. CCCE 2014
MY FIRST REACTION
Excitement to work on such an important piece of software
It must be really good, given the adoption
CloudStack is “the Bomb”!!

30. CCCE 2014
THEN I LOOKED UNDER THE HOOD
Code is cluttered
• Poor conventions (e.g. naming: ‘_*’, ‘s_*’)
• Commented out code
Poor domain model
• Long methods
• “God classes”
• Duplicated objects
• “Stringly typed” methods
Poor error handling
• “Pokémon” exception handling
Build system adoption was left halfway

31. CCCE 2014
THEN I LOOKED UNDER THE HOOD
Code is cluttered
• Poor conventions (e.g. naming: ‘_*’, ‘s_*’)
• Commented out code
Poor domain model
• Long methods
• “God classes”
• Duplicated objects
• “Stringly typed” methods
Barely any unit-testing
Poor error handling
• “Pokémon” exception handling
Build system adoption was left halfway

32. How can that be?

33. Code is my work environment

34. CCCE 2014
CLOUDSTACKWORK ENVIRONMENT
Is not a safe work environment
The feedback loop is way too long
It requires extensive amount of debugging
It lacks unit-tests (which keep me sane)

35. CCCE 2014
TESTING CHALLENGES
CloudStack code is tightly coupled to specific hardware devices
It has a database
Plug-in architecture
Distributed development teams

36. CCCE 2014
HOW DO YOU TEST SUCH A SYSTEM?
There are other plug-in system out there
Lets look at a study of test practices within the Eclipse project

37. CCCE 2014
TEST PRACTICES IN ECLIPSE
Test responsibilities
• “Tester and developer, that’s one person. (…)”
Unit testing
• “We have unit testing, and that’s where we put the main effort”
• “Ultimately, unit tests are our best friends, and everything else is already
difficult.”
• “What cannot be encapsulated is not tested.”
Influence of the plug-in architecture
• “The small glue code (..), that’s not tested, because it is just hard to test
that. And for these untested glue code parts we had the most bugs.”

38. CCCE 2014
GETTING BACK TO CLOUDSTACK
Why doesn’t it have more unit tests?
The code does not lend itself to
be unit-tested!

39. CCCE 2014
SOURCE CODE EVALUATION
According to model used for certifying software maintainability
Focus on simple low-level source code metrics
Aggregation of metrics to system level respecting statistical properties

40. CCCE 2014
MAINTAINABILITY RATING
http://www.sig.eu

41. CCCE 2014
VOLUME
Java 670 KLOC
JavaScript 70 KLOC
C# 44 KLOC
SQL 16 KLOC

42. CCCE 2014
DUPLICATION
Not Duplicated
Duplicated

43. CCCE 2014
UNIT SIZE
1 - 20
21 - 50
51 - 100
101 +

44. CCCE 2014
MODULE COUPLING
0 - 10
11 - 20
21 - 50
51 +

45. CCCE 2014
HOW DOES THIS TIE IN WITH UNIT-TESTING?

46. CCCE 2014
LET’S WRITE A UNIT TEST TOGETHER

47. CCCE 2014
WHAT DOES A UNIT-TEST LOOK LIKE
Low complexity (barely no branching at all)
3 blocks
• Setup: define the input
• Execution: call the method under test
• Verification: check expectations

48. CCCE 2014
UPDATE GUEST NETWORK
com.cloud.network.NetworkServiceImpl
updateGuestNetwork(…)
• 11 parameters,
• 286 lines of code
• 15 blocks of code (with header comments)
Unit test
• Setup: build/mock/abstract 11 input parameters (if you consider that 10 can be null, combinations = 2^10!)
• Execution: call updateGuestNetwork(…)
• Verification: what to expect from a 200+ line long method?
Easy answer: an updated guest network!

49. CCCE 2014
UPDATE GUEST NETWORK
In which state was it? Was it a system network? Was it any other kind of network?
Did it already have a name? Display name? Custom ID?
Did the network offering change?
Changing from VPC to non-VPC?
Network domain, IP reservation, …
Update life-cycle: shutdown all elements, make the update, turn back on

50. CCCE 2014
NOW WHAT?
I have some ideas…

51. CCCE 2014
APACHE IS ALL ABOUT THE COMMUNITY
There must be something the community can do
Wants:
• Better code, more tests, happier developers
Don’t wants:
• Strangle innovation, chase current developers away

52. CCCE 2014
LTS MODEL
Popular in several open-source projects
Allows for cutting-edge innovation
Allows for better structure and quality control

53. Freeze the current
master branch

54. Freeze the current
master branch
Automate code
quality checks

55. Refactor on a need-to
basis

56. Development as
usual in innovation
branches

57. When releasing in
LTS, code has to
meet quality standard

58. At some point, swap
devs between the
two teams

59. CCCE 2014
HOW TO MAKE IT WORK
Freeze the current release branch
Automate code quality standard
• Measure unit size, complexity, coupling, duplication, etc.
• Measure unit-testing coverage
Keep the code as is, but
• Every time the code is touched it is refactored according to the defined quality standard
• No new code gets in the LTS branch without meeting the tech quality requirements
Maintain two groups of developers (LTS and Innovation)
• As time progresses swap people between the two

60. CCCE 2014
SUMMARY
CloudStack is functionally quite good, but the code is not keeping up
Feedback loops for developers are way too long
More and better unit testing will help to increase quality and developer satisfaction
Only the CloudStack community has the power to change the game
Schuberg Philis is more than interested in helping

61. Thank you!

62. CCCE 2014
CREDITS
Skyscraper construction work photos by Charles C. Ebbets, source http://nowyorkcity.com/, © All rights reserved
Photo (and sand castle!) on slides 5 and 6, by Calvin Seibert, source http://www.flickr.com, © All rights reserved
Photo on slide 7, by Pedro Moura Pinheiro, source http://www.flickr.com, licensed under CC BY 2.0
Photo on slides 8 to 10, by Benjamin Esham, source http://www.flickr.com/, licensed under CC BY 2.0
Photo on slides 12 to 14, source http://legoquestkids.blogspot.nl/, © All rights reserved
Photo on slide 19, by Arthur John Picton, source http://www.flickr.com/, licensed under CC BY 2.0
Photo on slide 10, by Steven Zucker, source http://www.flickr.com/, licensed under CC BY 2.0
Photo on slide 22, by Murray Barnes, source http://www.flickr.com/, licensed under CC BY 2.0
Photo on slides 27 and 28, by Kristian, source http://www.flickr.com/, licensed under CC BY 2.0
Photo on slides 29 and 30, by Ben Wojdyla source http://jalopnik.com/5182381/adventures-with-wildlife-surprise-underhood-opossum-nest, ©
All rights reserved

63. CCCE 2014
CONTACTS
Miguel Ferreira, SBP, mferreira@schubergphilis.com, @miguel_f
Dennis Bijlsma, SIG, d.bijlsma@sig.eu
Ping us!