Talk presented at DevOpsDays in Auckland (2017) discussing how DevOps applies to Embedded Software Development. This talk discusses the approaches Navico have taken in the past and are planning to do going forward.

1. DevOps in an
Embedded World

2. Hello!I’m Sajeewa Dayaratne
sajeewa.com@sajeewa_d mail@sajeewa.com

3. Some of the products

4. Challenges in Embedded
Software Development
Hardware!
Resource Constrained
Deal with the
performance implications
Debugging
Learn to debug and deal
with
- OS bugs
- Hardware bug
- UFO bugs
Oscilloscope and JTAG is
your friend
Environment
- Thermal
- Moisture
- Power consumption

5. Deploying to Production

6. Why This Talk?

7. DevOps Community

8. DevOps in Embedded

9. Does CI/CD make sense for
Embedded Software

10. The Principles
High Frequency
Increase the ability to
release software to customers
faster
Reduce Overheads
Reduce the work required to
release features & products
to the market
Improve Defect Resolution
Enable finding defects faster
and as close to the developer
as possible
Automate
Automate all inefficient
manual tasks
Reduce Response Times
Enable developers to respond
to defects faster

11. A Case Study

16. Navico R&D Centres
our office

17. Our Challenges

18. 5,241,912Lines of Code
61Products and counting …

19. 12 12 12
19 19
23 23
28
32
48
50
61
48 49 50 51 52 53 54 55 56 57 58 59
Software Release
Products Supported

20. “
Our developers never release code.
Rather, it tends to escape, pillaging
the countryside all around.

21. Complex Networks

22. Architecture
◍ Old Codebase
◍ Backwards compatibility with older hardware
◍ Continuing to support advancements in new hardware
◍ Desire to provide value to customers (new features on
all products)
Legacy Architecture

23. All of this meant
◍ Defects were found late
◍ Software stabilisation was taking longer
◍ Manual testing couldn’t keep up with product growth
◍ Costs were increasing (including Opportunity Costs)
◍ We were impacting our ability to innovate
Time
Cost of Finding and Resolving Defects

25. Does CI/CD provide Answers?
High Frequency
◍ Reduce product release
timeframes
◍ Get new features to
customers faster
Reduce Overheads
◍ Become more efficient
◍ Reduce costs
Improve Defect Resolution
◍ Higher quality software
◍ Better brand perception
◍ More sales!
Automate
◍ Become more efficient
◍ Faster turnaround times
Reduce Response Times
◍ Prevent bugs on top of bugs
◍ Higher quality software

26. “
Continuous Delivery Sounds Great,
But It Won’t Work Here

28. Our Plan
Improve Tools & Architecture
Seek integration and flexibility over
compliance and process
Improve Build Speeds
Everything is depends on
faster builds so fix this.
Quality Control Incoming Code
Validate the quality of incoming code
automatically
Automated Testing
Manual testing can’t keep up, focus on
automating as much as possible

29. “You can have data without information, but you cannot
have information without data - Daniel Keys Moran

30. Previous VCS
◍ Proprietary tool with limited
support or updates
◍ Extremely limited integration
with other tools
◍ Lack of CI support
◍ No Code Review capability
Improve Tools
Bitbucket/Git
◍ Git!
◍ Integrated Code Reviews
◍ Significantly better
integration with CI
◍ Integration fully supported
with all tools

31. Improve Architecture
◍ Configurable codebase
◍ Fairly common HW platform
◍ Runtime feature flags

32. Improve Architecture
◍ Dependency matrix

33. Improve Architecture

34. Improve Architecture
◍ Dependency matrix
◍ Remove Cyclic Dependencies
◍ Eliminate Singletons
◍ Promote unit testability

35. Improve Branching
◍Many Branches – Long Lived – Mega Merges
Testing
Testing
Testing
Testing
Testing
Testing

36. Improve Branching
◍ Smaller Batch Sizes
◍ Easier to Merge/Test/Review
◍ Validation focused on a single branch
◍ Feature Flags
Testing

37. Improve Build Speeds

38. 8 Hours
For a Linux Platform Build
14+ Hours
To build and package a release
2 Hours
For each application build

39. Parallel BuildsIncrease Speed

40. Tools to look into
◍ ccache/clcache – for improving C++ compilation times
◍ IncrediBuild – faster parallel builds (tight
integration in Visual Studio)
◍ distcc – Open Source (free) alternative to IncrediBuild

41. 0
1
2
3
4
5
6
Average Time To Build (Hours)

42. Test Automation

46. So how did we approach this?
◍ Start with micro tests
- Unit Testing (Simulator)
- Unit Testing (Real Hardware)
◍ Build Tools
- Software Tools (N2K Simulator, Remote Control)
- Hardware Tools (Repurpose/Build)

47. UI Test Automation
◍ Build or Buy
◍ Functional Testing vs API Testing
◍ Utilise HW Test Tools
◍ Execute on Real HW as well as Simulators
◍ BAT vs Full Regression

48. CI Pipeline
◍ Code Commit -> Pull Request
◍ Automated Build / Unit Tests (on HW)
◍ Merge to Master
◍ Daily Integration Builds on Master for
- All HW platforms
- All Application Variations
◍ Ready for QA.

49. What’s Next?
◍ Configuration as Code
◍ Code Quality Tools
◍ Simulate More Hardware
◍ Increase Analytics and Reporting
◍ Fully Simulated Test Environments for Dev
◍ Scale – From internal infrastructure to the Cloud
◍ Grow the team (We need you)

50. CI Pipeline
◍ Code Commit -> Pull Request
◍ Automated Build / Unit Tests/ Functional UI Tests
◍ Automated Architecture / Code Formatting Checks
(SonarCube/clang)
◍ Merge to Master
◍ Daily Integration Builds on Master for
- All HW platforms
- All Application Variations
◍ Full Automated UI Test Coverage
◍ Ready for release.

51. Lessons Learnt
◍ Culture!
◍ Collect Data
◍ Get Executive Buy-in
◍ Change your Tools and Processes if needed
◍ Test Automation is key
- Invest in HW
- Simulate!
- Virtualise
◍ Focus on Good Software Design for Everything

52. Thanks!
sajeewa.com@sajeewa_d mail@sajeewa.com