"Accelerating Android Development and Delivery" presentation given at DESIGN West 2013.

1. Accelerating Android Development
David Rosen, Product Manager
drosen@electric-cloud.com
@adr0sen
Design West 2013

2. What’s this Android buzz all about?
2

3. Qualcomm Reference Design Program
<60 days TTM, and pushing for faster
Source: http://www.qualcomm.com/connect/analyst-relations/events/americas-analyst-summit-2012

4. Android Development Landscape
Real-world challenges at large APAC mobile device maker

5. Automotive
6%
Mobile
38%
Consumer
electronics
18%
Industrial
control and
automation
6%
Medical
4%
Military/aero
space
3%
Other
25%
Users of Android by Vertical
Electric Cloud Android/Embedded Survey (Q1 2013)
• Survey
• Focused on Android
development & delivery
• Global embedded contacts in
Electric Cloud marketing lists
• 350+ respondents
• 175+ Android

6. Android Delivery Process Today
Technical Challenges
• Slow Android Builds
• Slow, Complex CTS
Testing
• Complex B-T-R Matrix
• Custom Testing –
Volume, Complexity
Negative Business Impacts
• Slow Time to Market
• Poor Quality of products
• Poor productivity of
teams
Less than 1
hour, 22.5%
1-4
hours, 57.5
%
4-8
hours, 15.0
%
More than 8
hours, 5.0%
Avg. time to execute CTS
Less than 5
minutes, 7.8
%
5-20
minutes, 21.
0%
20 minutes -
1
hour, 28.8%
1-4
hours, 27.0
%
More than 4
hours, 15.3
%
Avg. release build time

7. Agenda
• Accelerating the Android Build
• Accelerating the Android CTS execution
• Accelerating the end-to-end Android Development Lifecycle

8. Accelerating the
Android build

9. Why look at Android builds?
• Android is everywhere
• 68% of global mobile phone market share1
• Explosion of non-phone applications
• Automotive (Saab, Renault)
• Entertainment (Ouya, Nvidia)
• Exercise equipment (NordicTrack)
• Rice cookers! (Panasonic)
• Android is everywhere
• … and that means thousands of developers building Android
• What if we can make those builds faster?
• How would your development process change?
• How much more could you achieve?
1 Source: Gartner, February 2013

10. What is (and is not) covered in this part of the talk?
• Build acceleration techniques fall into two broad categories:
• Hardware improvements like faster CPU’s and better disks
• Software improvements like smarter build tools and faster compilers
• In general, hardware and software improvements are complimentary
• If you want the fastest possible builds, leverage both!
• This talk is about software techniques for build acceleration
• Given a fixed hardware platform, how fast can we build Android?
• Or, what do you do when you reach the limits of your hardware?

11. 0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
14,000,000
16,000,000
18,000,000
Linux Kernel
Android
Mozilla Firefox
Chromium
Webkit
AOSP LOC Evolution
Jan 2009 – Jan 20131
1 Source: http://www.ohloh.net, http://en.wikipedia.org/wiki/Android_version_history (February 2013)

12. +15%
+123%
+19%
+2%
+30%
+12%
+17% +2%
0
2
4
6
8
10
12
14
16
18
00:00
00:14
00:28
00:43
00:57
01:12
01:26
01:40
2.2 2.3.4 4.0.1 4.1.1 4.2.2
Millions
Build Time
LOC
AOSP Build Evolution
Android Version Release Date LOC1 LOC
Growth %
GNU Make
Build Time2
Build Time
Growth %
2.2 May 2010 8,837,858 - 28m55s -
2.3.4 April 2011 11,492,324 30% 33m10s 15%
4.0.1 October 2011 12,827,330 12% 1h13m54s 123%
4.1.1 July 2012 15,028,331 17% 1h28m11s 19%
4.2.2 February 2013 15,266,8033 2% 1h32m56s 2%
1 http://www.ohloh.net, 2 Builds performed on 8-core 16GB RAM server, 3 LOC data as of December 2012

13. Common wisdom about Android builds
• I can just…
• … add more cores
• … use distcc
• … use ccache
• “The Android build is as optimized as it can be!”

14. The common wisdom is wrong.

15. Sidebar: how much faster is “interesting”?
• Some large mobile device makers run > 50K builds/week
• At 15 minutes per build that’s 12,500 hours of build time
• A reduction of just one minute would save 800 hours every week
• What about wait time?
• Faster builds = more iterations
• Faster builds = higher quality
• Faster builds = lower time to market

16. How fast is GNU make?
• Best time: ~15m
• 48-cores
• 128GB RAM
• No ccache

17. Problem: Visibility
• Knowing the end-to-end time is great but not actionable
• We need visibility
• Where is the time going?
• What are the bottlenecks?
• Are there opportunities for improvement?
• How do we get that visibility?
• Instrument the build… somehow! strace, echo, hack gmake, or…
• ElectricInsight
• Visualization and analysis of builds run with ElectricMake
• Let’s pump this build through emake and see how it looks!

18. Solution: ElectricInsight

19. What can we see from here?
• An ideal build would look like a tightly packed box
• Overall looks almost perfect – well done, Android team!
• But! a couple of things stand out:
• Crazy long parse time1
• Gaps at the end of the build, indicative of serializations
• We’ll look at both of these, but first: what if we just use more
cores?
1 emake parsing may be slower than gmake parsing

20. Longest serial chain
• Best possible runtime is about 15m
Best possible is about 15m

21. Projected runtime with more cores

22. What else can we learn?
• Lots of non-compile work in the build - ~25% of total time

23. Problem: long parse time
• Why do we build?
• To transform sources into deliverables (programs, docs, etc).
• Does parsing makefiles transform sources into
deliverables?
• Nope.
• Parsing makefiles is pure overhead
• But you have to tell make what to do somehow
• Unless you want to manually invoke the compiler, linker, etc.

24. Solution: parse avoidance
• What if we don’t parse (every time)?
• Makefiles don’t change very often, so why reparse every time?
• Reuse parse results from a previous build, as long as…
• Makefiles are unchanged (MD5)
• Command-line is unchanged
• Environment is unchanged
• How do we do it?
• Electric Make already has parse results in a reloadable form, just
need to add cache management
• GNU make doesn’t, but could be modified

25. Parse avoidance impact
Build time reduced to about 13m30s

26. Problem: serializations
• Gaps in visualization suggest serializations
• Q: How many dependencies are there in the
Android build?
• A: More than you think!

27. Android Dependency Graph
• Dump all makefile rules
• 100,000 files/targets
• 1,990,628 dependencies

28. Dependencies in Android
• ~19 dependencies per file: why so many?
• Consider a typical build structure:
• Some files will have many dependencies
• Most have only a few
• What is going on in Android?
lib.a: foo.o bar.o
foo.o: foo.c foo.h util.h
bar.o: bar.c bar.h util.h

29. Superfluous dependencies in Android
• Do we really need all 1.9M dependencies?
• The filesystem can tell us!
• Collect a list of files actually used to generate a target
• Compare to the list of prerequisites specified in the makefile
• Example:
• Why not run foo.txt and bar.txt in parallel?
foo.txt:
echo “Hello” > foo.txt
bar.txt: foo.txt
echo “World” > bar.txt

30. Actual dependencies in Android
• 100,000 files/targets
• 288,804 dependencies

31. Specified vs. Actual dependencies in Android
Specified Actual

32. Solution: dependency optimization
• Impossible to manually eliminate superfluous
dependencies
• Electric Make can prune them automatically
• If you use gmake, you can achieve the same effect:
• Remove existing dependency specifications
• Generate minimal dependency specifications from emake data
• Before:
• After:
$(OBJECTS): %.o: %.cpp $(ALL_HEADERS) $(OTHER_DEPS)
$(COMPILE.CC)
$(OBJECTS): %.o: %.cpp
$(COMPILE.CC)
foo.o: foo.h util.h
bar.o: bar.h util.h generated.h

33. Dependency optimization impact
Build time reduced to about 12m30s

34. Summary
• Google has done a great job of optimizing Android builds
• But there’s still room for improvement!

35. Electric Cloud
Android Software Delivery Solution

36. Test (CTS) Optimization
Upto 70% test time acceleration

37. Android Release Process Workflows
Out of the box best practices, integrations
Build
QA
Release
CTS
Reduce software delivery cycle time by 70%

38. MIPS/EC Webinar, Feb 2013

39. MIPS/EC Webinar, Feb 2013

40. MIPS/EC Webinar, Feb 2013

41. MIPS/EC Webinar, Feb 2013

42. MIPS/EC Webinar, Feb 2013

43. MIPS/EC Webinar, Feb 2013

44. MIPS/EC Webinar, Feb 2013

45. Android Software Delivery Solution - Key Benefits
Blazing Android Builds
Fast comprehensive CTS testing
End-to-end B-T-R process & visibility

46. Questions?
David Rosen, Product Manager
drosen@electric-cloud.com
@adr0sen