Every two weeks Facebook releases a new version of their flagship Android application which is one of the most widely used apps in the world. The codebase is worked on by a large engineering team. In this talk, Simon Stewart covers some of the technologies and approaches that Facebook use to allow their engineers to work with confidence and speed on the codebase, while at the same time ensuring solid and timely releases.

1. Building Android Apps
At Scale and Speed
Simon Stewart
@shs96c
Software Engineer, Facebook

2. One Guiding Principle

3. We don't want people waiting on computers
<

4. We want computers waiting on people
<

5. Scalable Source Organisation

6. Server
Sharing Code Today
Android
c++
iOS

7. Server
Sharing Code Today
AndroidiOS
c++c++

8. Server
Sharing Code Today
AndroidiOS
c++ c++c++

9. Sharing Code Tomorrow
fbsource
c++

10. All our source in a single
repository (a "monorepo")

11. Why a monorepo?
Easier: code sharing, versioning, contributing
Simpler: large scale changes
Consistent: uniform tooling

12. Scalable Source Control

13. Mercurial
>1M >100K
Source control commands
per day
commits
per week

14. Large Repos Are Slow
▪Lotsofcommitsaday
Pullsareslow
Checkoutsareslow
▪Yearsofhistory
Clonesareslow
Requireslotsofdiskspace

15. We want computers waiting on people
<

16. Shallow Checkouts
Clone just the latest version
Work locally without
complete history
Need more history?
Download on demand
HEAD
HEAD-1
HEAD-2
HEAD-3

17. Sparse Checkouts
Work on just the files you need
Build system integrations
knows how to check out more
~/fbsource
ios/...
common/...
~/fbsource/.hg

18. Watchman
OS independent filesystem change
notification
Integrates with mercurial
File checking operations scale
linearly with number of files
changed

19. Remote Filelog
Clone and pull just download metadata
Download files when required
Clones and pulls: 10x faster

20. Large Repos Are Fast
50
times faster than git
2000
hg improvements
Up to More than

21. Scalable Builds

22. Buck
▪Nextgenerationbuildsystem
▪Designedformodernhardware
▪SSD
▪PlentifulRAM
▪Manycores
▪Automaticallyparallelizesbuilds
▪Andit'sOpenSource

24. Build Files
java_binary(
name = 'example-binary',
deps = [
':example',
],
)
java_library(
name = 'example',
srcs = glob(['*.java']),
deps = [
'//third-party/guava:guava',
],
visibility = [
'//javatests/...',
],
)
Build file. Typically named BUCK

25. Build Files
java_binary(
name = 'example-binary',
deps = [
':example',
],
)
java_library(
name = 'example',
srcs = glob(['*.java']),
deps = [
'//third-party/guava:guava',
],
visibility = [
'//javatests/...',
],
)
Build file. Typically named BUCK
A build rule of type "java_binary"

26. Build Files
java_binary(
name = 'example-binary',
deps = [
':example',
],
)
java_library(
name = 'example',
srcs = glob(['*.java']),
deps = [
'//third-party/guava:guava',
],
visibility = [
'//javatests/...',
],
)
Build file. Typically named BUCK
A build rule of type "java_binary"
A target defined in the same build file

27. Build Files
java_binary(
name = 'example-binary',
deps = [
':example',
],
)
java_library(
name = 'example',
srcs = glob(['*.java']),
deps = [
'//third-party/guava:guava',
],
visibility = [
'//javatests/...',
],
)
Build file. Typically named BUCK
A build rule of type "java_binary"
A target defined in the same build file
Build rule of type "java_library"

28. Build Files
java_binary(
name = 'example-binary',
deps = [
':example',
],
)
java_library(
name = 'example',
srcs = glob(['*.java']),
deps = [
'//third-party/guava:guava',
],
visibility = [
'//javatests/...',
],
)
Build file. Typically named BUCK
A build rule of type "java_binary"
A target defined in the same build file
A target defined in another build file
Build rule of type "java_library"

29. Build Files
java_binary(
name = 'example-binary',
deps = [
':example',
],
)
java_library(
name = 'example',
srcs = glob(['*.java']),
deps = [
'//third-party/guava:guava',
],
visibility = [
'//javatests/...',
],
)
Build file. Typically named BUCK
A build rule of type "java_binary"
A target defined in the same build file
A target defined in another build file
Build rule of type "java_library"
All targets have visibility

30. Target and Action Graph
Target Graph: a naive representation of targets defined in build files.
Action Graph: an optimized graph of actions to perform as the build

31. Language-Aware Smarts
public class Foo {
private String someField;
public void doSomething(int times) {
System.out.println("Wow! " + times);
}
}
Buck calculates the ABI, and keys rebuilds of dependencies off that
Packaging rules that depend on this will always be re-run.

32. Language-Aware Smarts
public class Foo {
private boolean theCakeIsALie = true;
public void doSomething(int times) {
System.out.println("Wow! " + times);
}
}
Private field changed: ABI remains the same as before

33. Language-Aware Smarts
public class Foo {
private boolean theCakeIsALie = true;
public void doSomething(int num) {
System.out.println("Wow! " + num);
}
}
Parameter name changed: ABI remains the same as before.

34. Language-Aware Smarts
public class Foo {
private boolean theCakeIsALie = true;
public void doSomething(String num) {
System.out.println("Wow! " + num);
}
}
Parameter type changed: ABI changes too!

35. Language-Aware Smarts
public class Foo {
private boolean theCakeIsALie = true;
public void doSomething(String num) {
System.err.println("Wow! " + num);
}
}
Body of method changed: ABI remains the same.

36. Pervasive Caching
• Only3%ofeachbuildisunique.
• Localanddistributedcaches
slashbuildtimes.

37. Exopackage
▪Exopackageisan
optimizationfor
buildingandinstalling
developmentbuildsof
Androidapps.

38. Buck Smarts
Build rule knows all of the inputs that can affect its output
Changes to dx and proguard for faster compilation
Java ABI smarts
Android resource smarts
Daemon to watch file changes

39. Scalable Continuous Integration

40. <10Minutes
Feedback to Developers
>1000
test results per second
5Years
machine work per day

41. Life Cycle of a Diff
Write Code Test Locally Diff and review Sandcastle Land

42. Sandcastle
RapidHigh-Signal Frequent

43. Sandcastle Design
Single Master
Master
Worker Worker WorkerWorker

44. Sandcastle Design
Single Master Distributed Queue
Master
Worker Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Distributed
Queue

46. Test Runs
Flakiness SpeedCoverage

47. Landcastle
Land
Target Determinator
Test Test Test
Push

48. We don't want people waiting on computers
<

49. We want computers waiting on people
<

50. Summary
Tool OSS? Link!
Buck Yes https://facebook.github.io/buck/
Mercurial Yes http://mercurial.selenic.com/
Watchman Yes https://facebook.github.io/watchman/
hgwatchman Yes https://bitbucket.org/facebook/hgwatchman
remote filelog Yes https://bitbucket.org/facebook/remotefilelog
Sandcastle No