White Paper for the Adobe Systems' presentation by Jethro Villegas

1. High-Perform ance Coding, Building & Testing for Multiple Platform s
and Devices
Jethro Villegas
jet@adobe.com
Overview
Today’s software development landscape requires that all the associated costs yield the maximum benefit
for large user bases across increasingly divergent hardware and software platforms. In real terms, can the
value of a single line of HTML markup be weighed against a single line of ARM5 assembly language?
This talk aims to show how careful management of a large code base that targets multiple platforms can be
used to lower the cost of generic cross-platform core code, and maximize the return on investment on
expensive but competitively advantageous platform-specific source code. The real-world data set for this
talk will be the Adobe Flash Runtime products. This talk will describe the problems we had to solve, the
factors that influenced our decisions, the solutions we came up with, what worked, what didn't, and our
plans for the future.
Brief History
The Flash Runtime is actually several different products and technologies that evolved over 12 years. The
Flash Player, Adobe Integrated Runtime (AIR), and Flash Lite (Mobile) products were built on several
codelines and were becoming very hard to manage. An enormous amount of effort was being spent porting
features and fixes from one product to the other. Our ship schedules were in disarray, and it was very
difficult to predict when a product would be ready to ship. In 2008, a goal was set to consolidate the
codebases into one common trunk now known as FlashRuntime/Main.
"Flash Player 10 on all devices" was the mission. The promise was to provide the complete Internet to
every supported device. At the time, our mobile binaries were built on an older code base that did not
include the latest Flash Player features. By the time we finally shipped Flash Player 10.1 on the desktop
and mobile platforms, we actually made more code changes than when we went from Flash Player 9 to 10.
In 2008, the mobile computing landscape was in a confusing state. It was unclear which devices would be
capable of supporting the full Flash Player 10 experience. The availability of the 1GHz Qualcomm
Snapdragon chip later that year made it viable to support plans to bring our mobile codebase up to the same
level as the desktop.
The Mobile & Desktop Flash Player and AIR products are now built from a single highly portable unified
code base, separated into platform and core components. The same code base produces hundreds of
different binaries on dozens of software and hardware platforms, from low-power phones to high-end 64-
bit workstations. Every time a developer checks in to the shared mainline trunk, her code must build, run,
and pass quality tests on hundreds of different devices. She must write code that passes a rigorous code
review process, and adhere to an architectural model that allows the core code to evolve while minimizing
divergent behavior across the different runtime environments.
• All features are supported on the main desktop platforms, at parity.
• All features are supported in software. Hardware is optionally used for acceleration, but not
differing functionality.
• All code compiles as straight C++, without a requirement for assembly. (Although we certainly
optimize using machine code.)
• The SDK and mobile player supports full web browsing, although at a loss of fidelity where
appropriate, but not in a way that breaks functional compatibility.

2. Careful consideration was made for the planned proliferation of mobile platforms. The entire source tree is
divided into 3 directories:
• code
The code folder contains all the source code to build all the products. This directory is further
divided into several module areas, under 3 major categories:
1. core – this directory contains code that is common to any Flash product on any platform. For
example, the parsing of the SWF file format is in this directory. Most of the Flash Player is
core code.
2. platform – this directory contains the platform-specific code for each Flash product. For
example, the code to rasterize specifically on DirectX on Windows could be found here.
3. third_party – this directory contains code we license and use from other organizations. Video
and audio codecs are typically found here.
• tools
The tools directory contains utilities and other non-code components of the Flash Runtime
products. This folder is under version control to ensure that the non-code components are in
lock=step with the matching code folder.
• buildconfig
This directory holds the configuration data for the build system. This folder is also under source
control so that we can build and ship from any code line with the same build system.
Given the diverse build and deployment configurations that Flash needs to run on, the developer doesn’t
typically have every type of device or development environment available. Writing code for dozens of
products requires a system that can validate code changes across all the supported configurations. The
system shouldn't require the developer to have every supported build environment and toolchain available
locally. The system also has to build and test the new code in a limited amount of time.

3. A sandbox build and test system is used to validate her code before the check-in into the mainline trunk.
She checks her code into her branch and points the sandbox build system at her code, tools, and buildconfig
directories. This gives her maximum flexibility and reduces the need to branch all 3 main directories, as she
can use her personal code branch, while using production tools and buildconfig directories.
She authenticates with her LDAP username and password, then supplies the Perforce info needed by the
build system to pull her code into each instance of a builder that will compile, link, and test.
She can then select all the targets that her code will be built and tested against.

4. The developer specifies the Perforce path and changelist from which to build her code. She can also specify
the tools used for the build, and the parent branch used to run tests against her code. Because Flash is
always shipping, we typically have 3 to 6 production branches in addition to the mainline. Each production
branch may have different features or fixes, and the tests to run need to account for that. After making her
selections and choosing ‘Start Build’, dozens of computers spin up to either build or test her code. After 1
to 2 hours, she can check the results of her build on the status page.

5. Other developers keep their own sandbox branches and often share branches with each other to conserve
computing resources and minimize integration time.
Nightly, Continuous, Sandbox Build Configurations
The Flash Runtime team supports 3 levels of configuration:
Sandbox. The sandbox configuration was described above, and is typically run on new, untested, or
experimental code.
Continuous. Our production branches are continuously built and tested 24 hours a day to validate code
checked in. These builds are synced incrementally and built about once an hour.

6. Nightly. A scheduled script increments the version number in Perforce prior to building the official
production builds. This occurs once a day on a typical release. Once a build becomes a Release Candidate
(RC) the nightly builds stop and official builds are built on demand.
Tracking quality and performance metrics
All new code has to be run against a set of tests that are appropriate for the type of builds that are running.
A sandbox or continuous build is tested against a quick set of automated tests that exercise core
functionality in under 10 minutes. An official nightly build is tested against an automated battery of tests
that run in about 30 minutes. A release candidate is typically tested with automated and manual tests that
run between 48 and 72 hours.
As new devices are added to the test matrix, the system has to be made aware of their availability and
capabilities. Even as some devices may lack certain features, a core set of capabilities need to be validated

7. in an automated manner. Graphics, audio, and video tests are difficult to automate and some tests have to
be done manually.
References:
Lee Thomason & Jim Corbett – Flash Player Architecture Info
Leah Zagreus & Alvin Luison – Flash Player Testing Info