White Paper: Creating a World-Class Build System, and Getting It Right

Copyright Citrix Systems, Inc. All rights reserved.

1

Creating
a
World-‐Class
Build
System,
and
Getting
It

Right

Scott
Estes

Citrix
Systems,
Inc.


2

1 Table
of
Contents

2
Introduction
.........................................................................................................................................
4

3
SCM
and
Build
are
Critical

....................................................................................................................
4

4
Who
Are
the
Vision
Stakeholders?
......................................................................................................
4

5
What
are
the
Requirements?
..............................................................................................................
5

6
Evolution
versus
Revolution
................................................................................................................
6

7
Build
versus
Buy?
.................................................................................................................................
7

8
Leveraging
Others
to
Accomplish
Common
Goals
...............................................................................
7

9
SCM
and
the
Build
System
at
Citrix

......................................................................................................
8

9.1
Engineering
Structure
..................................................................................................................
8

9.2
XenApp/Desktop
Component
Structure
......................................................................................
9

9.3
PLANB
Structure
..........................................................................................................................
9

10
The
Flow
of
the
Build
.....................................................................................................................
11

10.1
Input
..........................................................................................................................................
12

10.1.1
Automation
at
Citrix
..........................................................................................................
12

10.2
Pre-‐Build
Steps
...........................................................................................................................
13

10.2.1
Automation
at
Citrix
..........................................................................................................
13

10.3
Build
...........................................................................................................................................
14

10.3.1
Automation
at
Citrix
..........................................................................................................
15

10.4
Post-‐Build
Steps
.........................................................................................................................
16

10.4.1
Automation
at
Citrix
..........................................................................................................
17

10.5
Output

........................................................................................................................................
18

10.5.1
Automation
at
Citrix
..........................................................................................................
19

11
Optimizing
the
System
...................................................................................................................
20

11.1
Keep
it
Deterministic
.................................................................................................................
20

11.2
Keep
it
fast
.................................................................................................................................
22

11.3
Keep
it
simple
............................................................................................................................
25

11.4
Keep
to
standards
......................................................................................................................
25

12
Scheduling
Updates
and
Maintenance
..........................................................................................
25

12.1
How
to
do
it
...............................................................................................................................
25

12.2
When
to
do
it
.............................................................................................................................
26


3

13
Final
Thoughts
...............................................................................................................................
26


4

2 Introduction

Anything
worth
doing
has
to
begin
with
a
vision,
regardless
of
how
mundane
it
may
seem.

Without
a

vision,
whatever
it
is
we’re
trying
to
do
is
inevitably
doomed
to
either
go
supernova
in
the
short
term,
or

die
a
drawn
out,
laborious
death
over
a
longer
period.

In
both
scenarios,
the
collateral
damage
is

devastating
and
usually,
the
only
way
to
recover
is
to
trash
everything
and
start
over
at
the
beginning

again.

An
interesting
study
could
be
done
to
show
how
much
time,
money
and
resources
are
wasted
by

various
industries,
but
that
is
a
topic
for
another
time.

What
is
explained
and
demonstrated
in
this

paper
is
how
to
define,
implement
and
maintain
an
optimized
Build/SCM/ALM
vision,
and
avoid

common
(and
not
so
common)
pitfalls
along
the
way.

3 SCM
and
Build
are
Critical

It’s
been
said
that
the
Build
is
the
heartbeat
of
a
project.

This
is
a
fundamental
truth,
yet
it’s
amazing

how
more
often
than
not,
person-‐months
are
invested
in
the
Develop
and
Test
visions
and
strategies
of

a
project,
and
scant
person-‐hours,
(if
that)
are
given
to
how
the
project
code
base
will
be
managed
and

built,
until
the
very
last
possible
point
in
time.

This
“Build
Avoidance”
is
understandable
because

everyone
is
usually
focused
on
customer
requirements,
product
features,
schedules
and
a
myriad
of

other
things
that
siphon
attention
away
from
SCM
and
Build.

All
too
often,
the
Product
team
will
simply

assume
that
the
product
will
magically
coalesce,
or
that
someone
else
is
thinking
about
how
to
build
it.

This
is
the
point
where
the
savvy
Build
Visionary
can,
with
the
proper
support
and
resources,
create

order
from
the
technical
chaos,
and
provide
a
quality-‐built
product
that
meets
everyone’s
expectations.

It
must
also
be
stressed
that
there
must
be
executive
level
support
for
SCM.

If
the
senior
management

staff,
and
those
in
charge
of
driving
the
overall
Engineering
roadmap(s)
are
not
bought
in
to
the
Build

Vision,
there
is
little
to
no
chance
of
obtaining
the
resources
required
to
create
a
successful
SCM
and

Build
infrastructure.

4 Who
Are
the
Vision
Stakeholders?

Identifying
and
knowing
all
the
stakeholders
of
the
Build
system
is
the
first
step
to
eliminating
any
“blind

spots”
in
the
Build
vision.

Understanding
the
motives
and
mindsets
of
the
stakeholders
also
goes
a
long

way
towards
ensuring
that
the
Build
system
is
optimized
and
meeting
all
the
requirements.

At
the
high
level,
the
Build
stakeholders
can
be
organized
as
follows,

• Customers

• Project
Management

• Product
Development

• Finance

• Product
Marketing/Sales

• Quality
Assurance/Test


5

• Lifecycle
Maintenance/Customer
Support

• Build

These
stakeholders
will
have
varying
levels
of
input
and
requirements
to
the
Build
vision
and
some
may

not
immediately
realize
that
they
are
stakeholders.

This
is
acceptable
as
long
as
the
Build
Visionaries

remain
aware
of
all
the
stakeholders
and
are
proactive
at
meeting
their
requirements.

As
an
example,

the
customer
is
as
key
a
stakeholder
as
is
Product
Development.

That
being
said,
it
might
be
considered

a
bit
odd
to
find
a
Build
engineer
consulting
with
a
customer
on
how
the
product
should
be
built.

Here,

the
forward
thinking
Build
visionary
should
be
looking
at
project
schedules
and
planning
how
the
Build

vision
can
facilitate
delivering
a
quality
product
on
time
to
the
customer.

Conversely,
the
Build
Visionary

should
be
in
constant
communication
with
Product
Development
to
ensure
that
their
requirements
are

being
accounted
for
in
the
vision.

5 What
are
the
Requirements?

Once
the
stakeholders
have
been
identified,
the
next
step
is
to
gather
their
requirements.

The
following

table
defines,
at
the
high
level,
some
of
these
requirements.

Stakeholder

Requirements

Customers
• Quality
product

• Delivered
on
time

• Affordable

• Maintainable

Project/Release

Management

• Deterministic
builds

• On
schedule

• Of
quality

Product
Development
• Fast
builds

• Efficient
Build
system

• Fully
supported

Finance
• Adherence
to
budget

• Proper
forecasting

Product
Marketing/Sales
• Features

• Quickly
available

Quality
Assurance/Test
• Fast
builds

• Automated
delivery
and
testing

• Fully
repeatable

Lifecycle
Maintenance
• Escrow-‐able,
(“goldenized”)
builds

• Fully
repeatable

• Patchable

• Business
Continuity

Build
• Stakeholder
requirements

• Adhere
to
SCM
standards

• Resourced
and
supported


6

There
may
be
additional
requirements
to
what
is
listed
here,
but
having
the
ones
listed
here,
well

defined,
will
ensure
a
smooth
departure
on
the
path
towards
the
world-‐class
build
system.

There
is
one
additional
stakeholder
with
requirements
dependent
on
the
maturity
level
of
the
company,

and
that
is
Security.

The
requirements
here
range
from
simply
protecting
the
Intellectual
Property
of

the
company
to
full-‐scale
security
audits,
threat
models
and
so
on.

These
requirements
can
be
a
stand-‐
alone
topic
by
themselves,
and
as
such,
left
for
another
time.

6 Evolution
versus
Revolution

At
this
point,
the
Build
Visionary
should
have,

a) Executive
sponsorship

b) Identified
stakeholders

c) Stakeholder
requirements

With
these,
he/she
can
then
decide
which
of
the
two
approaches,
(Evolution
or
Revolution)
best
suit
the

needs
of
the
product
and
company
as
a
whole.

Each
approach
has
associated
pros
and
cons,
and
often

a
combination
of
the
two
is
what
proves
to
be
most
effective
moving
the
vision
forward.

The
high-‐level

pros/cons
are
listed
in
the
following
tables,

Evolution

Pros

Cons

• Usually
a
slow,
stable
rate
of
change

• Can
be
too
slow
to
affect
meaningful

change

• Training/documentation
can
keep
pace

• Impetus
can
be
lost
as
others
vie
for

resources

Revolution

Pros

Cons

• Shortcut/mitigate
barriers
to
change

• Amount
of
change
can
adversely
affect

stakeholders’
ability
to
absorb
it.

• Can
open
avenues
for
other
shift
changes.

• Training/documentation
will
usually
not

keep
pace.

• Can
cause
unintended
side-‐effects
if
not

carefully
planned.

Focusing
exclusively
on
only
one
of
these
two
approaches
is
going
to
result
in
failure
of
the
Build
Vision

at
some
point,
either
from
stakeholder
exhaustion
or
loss
of
confidence
in
the
Vision/Visionary.

While

there
are
no
hard
and
fast
“rules”
on
how
to
combine
these
into
an
effective
vision,
experience
indicates

that
Revolution
works
best
with
small,
experienced
teams
that
can
then
act
as
“ambassadors”
to
the

larger
organization
when
the
vision
is
introduced
at
that
scale.

Evolution
can,
in
fact,
harness
these


7

“mini”
revolutions
such
that
the
larger
teams
can
absorb
the
changes
as
schedules
and
resources

permit.

The
most
important
task
for
the
Build
Visionary
is
to
stay
on
top
of
the
teams
to
ensure
that

they
remain
committed
to
the
Vision.

7 Build
versus
Buy?

There
is
no
magic
answer
to
this
perennial
question;
it
can
be
different
to
varying
degrees,
for
every

application
at
every
company.

The
first
major
challenge
the
Build
Visionary
must
overcome,
is
to

adequately
evaluate
the
available
product
offerings
to
see
which
ones
could
fit
into
the
Build
Vision

compared
to
building
and
supporting
the
infrastructure
in-‐house.

It’s
not
enough
simply
to
determine
if

the
tools
work
technically,
the
evaluation
and
resulting
recommendation
has
to
include
the
overall

return
on
investment,
(ROI)
and
ongoing
total
cost
of
ownership,
(TCO).

The
recommendation
will
be

based
on,

• The
amount
of
funds/resources
available,
and

• The
amount
of
time
allocated
for
the
evaluations.

The
second
major
challenge
for
the
Visionary
is
to
convince
the
company
to
allocate
the
amount
of
time

needed
to
properly
answer
this
question,
while
keeping
pace
with
the
ongoing
requirements
of
the

product
release
cycle(s).

8 Leveraging
Others
to
Accomplish
Common
Goals

Very
rarely,
if
ever,
will
there
be
enough
SCM/Build
resources
to
singularly
accomplish
the
Vision.

Therefore,
the
savvy
Build
Visionary
will
look
to
others
outside
the
immediate
Build
team
to
assist.

This,

aside
from
moving
the
Vision
forward,
has
two
important
side
effects,

1. People
who
help
feel
like
their
ideas
are
being
heard
and
valued.

As
people
see
the
results
of

their
input
and
work,
they
will
become
more
comfortable
with
continuing
to
present
new
ideas

and
processes.

When
channeled
properly,
this
can
become
a
self-‐sustaining
cycle
that
further

ensures
the
relevance
of
the
Vision.

2. Those
who
help
in
turn
become
“Build
Ambassadors”
to
the
rest
of
their
respective
teams.

Generally
speaking,
the
best
customer
of
an
invention
is
the
inventor.

If
a
person
on
a

development
team
has
contributed
to
the
Build
Vision,
that
person
will
then
most
likely
take
the

vision
and
spread
it
among
the
rest
of
the
team,
thus
ensuring
that
the
Vision
is
widely
accepted

without
relying
on
the
Visionary
to
do
all
the
“selling”.

There
are
a
couple
of
“gotchas”
that
must
be
mitigated
when
leveraging
resources
from
other
teams,

1. Don’t
interfere
with
other
teams’
mandates
and
objectives.

Most
development
managers
like

to
keep
their
teams
100%
focused
on
the
team’s
particular
roadmap,
leaving
no
time
for
other

pursuits.

Most
likely,
the
help
provided
will
be
in
the
form
of
“skunk-‐works”,
or
after-‐hours
type

projects.

Sometimes
the
benefit
to
the
overall
group
will
be
significant
such
that
a
Development


8

manager
will
assign
a
certain
percentage
of
his/her
resources
to
formally
helping
on
SCM/Build

objectives.

In
any
scenario,
the
Build
Visionary
must
take
extreme
care
to
balance
the
Build

needs/wants
with
those
of
the
supporting
team(s).

2. Make
certain
to
manage
expectations.

The
last
thing
any
of
the
Build
stakeholders
want
is
to

feel,
(whether
real
or
imagined)
that
their
ROI
is
not
realized.

Failure
to
manage
the

expectations
is
one
of
the
quickest
ways
to
destroy
the
credibility
and
success
of
the
Vision.

In
summary,
the
Build
Visionary
must
exert
all
his/her
diplomatic
abilities
when
working
with
extended

teams
and
other
Stakeholders
to
ensure
continued
support
for
the
Build
Vision.

9 SCM
and
the
Build
System
at
Citrix

Up
to
this
point,
the
discussion
has
focused
primarily
on
non-‐technical,
logistical
and/or
philosophical

aspects
that
must
be
considered
when
creating
an
ideal
Build
Vision.

In
parallel,
the
Build
Visionary

should
be
constructing
the
building
blocks
that
will
comprise
the
system
that
will
execute
as
mandated

by
the
Vision.

The
core
of
the
ideal
Build
system
should
revolve
around
well-‐defined
processes
and
tools
needed
to

facilitate
the
fundamentals
of
good
SCM
practices
such
as,

• “Determinism”
(
the
ability
to
determine
attributes
of
a
build)

• Branching/Merging

• Parallel
development

• Lifecycle
maintenance

• “Goldenization”/escrow

• Business
Continuity

• Some
level
of
Development
automony/self-‐service

• Etc.

Discussions
around
some
of
these
areas
can
be
separate
topics,
and
as
such
will
not
necessarily
be

discussed
in
detail
here,
other
than
to
show
how
the
Citrix
Enterprise
and
Desktop
Applications
Build

team
have
leveraged
the
power
of
Perforce
to
facilitate
the
realization
of
our
ideal
Build
Vision.

Before
the
technical
details
of
the
build
system
are
discussed,
the
composition
of
the
Engineering
teams

responsible
for
the
product,
and
the
structure
of
the
product
itself,
will
be
explained
to
show
how
Citrix’

ideal
Build
vision
has
evolved
to
meet
the
requirements
of
the
stakeholders.

9.1 Engineering
Structure

There
are
over
500
engineers
organized
in
development
and
supporting
teams
located
in
the
following

sites,

• Bangalore,
India

• Bedford,
Massachusetts


9

• Cambourne
and
Chalfont,
UK

• Fort
Lauderdale,
Florida

• Santa
Clara,
California

• Sydney,
Australia

• Tokyo,
Japan

The
global
Build
team
consists
of,

• 1
engineer
in
Bangalore

• 1
engineer
in
Bedford

• 5
engineers
in
UK

• 4
engineers
in
Fort
Lauderdale

• 1
engineer
in
Santa
Clara

The
Build
engineers
are
primarily
responsible
for
maintaining
the
Build
Vision
and
Systems
at
the
site

where
they
are
located.

Additionally,
the
team
can
leverage
a
“follow-‐the-‐sun”
work
schedule
to
be

able
to
maximize
Build
resources
and
coverage
without
requiring
any
one
of
the
teams
to
provide

support
outside
of
normal
working
hours
at
each
site.

9.2 XenApp/Desktop
Component
Structure

The
code
base
that
comprises
the
product
is
structured
along
core
technology
boundaries,
known
as

Technology
Components,
(TCs),
(e.g.
Client,
Server,
protocols,
etc.).

The
engineers
are
assigned
into
TC

teams,
and
focus
on
their
technology
instead
of
a
particular
release
of
the
overall
product.

Each
TC

team
has
a
junior
architect,
(or
Principal
Design
Engineer)
who
is
responsible
for
the
evolution
of
the

technology,
and
coordinating
what
features
can
be
delivered
for
a
particular
product
release
with
the
TC

Manager,
Release
and
Product
Management
teams.

The
deliverables
of
the
TCs
are
released
as
SDKs

which
can
consume/be
consumed
in
such
a
way
as
to
create
a
complete
product.

There’s
a
lot
more

that
could
be
discussed
about
the
TC
methodology,
and
its
advantages
and
disadvantages.

In
synopsis
for
this
discussion,
the
main
advantage
of
this
approach
is
that
a
huge
code
base
is
organized

into
small,
manageable
chunks
that
can
be
“snapped”
together
to
create
a
product
release.

The

disadvantage
is
that
there
is
more
responsibility
placed
on
the
architects,
managers
and
Release
Team

to
ensure
that
the
TC
teams
are
communicating,
and
any
component
dependencies
are
aligned
for
a

particular
release.

9.3 PLANB
Structure

The
build
system
at
Citrix
is
called
the
Perforce
Linked
Automated
Network
Builder,
(or
PLANB).

It
is

comprised
of
six
distinct
components
that
communicate
with
the
Perforce
servers,
and
are
responsible

for,

• Determining
when
to
build

• Creating
the
Build
environment

• Building
the
binaries


10

• Creating/updating/publishing
the
TC
SDKs

• Creating
the
installers/DVD
layout

• Publishing
the
product
DVD

There
are
two
types
of
builds
that
PLANB
controls,

• SDKs

• DVD
layouts

When
building
SDKs,
there
is
an
optional,
manual
SDK
updater
component,
(using
ASP.net)
provided
to

“publish”
the
output
as
determined
by
the
TC
owner.

This
process
is
intended
to
be
used
when
the
TC

output
is
at
some
level
of
quality
as
to
be
considered
“locked”
for
the
overall
product
DVD
release.

If
a

TC
owner
chooses
to
not
publish
his/her
SDK,
then
the
latest
successful
build
of
the
output
is
used
for

the
DVD
layout.

A
Watcher
service
(using
WCF
functionality)
polls
the
Perforce
servers
for
code
submissions.

When
a

submission
is
detected,
the
service
determines
if
any
client
spec
template
changes
are
required,
and

make
any
needed
changes
before
passing
the
change
list(s)
to
be
built,
and
control
to
the
next

component.

The
Build
VM
creator
(also
using
WCF),
seeks
out
an
available
“bare-‐metal”
machine,
starts
up
a

XenServer-‐based
virtual
machine
template
on
it,
and
passes
control
to
the
Building
component.

The
Binaries
Builder,
(using
C#
and
MSBuild
functionality)
synchronizes
the
code
using
the
change
lists

sent
from
the
Watcher,
compiles
the
code,
creates
the
SDK(s)
and/or
other
components
into
a
“save”

area
which
can
either
be
used
as
input
to
publish
a
TC’s
component(s)
or
as
direct
input
into
the

Installer/DVD
creator.

The
Installer/DVD
creator,
(using
PowerShell
scripts),
collects
the
binaries,
(SDKs,
installers,
etc.)
from

Perforce
and
the
“save”
area,
and
creates
the
DVD
layout
that
will
be
published
to
Test.

The
Publisher,
(also
using
PowerShell
cmdlets)
then
copies
the
DVD
image
out
to
an
area
where
can
be

consumed
by
the
various
Engineering
groups,
and
sends
a
semaphore
to
the
Test
infrastructure
which

initiates
automated
Build
Verification
Tests
and
subsequent
Regression
test
suites.

When
defining
these
components,
it
was
realized
that
the
overall
system
needed
to
be
broken
into

smaller
pieces
or
projects
such
that
the
architecture
of
each
component
could
be
properly
designed

without
getting
lost
in
the
larger
scope
of
the
entire
system.

The
projects
were
aligned
as
follows,

• Cumulonimbus,
responsible
for
the
Watcher
and
Build
VM
creator
components,
as
well
as
the

code
synching
process
of
the
Binaries
builder
component.

Utilizes
a
SQL
database
to
store
build

information.

• Piston
is
a
“toolbox”
of
PowerShell
scripts
used
to
create
DVD
layouts
and
associated
artifacts.

• Vulcan
is
the
core
build
tool.

It
hooks
into
Visual
Studio’s
MSBuild
engine
to
compile
Windows


11

code.

Figure
9.3.1
maps
out
the
components,
and
the
interactions
between
them.

Additional
details
of
each

component
and
the
associated
projects
will
be
discussed
in
sections
10.1-‐10.5
below.

Figure
9.3.1:

PLANB
(Project)
Piston
(Project)
Cumulonimbus
Watcher Build
VM
Creator
(Project)
Vulcan
Binaries
Builder Installer/DVD

Creator
Publisher
Synchs
code
Notified
of
submissions
and
updates
client
views Synchs
SDKs
SDK
Updater
manual
process
Updates
SDKs
“Save”
area
Perforce
Svr
1 Svr
2 Svr
3 Svr
n
...

10 The
Flow
of
the
Build

Regardless
of
the
complexity
of
the
product/project,
the
flow
of
the
Build
should
always
follow
the
basic

flow
of
control
in
the
following
diagram,

Figure
10.1:

Input Pre-‐Build
Step(s) Build Post-‐Build
Step(s) Output

This
diagram
illustrates
the
flow
of
control
at
its
most
componentized
level.

The
Build
Visionary
should

clearly
define
each
of
these
components
with
the
following
attributes,

• Owner(s)

• Stakeholder(s)

• Input(s)

• Control(s)

• Output(s)


12

Keep
in
mind
that
there
may
be
multiple
builds
following
this
model
that
feed
as
inputs
into
other
builds

and/or
the
ultimate
layout
of
the
product.

Having
the
attributes
for
each
“layer”
of
build
well
defined
is

key
to
ensuring
a
deterministic,
simple
and
efficient
build
system.

10.1 Input

Input
can
be
loosely
defined
as
any
new
“thing”
that
would
cause
a
subsequent
build
to
be
different

than
the
current
build.

Narrowing
this
down
a
bit,
an
input
is
usually
a
direct
code
submission,
an
SDK

or
some
other
“black
box”
component
update.

A
“black
box”
update
is,
at
some
point,
going
to
be
the

result
of
a
direct
code
submission.

Multiple
or
tiered
Build
systems,
(which
can
be
from
where
“black

box”
updates
come
will
be
discussed
later).

Applying
the
previously
defined
attributes
to
the
Input,
defines
the
component
as
follows,

Table
10.1.1:

Attribute
Details
Comments

Owner(s)
• Project/Release

Management

Responsible
for
ensuring
builds
are
released
as

per
schedule.

Stakeholder(s)
• Project/Release

Management

• Build

• Product
Development

• Quality
Assurance/Test

• Product
Marketing/Sales

Everyone
in
this
list
is
vested
in
ensuring
that
the

build
is
successful
at
all
stages
of
the
process.

Input(s)
• Code
submission(s)

• “black-‐box”
update

• Previous
build
reports

Any
new
thing
that
needed
in
the
build
as
well
as

any
historical
artifacts.

Control(s)
• “Pre-‐flight”
checks

• Triggers

• “Determinism”

Ensures
quality
and
traceability
of
submitted

code.

Output(s)
• “Of
quality”
code
Code
has
passed
all
controls
and
can
be
built.

10.1.1 Automation
at
Citrix

The
watcher
service,
(part
of
project
Cumulonimbus),
monitors
the
Perforce
ports
for
submissions,
and

automatically
queues
them
for
building
in
the
build
farm.

When
a
submission
is
detected,
the
service

gathers
the
change
list(s)
that
comprises
the
submission,
and
then
checks
the
project’s
configuration
file

to
see
if
any
changes
in
it
require
updates
to
the
client
view
templates,
(e.g.
a
new
SDK
is
needed
by
the

project).

If
any
such
changes
are
detected,
the
service
will
update
the
associated
templates,
before

passing
the
change
list(s),
and
a
job
ID
associated
with
the
change
lists
to
the
Build
VM
creator.

This

interaction
is
shown
in
Figure
10.1.1.

Figure
10.1.1:


13

Watcher
Code
Submission
Perforce
Submission

notification
received
Project
config

change?
Yes
Call
VM
Creator
No
Update
Client
spec

templates
Get
Change
List(s)

and
job
ID
Cb
database
Job
ID

10.2 Pre-‐Build
Steps

There
usually
aren’t
too
many
steps
at
this
point;
usually
consisting
of
synching
the
code,
and
any
build

machine
preparation
needed
prior
to
building
the
code.
The
attributes
for
Pre-‐Build
steps
are;

Table
10.2.1:

Attribute
Details
Comments

Owner(s)
• Build
Responsible
for
ensuring
the
process
and

infrastructure
meets
stakeholder
requirements.

Stakeholder(s)
• Project/Release

Management

• Build

• Product
Development

• Quality
Assurance/Test

• Product
Marketing/Sales

Everyone
in
this
list
is
vested
in
ensuring
that
the

build
is
successful
at
all
stages
of
the
process.

Input(s)
• “Of
quality”
code

• Build
machine/infrastructure

Code
and
infrastructure
on
which
to
build
it.

Control(s)
• Available
infrastructure

• System
resiliency

Must
have
reliable
and
available
infrastructure.

Output(s)
• Buildable
code

• Build
environment

Code
is
now
ready
to
be
built.

10.2.1 Automation
at
Citrix

The
Build
VM
creator
component,
(also
part
of
project
Cumulonimbus),
establishes
and
maintains
a

load
balanced
build
farm
consisting
of
dynamic
and
consistent
build
machines
leveraging
Citrix

XenServer®
technology.

When
the
Watcher
service
sends
a
cue
to
build,
the
creator
determines
which
virtual
machine,
(from

pool)
to
use
based
on
the
job
ID
sent
from
the
Watcher.

It
then
determines
if
the
virtual
machine
is


14

available
in
the
pool.

If
it
is
not,
(i.e.
it
is
building
another
job),
the
creator
places
the
job
in
a
queue
to

build;
otherwise,
it
then
calls
into
the
Xen
API
functions
to
connect
a
data
and
tools
drives
to
the
virtual

machine,
assigns
the
machine
to
a
host
and
starts
it.

The
final
step
in
this
part
of
the
Build
process
is
a

build
script,
(which
resides
in
the
Windows
Startup
folder)
initiates
to
begin
synching
the
code
base.

The

control
flow
is
shown
in
Figure
10.2.1.1
below.

One
additional
note,
at
this
point
the
data
and
tools
drives
are
the
storage
areas
for
the
code
base
and

build
tools
required
for
building
the
code
base,
respectively.

These
drives
reside
on
a
NAS
infrastructure

which
will
be
discussed
further
in
section
11.2.

Figure
10.2.1.1:

Build
VM
creator
Cue
to
build Determine
VM
type
Connect
Data
and

Tools
drives
to
VM
Is
a
VM

available?
Wait
for
VM
to

become
available
No
Yes
Cb
database
VM
type
Start
VM/
Build
script
Assign
VM
to
host

10.3 Build

The
Build
Visionary
has
to
be
careful
that
this
part
of
the
system
is
maintained
properly
to
remain
in
line

with
the
Vision.

Competing
requirements
from
the
development
teams
that
for
the
most
part,
own
the

component
build
files
(solutions,
make
files,
etc.)
are
often
introduced
as
shortcuts/hacks
to
accomplish

short-‐term
goals,
(e.g.
deadlines/milestones)
at
the
component
level.

Failure
to
review
and
make

necessary
updates
to
the
build
files
will
result
in
a
non-‐optimized
product
build.

The
following
table

defines
the
attributes
for
Build,

Table
10.3.1:

Attribute
Details
Comments

Owner(s)
• Build

• Product
Development

Responsible
for
jointly
maintaining
the

component
build
tasks,
solution
files,
makefiles,

etc.


15

Stakeholder(s)
• Project/Release

Management

• Build

• Product
Development

• Quality
Assurance/Test

Everyone
in
this
list
is
vested
in
ensuring
that
the

build
is
successful
at
all
stages
of
the
process.

Test
will
mostly
be
concerned
with
build
times

and
availability.

Input(s)
• Buildable
code

• Build
environment

Control(s)
• Build
standards
Component
build
scripts
and
associated
files

must
conform
to
company
standards.

Output(s)
• Binaries

• Installers*

Binaries
ready
to
be
signed,
packaged,
etc.
in

preparation
for
release
to
test.

*Installers
may
be
considered
part
of
the
Post-‐
Build
steps.

10.3.1 Automation
at
Citrix

Project
Vulcan
provides
a
build
framework
around
Microsoft’s
Visual
Studio
MSBuild
engine
into
which

Windows
component
solution
files
are
fed.

This
framework,
shown
in
Figure
10.3.1.1
is
kept
as
simple

as
possible
for
maintenance
purposes.

Only
the
Input,
Post-‐Build
and
Output
steps
are
developed
and

maintained
in-‐house.

The
Pre-‐Build
and
Build
steps
are
all
MSBuild-‐centric,
and
no
additional

maintenance
is
required
outside
of
the
development
teams
maintaining
the
solution
and
project
files.

That
being
said,
the
Build
team
has
developed
a
Citrix-‐specific
add-‐in,
including
project
templates,

(based
on
the
out-‐of-‐the-‐box
templates
provided
with
Visual
Studio),
to
Visual
Studio
which
provides

support
for
Engineering-‐wide
requirements
such
as,

• Enforcement
of
banned
Microsoft
functions,
(e.g.
strcpy()
versus
sstrcpy()).

• Consistent
signing
of
binaries.

• Automatic
inclusion
of
test
frameworks,
(e.g.
nUnit
and
code
coverage).

• Hooks
into
in-‐house
developed
tools
used
in
the
build,
(e.g.
custom
graphics
generator)


16

Figure
10.3.1.1:

Input Pre-‐Build
Steps Post-‐Build
Steps Output
Build
Synch
code
Create
SDKs,

installers,
reports,

etc.
ID
components

to
build
Check
and

build
deps
Compile
code
Sign,
symbol

index,
copy

output,
etc.
Compile

successful?
Log
error
and

continue
No
More
to
build?
Yes
Configuration

.proj/search
Yes
No

Non-‐Windows
builds
utilize
standard
make
files
with
a
similar
framework.

The
non-‐windows
build

system
does
not
support
all
the
functionality
that
MSBuild
provides,
which
means
it
is
somewhat

simpler,
yet
requires
more
developer
support
to
ensure
the
make
files
are
kept
as
clean
as
possible,

(especially
in
the
case
of
dependencies),
to
ensure
fast,
accurate
builds.

Additionally,
system
has
not

been
updated
to
support
the
enhanced
dependency
checking
and
reporting
capabilities
found
in
the

Windows
system.

This
does
not
currently
present
a
burden
to
the
Development
and
Build
teams,
as
the

non-‐Windows
code
base
consists
of
reasonably
small
client
side
applications.

The
adequate
amount
of

Determinism
is
provided
by
the
Perforce
filelog
functionality.

The
framework
is
shown
in
Figure

10.3.1.2.

Figure
10.3.1.2:

Input Post-‐Build
Steps Output
Build
Synch
code
Create
SDKs,

installers,
reports,

etc.
Search
for

makefiles
Compile/Link

code
Sign,
symbol

index,
copy

output,
etc.
Compile/Link

successful?
No
More
to
build?
Yes
Yes
No
Log
error
and
die

10.4 Post-‐Build
Steps

At
this
point,
all
binary
components
needed
for
the
product
build
should
be
available
for
additional
post-‐
build
processing.

This
post-‐build
processing
can
consist
of
binary
signing,
installer
creation,
(some
teams

may
choose
to
make
these
part
of
the
Build
steps),
“layout”
(also
known
as
the
DVD)
creation,
gathering

data
for
the
Build
report,
and
any
other
tasks
in
preparation
for
releasing
the
build
to
Quality


17

Assurance/Test.

The
attributes
are
as
follows,

Table
10.4.1

Attribute
Details
Comments

Owner(s)
• Build

• Product
Development

Responsible
for
jointly
maintaining
the
install

tasks.

Build
responsible
for
ensuring
that
the

build
is
readied
for
release.

Stakeholder(s)
• Project/Release

Management

• Build

• Product
Development

• Quality
Assurance/Test

Everyone
in
this
list
is
vested
in
ensuring
that
the

build
is
successful
at
all
stages
of
the
process.

Test
will
mostly
be
concerned
with
build
times

and
availability.

Input(s)
• Binaries

• Installers
(if
any)

Control(s)
• Build
standards
Install
and
“layout”
scripts
must
conform
to

company
standards.

Output(s)
• Installable
product
image

• Build
reports

• Email
notification

• Semaphore
for
automated

testing

Everything
needed
to
continue
with
Build

Verification
and
Regression
Testing.

10.4.1 Automation
at
Citrix

Project
Piston
provides
a
“toolbox”
of
PowerShell
scripts
which
are
used
to
create
the
product
DVD.

The

request
to
create
a
DVD
can
either
come
from
a
schedule
semaphore,
or
be
manually
requested
from

the
Build
Portal,
(more
on
the
portal
later).

When
the
request
to
create
a
layout
is
received,
the
main
PowerShell
script
extracts
the
data
identifying

the
components
needed
for
the
product
DVD
from
the
project’s
layout
configuration
file.

This
data
is

then
passed
to
the
next
steps
of
creating
the
DVD
image.

Components
that
comprise
the
DVD
contents

are,

• Merge
modules
and
installers
from
the
various
stand-‐alone
sub-‐components.

• Help
files
and
supporting
documentation.

• Meta-‐installers.

• Non-‐Windows
clients

The
next
step
in
the
process
is
to
update
the
debug
symbols
server
with
the
symbols
generated
during

the
build.

The
scripts
then
generate
the
various
reports
needed
to
provide
the
desired
level
of

Determinism
for
the
build,

• Location
of
the
build

• What
changes
went
into
the
build


18

• Build
time

• Any
major
updates

• Checksum
for
comparison
with
previous
build.

• What
binaries
changed

• Etc.

The
final
stage
of
the
process
is
to
auto-‐generate
and
send
an
email
notifying
of
the
build’s
availability,

and
sending
a
semaphore
to
the
Test
infrastructure
to
initiate
automated
testing.

The
process
is
shown

in
Figure
10.4.1.1.

Figure
10.4.1.1:

Piston
On-‐Demand/
per
schedule
Get
layout

information
Layout

configuration

file
Release
DVD
for

testing
Create
DVD
image
Update
Symbol

server
Symbol
Server
Create
reports

10.5 Output

The
final
stage
of
the
process,
the
build
is
now
ready
for
Quality
Assurance/Test
to
begin
executing
a
set

of
“smoke
tests”
or
Build
Verification
Test
(BVT),
to
ensure
the
build
will
perform
fundamental

activities,
(e.g.
install).

Based
on
the
successful
completion
of
the
BVT,
more
extensive
test
suites
will
be

executed
on
the
build.

Often
the
results
of
these
tests
will
serve
as
input
for
subsequent
builds.

The

attribute
table
for
Output
is
as
follows,

Table
10.5.1

Attribute
Details
Comments

Owner(s)
• Quality
Assurance/Test
Responsible
for
jointly
maintaining
the
install

tasks.

Build
responsible
for
ensuring
that
the

build
is
readied
for
release.


19

Stakeholder(s)
• Project/Release

Management

• Build

• Product
Development

• Quality
Assurance/Test

• Product
Marketing/Sales

Everyone
in
this
list
is
vested
in
ensuring
that
the

build
is
successful
at
all
stages
of
the
process.

Input(s)
• Installable
product
image

• Build
reports

• Semaphore
for
automated

testing

Control(s)
• Test
standards
Test
suites
must
conform
to
company
standards.

Output(s)
• Test
results

• Bug
reports

10.5.1 Automation
at
Citrix

The
Citrix
Automation
Development
Infrastructure,
(CADI)
is
a
SDK
which
provides
a
Test
Automation

framework
for
creating
test
suites
which
can
then
be
executed
in
various
test
environments.

Skynet
is
a

NAS-‐based
cloud
infrastructure
that
provides
a
highly
available
virtualization
infrastructure
that
will

allow
for
fast,
on-‐demand
access
to
virtual
machines
that
can
be
acquired
by
CADI.

(Skynet
is
part
of
a

larger
quota-‐based
storage
infrastructure
and
will
not
be
discussed
in
detail
here).

CADI
polls
the
build
release
area
for
a
semaphore
file
which
indicates
a
build
is
available.

When
it
finds

a
file,
it
provisions
a
test
infrastructure
based
on
the
product
build
requirements
extracted
from
the
Test

infrastructure
database.

CADI
then
executes
the
BVT,
which
determines
if
the
build
is
good
enough
for

more
exhaustive
testing.

If
the
BVT
reports
a
failure,
testing
stops
on
the
build,
and
it
is
marked
as
a

failed
or
bad
build.
If
the
BVT
reports
success,
the
build
is
then
submitted
for
additional
testing
to

determine,

• Regressions

• Performance

• Scalability

• Overall
stability

Data
from
these
tests
is
used
to
generate
test,
bug
and
other
useful
reports
that
indicate
the
state
of
the

product
and
project.

The
high-‐level
flow
is
shown
in
Figure
10.5.1.1.


20

Figure
10.5.1.1:

CADI
Build
is
ready
Provision
Test

Environment
Create
reports,

generate
CPRs,
etc.
Execute
BVT
Execute
additional

test
suites
No
BVT
Pass? Yes
Report
and
die
Skynet
Test

infrastructure

11 Optimizing
the
System

The
Build
Vision
must
ensure
the
build
system
is
optimized
for
the
following,

• Determinism

• Performance

• Self-‐service

• Standards

Having
these
areas
optimized
will
ensure
that
the
Build
“Customers”
are
happy,
and
the
overall
Build

needs
and
expectation
of
the
organization
are
being
met
and
exceeded.

11.1 Keep
it
Deterministic

The
term
“Determinism”
is
overloaded
from
its
usual,
“fatalistic”
definition,
in
that
it
means
that
the

capability
within
the
Build
system
exists
to
know
everything
about
a
product
build.

Not
only
does
it

have
to
exist,
it
has
to
be
as
efficient
as
possible.

For
example,
the
Perforce
command,
filelog
will

accurately
report
what
changes
went
into
a
build.

Having
the
additional
capability
to
determine
which

code
submissions
caused
which
binaries
to
change
is
not
necessarily
as
simple
yet
is
critical
for
Quality

Assurance/Test
to
be
able
to
determine
quickly,
in
order
to
properly
seed
their
test
bed.


21

Example:
Historically,
when
a
XenApp/XenDesktop
build
was
released
to
Test,
there
was
no

capability
to
accurately
determine
which
binaries
changed
from
build
to
build,
(checksums

and
binary
comparison
tools,
while
useful,
could
not
provide
the
levels
of
detail
required

for
targeted
testing).

As
a
result,
test
cycles
for
intermediate
builds
required
the
same

amount
of
time
at
those
for
milestone
builds,
which
slowed
the
entire
release
cycle.

An
in-‐house
developed
tool
called
the
Binary
Sleuth
leverages
the
information
stored
in

the
debugging
symbols
created
by
the
Windows
Symbol
Indexing
functionality,
(for

Windows
builds
only),
to
accurately
determine
which
binaries
change
from
build
to
build.

This
has
enabled
Test
to
target
specific
areas
to
test
based
on
the
scope
of
the
changes
to

obtain
results
much
more
quickly
than
by
starting
at
the
beginning
of
the
test
suites
and

working
down
to
the
areas
that
need
quick
turnaround
verification.

The
format
of
the

report
is
as
follows,

Figure
11.1.1:

Being
able
to
track
and
report
Build
data
as
efficiently
as
possible
is
critical
to
the
overall
speed
of
the

product
release
cycle.


22

Example:
The
XenApp/XenDesktop
product
has
evolved
from
a
monolithic
build
to
a
compilation
of

30
major
components,
(with
over
64
subcomponents).

There
are
many
different
data
sets

that
need
to
be
examined
on
a
per
build
basis,
and
having
these
automatically
generated
in

report
format
is
critical
for
the
Release
team
to
keep
development
notified
of
the
schedule.

A
list
of
the
currently
available
reports
is
as
follows,

Figure
11.1.2:

11.2 Keep
it
fast

The
ideal
goal
here
is
to
provide
builds
faster
than
they
can
be
consumed.

While
this
may
be
unrealistic,

keeping
the
build
times
as
low
as
possible
will
definitely
mean
that
the
consumers
will
remain
happy
and

able
to
perform
their
respective
duties
without
having
to
wait
for
the
build.


23

Example:
At
one
point
before
the
Technology
Component
methodology
was
introduced
at
Citrix,
the

build
time
for
a
monolithic
XenApp
build
was
approximately
16
hours.

These
extremely

long
build
times
made
it
very
challenging
to
keep
to
aggressive
Test
and
Release
schedules.

Several
factors
contributed
to
the
time,

• Compiling
code
multiple
times

• Compiling
“dead”
code

• Build
infrastructure
not
optimized.

• Dependencies
not
well
defined.

• Etc.

Over
time
and
with
concerted
effort,
the
build
times
have
been
reduced
significantly
to

approximately
5.5
hours
to
build
all
the
components
in
serial.

Having
said
that,
build
times

have
been
further
reduced
as
a
result
of
SDKs
and
components
releasing
semi-‐
autonomously,
(as
per
the
TC
model)
and
simply
being
included
as
part
of
the
DVD
layout.

Additional
development
work
to
simplify
the
component
dependencies
have
reduced
the

times
as
well
as
optimized
incremental
build
times.

Build
times
reports
have
been
added

to
the
Build
portal
and
can
now
be
reviewed
to
determine
the
cause
for
any
significant

changes.

A
sample
report
is
shown
in
Figure
11.2.1.

Figure
11.2.1:


24

Leveraging
Perforce
functionality,
like
P4Proxy,
as
well
as
isolated
network
segments,
fast
hardware,
etc.

are
starting
points
to
a
fast
build.

The
Citrix
XenApp/XenDesktop
build
farm
topology
shown
in
Figure

11.2.2
below,
utilizes
these
optimizations
as
follows,

• Uses
fast
Bare-‐Metal
machines,
(64GB
RAM,
Quad
core
half-‐height
blades)
paired
with
a
NetApp

storage
device.

• Maintains
multiple
and
dedicated
Perforce
proxy
servers
across
the
sites
to
increase

performance.

• Isolates
the
Build
farm
on
its
own
optimized
10GB
network
segment
to
avoid
traffic
“jams”.

• Maintains
a
well-‐defined
Business
Continuity
process
to
ensure
that
build
jobs
can
be
quickly

moved
from
site
to
site.

Figure
11.2.2:

UK
Cam
Devs
US
Business
Continuity
Center
Build
Farm
NAS
1GB
10GB
Servers
Build
machines
P4
Proxy
Data
P4
Proxy
Perforce
Svr
1
Builds
NAS
Perforce
Svr
2
Devs
1GB
Perforce
Svr1/
Proxy
ports
Servers
Build
machines

(+BC
for
Chf)
Chf
Devs
Servers
Build
machines
(+BC
for
Cam)
1GB
Perforce
Svr1/
Proxy
ports
Builds
NAS
(+BC
for
Chf)
Builds
NAS
(+BC
for
Cam)


25

Additional
tweaks,
like
incremental
builds,
parallel
component
builds
will
also
further
reduce
build

times.

Some
compilers,
such
as
Microsoft’s
MSBuild
engine
capitalize
on
the
multi-‐processor

capabilities
of
current
hardware
to
spawn
non-‐dependent
components
across
the
available
processors

to
more
fully
optimize
build
times.

11.3 Keep
it
simple

Most
people
enjoy
a
good
puzzle
for
fun.

Most
software
engineers
don’t
have
the
same
feeling
when

they’re
trying
to
meet
feature
milestones
and
project
milestones,
and
find
themselves
lost
in
an

intricate,
cryptic
build
system.

The
cardinal
rule
of
build
systems
is
to
keep
the
user
experience
as
close

to
an
“out
of
the
box”
install
as
possible.

This
means,
don’t
create
a
set
of
elaborate
scripts/steps
that

modify
the
build
environment
needlessly.

Recall
Microsoft’s
“Razzle”
build
system?

It
worked
great
as

long
as
nothing
changed,
and
when
something
within
the
environment
did,
then
the
results
were

unknown,
and
there
was
not
an
obvious
way
to
debug.

Happily,
Microsoft
learned
their
lesson,
and
the

current
Visual
Studio/MSBuild
system
is
very
extensible
to
meet
developers’
needs.

The
ideal
Build

Vision
should
support
a
core
build
“framework”
into
which
additional
layers
or
components
can
plug
to

facilitate
the
organization’s
requirements,
(e.g.
binary
signing,
security
settings,
etc.)

At
Citrix,
for
Windows-‐based
builds,
Visual
Studio
and
Perforce
comprise
the
core
framework,
and
as

few
MSBuild
plug-‐ins
as
required
are
leveraged
to
create
the
flagship
XenAppXenDesktop
products.

The
details
of
the
Vulcan
Build
system
were
previously
discussed
in
section
10.3.1.

11.4 Keep
to
standards

Use
standards
where
it
makes
sense
to
do
so,
and
resist
the
dark
embrace
of
the
“cookie
cutter”

approach
because
this
is
ultimately
doomed
to
fail.

How
the
Build
Visionary
chooses
to
define
the

standards
should
be
based
on
development
feedback
and
the
problems
they
are
trying
to
solve.

For

example,
there
could
be
a
standard
way
to
build
all
C#/Managed
code
components,
or
a
way
that
all

Windows-‐based
components
are
signed.

The
reality
is
that
the
combinations
are
many,
and
to
reiterate

should
be
based
on
the
needs
of
all
the
Build
stakeholders.

12 Scheduling
Updates
and
Maintenance

12.1 How
to
do
it

The
Build
Vision
should
account
for
system
maintenance
and
updates,
as
this
is
an
important
part
of

keeping
the
Build
system
up
to
date
and
meeting
new
requirements.

As
mentioned
earlier,
keeping
the

various
parts
of
the
system
componentized
as
plug-‐ins
to
the
core
framework
will
facilitate
ongoing

maintenance.

Additionally,
the
following
guidelines
further
ensure
seamless
transitions
as
a
result
of

updates,

• Stage
updates
to
get
maximum
ROI.

Group
similar
updates
together
to
minimize
the
amount
of

change
to
the
system.

This
may
seem
obvious,
but
is
often
overlooked
by
the
overzealous
Build


26

engineer.

• Automate
as
much
as
possible.

scripting
updates
such
that
they
require
minimal
manual

intervention
also
mitigates
any
potential
human
error

• Test
to
trust.

This
is
the
most
critical
part
of
maintenance.

Nothing
will
destroy
the
credibility

of
the
Build
team
quicker
than
a
botched
upgrade
resulting
in
downtime
for
Development

teams.

Trust
between
the
stakeholders
is
a
requirement
for
a
successful
Build
Vision,
so
test,

test
and
test
some
more
before
implementing
updates
into
production.

• Keep
the
Stakeholders
in
the
loop
on
progress.

Notification
of
key
events
during
the

maintenance,
(start
time,
end
time,
benefits,
changes
in
behavior,
etc.)
will
also
shore
up
trust
in

the
Build
team
and
Vision.

The
Stakeholders
will
feel
like
they
are
“in
on”
events,
even
if
they

don’t
need
to
be,
plus,
the
Build
visibility
doesn’t
hurt
the
reputation
of
the
team
either.

12.2 When
to
do
it

Updates
come
with
some
overhead
in
that
they
can,
(and
usually
do)
come
with
some
amount
of
change

to
the
system.

It
is
critical
that
they
are
scheduled
with
the
knowledge
and
consensus
of
the
Build

stakeholders.

Following
the
following
general
guidelines
will
ensure
that
there
will
be
minimal

confusion
as
the
result
of
Build
System
maintenance,

• Align
updates,
(as
much
as
possible)
with
project
schedules
and
milestones.

The
last
thing
the

Build
Visionary
wants
is
to
conduct
maintenance
during
a
critical
project
milestone,
(e.g.
the
last

weekend
before
the
project
achieves
final
function/code
freeze).

• Attach
updates
to
a
particular
project.

At
the
end
of
the
day,
it’s
really
all
about
getting
the

product
to
the
customer,
so
having
a
Build
Vision
in
general,
(in
addition
to
the
maintenance),

attached
to
a
project
ensures
that
the
Vision
will
be
funded
properly.

The
savvy
Build
Visionary

will
work
closely
with
the
Release
and
Development
teams
to
embed
needed
Build
System

updates
into
the
project
schedule.

13 Final
Thoughts

Like
everything
else
in
the
technical
industry,
the
Build
Vision
never
ends;
it
is
always
evolving
to
meet

new
requirements,
tools
and
processes.

As
we
move
forward
into
whatever
the
next
new
“thing”
that

will
become
the
norm
in
Software
Development,
and
companies
increasingly
focus
on
trimming
costs

and
increasing
revenues,
the
Build
Visionary
must
take
care
to
nurture
the
Vision,
keep
it
looking

forward,
and
constantly
demonstrating
the
ROI
to
everyone
who
will
listen.

This
will
help
secure
the

Build
Vision’s
position
as
a
critical,
core
team
within
the
organization.

The
Build
Visionary
will
find

himself/herself
wearing
many
hats,
from
the
technical
to
the
salesperson,
ensuring
longevity
of
the

Build
Vision.

White Paper: Creating a World-Class Build System, and Getting It Right

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (7)

Semelhante a White Paper: Creating a World-Class Build System, and Getting It Right

Semelhante a White Paper: Creating a World-Class Build System, and Getting It Right (20)

Mais de Perforce

Mais de Perforce (20)

White Paper: Creating a World-Class Build System, and Getting It Right