1. Scaling Source Control for the Next
Generation of Game Development
Mike Sundy Toby Roberts
2007 Perforce User’s Conference
Scaling Source Control for NextGen Game Development 1

2. Overview
• Current Gen vs. NextGen
• Test Methodology
• Test Results
• Lessons Learned
• Future of P4 at EA
Scaling Source Control for NextGen Game Development 2

3. Challenges in Game Industry
• CurrentGen vs. NextGen
• 9 GB for DVD's for CurrentGen. (Xbox, PS2)
• 30-50 GB for NextGen (Blu-Ray, HD-DVD – XBox 360,
PS3) - 3-5x increase in storage capacity
• Asset Volume Explosion
• 50k clientspec grew to 500k clientspec
• Build Farms
• 20-30 machines grew to 100+
• Team Sizes
• Outsourcing
Scaling Source Control for NextGen Game Development 3

4. EA Redwood Shores (EARS)
P4 Installation Metrics
• 11 Perforce Servers, 1200 users (90+ servers, 4,000+ users at EA)
• 7 million files on one server, 80 GB db.have
• Store all source code and source data in P4
• 1.5 TB on one server, 7 TB total P4 storage
• Avg. client workspace – 80-120GB and 250k-500k files
• 2,500 – 66,000 syncs per day
• Titles include Sims, Godfather, Simpsons, and Spore.
Scaling Source Control for NextGen Game Development 4

5. Problems with Windows
• 2 GB per process memory limit
• Filehandle leak (crashed up to 6 times per day)
• Performance during large concurrent syncs
• 40 GB sync went from 40 min. to 3+ hours
• NTFS
• Threads and troubleshooting
• Patching
Scaling Source Control for NextGen Game Development 5

6. Goals of Project
• Improve stability
• Improve scalability
• Solution that works for our dev environment,
infrastructure, and team cultures
Scaling Source Control for NextGen Game Development 6

7. Parameters of Project
• Dell and STK hardware
• Intel
• RedHat
• Windows Desktops
• 95% of user machines
• Gig network
Scaling Source Control for NextGen Game Development 7

8. Test Methodology
• Single operation vs. load
• P4 benchmarking tools
• fstst and locktest
• Homegrown scripts
• Force sync of 39 GB data, 180k files
• p4.exe on Windows
Scaling Source Control for NextGen Game Development 8

9. Configuration Overview
• Tested 21 different server configurations.
• Windows vs. Linux
• 32-bit vs. 64-bit
• NTFS v. ext3 v. ReiserFS v. XFS
• 10k vs. 15k drives
Scaling Source Control for NextGen Game Development 9

10. db locktest
6
5
4
Seconds
3
2
1
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
config
Scaling Source Control for NextGen Game Development 10

11. db 8k read-write
600
500
400
MBs
300
200
100
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
config
Scaling Source Control for NextGen Game Development 11

12. rcs 64k write
900
800
700
600
500
MBs
400
300
200
100
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
config
Scaling Source Control for NextGen Game Development 12

13. rcs 64k read
3500
3000
2500
2000
MBs
1500
1000
500
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
config
Scaling Source Control for NextGen Game Development 13

14. 1 force sync avg.
21
19
17
15
13
config
11
9
7
5
3
1
0 5 10 15 20 25 30 35 40 45 50
minutes
Scaling Source Control for NextGen Game Development 14

15. 3 force syncs avg.
21
19
17
15
13
config
11
9
7
5
3
1
0 10 20 30 40 50 60 70 80
minutes
Scaling Source Control for NextGen Game Development 15

16. 5 force syncs avg.
21
19
17
15
13
config
11
9
7
5
3
1
0 10 20 30 40 50 60 70 80
minutes
Scaling Source Control for NextGen Game Development 16

17. % degradation from 1 to 3 syncs
21
19
17
15
13
config
11
9
7
5
3
1
0 0.1 0.2 0.3 0.4 0.5 0.6
percent
Scaling Source Control for NextGen Game Development 17

18. edit loop avg. (176 files)
21
19
17
15
13
config
11
9
7
5
3
1
0 0.2 0.4 0.6 0.8 1 1.2
seconds
Scaling Source Control for NextGen Game Development 18

19. Test Results Summary
• DB operations 5-10x faster on 64-bit Linux on XFS than
Windows 32-bit.
• Linux sync performance under load degraded only 5%
instead of the 450% we saw with Windows.
• XFS outperforms ext3 and even edges ReiserFS for data
syncs from the repository volume.
Scaling Source Control for NextGen Game Development 19

20. dm-CommitSubmit
config linux-gf1_1666
win-gf1_1777
win-gf1_1666
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
seconds
Scaling Source Control for NextGen Game Development 20

21. avg. edit
config linux-gf1_1666
win-gf1_1777
win-gf1_1666
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
seconds
Scaling Source Control for NextGen Game Development 21

22. avg. fstat
config linux-gf1_1666
win-gf1_1777
win-gf1_1666
0 0.1 0.2 0.3 0.4 0.5 0.6
seconds
Scaling Source Control for NextGen Game Development 22

23. avg. sync compute phase
config linux-gf1_1666
win-gf1_1777
win-gf1_1666
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
seconds
Scaling Source Control for NextGen Game Development 23

24. Configuration Lessons Learned
• OS
• 2.6 kernel was 2 – 4x faster than 2.4 in some tests.
• RAID
• RAID 5 best for RCS due to the high number of syncs/reads.
• DB benefited from RAID 1 or RAID 10.
• Filesystem
• XFS proved to be significantly faster than ext3 and beat Reiser by
30% in some tests.
• Drive Speed
• 20% less degradation with 15k drives during multiple concurrent
forced syncs due to higher I/O capacity.
Scaling Source Control for NextGen Game Development 24

25. Performance Principles
• Keep your db’s on their own disks
• Proxy heavy sync users such as build farms or remote
locations
• Best place to improve hardware performance is memory
and disk I/O for db’s
• There is a big difference in filesystem speed
Scaling Source Control for NextGen Game Development 25

26. Success Stories
• New standard has had a huge impact on our local team
productivity and P4 performance across EA.
• New platform has scaled well for 1,000 user team and
could scale to larger teams.
• Stability is great!
• No more memory or file descriptor limits
• Can identify and kill individual threads.
• $5 million saved per year in productivity across EA.
Scaling Source Control for NextGen Game Development 26

27. NextGen EA P4 Architecture
• P4 Server setup
• Distributed Development
• Proxy Build Farm (30% gain)
• Case-insensitivity
• Disk to disk mirroring
Scaling Source Control for NextGen Game Development 27

28. The Future of Perforce at EA
• RAID 50 (2x faster locking than RAID 10)
• SAS SFF drives
• AD2P4
• Multi-Node Cluster
• Perforce + binary delta copy technology
• Workspace mirroring
• Single View of Perforce Servers
Scaling Source Control for NextGen Game Development 28

29. Thanks
• Michael Shields
• Brett Taylor
• Kenny Wu
Scaling Source Control for NextGen Game Development 29

30. Trivia for Schwag
Questions?
Scaling Source Control for NextGen Game Development 30