1.
MERGE 2013 THE PERFORCE CONFERENCE SAN FRANCISCO • APRIL 24−26
Perforce Merge Conference, April 2013
To facilitate transporting large amounts of data
between global sites, a Perforce proxy with an alternate
transport can be used to work around latency or
bandwidth limitations inherent with TCP/IP networks.
Using a Perforce Proxy with
Alternate Transports
Overcoming High-Latency or Low-Bandwidth Networks
Matthew Janulewicz, Lucasfilm Ltd.

2. 2 Using a Perforce Proxy with Alternate Transports
This is text for annotations in footer. Similar to footnotes treatment.
What is a Perforce Proxy?
A Perforce proxy is a server that collects and maintains a copy of archive files to be used by
remote teams that are geographically separated from the Perforce server. The purpose of a
Perforce proxy is to “cache” frequently used data on a server local to the remote users to
prevent repeated transfer of commonly used data over long distances.
A Perforce proxy is stateless in that it does not track the state or presence of archive files
through a database. Sync requests are passed through a proxy to the associated Perforce
server. Once the server calculates the file revisions required for the sync request, they are
delivered from the Perforce proxy if they are available. If they are not available, files are
delivered from the Perforce server and a copy is deposited on the Perforce proxy to be used in
subsequent requests for that file revision. A Perforce proxy starts empty and the file revisions
available from it are built over time based on sync requests.
When new file revisions are submitted through a Perforce proxy, a copy is deposited in the
proxy and does not have to be resynced to the proxy.
Some sites use an automated, scheduled “pre-cache” script to sync updated files to the
Perforce proxy at set intervals. This way they can update the proxy with new revisions before
local users need them. Using ‘p4 –Zproxyload’ will load files into the Perforce proxy without
having to physically sync them to a workspace.
Older file revisions in the Perforce proxy can be removed without penalty because the proxy
merely checks for the presence of the file when it is requested. If someone specifically
requests a removed file, the proxy will simply download it from the Perforce server again. The
archive file layout on the Perforce proxy mirrors the layout on the Perforce server.
Why Not Use a Replica?
Perforce replicas do have some advantages over proxies. In the case of a forwarding replica,
the replica holds a copy of the database so certain queries (sync calculations, fstat calls, etc.)
return results faster because the request does not have to go all the way to the originating
server. The burden on the master server is greatly reduced by using a forwarding replica
instead of a Perforce proxy.
The drawback to any replica is that it replicates an entire server, including all archive file
revisions (up to p4d 2012.2). In Lucasfilm’s case, the server we wanted to share data from has
more than 6 TB of archive data in it (and is growing). The files we need to collaborate on in
remote offices take up about 1.5 TB of that. The cost of building out a system to handle that
much storage, storage that will mostly not be used, was not something we were willing to
commit to.
A replica basically serves the same purpose of an “actual” server, in that it has to handle all the
same requests to the database that a regular server might. You essentially have to build out a
new server with specs to match. A proxy, on the other hand, is primarily a file cache and will
use fewer resources than a full Perforce server or replica.
The size of the stored data on a Perforce proxy is also easier to manage. As mentioned earlier,
you do not manage archive storage on a replica as much as you simply provide it with an
equal amount of storage as the master server it replicates. You can, however, remove old
revisions of files from a proxy with no penalty. A basic tactic in managing the required size of
storage in a proxy might be to remove any files that have not been accessed within a certain

3. 3 Using a Perforce Proxy with Alternate Transports
This is text for annotations in footer. Similar to footnotes treatment.
number of days. You could also be more aggressive, depending on need, by writing a short
script to find files that have more than one revision and removing any that are not the current
#head or perhaps, additionally, #head-1. Most administrators will likely want to strike a balance
between the two approaches.
Lastly, as this white paper will outline, you are not limited to a TCP/IP transport when working
with a Perforce proxy. A Perforce proxy has no knowledge about how or why an archive file is
present; it only checks that it is there.
The Enemy: Network Latency
Communication between Perforce clients and servers is handled through TCP/IP. TCP/IP is a
fine protocol but it is “chatty” and is adversely affected by high-latency networks. Our ping
latency between California and the Pacific Rim is typically 200-300 ms. Ping results from
Perforce itself (p4 ping) were even slower (see Figure 1).
Figure 1: Response time from ‘p4 ping’: local average = 0.000s, remote average = 1.953s
Although we have high bandwidth (1 GB), the high latency can wreak havoc on large data
transfers over the Pacific Ocean. More latency means slower response times and more delays
for each packet going through. For short bursts it is unnoticeable, but with file transfers of even
a few GB, you can see a significant drop in expected transfer speeds, especially during times

4. 4 Using a Perforce Proxy with Alternate Transports
This is text for annotations in footer. Similar to footnotes treatment.
of high network usage. On our network there was a practical cap of well under .5 MB/s when
transporting large datasets between remote offices. Our initial test and production servers ran
on CentOS with a Linux 2.6.24 kernel (see Figure 2).
Figure 2: Initial transfer throughput (in MB/s) for Centos 5.4 and default TCP/IP settings using a P4 proxy
Significant improvements in throughput where achieved by upgrading the OS (CentOS) and
tweaking the TCP/IP stack on the server and associated Perforce proxy. These results are
from CentOS 5.5 with a Linux 2.6.32 kernel (see Figure 3). Additional TCP/IP tweaks were
applied as outlined in the Appendix.

5. 5 Using a Perforce Proxy with Alternate Transports
This is text for annotations in footer. Similar to footnotes treatment.
Figure 3: Initial transfer throughput (in MB/s) for Centos 5.5 and optimized TCP/IP settings using a P4
proxy
This improvement was still not enough. Considering the size of datasets we might need to
share during game development, especially in the time leading up to a milestone, we were
seeing too much delay prior to all our data arriving in the remote office. The end of our day,
when most work is submitted, coincided with the beginning of their day. It would not be
unusual for the remote office to have to wait until the middle of their day, or even the next day,
to begin work on the items we had updated the day before.
As an example, sending a 1 GB file with our default OS and settings could take an entire day
or more to transfer. With the updated OS/kernel and optimized TCP/IP settings, this transfer
was cut to 4 hours. This was a great improvement but still nowhere near approaching real
time. For collaboration on large datasets, we needed to improve transfer times even more.
Solution: Dispense with TCP/IP
As mentioned earlier in this paper, the layout of the archive files in a Perforce proxy mirrors
that of the associated Perforce server. Because the proxy is not “smart” (stateless) and does
not know what files it houses at any given time, it also has no knowledge of how files were
delivered to it in the first place. It checks for the presence of files during each request. A
common way to seed a large Perforce proxy is to make a backup (tar, CD, tape) of significant
files from the server and restore it to the associated proxy server.
You can take advantage of this statelessness by using a different transport protocol, in our
case User Datagram Protocol (UDP). UDP is a lightweight transfer protocol that when

6. 6 Using a Perforce Proxy with Alternate Transports
This is text for annotations in footer. Similar to footnotes treatment.
compared to TCP/IP has less (no) error checking, no concept of retransmitting dropped
packets, no ordering of packets, and no congestion control.
Open source and commercial UDP applications have added support for error checking,
security, and bandwidth control. Lucasfilm already had experience with the fasp technology
from Asperasoft and decided to use it to help overcome the increasing transfer speed
limitations we encountered over time while our Perforce proxy grew.
We already owned licenses for its point-to-point server product, which essentially provides scp-
like UDP transfers for single files at a time. We have been using this for several internal tools
for some years. However, the nature of those tools is that the files/packages being transferred
are known. We were not doing any type of “mirroring” of file systems but were simply
transferring files on a per-user basis.
To effectively mirror a file system, or parts thereof, we used Asperasoft’s Aspera Sync (async)
product, which provides a drop-in replacement for rsync. We were then able to set up a simple
one-way mirroring of relevant file paths between our Perforce server and associated proxy.
Because async allows you to throttle the utilized throughput, we could realize just about any
jump in throughput we chose. We found that between 20-25 MB/s was ideal because it gave
us a significant enough improvement without affecting the other transfers that might be
happening at any given time.
At these rates, the time to send a 1 GB file improved from 4 hours to a few minutes. When
needed, we can coordinate with other UDP users to utilize more bandwidth to send content to
remote offices even faster.
Conclusion
Under certain circumstances a Perforce proxy is preferable to a Perforce replica. The inherent
statelessness of a proxy lends itself to clever solutions for overcoming suboptimal bandwidth
or latency in your global network. You can effectively take TCP/IP out of the equation to realize
orders of magnitude improvements in transfer rates for Perforce proxy assets.
Appendix
The following settings were tweaked to realize modest improvements in TCP/IP transfer rates.
In CentOS 5, the settings are in /etc/sysctl.conf
### New 2.6.x TCP tuning:
net.ipv4.tcp_adv_win_scale=7
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_congestion_control=bic
net.ipv4.tcp_moderate_rcvbuf=1
### IPV4-TCP specific settings
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_rmem = 16777216 16777216 16777216
net.ipv4.tcp_wmem = 16777216 16777216 16777216

7. 7 Using a Perforce Proxy with Alternate Transports
This is text for annotations in footer. Similar to footnotes treatment.
net.ipv4.tcp_mem = 16777216 16777216 16777216
net.ipv4.ipfrag_high_thresh = 83886080
net.ipv4.ipfrag_low_thresh = 41943040
net.ipv4.ipfrag_time = 10
net.ipv4.tcp_low_latency = 0
### CORE settings (mostly for socket and UDP effect)
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 16777216
net.core.netdev_max_backlog = 3000000
# disable swapping if at all possible:
vm.swappiness = 0
# Initial 9.3 rev. Increase max shared memory segment size.
# Disable response to broadcasts.
# You don't want yourself becoming a Smurf amplifier.
net.ipv4.icmp_echo_ignore_broadcasts = 1
# enable route verification on all interfaces
net.ipv4.conf.all.rp_filter = 1
kernel.shmmax = 536870912
# modify overcommit settings to allow large zeno processes to fork.
vm.overcommit_memory = 2
vm.overcommit_ratio = 200