A brief talk I gave at Scale Camp, hosted by The Guardian. http://www.scalecamp.org.uk

1. Scaling war stories
• Last.fm vs Xbox360
• Quick and hacky profiling
• Discuss
Wednesday, 26 May 2010

2. What happens when
scale is forced on you?
• 24 Million Xbox360 users
• Estimated ~5 times peak traffic increase
• No control over source of client
• External partner, with their own deadlines
• Must be reliable at launch. Err? (The PR
problem)
Wednesday, 26 May 2010

3. A dose of reality
• You won’t have the resources you’d like
• People (skills, availability,
• Budget (new servers would be nice)
• Time (more of this would be nice too)
Wednesday, 26 May 2010

4. B-b-but...
• It’s not an ideal world
• I’m not a systems developer (I spend half
my time on the front end)
• Quite hacky. Open to suggestions,
alternatives, your experiences
Wednesday, 26 May 2010

5. HTTP profiling
• Had a development Xbox
• Stuck a proxy box in the middle
• Profiled traffic with Wireshark
Wednesday, 26 May 2010

6. Wireshark
Wednesday, 26 May 2010

7. Making a list...
• API calls made
• In order, as uses cases
• Bad uses that can be factored out
• What breaks when call X is disbabled
• Proxy routed to a dev version of API
Wednesday, 26 May 2010

8. Xbox Beta
• Launched to 1,000 people (At midnight on
a Friday). Then to a further 9,000 people
• Gave a nice real usage sample
• Great, the most common calls are...
completely pointless.
• frequent != expensive (necessarily)
Wednesday, 26 May 2010

9. Custom Logging gen-time
LogFormat "%h %l %u %t "%r"
%>s %b "%
{Referer}i" "%{User-Agent}i"
%{gentime}n"
<?php
apache_note("gen-time", $x);
Wednesday, 26 May 2010

10. Abusing bash
Text
awk 'BEGIN
{FS="t"} {a[$2]++;b[$2]=b[$2]+$1}
END
{for (i in a) printf("%.2f %sn", b[i]/a[i]/
1000000, i)} '
Wednesday, 26 May 2010

11. Stats
Count Total Average
71638 trackgetInfo 15789 0.22
53941 artistgetImages 19647 0.36
15150 radiogetPlaylist 6962 0.46
7308 librarygetArtists 2402 0.33
5020 usergetRecentStations 1674 0.33
4979 adsgetvideos 1810 0.36
4205 radiotune 923 0.22
3155 tracklove 0 0.00
1507 artistgetinfo 440 0.29
1258 usergetRecommendedArtists 1457 1.16
1135 usergetInfo 171 0.15
1130 geogetTopArtists 575 0.51
1128 radiogamerstations 298 0.26
1102 taggettopartists 1488 1.35
1021 trackban 0 0.00
1006 usergetLovedTracks 271 0.27
Wednesday, 26 May 2010

12. zomg, graphs!
Wednesday, 26 May 2010

13. Wednesday, 26 May 2010

14. Wednesday, 26 May 2010

15. Wednesday, 26 May 2010

16. Profiling with Xdebug
• Profile PHP, stack traces, memory usage, etc
• http://xdebug.org
extension=xdebug.so
xdebug.max_nesting_level=500
xdebug.default_enable=0
xdebug.profiler_enable_trigger=1
xdebug.profiler_append=1
xdebug.trace_output_name=
%R.cache.out
Wednesday, 26 May 2010

17. Making sense of it
Wednesday, 26 May 2010

18. Contingencies
• From invisible, to user affecting
• Disable DB writes, speed up and prevent
blocking on failure
• Suppress badly architected Flash Player (and
less evil taste-o-meter)
• Kill % of method calls at perlbal
Wednesday, 26 May 2010

19. Wednesday, 26 May 2010

20. Monitoring porn
• Launch control with: 3 projectors, 5 x 24”
monitors, a dozen dev/sysops with laptops
• A moose, a cow and a fail train
Wednesday, 26 May 2010