Opera Software is mainly known for its web browser, but Opera also created and develops a growing set of web applications like its own community dedicated to the browser, my.opera.com, started in 2001. Since then, the community grew bigger and bigger, and had the "usual" scalability problems. The application has been rewritten 3 times and the current developers team is at a turning point now. How to make the application and systems scalable with the growing amount of traffic and users?

1. my.opera.com scalability v1
Italian Perl Workshop ~ Pisa 2008
cosimo streppone <cosimo@cpan.org>

4. Scalability
==
Performance

5. Scalability
!=
Performance

6. Scalability
==
memcached?

7. Scalability
!=
memcached

8. My Opera Community
http://my.opera.com
blogs shoutbox
Opera browser promotion
photo albums
RSS-Atom feeds
slideshows
custom designs
Public APIs
xml/json forums

9. 2001

10. 2003

11. 2005

12. 2006

14. 2008

15. my.opera.com
small (un)known web sites

16. Users (k)
2.500 *
1.640 IPW2008
887
430
257 205
1 10 50
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

17. Users (k)
Servers
2.500 *
IPW2008
1.640
887
430
257 205
1 10 50
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

18. Users (k)
Servers
Dyn req/s
2.500*
1.640
887
257 205 430
1 10 50
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

19. proxy errors

20. the zen way to
indefinite scaling

21. Start
Y
Does it run?
Does it N
N scale?
No problem...
Identify and fix
Wait and restart Y bottlenecks
End

22. /* recommendations */
server-side statistics
good stress testing tools

24. shared-nothing
architecture
(“zero tolerance for sharing”)

25. lvs +
lw httpd
mod_perl
.....
backends
nfs server
disk cache
users store

26. lvs +
lw httpd
mod_perl
.....
backends
nfs server
disk cache
users store

27. lvs +
lw httpd
mod_perl
.....
backends
nfs server
disk cache
users store

28. ... not so easy ...

29. dev.opera.com
articles cache
(or “don't guess, profile!”)

30. /* problem */
1 function
(XWA::Quote2Dev::Quote)
~95% req time

31. /* solution */
Start
Y Great. Serve
In Cache? from cache.
N
Query DB, etc... Done!
Put it in cache
this time!

32. /* solution */
Yes, that simple!
Profiling is key.
try:
m{^ Devel:: .* Prof .*}x

34. static avatars
(or “put your http servers at work”)

36. /* problem */
a single forum page
~~
50 avatars
(crappy CGI backend requests)

38. new storage subsystem
pools, servers
fault tolerance, redundancy
webdav, http, ftp,
scp, mogilefs?, ...

39. user uploads use case
# Create resource object for avatar
my $res = MyOpera::Storage::Resource::Avatar->new(
owner => '{userid}',
content => '{binary data}',
);
# Main storage subsystem handle
my $storage = MyOpera::Storage->new();
# Upload on pools of servers all at once
my $ok = $storage->upload($res);

40. resources
(user uploads, binary blobs, ...)
pools, servers
URLs
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_o.png
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_t.jpg
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_m.jpg
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_l.jpg

41. /* results */
saved ~500k backend req/day
browser cache used!

42. dogpile effect
(or the cache “storms”)

43. /* problem */
very high load spikes
on DB, backends

44. /* cause */
many concurrent clients
build same cached items

45. /* solution */
cache contention algorithm
NFS-resistant
optimistic file locking

46. /* solution */
ideas taken from
HTML::Mason, Django, ...
patched File::NFSLock

47. /* further ideas */
LOCK_SPIN_TIME
distributed cache
on backends

48. Opera releases

49. /* problem */
10-30x normal load
handling of peak load?

50. /* try 1 */
load-balanced
database profiles

51. /* try 2 */
url hot-list

52. /* solution? */
no solution this time,
sorry.

53. /* further ideas */
Amazon EC2 ?
Akamai CDN services?
... ?

54. soft counters
(didn't work as expected, in progress...)
sacrifice speed for concurrency
(mysql's myisam locks all table on update)

55. /* future directions */
database partitioning?
modperl backend optimization?
threading sucks in mp2. debian anyone?
improve memory sharing?
sysadmin-level optimizations?

56. take-aways
find your PKIs

57. take-aways
always compare
before and after

58. take-aways
stress testing in your
release cycle

59. take-aways
have fun!
;-)

60. ?questions?

62. thanks!