Presentation at the New York Scala Meetup - 10/20/2010

1. Scala @foursquare
Jorge Ortiz
@jorgeortiz85

2. foursquare runs on...
• Scala 2.8.0
• Lift 2.1
• MongoDB
• Twitter open source projects
• ~46k lines of Scala

3. foursquare server team
• 9 out of 11 had no major prior Scala experience
• Previously: Java, Python, C#
• Most exposed to: ML, Lisp

4. foursquare loves Scala
• Pimped: Option, List, Map
• Type-safe equality: =?
• BackgroundOperation { ... }
• Throttle(“secret feature!”) { ... }
• Any Java library

5. 2.7.7 => 2.8.0
(painless migration)
• Compile: 2.7.7 with -deprecation
• Branch
• Compile 2.8.0 without -deprecation
• Compile 2.8.0 with -Xmigration
• Compile 2.8.0 with -Xwarninit
• Switch

7. foursquare on Lift
• REST API
• Out-of-the-box:
• Rich Ajax/Comet support
• ORMs for PostgresQL, MongoDB
• Random: Email, Facebook, JSON, utils...

8. Lift Deployment
• nginx
• HAProxy
• API machines - stateless
• WWW machines - stateful

9. GC issues
• Stateless API machines
• Tens of millions of requests per day
• Only a dozen or so servers
• 12 GB heap => 15sec GCs *ouch*

10. GC issues
• GC tuning?
• “There's no way to get around the fact that
we're allocating faster than we can collect.”
• Allocating ~200 MB/s
• Opportunity to reduce wasteful allocations
• Better load-balancing strategy
• Cap # of concurrent connections

11. GC issues
• Stateful WWW machines
• Stateful Lift + Scala closures + Java GC
• Lots of variables getting caught in Session
• val venues: RequestCache[List[Venue]] =
RequestCache(Venue.findAll(...))

12. MongoDB in Scala
• Some support built-in to Lift
• No expressive query language, so we wrote one:
Venue where (_.mayor eqs userid)
modify (_.tags push “I’m mayor!”)
updateMulti

13. Twitter open source
• Kestrel
• Ostrich
• GrabbyHands
• (probably more soon...)

14. We’re hiring
http://foursquare.com/jobs

15. Questions?