2. About me
Manish Pandit
Director of Engineering, IGN
@lobster1234
linkedin.com/in/mpandit
3. About IGN
We are a leading online media and services company
obsessed with gaming and entertainment.
56MM Monthly Unique Views
737MM Monthly Page Views
20MM Monthly Video Views
4. My Objective
Is:
To get you excited about building APIs in Scala
Isn't:
To incite a functional vs. imperative, or Scala vs.
____ debate
5. Agenda
• APIs at IGN
• API Evolution
• Why Scala
• API Development with Scala
• Monitoring, Maintaining, Evolving
• Performance Management
6. APIs at IGN
• Offer SOA advantages
• Isolate the back end systems from the FE
• IGN is going where the audience is, fast
– Mobile Devices, Consoles, TVs, Aggregators
• APIs are a specialized skillset
• Expand the development to outside
developers
7. APIs at IGN : Numbers
• ~5 Billion requests a month
• Average Response time of under 20ms on
cache misses
• 17 APIs between Social, OAuth, and Content
• Hardest hit APIs doing about 25K RPM at peak
9. The API Evolution : V1 [2006]
• The CMS and the desktop site calling Java
Services, which wrapped Oracle DB calls
behind Hibernate.
• Caching with Memcache
– Near-far Cache Architecture to address hot spots
• Not all content types were supported
• Schema was still too complex
10. The API Evolution : V2 [2009]
• Rails based Content API, Java based Social API
– Standards around request/response
– Automated Testing
– MongoDB NoSQL storage to simplify the schema
– Solr
• Video and Blogs served by V2 API
• Games and Articles still used V1
• Mixed-infrastructure with Rails and Java
echosystem
• Performance sucked royally
11. The API Evolution : V3 [2012]
• A fresh look at the APIs and IGN’s direction
• First attempt to
– Integrate all content types
– Open up the APIs to external consumers
– Evolve the CMS, traditionally a monolithic system
– ElasticSearch
– Varnish Response Caching
• A learning opportunity for traditional Java
stack engineers
12. API Development
• MVC Model for APIs
– View, which marshals/unmarshals request and
response data
– Controller, which picks the right model to serve
the request
– Model, which has the necessary logic to talk to the
database, or other APIs to fetch data
13. Why Scala
• Performance tied to cores than processor
speed, i.e. concurrency
• Ideal for API development – (relatively) simple
to handle concurrency and immutability
• JVM based = Performance
• Re-use Java toolkits and libraries
• Concise, expressive code (1:2.5 with Java)
• Actor model makes the difficult parts easy
14. Why Scala
• Richer Collections API
• Traits enabling Mix-Ins and behavior re-use
• Statically Typed w/Type inference
• Functional
• Growing developer community
• REPL
• Culture – learning, hacking and achieving
16. The Biggest Reason
• Just makes sense
– Pattern matching
– Implicit conversions
– Control abstractions for collection mapping,
filtering, iterating
– Guaranteed Singletons at the classloader level
– Almost non-existent boilerplate
….Productive, and fun!
17. Picking up Scala : Week 1
• Scala for Java Developers fast-tracked the
learning
• Syntax Familiarity
• Java without semicolons?
def total(nums: List[Int]) :Int = {
var sum:Int=0
for(num<-nums) {
sum+=num
}
sum
}
18. Picking up Scala : Week 4
• Reduce the lines of code, embrace
immutability, use the functional paradigms
built into the language
• Unmystrify the implicits magic
nums.foldLeft(0)((n,c) => n+c)
19. Picking up Scala : Today
• Focus on immutability
• Think functional – with the knowledge gained
from using the functional aspects of the
language, build control abstractions
• Keep learning and applying
20. API Components
• Scalatra (View, Controller)
• Lift-MongoRecord for MongoDB (Model)
• Casbah for MongoDB
• Play! Anorm for MySQL
• Actors for tasks like syndication
• Maven for Builds
• MongoDB for persistance
21. Other API Components
• Varnish Cache
• Elasticsearch
• Yammer Metrics (ping, healthcheck)
• Swagger (self-documenting RESTful APIs)
• 3Scale for Partner APIs
• IntelliJ IDEA and Eclipse with ScalaIDE
22. Challenges with Scala
• Learning Curve depends on the culture
• TPS Cover Sheets, vs. Hack Nights
• Scala is what you make of it
24. Challenges with Scala
• Scala is what you make of it
– Middle ground – simple is beautiful
• Tooling not as rich as Java
– Changing with Typesafe investing in ScalaIDE
25. Challenges with Scala
• Slow compilation based on the source
– How well the code matches up to the JVM
constructs
• No (binary) compatibility of dependencies for
the major releases (2.8, 2.9, 2.10(?))
• Monitoring
– No native instrumentation agents as they exist for
Java
26. API Development : Deploying
• Continuous Deployment
• Jenkins hooks to github
• Pushes artifact(s) to Nexus
• Fabric to push code to production (to be
automated)
27. API Performance Management
• As far as Scala goes, the tips are not much
different than Java
– GC Tuning
– Avoid shared state
– Parallel collections
28. API Performance Management
• For the overall APIs, the performance tuning has
many knobs to work around with
– Stateless services
– Async calls with Akka Agents
– Response level cache
• Keep an eye on the database and network
– Control the payload size
– Use reasonable defaults
– Allow field filtering
– Paginate your response
29. Monitoring and Maintenance
• Newrelic
– JVM stats, SQL trace
– Supports Scala via annotations, but not as rich as
Java
• Typesafe Console for Akka
30. Future of APIs at IGN
• Opening up the APIs
• Tests, tests, tests..
• Move towards simplifying the stack
– sbt vs. maven
– Typesafe stack
32. Resources
• Do NOT use scaladoc as your 1st reference, at
least initially
• StackOverflow
• Twitter’s Scala School
• Online blogs and tutorials
• Book: Scala for the Impatient
• Scala Meetup Groups in your area
• Coursera course on FP by Martin Odersky