This document discusses Netflix's API ecosystem built using Scala, Scalatra, and Swagger. It summarizes Netflix's use of these technologies to build APIs that power their consumer electronics partner portal and enable certification of Netflix ready devices. It describes how the APIs provide a single source of truth for all device data at Netflix and correlate streaming quality metrics. It then discusses aspects of the architecture including the manager layer containing business logic, HTTP layer for handling requests/responses, and use of Scala, Scalatra, Swagger, and deployment process including immutable infrastructure.

1. API ecosystem with
Scala, Scalatra, and Swagger
at Netflix
Manish Pandit
@lobster1234

2. Manish Pandit
Engineering Manager, Streaming Platforms
Netflix
@lobster1234

3. Thank you, OSCON!

7. 50 M+ Streaming Members
40 countries and counting
Thousands of device types

8. Power the Consumer Electronics Partner Portal
Enable Certification of Netflix Ready Devices
Source of truth for all Device Data at Netflix
Correlate Streaming Quality Metrics
Our APIs

10. Devices
Smart devices
+
Certification
=
Lots of Device Metadata!

11. Model
Firmware
Screen Resolution
Subtitle Support
3D
DRM
Remote Control
Netflix SDK
4K HD
…
Everything matters..

12. 58 Resources
~10 methods per resource

13. Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor
Architecture
HTTP
Layer,
and
Manager
Layer
Cassandra
EVCache
Crowd/SSO
RDS
Astyanax
Netflix OSS Cloud Components

14. Manager Layer = Business Code
Protocol Independent
Package-able as an artifact

15. HTTP Layer = Protocol Wrapper
Map HTTP methods to Manager functions
Transform JSON <=> Case Classes

16. The road trip from your IDE to a Production Node

18. Yes, it can be fun!

19. Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor

20. Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor
Open Source Infrastructure Components
Libraries
Tools
Frameworks
PaaS with AWS IaaS

21. Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor

22. Scala
Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor

23. Immutable by default
Concise, yet expressive
Type inference
Multi paradigm
Java ecosystem
Functional

24. /**
* Map results from a java.sql.ResultSet to a list of objects of another type
* @param rs The result set
* @param f Function that takes the result set and returns an instance of T
* @tparam A The type
* @return Sequence of instances of type A
*/
def mapResults[A](rs: ResultSet, f: ResultSet => A) : Seq[A] = {
val array = ArrayBuffer[A]()
while (rs.next) {
array += f(rs)
}
array.toSeq
}

25. /**
* Function to return a cached value, or fetch it per the parameter f
* @param key The key to look up in cache, or set against this key if not found
* @param f The function that fetches the instance from some persistent store
* @tparam A The type
* @return None if not found in cache and persistent store, else Some[A]
*/
def withCache[A](key: String)(f: => Option[A]): Option[A] = {
EVCacheGateway.get[A](key) match {
case None =>
val data = f
if (!data.isEmpty) EVCacheGateway.set(key, data.get)
data
case Some(x) =>
Some(x)
}
}

26. /**
* Get a user given an ID
* @param id The User ID
* @return None if user is not found, else the user
*/
def getById(id: Long): Option[User] = {
withCache[User](s"User$id") {
val query = "select * from users where user_id = ?"
DBHelper.queryWith(query, extract, id).headOption
}
}

27. Scalatra
Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor

28. Lightweight HTTP wrapper
.war based deployment
Swagger support
ScalatraSpec

29. /**
* Get all the chipsets ordered by name.
*/
get("/providers", operation(getChipsetProviders)) {
Ok( ChipsetManager.getProviders(0,20) )
}

30. ScalaTest
Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor

31. Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor
Simple
Intuitive
English-like
Rich
Promotes BDD

32. it should "return the chipset provider partner orgs" in {
val data = ChipsetManager.getProviders(0,20)
data.records.isEmpty should be (false)
data.records.size should be (20)
data.total should be >= 21l
}

33. ScalatraSpec
Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor
Traits to take ScalaTest to the next level
Helpers for JSON parsing
Plenty of wrappers (body, headers..)

34. addServlet(new ChipsetService, "/*")
it should "get a list of chipset providers" in {
getWithToken("/providers")(TestUserTokens.testToken) {
status should equal (200)
body should include (""""start":0,"end":19,"count":20""")
}
}

35. Swagger
Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor

36. Excellent support within Scalatra
Type-safe documentation
Contract First API Sandbox
Machine Readable

37. get("/providers", operation(getAllChipsetProviders)) {
//code
}
case class PartnerList(data: Seq[PartnerOrg], start: Int, end: Int, count: Long, total:
Long)
def getAllChipsetProviders = authApiOperation[PartnerList]("getChipsetProviders", "Get all
chipset providers")
.parameter(queryParam[Option[Int]]("start").description("Starting record count,
defaults to 0"))
.parameter(queryParam[Option[Int]]("count").description("The number of records
requested, defaults to 20"))

38. Swagger

40. Deployment
Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor
Push to dev
Jenkins runs dev
build, tests,
merges to
master
Jenkins runs
master build,
makes a .deb
Aminator bakes
an AMI from
the .deb
Asgard deploys
the AMI in
staging cloud

41. Production?
Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor

42. lgml-mpandit:nrd-portal-api mpandit$ git status
On branch dev
Your branch is up-to-date with 'origin/dev'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: api/src/main/scala/com/netflix/nrdportal/http/ChipsetService.scala
modified: api/src/test/scala/com/netflix/nrdportal/http/ChipsetServiceSpec.scala
modified: api/src/main/scala/com/netflix/nrdportal/manager/ChipsetManager.scala
modified: api/src/test/scala/com/netflix/nrdportal/manager/ChipsetManagerSpec.scala
no changes added to commit (use "git add" and/or "git commit -a")
Check in

43. [info] Passed: Total 1386, Failed 0, Errors 0, Passed 1386
[success] Total time: 1273 s, completed Jul 19, 2014 8:39:54 PM
Build step 'Build using sbt' changed build result to SUCCESS
Pushing HEAD to branch master of origin repository
Returning node parameter for ssh-dynaslave-1291624c
Triggering a new build of PPD-NRD-PORTAL-API-MASTER #1172
Notifying upstream projects of job completion
Finished: SUCCESS
Dev Build

44. Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor
Integration Tests
Hitting Real DB
Refreshed every night with Production

45. Build Artifacts:
nrd-portal-api_1.0-h1172.880e187_all.deb 127.71 MB [fingerprint]
Changes:
Added an endpoint to return a list of chipset provider partner orgs.
NFLX-5514 (detail)
Started by upstream project PPD-NRD-PORTAL-API-DEV build number 1923
originally caused by:
Started by an SCM change
Revision: 880e1873fef63d278b3180b49af7434e234dee40
origin/master
Master Build

46. Finally!

47. “A node, once deployed, cannot
be changed.”
Immutable Deployments

48. Easy rollbacks
Consistency
Predictability
Immutable Deployments

49. Summary
Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor
Bridge the gap between dev and deploy
No such thing as too many tests
Automate everything
Document your APIs
Best solutions are implementation
agnostic

50. Netflix OSS - http://netflix.github.io/#repo
Scalatra - http://www.scalatra.org/
Swagger - https://helloreverb.com/developers/swagger
ScalaTest - http://www.scalatest.org/

51. Session Feedback and Rating
slideshare.net/lobster1234
speakerdeck.com/mpandit