Life on the edge is innovating a system that is ultimately responsible for 1/3 of all internet bandwidth usage in the US. Ever wonder how Netflix does it? This talk will overview a combination of services and open source projects that comprise the edge network, including Neflix Denominator. Discussions will include how we deal with DNS, distributed load balancing as well as discovery of mid-tier services. You'll understand which service components are involved with edge traffic management. When you leave, you'll have enough context to create your own life on the Edge with NetflixOSS!

1. Living on the Edge
June 2013
Adrian Cole
@adrianfcole #netflixoss @denominatorOSS
http://www.linkedin.com/in/adrianforrestcole

2. How Netflix Streaming Works
Quick DNS Primer
Denominator Deep Dive
Cloud Prize

3. Netflix Member Web Site Home Page
Personalization Driven – What goes on to make this?

4. How Netflix Streaming Works
Customer Device
(PC, PS3, TV…)
Web Site or
Discovery API
User Data
Personalization
Streaming API
DRM
QoS Logging
OpenConnect
CDN Boxes
CDN
Management and
Steering
Content Encoding
Consumer
Electronics
AWS Cloud
Services
CDN Edge
Locations

5. Content Delivery Service
Open Source Hardware Design + FreeBSD, bird, nginx

6. November 2012 Traffic

7. Real Web Server Dependencies Flow
(Netflix Home page business transaction as seen by AppDynamics)
Start Here
memcached
Cassandra
Web service
S3 bucket
Three Personalization movie group
choosers (for US, Canada and Latam)
Each icon is
three to a few
hundred
instances
across three
AWS zones

8. Asgard
http://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html

9. Portable DNS Control
Cassandra Replicas
Zone A
Cassandra Replicas
Zone B
Cassandra Replicas
Zone C
Regional Load Balancers
Cassandra Replicas
Zone A
Cassandra Replicas
Zone B
Cassandra Replicas
Zone C
Regional Load Balancers
UltraDNS
AWS
Route53
A portable way to manage multiple DNS providers from Java
DenominatorDynECT

10. PORTABLE CONTROL OF DNS CLOUDS

11. A Cloud Native Open Source Platform

13. Feature Set
• Do stuff in batches
• Cleanly handle advanced records
• Play nice with persistence
• Don’t do too much
• … Use cool things

14. Hello Denominator
mgr = Denominator.create("ultradns”,
credentials(username, password))
for (Iterator<String> zone =
mgr.getApi().getZoneApi().list();
zone.hasNext();) {
processZone(zone.next());
}

15. DNS Lingo
• Zone
– Name (ex. denominator.io.)
• Records
– Name (ex. www.denominator.io.)
– Type (ex. CNAME)
– TTL (ex. 300)
– RData (ex. myLB-1234567890.us-east-
1.elb.amazonaws.com)

16. Record Set
• Records visible to the resolver that have the
same name and type (also TTL).
• Ex. If www.denominator.io has 4 ip
addresses, they can be in the same recordset.
• Concept is helpful for DNSSEC

17. DNS Things
• Nameserver
– Server that listens on port 53 for queries
• Resolver
– Client that makes queries
• API
– Creates and controls configuration and data on
the nameservers.

18. http://square.github.io/dagger/

19. We wire with Dagger
@Provides
@Singleton
Route53Api provideApi(Supplier<Credentials>
credentials) {
return ContextBuilder.newBuilder(“route53”)
.credentialsSupplier(credentials)
.buildApi(Route53Api.class);
}

20. We adapt vendor-specifics
• Backends are jclouds apis, except mock.
• The back the denominator model
@Provides
@Singleton
ZoneApi provideZoneApi(DynECTApi jclouds) {
return new DynECTZoneApi(jclouds);
}

21. Denominator Model
Extensible types are Forwarding Maps
mxData.getPreference()
mxData.get("preference”)
profile.getType()
profile.get(“type”)

22. ResourceRecordSet Profiles
Server-side configuration for advanced
features like health-checks and visibility.
rrset.getProfiles().size() == 0 // not special
rrset.getProfiles().get(0).get(“type”)
// could be “geo”

24. Availability by reshaping traffic

25. Geo Profile
Visible to clients in specific territories
geo.getType() == “geo” // or geo.get(”type”)
geo.getGroup() // like US-West
geo.getTerritories() // subset of territories

26. Example code
// select the existing territories in US
existing = geoApi.getByNameTypeAndGroup("www.mysite.com.", "CNAME", "US");
// refine to exclude california
Geo existingGeo = toProfile(Geo.class).apply(existing);
Multimap<String, String> update = filterValues(existingGeo.getRegions(), not(equalTo("California")));
// apply the update
geoApi.applyRegionsToNameTypeAndGroup(update, "www.mysite.com.", "CNAME", "US");

27. Airline
https://github.com/airlift/airline

28. Our CLI is Airline
./denominator -p route53 record --zone
foo.com. add --name hostname.foo.com. --type
A --ec2-public-ipv4

29. Airline is a git-like CLI builder
We use a process called “really executable jar”
so that ./denominator works.
class GeoResourceRecordSetCommand extends
DenominatorCommand {
@Option(type = OptionType.GROUP,
required = true,
name = { "-z", "--zone" },
description = "zone name to affect. ex. netflix.com.")
public String zoneName;
}

30. RANDOM CLICKING AROUND TIME!

31. Netflix Cloud Prize
Boosting the @denominatorOSS
Ecosystem

33. Entrants
Netflix
Engineering
Judges Winners
Nominations
Conforms to
Rules
Working
Code
Community
Traction
Categories
Registration
Opened
March 13
Github
Apache
Licensed
Contributions
Github
Close Entries
September 15
Github
Award
Ceremony
Dinner
November
AWS
Re:Invent
Ten Prize
Categories
$10K cash
$5K AWS
AWS
Re:Invent
Tickets
Trophy

34. Functionality and scale now, portability coming
Moving from parts to a platform in 2013
Netflix is fostering an ecosystem
Rapid Evolution - Low MTBIAMSH
(Mean Time Between Idea And Making Stuff Happen)

35. Takeaway
Denominator is a multi-cloud DNS abstraction built as a library and a cli.
Fixing DNS APIs, one vendor at a time
https://github.com/Netflix/denominator
https://groups.google.com/forum/#!forum/denominator-dev
https://groups.google.com/forum/#!forum/denominator-user
http://www.linkedin.com/in/adrianforrestcole
@adrianfcole #netflixoss @denominatorOSS