User & Device Identity for Microservices @ Netflix Scale

User & Device Identity
For Microservices
@ Netflix Scale
Satyajit Thadeshwar
QCon San Francisco 2019

InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-user-identity/

Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

User & Device Identity for Microservices @ Netflix Scale
Satyajit Thadeshwar
Logged out?
#$%&!

Satyajit Thadeshwar
Time
CoreStreamingMetric
Current Last Week

Satyajit Thadeshwar
Satyajit Thadeshwar
Product Edge Access Systems
sthadeshwar@netflix.com

Satyajit Thadeshwar
Complicated

Satyajit Thadeshwar

Satyajit Thadeshwar
9 teams
57 watchers

Netflix subscribers and the devices that they use
Satyajit Thadeshwar

Satyajit Thadeshwar
Where we were
What we did
Wins

Where we were
Satyajit Thadeshwar

Satyajit Thadeshwar
Zuul
EDGE
Email: jsmith@gmail.com
Password: ********
ESN: LGTV20165-193456G568
User Login

Satyajit Thadeshwar
Zuul API
EDGE ORIGIN
Password: ********
ESN: LGTV20165-193456G568 /login
User Login

Satyajit Thadeshwar
Zuul API
Netflix
Microservices
auth
service
EDGE ORIGIN MID-TIER SERVICES
Password: ********
success
User Login

Satyajit Thadeshwar
Zuul API
Netflix
Microservices
auth
service
Password: ********
success
User Login
customerId: 10192378
ESN: LGTV20165-193456G568
Expires: In 8 hours

Satyajit Thadeshwar
Zuul API
Netflix
Microservices
auth
service
Password: ********
successSet-Cookie
User Login
ESN: LGTV20165-193456G568
Expires: In 8 hours

Satyajit Thadeshwar
Zuul
EDGE
Authenticate Request
/browse

Satyajit Thadeshwar
Zuul API
EDGE ORIGIN
/browse
/browse

Satyajit Thadeshwar
Zuul API
EDGE ORIGIN
/browse
success
KEY MANAGEMENT
SERVICE
/browse

Satyajit Thadeshwar
Zuul API
Netflix
Microservices
EDGE ORIGIN
/browse
success
MID-TIER
SERVICES
ESN: LGTV20165-193456G568
KEY MANAGEMENT
SERVICE
/browse

More than one service consuming cookies
Satyajit Thadeshwar

Satyajit Thadeshwar
Zuul
API
Device
Auth
Service
Legacy
API
Netflix
Microservices
SIGNUP
FLOW
SERVICE
subscriber
auth
service
lolomo / Search
DRM
Other services
EDGE ORIGINS MID-TIER SERVICES

Satyajit Thadeshwar
Zuul
API
Device
Auth
Service
Legacy
API
Netflix
Microservices
SIGNUP
FLOW
SERVICE
subscriber
auth
service
lolomo / Search
DRM
Other services
EDGE ORIGINS MID-TIER SERVICES/ios
/android
/atv
...

At massive scale
Satyajit Thadeshwar

Satyajit Thadeshwar
Netflix
158M+ subscribers

Satyajit Thadeshwar
Netflix
158M+ subscribers
1B+ devices

Satyajit Thadeshwar
Netflix
158M+ subscribers
1B+ devices
2M peak RPS

Satyajit Thadeshwar
Authenticate Request / Extract Identity
API
ORIGIN
KEY MANAGEMENT
SERVICE
= 2 million Requests Per Second

More than one token type
Satyajit Thadeshwar

Cookies
Satyajit Thadeshwar

Cookies
Satyajit Thadeshwar
- Signup

Cookies
Satyajit Thadeshwar
- Signup
- Login

Cookies
Satyajit Thadeshwar
- Signup
- Login
- Discovery

MSL Tokens
Satyajit Thadeshwar
- Device authentication
- Encryption
Message Security Layer (MSL)
https://www.infoq.com/news/2014/11/netflix-msl/

MSL Tokens
Satyajit Thadeshwar
- License
- Playback

CTicket
Satyajit Thadeshwar
- Legacy devices

Partner
Tokens
Satyajit Thadeshwar
- JWS, JWE
- Non-member
experiences

- Signup
- Sign-in
- Discovery
- License
- Playback
- Legacy
devices
- Non-member
experience
Cookies
MSL Tokens
CTicket
Partner
Tokens
(JWS, JWE)
Satyajit Thadeshwar

- Multiple services consuming auth tokens
- Multiple types of auth tokens
- Massive scale
- Inefficient, insecure & complicated
Where we were
Satyajit Thadeshwar

Satyajit Thadeshwar
Zuul
API
Device
Auth
Service
Legacy
API
Netflix
Microservices
SIGNUP
FLOW
SERVICE
subscriber
auth
service
NodeJS
Services
Lolomo / Search
DRM
Other services
Discovery
API
Playback
API

What we didUser & Device Identity for Microservices @ Netflix Scale
Satyajit Thadeshwar

Moved authentication to the edge
Satyajit Thadeshwar

Satyajit Thadeshwar
Zuul
API
Device
Auth
Service
Legacy
API
Netflix
Microservices
SIGNUP
FLOW
SERVICE
subscriber
auth
service
NodeJS
Services
Lolomo / Search
DRM
Other services
Discovery
API
Playback
APICookie
Service
MSL
Service
Partner
Service
EAS

Satyajit Thadeshwar
Zuul
API
Device
Auth
Service
Legacy
API
Netflix
Microservices
SIGNUP
FLOW
SERVICE
subscriber
auth
service
NodeJS
Services
Lolomo / Search
DRM
Other services
Discovery
API
Playback
APICookie
Service
MSL
Service
Partner
Service
EAS
EDGE AUTHENTICATION SERVICES

Satyajit Thadeshwar
Zuul
EDGE
EAS
renewal / device auth / key exchange
Cookie
Service
MSL
Service
Partner
Service
valid and not expired
95%
5%

Satyajit Thadeshwar
Zuul
EDGE
Cookie
Service
EAS
valid but expired
renewal call

Satyajit Thadeshwar
Zuul
EDGE
Cookie
Service
EAS
valid but expired
renewal call failed

Satyajit Thadeshwar
Zuul
EDGE
Cookie
Service
EAS
valid but expired
renewal call rescheduled
resolved identity

Satyajit Thadeshwar
Zuul
EDGE
Cookie
Service
EAS
valid but expired
renewal call rescheduled
rescheduled cookie
resolved identity

Satyajit Thadeshwar
Zuul
API
Device
Auth
Service
Legacy
API
Netflix
Microservices
SIGNUP
FLOW
SERVICE
subscriber
auth
service
NodeJS
Services
Lolomo /
Search
DRM
Other services
Discovery
API
Playback
APICookie
Service
MSL
Service
Partner
Service
EAS
EDGE AUTHENTICATION SERVICES

Satyajit Thadeshwar
Passport

Satyajit Thadeshwar
Passport
- Identity structure created at the edge for each request

Satyajit Thadeshwar
Passport
- Contains user & device identity

Satyajit Thadeshwar
Passport
- Internal to Netflix ecosystem

Satyajit Thadeshwar
Passport
- Integrity protected by HMAC

Satyajit Thadeshwar
Passport
- Integrity protected by HMAC
- Protobuf format

Satyajit Thadeshwar
Passport
message Passport {
Header header = 1;
UserInfo user_info = 2;
DeviceInfo device_info = 3;
Integrity user_integrity = 4;
Integrity device_integrity = 5;
}

Satyajit Thadeshwar
Passport
message Passport {
Header header = 1;
}
message Header {
string originator = 1;
}

Satyajit Thadeshwar
Passport
message Passport {
Header header = 1;
}
message UserInfo {
Source source = 1;
AuthenticationLevel auth_level = 2;
Int64Wrapper customer_id = 3;
Int64Wrapper account_owner_id = 4;
repeated UserAction actions = ;
}

Satyajit Thadeshwar
Passport
message Passport {
Header header = 1;
}
message DeviceInfo {
Source source = 1;
StringValue esn = 3;
Int32Value device_type = 4;
repeated DeviceAction actions = 5;
}

Satyajit Thadeshwar
Passport
message UserInfo {
Source source = 1;
}
Source source = 1;
}

Satyajit Thadeshwar
Passport
message UserInfo {
Source source = 1;
}
Source source = 1;
} enum Source {
COOKIE = 1;
MSL = 2;
PARTNER_TOKEN = 3;
CTICKET = 4;
}

Satyajit Thadeshwar
Passport
message UserInfo {
Source source = 1;
}
Source source = 1;
} enum AuthenticationLevel {
LOW = 1; // untrusted transport
HIGH = 2; // secure tokens over TLS
HIGHEST = 3; // MSL or user credentials
}

Satyajit Thadeshwar
Passport
message Passport {
Header header = 1;
}
message Integrity {
string key_name = 1;
bytes hmac = 2;
}

Satyajit Thadeshwar
Passport Introspector
- Wrapper over passport
binary data

Satyajit Thadeshwar
binary data
public interface PassportIntrospector {
Long getCustomerId();
Long getAccountOwnerId();
String getEsn();
String getPassportAsString();
...
}

Satyajit Thadeshwar
binary data
public interface PassportIntrospector {
Long getCustomerId();
Long getAccountOwnerId();
String getEsn();
String getPassportAsString();
...
}
- Consumers create
passportIntrospector
from binary passport data
factory.createIntrospector(passport);

Satyajit Thadeshwar
Tooling
Self-service tool for
teams to decrypt
passport

Satyajit Thadeshwar
Passport Actions
message UserInfo {
repeated UserAction actions = 6;
...
}
...
}

Satyajit Thadeshwar
Passport Actions
message UserInfo {
...
}
...
}
- Explicit signal sent by the
downstream services, when an update
to user or device identity has been
performed

Satyajit Thadeshwar
Passport Actions
message UserInfo {
...
}
...
}
- Explicit signal sent by the
downstream services, when an update
to user or device identity has been
performed
- This "signal" is used by EAS to either
create or update the corresponding
type of token

Satyajit Thadeshwar
Passport Action

Satyajit Thadeshwar
Passport Action: User Login

Satyajit Thadeshwar
Zuul
EDGE
Password: ********
ESN: LGTV20165-193456G568

Satyajit Thadeshwar
Zuul API
EDGE ORIGIN
Password: ********
(Device
Bound)

Satyajit Thadeshwar
Zuul API
Netflix
Microservices
auth
service
Password: ********
success
(Device
Bound)

Satyajit Thadeshwar
Zuul API
Netflix
Microservices
auth
service
Password: ********
success
(Device
Bound)
user loginuser login

Satyajit Thadeshwar
Zuul API
Netflix
Microservices
auth
service
Password: ********
successSet-Cookie
Cookie
Service
(Device
Bound)
user loginuser login

Satyajit Thadeshwar
Passport Action: Profile Switch

Satyajit Thadeshwar
- Each profile has its own
identity

Satyajit Thadeshwar
- Each profile has its own
identity
- Switched profile tokens
sent back to the device

Satyajit Thadeshwar
Passport Actions
Separation Of
Concerns
Increased
Visibility

- Moved authentication to the edge
- Streamlined the identity resolution and mutation path
- Making consumption of user & device identity
- Efficient, secure & simple
What we did
Satyajit Thadeshwar

WinsUser & Device Identity for Microservices @ Netflix Scale
Satyajit Thadeshwar

Satyajit Thadeshwar
Token Agnostic
Identity
Downstream systems don't have
to worry about authentication
concerns

Satyajit Thadeshwar
Simplified
Authorization
Downstream services use
authentication level for
authorization decisions

Satyajit Thadeshwar
Simplified
Authorization
Before:
long customerId = 2123125603L;
String ESN = "NFXBOX-235F…";

Satyajit Thadeshwar
Extensible
Identity Model
New attributes about user or
device can be added

Satyajit Thadeshwar
Local cache for
up to date
subscriber data
message UserInfo {
BytesValue subscriber_account
...
}
Placeholder for local cache of
subscriber data

Satyajit Thadeshwar
Offloaded &
Fine Tuned
Offloaded token processing which
resulted into significant gains for
- CPU
- Request Latency
- GC
- Cluster Footprint
We were able to fine-tune EAS
systems based on the token
processing profile

Satyajit Thadeshwar
Offloaded &
Fine Tuned
Offloaded token processing which
resulted into significant gains for
- CPU
- Request Latency
- GC
- Cluster Footprint
We were able to fine tune EAS
systems based on the token
processing profile

Satyajit Thadeshwar
Offloaded &
Fine Tuned
- 30% reduction in CPU
cost per request
- 40% reduction in load
average
CPU to RPS ratio for API instance

Satyajit Thadeshwar
Offloaded &
Fine Tuned
- 30% reduction in
average latency
- 99th
percentile latency
dropping by 20%
Response time for API instance

Satyajit Thadeshwar
Offloaded &
Fine Tuned
- Significant reduction in
GC pressure and GC
pause times
Stop the world GC for API cluster

Satyajit Thadeshwar
Increased
Visibility
Increased visibility into identities
flowing in and out of Netflix
ecosystem
...and into the identity mutations
happening in a request

Satyajit Thadeshwar
Developer
Velocity
Greatly increased developer
velocity for authentication related
changes

Satyajit Thadeshwar
Team focused on
security
Separation of concerns among the
teams

Satyajit Thadeshwar
Key Takeaways
- Token agnostic identity model
- Simplified authorization
- Extensible identity model
- Offloaded all the token processing from many systems
- Fine tuned individual microservices to suit the token processing profile
- Increased visibility into identities flowing and corresponding mutations
- Increased developer velocity for authentication & identity related changes
- Team focused on security

Thank
You.
Satyajit Thadeshwar
sthadeshwar@netflix.com
https://www.linkedin.com/in/satyajit-thadeshwar

Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-user-identity/

User & Device Identity for Microservices @ Netflix Scale

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to User & Device Identity for Microservices @ Netflix Scale

Similar to User & Device Identity for Microservices @ Netflix Scale (20)

More from C4Media

More from C4Media (20)

Recently uploaded

Recently uploaded (20)

User & Device Identity for Microservices @ Netflix Scale