Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale
1. Ad Serving at Spotify Scale
A journey of incremental full stack overhaul
Kinshuk Mishra, Director of Engineering
kinshuk@spotify.com
@_kinshukmishra
12. What I do
● Founded ads engineering team at Spotify in 2011
● Build all things ads engineering - team & software
● Major focus areas :
○ Ad delivery (Backend and Web)
○ Multi-platform native ads (Client Platform)
○ Ad performance (ML and Data)
30. In essence it is pretty simple
Client
User Profile database
Ad Server
Campaign Management
Portal
Billing/
Reporting
Ad campaign
database
Data
Collection
System
31. Spotify Ads infrastructure in 2011
Edge
Service
Desktop
Log
Delivery
HDFS
User
Profile
Batch
Basic Ad Server
Campaign
Management
Billing/
Reporting
32. Spotify Ads infrastructure in 2017
iOS
Edge
Service
Android
Ads SDK
Desktop
Web
Chromecast/
Playstation/
FireTV
Ad
Aggregation
Service
Log
Delivery
GCS
User
Profile
Targeting
Service
DMP
Stream Batch
Ad Server
Decision Delivery
Ad
Exchanges
Campaign
Management
Optimization
Modeling
Self-Serve Portal
Creative
Generation
Payments
Billing/
Reporting
42. Future needs
●
● Growth in scale
● Emergence of new client platforms
● Cheap cloud computing
● New products to meet business objectives
● Technical debt
45. Original ad server design
Edge Service
Router
hash(userid)
Ad server ring with
partitions
Ad server instance
Memcache
Memcache
Memcache
Memcache
Campaign DB
User DB
Desktop
Rendering Ad trigger
decisioning
Ads
Ranking
Ads
Caching
Ad batching & fetch communication
55. The ad server transition
Edge
Service
Log
Delivery
HDFS
User
Profile
Batch
Smart Ad
Server
Campaign
Management
Billing/
Reporting
Ad Server
Proxy
(routing) Basic Ad
Server
Gradual transition from basic to smart ad
serving
Desktop
Rendering Ad trigger
decisioning
Ads
Ranking
Ads
Caching
Ad batching & fetch
communication
56. After the ad server transition
Proxy
Service
Log
Delivery
HDFS
User
Profile
Batch
Campaign
Management
Billing/
Reporting
Smart Ad Server
Desktop
Rendering Ad trigger
decisioning
Ads
Ranking
Ads
Caching
Ad batching & fetch
communication
58. Division of responsibilities
Desktop iOS
Android
Ads SDK
Desktop
Web
Rendering
Ad trigger
decisioning
Ads
Ranking
Ads
Caching
Ad batching & fetch communication
Ad
decisioning
Ad fetch
orchestration
Client context
Ad Trigger & Render
Before After
69. Remember division of responsibilities?
Desktop iOS
Android
Ads SDK
Desktop
Web
Rendering
Ad trigger
decisioning
Ads
Ranking
Ads
Caching
Ad batching & fetch communication
Ad
decisioning
Ad fetch
orchestration
Client context
Ad Trigger & Render
BAD GOOD
70. Multiplatform Client design
iOS
Proxy
Service
Android
Ads SDK
Desktop
Web
Chromecast/
Playstation/
FireTV
Ad
Aggregation
Service
Log
Delivery
GCS
User
Profile
Targeting
Service
DMP
Stream Batch
Ad Server
Decision Delivery
Ad
Exchanges
Campaign
Management
Modeling
Self-Serve Service
Creative
Generation
Payments
Billing/
Reporting