Databus - Abhishek Bhargava & Maheswaran Veluchamy - DevOps Bangalore Meetup March 28th 2015

1. Recruiting SolutionsRecruiting SolutionsRecruiting Solutions
Databus
"
A low latency change capture system
Databus Team @ Linkedin
Abhishek Bhargava
Maheswaran Veluchamy

2. DATABUS
§ Databus - a real-time change data capture system
§ Developed in 2005 by LinkedIn
§ Scalable and highly available
§ Long look-back with no impact to the source
§ Guarantee delivery of messages

3. Talking Points
§ Why Databus
§ Evolution
§ Architecture
§ Components of Databus
§ Demo
§ Questions

4. db
app
app
app
app
app
app
app
JKL
!!!!!!!!!!!!
A World without databus …

5. Why Databus
§ Data flow is essential
§ Data consistency is critical
§ Apps need to be able to scale
§ Caches need to be kept up to date
§ Database should not be overloaded

6. Extract changes from
database commit log
Tough but possible
Consistent!!!
Application code dual
writes to database
and pub-sub system
Easy on the surface
Consistent?
Two Ways

7. Change Extract: Databus
7
Primary
Data Store
Data Change Events
Standard
ization
Standard
ization
Standard
ization
Standard
ization
Standard
ization
Search
Index
Standard
ization
Standard
ization
Graph
Index
Standard
ization
Standard
ization
Read
Replicas
Updates
Databus

8. Databus Eco-system: Participants
Primary
Data Store
Source Databus
Consumer
Application
Change
Data
Capture
Change Event
Stream
events
events
change
data
• Support
transactions
• Extract changed
data of committed
transactions
• Transform to ‘user-
space’ events
• Preserve atomicity
• Receive change
events quickly
• Preserve
consistency with
source

9. db
relay
relay
relay
relay
v
i
p
app
app
app
app
app
app
app
• Only … Relay…
• High possibility of all clients ‘falling off’
• Databases are Overloaded
• Requires a lot of human effort
Databus V1

10. databus v2
§ Relay
§ Bootstrap
– Producer
– Mysql
– Server
§ Client
How does it all work??

11. db
relay
relay
relay
v
i
p
app
app
app
app
bootstrap bootstrap
vip
mysqlmysql
producer producerserver server
app

12. Databus-ifying the Source
§ logic to extract changes from source from specified SCN
§ Implementations
– Oracle
§ Trigger-based
§ Commit ordering
§ Special instrumentation required

13. Flow within the source
TXNSALNAME
AA 100 NULLAA 200 221
Trigger
Oracle
Sequence
Change Tracking Table
-------------------------------
Txn scn mask ts
221 99999
Database job
221 1000 10 xx
Consumers ::
Relay
Databus Clients

14. Change Data Capture in Oracle
Subscribe:
ABC
Subscribe:
XYZ
ABC XYZ
Change Tracking Table

15. The Databus Relay
Change
Capture
Event Buffer
(In Memory)
Relay
Database Schemas
Src
Meta-
data
• Encapsulates change capture logic and
change event stream
• Source aware, schema aware
• Multi-tenant: Multiple Event Buffers
representing change events of different
databases
• Optimizations
• Index on SCN exists to quickly
locate physical offset in EventBuffer
• Locally stores SCN per source for
efficient restarts
• Large Event Buffers possible (> 2G)
SCN
store
API

16. Scaling Databus Relay
db
relay
relay
relay
relay v
i
p
app
app
app
app
app
app
app
app
relay
relay
relay
v
i
p
relay
relay
relay

17. The Components of Databus
17
DB
Change
Capture
Event Buffer
(In Memory)
change data
Consumer
Relay
Databus
Client
Application
online changes
Bootstrap
New
ApplicationConsistent
snapshot
Log Store
Snapshot
Store
online changes
Bootstrap
Consumer
older changes
Slow
Application
Metadata

18. Databus: Current Implementation
§ OS - Linux, written in Java , runs Java - 6,7,8
§ All components have http interfaces
§ Databus Client: Java
– Other language bindings possible
– All communication with change stream via http
§ More info - https://github.com/linkedin/databus/

19. Questions
19