Recently, Apache Phoenix has been integrated with Apache (incubator) Omid transaction processing service, to provide ultra-high system throughput with ultra-low latency overhead. Phoenix has been shown to scale beyond 0.5M transactions per second with sub-5ms latency for short transactions on industry-standard hardware. On the other hand, Omid has been extended to support secondary indexes, multi-snapshot SQL queries, and massive-write transactions.
These innovative features make Phoenix an excellent choice for translytics applications, which allow converged transaction processing and analytics. We share the story of building the next-gen data tier for advertising platforms at Verizon Media that exploits Phoenix and Omid to support multi-feed real-time ingestion and AI pipelines in one place, and discuss the lessons learned.
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
1. Scaling Cloud-Scale Translytics Workloads
with Omid and Phoenix
Ohad Shacham
Yahoo Research
Edward Bortnikov
Yahoo Research
RESEARCH
Yonatan Gottesman
Yahoo Research
4. The Applications Perspective
4
Event-to-action/insight latency becomes king
Stream processing, asynchronous execution
Data consistency becomes nontrivial
Complex processing patterns (online reporting to AI)
Data integration across multiple feeds and schemas
OLTP World
Analytics World
5. Translytics Platforms Vision
5
The best of all worlds: OLTP and Analytics all-in-one
Enable complex, consistent, real-time data processing
Simple API’s with strong guarantees
Built to scale on top of NoSQL data platforms
6. OLTP Coming to NoSQL
6
Traditional NoSQL guarantees row-level atomicity
Translytics applications often bundle reads and writes
Asynchronous design patterns drive concurrency
Without ACID guarantees, chaos rules!
7. ACID transactions
Multiple data accesses in a single logical operation
Atomic
“All or nothing” – no partial effect observable
Consistent
The DB transitions from one valid state to another
Isolated
Appear to execute in isolation
Durable
Committed data cannot disappear
8. Use Case: Audience Targeting for Ads
8
Advertisers optimize campaigns to reach the right user audiences
Ad-tech platforms build and sell audience segments (identity sets)
Segmentation is based on user features (demographics, behavior, …)
Algorithms vary from rule-based heuristics to AI classification
Timeliness directly affects revenue
9. Real-Time Targeting Platform
9
Storm for Compute
Audience segmentation algorithms embedded in bolts
HBase for Storage
User Profiles (U), Segments (S), and U ↔ S relationships
Kafka for Messaging
Scale: trillions of touchpoints/month
10. Challenge: Keeping the Data Consistent
10
Shared data is accessed in parallel by multiple bolts
Access patterns are complex
User profile update: read+compute+write
User↔Segment mapping update: two writes
Segment query (scan): read multiple rows
HBase read/write API does not provide atomic guarantees
11. Omid Comes to Help
11
Transaction Processing layer for Apache HBase
Apache Incubation (started 2015, graduation planned 2019)
Easy-to-use API (good old NoSQL)
Popular consistency model (snapshot isolation)
Battle tested (in prod @Yahoo since 2015, new customers onboarding)
12. Omid Programming
12
TransactionManager tm = HBaseTransactionManager.newInstance();
TTable txTable = new TTable("MY_TX_TABLE”);
Transaction tx = tm.begin(); // Control path
Put row1 = new Put(Bytes.toBytes("EXAMPLE_ROW1"));
row1.add(family, qualifier, Bytes.toBytes("val1"));
txTable.put(tx, row1); // Data path
Put row2 = new Put(Bytes.toBytes("EXAMPLE_ROW2"));
row2.add(family, qualifier, Bytes.toBytes("val2"));
txTable.put(tx, row2); // Data path
tm.commit(tx); // Control path
13. SQL Coming to NoSQL
13
NoSQL API is simple but crude and non-standardized
Hard to manage complex schemas (low-level data abstraction)
Hard to implement analytics queries (low-level access primitives)
Hard to optimize for speed (server-side programming required)
Hard to integrate with relational data sources
14. Use Case: Real-Time Ad Inventory Ingestion
14
Advertisers deploy campaign content & metadata in the marketplace
SQL-speaking external client
Complex schema (many campaign types and optimization goals)
High scalability (growing market)
Campaign operations run multidimensional inventory analytics
Aggregate queries by advertiser, product, time, etc.
ML pipeline learns recommendation models for new campaigns
NoSQL-style access to data
15. Phoenix comes to Help
15
OLTP and Real-Time Analytics for HBase
Query optimizer transforms SQL to native HBase API calls
Standard SQL interface with JDBC API’s
High level data abstractions (e.g., secondary indexes)
High performance (leverages server-side coprocessors)
16. Phoenix/Omid Integration
16
Phoenix is designed for public-cloud scale (>10K query servers)
Omid is extremely scalable (>600k tps), low-latency (<5ms), and HA
New Omid release (1.0.1) - SQL features, improved performance
Supports secondary indexes, extended Snapshot Isolation, downstream
filters
Phoenix releases 4.15 and 5.1 include Omid as Phoenix Tps
Phoenix refactored to support multiple TP backends (Omid is default)
18. Why do we care?
18
SQL transactions
SELECT * FROM my_table; -- This will start a transaction
UPSERT INTO my_table VALUES (1,'A’);
SELECT count(*) FROM my_table WHERE k=1;
DELETE FROM my_other_table WHERE k=2;
!commit -- Other transactions will now see your updates and you will see theirs
19. Why do we care?
1919
Non-transactional secondary index update might breaks consistency
(k1, [v1,v2,v3])
Table Index
(v1, k1)
Write (k1, [v1,v2,v3])
20. Why do we care?
20
Updating the secondary index fails
Out of handlers
Many jiras discuss this issue
20
(k1, [v1,v2,v3])
Table Index
Write (k1, [v1,v2,v3])
21. Transactions and snapshot isolation
Aborts only on write-write conflicts
Read
point
Write
point
begin commitread(x) write(y) write(x) read(y)
30. What about high availability?
Client
Begin/Commit
Data Data Data
Commit
Table
Persist
Commit
Verify commitRead/Write
Single
point of
failure
30
Transaction
Manager
Results/Timestamp
31. High availability
Client
Begin/Commit
Data Data Data
Commit
Table
Verify commitRead/Write
31
Results/Timestamp
Transaction
Manager
(TSO)
Transaction
Manager
(TSO)
Recovery
state
Force abortPersist
Commit
33. New scenarios for Omid
33
Secondary Indexes
Atomic Updates
How can we update metadata?
On-the-Fly Index Creation
What should we do with inflight transaction?
Extended Snapshot Isolation
Read-Your-Own-Writes Queries
Does not match to snapshot isolation
34. Secondary index: creation and maintenance
34
T1
T2
T3
CREATE
INDEX
started
T4
CREATE
INDEX
complete
T5
T6
35. Secondary index: creation and maintenance
35
T1
T2
T3
CREATE
INDEX
started
T4
CREATE
INDEX
complete
T5
T6
Bulk-Insert
into index
Abort
(enforced
upon
commit)
Added by
a
coproces
sor
Added by
a
coproces
sor
Index
update
(stored
procedure)
37. Moving snapshot implementation
37
Checkpoint for
Statement 1
Checkpoint for
Statement 2
Writes by
Statement 1
Timestamps allocated by TM in blocks.
Client promotes the checkpoint.
38. Summary
38
Apache Phoenix is a relational database layer for HBase
Apache Phoenix need a scalable and HA Tps
Omid is Battle-Tested, Highly Scalable, Low-Latency Tps
Phoenix-Omid integration provides an efficient OLTP for Hadoop
Cloud-scale use cases in Yahoo