Companies doing any kind of advertising typically have an attribution process that joins users’ conversions with the impressions that they were served or that they clicked on. The standard workflow is typically a batch job that runs every few hours or once a day. However, as technology gets more sophisticated, advertisers are looking for more real-time reporting and results. This talk presents an example of a foundational architecture for near real-time attribution and advanced analytics against real-time impression and conversion data using Structured Streaming and Databricks Delta.

1. Caryl Yuhas, Databricks
Real-Time Attribution with
Structured Streaming and
Databricks Delta
#ExpSAIS13

2. Introduction
2#ExpSAIS13
• Goal:
Provide tools and information
that can help you build more
real-time / lower latency
attribution pipelines
• Crawl, Walk, Run: Pull Model
Carylpreviously MediaMath / SE / PM
for Attribution, SA for Databricks

3. Getting Started
3#ExpSAIS13
• What is Attribution?
Image Source: www.mediamath.com

4. Introduction
What is Databricks Delta?
Delta is a data management capability that
brings data reliability and performance
optimizations to the cloud data lake.
4#ExpSAIS13

5. Stream-to-Sink BEFORE
5#ExpSAIS13
Reprocessing
Data Lake
λ-arch
λ-arch
Streaming
Analytics
Reporting
Events
Validation
λ-arch
Validation
Reprocessing
Compaction
Partitioned
Compact
Small Files
Scheduled to
Avoid Compaction
1
2
3
1
1
3
4
4
4
2

6. Stream-to-Sink AFTER
6#ExpSAIS13
Reprocessing
λ-arch
Streaming
Analytics
Reporting
Events
Validation
λ-arch
Validation
Reprocessing
Compaction
Partitioned
1
2
Compact
Small Files
3
1
4
4
2
DELTA
Optimize
3
ZOrder
3

7. Attribution in Practice
7
impressions conversionsJOIN
#ExpSAIS13
attributed impressions

8. Attribution Challenges
Scale
• Often dealing with millions to billions of data
points per attribution window
Complexity
• Simple, last-click model is still common
• MTA and more sophisticated attribution on rise
8#ExpSAIS13

9. High Level Attribution Pipeline
9#ExpSAIS13

10. Attribution in Practice
10
impressions conversionsJOIN
#ExpSAIS13
attributed impressions

11. Data Architecture
11#ExpSAIS13
impression stream
conversion stream conversions table
impressions table
attributed table
last touch
attributed table
weighted
attribution views
(filters, logic, etc.)

12. System Architecture
12#ExpSAIS13
STRUCTURED
STREAMING
Amazon
Kinesis

13. Unification of Streaming + Batch
DEMO
13#ExpSAIS13

14. • How can we optimize performance?
• Levers:
– Delta Tools
• Optimize
• ZOrder
• Caching
• Data Skipping
– Join on Stream
– Cluster Size
Managing Performance
14#ExpSAIS13

15. Handling Complexity
• Flexibility with Complex Logic
– Forking streams
– Logic on query vs. in-stream
• Late or Corrected Data
– Upserts
– Views automatically update when raw data changed
15#ExpSAIS13

16. Conclusion
• Unification of Batch & Streaming
• Easy APIs for Managing Performance
• Flexible and Scalable Analytics on Near
Real-Time Data
16#ExpSAIS13