Slides from our talk at the Streams processing meetup (https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/) about Concourse. Concourse is a scalable near real time content generation and fanout platform which is used to power multiple notifications at LinkedIn. Concourse supports fanout based on a chosen graph and/or other rule based targeting logic and each notification candidate is scored and filtered to generate the most relevant notifications.

1. Concourse : Near real time
notifications platform at Linkedin
​Jeff Weiner
​Chief Executive Officer
​Ajith Muralidharan
​Staff Engineer, Data/Relevance
​LinkedIn
​Vivek Nelamangala
​Staff Engineer, Data/ Relevance-infra
​ LinkedIn
1

2. Outline
Background
Nearline non-transactional notifications
Concourse Nuts and Bolts
Performance optimization
Results
2

3. Outline
Background
Nearline non-transactional notifications
Concourse Nuts and Bolts
Performance optimization
Results
3

4. Notifications at Linkedin
• Communicate timely and relevant information to our members
• Bring users to LinkedIn to engage with content/events that they
can’t miss
• Our mission : Deliver right content to the right people at
right time via the right channel (email, push and in-app) for
all members.
Jeff Weiner, is in the news: “The top
CEOs of 2017”
Ajith Muralidharan has sent a
connection invite
4

5. App based notifications at Linkedin
• UI push
• Badge update
In both cases, the notifications can be found in the relevant tab on the
app
5

6. M2M
(Invites, Messages, etc.)
Air Traffic
Controller
(ATC)
Concourse
Near real-time
generation
Beehive
Offline generation
Misc.
In-app
Tab badge
App badge
UI push
Email
Notifications
Service
Flock
LinkedIn’s Notification Ecosystem

7. Outline
Background
Nearline non-transactional notifications
Concourse Nuts and Bolts
Performance optimization
Results
7

8. Nearline non-transactional notifications
Proactively reach right audience with highly relevant
content in a timely manner, so that LinkedIn members
stay well-informed and easily engaged with professional
communities to boost their professional goals.
8
Examples : Activity based notifications, job change, daily news

9. Non-transactional notifications
• keep members informed about their
network
• surface LinkedIn’s value to the
members
• make content contributors feel heard
• move recipients through the
engagement funnel
Benefits Challenges
• irrelevant notifications will annoy
members
• disablement / unsubscription /
complaint
• bad reputation for LinkedIn
9

10. Nearline non-transactional notifications
Vivek creates a
LinkedIn post
Vivek’s connections receive a
notification
Example Use-Case
Activity based Notification
10

11. Benefits of nearline processing
Legacy offline workflow
○ 4 hour time delay to notify a user.
○ Limited targeting capability - can only scale to a
subset of the targeting.
Nearline processing is required to generate a snappy
member experience.
11

12. Outline
Background
Nearline non-transactional notifications
Concourse Nuts and Bolts
Performance optimization
Results
12

13. Nearline non-transactional notifications
1. Vivek creates a
LinkedIn post
2. Concourse fans out content to eligible
recipients: connections, followers
4. Concourse prunes low-quality
candidates based on score
3. Concourse scores each candidate
5. Remaining candidates trigger
notifications to ATC, where they are scored
again (second pass scoring) and optimized
(aggregation, capping, etc.)
Concourse
1ATC
13
1
1

14. 14
Samza Workflow
Kafka Stream
Rest API

15. Workflow manager (Repartitioner)
15
Samza Workflow
Kafka Stream

16. Fanout
16
Fanout
***
Recipient
Stream 1
To Scorer
Recipient
Stream 2
Recipient
Stream N
Update data
Fanout data
Targeting attributes
Actor partitioned
*
Recipient
partitioned**
Connections
Graph
Follows
Graph
Other Graphs

17. Scorer
17

18. Features processor
Features-
Processor
Aggregated
Feature
Stream (**)
Feature
Stream N
Feature
Stream 2
Feature
Stream 1
Data
Transfor-
mation
HDFS
Dataset 1
Data
Transfor-
mation
HDFS
Dataset 2
Data
Transfor-
mation
Input Kafka
Stream
Scorer
18
Samza Workflow
Kafka Stream
Recipient Partitioned
Hadoop Workflow
**

19. Relevance support
● Access item/actor features from existing rest endpoints
● Member/edge features ingested into rocksDB
● Data collection for model training
● Scoring and filtering
○ Support for single & multi utility models.
19

20. Concourse as a platform
Onboarding process
● Integrate new content type in the workflow manager
● Fetch fanout/targeting from existing/new external REST
endpoints
● Onboard new item features
● Reuse existing member features and/or onboard new ones
● Specify and onboard relevance model, collect training data
and iterate on models.
20

21. Outline
Background
Nearline non-transactional notifications
Concourse Nuts and Bolts
Performance optimization
Results
21

22. Scale
● 500K notifications scored per second
● ~2000 notifications sent per second
● 34 Billion feature records (2TB) ingested into Samza
local stores
● 1024 samza tasks
● 60 seconds P90 delay
22

23. Scale Optimization
Feature optimization
● Strongly typed schema 4x size reduction
● Delta feature push with a smoothed full feature push
● Rate Limiting to reduce kafka load
23

24. Scale Optimization
Samza native optimizations
● Host-affinity in samza for faster deploys
● Dedicated samza cluster
● Samza Local State to make high QPS scoring possible
● Distributed traffic across data centers
● Side inputs feature to minimize state footprint
24

25. Outline
Background
Nearline non-transactional notifications
Concourse Nuts and Bolts
Performance optimization
Results
25

26. Offline to nearine migrations
Activity-based notifications
● Increased engagement/sessions and a p90 delay of a few
seconds (from 4 hours)
● unlocked much larger additional gains from expanded fanout and
enhanced targeting
Job change
● Increase in engagement and response rates and a p90 delay of
a few seconds (from 10 hours)
26

27. Concourse as a platform
Hosts growing set of use cases
○ Activity based notifications
○ Job change notifications
○ Notifications about conversations on your posts
○ Group activity notifications
○ ...
27

28. Thank you
28
Blog post : https://engineering.linkedin.com/blog/2018/05/concourse--generating-personalized-content-notifications-in-near