RedisDay London 2018

1. How to count at scale

2. What is Pusher Channels?

3. It’s like Redis Pub/Sub ...

4. It’s like Redis Pub/Sub ...
… with millionsof concurrentclients
… on the Internet
… using WebSocketconnections
… hosted at
pusher.com
… plus extra features (presence,
encryption, debug console …)

5. <script src="//js.pusher.com/4.3/pusher.min.js"></script>
<script>
var pusher = new Pusher('abc123');
var channel = pusher.subscribe('donuts');
channel.bind('new-price',
price => alert('New price: ' + price));
</script>

6. <?php
$pusher = new Pusher('abc123', 'd34db33', '123456');
$pusher->trigger('donuts', 'new-price', '0.99');

7. $ pusher channels channel-info --channel donuts
--app-id 180207
{
"Name": "donuts",
"occupied": true,
"subscription_count": 2
}

8. How to count subscriptions

9. We start with clients.
client client

10. These clients connect to a Channels service.
The clients get “socket ids”.
client 2345
Channels
client 234

11. Clients create subscriptions to channels.
client 2345
user-jim
Channels
donuts
client 234
donuts btc-usd

12. The Channels box must keep a map of
channel name to clients.
client 2345
user-jim
Channels
donuts
client 234
donuts btc-usd
user-jim
donuts
btc-usd

13. client 2345
user-jim
Channels
donuts
client 234
donuts btc-usd
user-jim
donuts
btc-usd
Let’s count subscriptions.
How many subscriptions are there for the
channel “donuts”?
Easy: find the “donuts” key of the map, then
count the number of connections.
(The answer is: 2.)

14. Scaling out

15. client 2345
user-jim
Channels
donuts
client 234
donuts btc-usd
user-jim
donuts
btc-usd
This is a monolith!
It can’t be scaled in this form.

16. client 123.2345
user-jim
socket 123 socket 124 socket 125
donuts
client 123.234
donuts
client 124.23
donuts btc-usd
client 125.1
donutsbtc-usd
user-jim
donuts
btc-usd
donuts
btc-usd
donuts
To scale out: add more processes!
We call these “socket processes”.

17. client 123.2345
user-jim
socket 123 socket 124 socket 125
donuts
client 123.234
donuts
client 124.23
donuts btc-usd
client 125.1
donutsbtc-usd
user-jim
donuts
btc-usd
donuts
btc-usd
donuts
redis-pubsub
After adding more processes, where
should you publish to?
These socket processes need to
communicate somehow.
For this, we use Redis.

18. client 123.2345
user-jim
socket 123 socket 124 socket 125
donuts
client 123.234
donuts
client 124.23
donuts btc-usd
client 125.1
donutsbtc-usd
user-jim
donuts
btc-usd
donuts
btc-usd
donuts
redis-pubsub
user-jim
donuts
btc-usd
Specifically, we use the Redis Pub/Sub
feature. Hence, we call this Redis
“redis-pubsub”.
Notice redis-pubsub looks a lot like a
socket process! Redis Pub/Sub provides a
very similar mapping from “channel
names” to connections.

19. Counting subscriptions across boxes

20. client 123.2345
user-jim
socket 123 socket 124 socket 125
donuts
client 123.234
donuts
client 124.23
donuts btc-usd
client 125.1
donutsbtc-usd
user-jim
donuts
btc-usd
donuts
btc-usd
donuts
redis-pubsub
user-jim
donuts
btc-usd
Let’s count subscriptions again.
How many subscriptions are there for the
channel “donuts”? We can ask Redis:
127.0.0.1:6381[2]> pubsub numsub donuts
1) "donuts"
2) "3"
Wrong! Socket 123 has two end-user
subscriptions for “donuts”, so the answer
should be “4”.

21. client 123.2345
user-jim
socket 123 socket 124 socket 125
donuts
client 123.234
donuts
client 124.23
donuts btc-usd
client 125.1
donutsbtc-usd
user-jim
donuts
btc-usd
donuts
btc-usd
donuts
redis-pubsub
user-jim
donuts
btc-usd
Instead of counting connections to socket
processes, we must keep separate counters
for the number of connections to clients.
Unfortunately, Redis Pub/Sub doesn’t
provide this hypothetical feature, so ...
1 2 4

22. redis-main
client 123.2345
user-jim
socket 123 socket 124 socket 125
donuts
client 123.234
donuts
client 124.23
donuts btc-usd
client 125.1
donutsbtc-usd
user-jim
donuts
btc-usd
donuts
btc-usd
donuts
redis-pubsub
user-jim
donuts
btc-usd
… for subscription
counts, we have
another Redis we call
“redis-main”.
All socket processes
connect to this redis,
too.

23. redis-main
client 123.2345
user-jim
socket 123 socket 124 socket 125
donuts
client 123.234
donuts
client 124.23
donuts btc-usd
client 125.1
donutsbtc-usd
user-jim
donuts
btc-usd
donuts
btc-usd
donuts
redis-pubsub
user-jim
donuts
btc-usd
user-jim
donuts
btc-usd
1
4
2
channels:count:global
redis-main keeps the
per-channel subscription
counts in a Redis hash
“channels:count:global”.
When a subscription to
“donuts” is added or
removed on a socket
box, we do:
HINCRBY channels:count:global donuts 1
HINCRBY channels:count:global donuts -1

24. What happens when
a socket process goes away?

25. redis-main
client 123.2345
user-jim
socket 123 socket 124 socket 125
donuts
client 123.234
donuts
client 124.23
donuts btc-usd
client 125.1
donutsbtc-usd
user-jim
donuts
btc-usd
donuts
btc-usd
donuts
redis-pubsub
user-jim
donuts
btc-usd
user-jim
donuts
btc-usd
1
4
2
channels:count:global
Consider this situation.

26. redis-main
client 123.2345
user-jim
socket 123 socket 124 socket 125
donuts
client 123.234
donuts
client 124.23
donuts btc-usd
client 125.1
donutsbtc-usd
user-jim
donuts
btc-usd
donuts
btc-usd
donuts
redis-pubsub
user-jim
donuts
btc-usd
user-jim
donuts
btc-usd
1
4
2
channels:count:global
Suddenly, socket 123
goes away!
Along with all of its
clients ...

27. redis-main
socket 124 socket 125
client 124.23
donuts btc-usd
client 125.1
donuts
donuts
btc-usd
donuts
redis-pubsub
donuts
btc-usd
user-jim
donuts
btc-usd
1
4
2
channels:count:global
Redis-pubsub was able to
clean up, but what about
redis-main? Let’s count
subscriptions again.
How many subscriptions
are there for the channel
“donuts”?
Not four, but two! The
per-channel subscription
counters are wrong!
We need to subtract the
two client subscriptions to
“donuts” from the counter,
but that number is now
lost ...

28. redis-main
client 123.2345
user-jim
socket 123 socket 124 socket 125
donuts
client 123.234
donuts
client 124.23
donuts btc-usd
client 125.1
donutsbtc-usd
user-jim
donuts
btc-usd
donuts
btc-usd
donuts
redis-pubsub
user-jim
donuts
btc-usd
user-jim
donuts
btc-usd
1
4
2
channels:count:global
user-jim
donuts
btc-usd
1
2
1
channels:count:123
donuts
btc-usd
2
1
channels:count:124
donuts 1
channels:count:125
We must keep the
per-channel connection
counts for each socket
process, so that when
the socket process
dies, its counts can be
subtracted.
For each socket
process, we keep
another Redis hash of
its subscription counts.
For example, socket
process 123 has the
hash
“channels:count:123”.

29. MULTI
HINCRBY channels:count:global donuts 1
HINCRBY channels:count:123 donuts 1
EXEC

30. redis-main
client 123.2345
user-jim
socket 123 socket 124 socket 125
donuts
client 123.234
donuts
client 124.23
donuts btc-usd
client 125.1
donutsbtc-usd
user-jim
donuts
btc-usd
donuts
btc-usd
donuts
redis-pubsub
user-jim
donuts
btc-usd
user-jim
donuts
btc-usd
1
4
2
channels:count:global
user-jim
donuts
btc-usd
1
2
1
channels:count:123
donuts
btc-usd
2
1
channels:count:124
donuts 1
channels:count:125
Is this enough for
cleaning up? See what
happens when socket
123 goes away ...

31. redis-main
socket 124 socket 125
client 124.23
donuts btc-usd
client 125.1
donuts
donuts
btc-usd
donuts
redis-pubsub
donuts
btc-usd
user-jim
donuts
btc-usd
1
4
2
channels:count:global
user-jim
donuts
btc-usd
1
2
1
channels:count:123
donuts
btc-usd
2
1
channels:count:124
donuts 1
channels:count:125
Redis-pubsub is able to
clean up, as before.
But redis-main still
doesn’t do anything!
Why?
Unfortunately,
redis-main doesn’t
know the relationship
between the lost
connection and the id
“123”.
Instead, we must
detect dead socket
processes ourselves,
and clean up after
them.

32. Detecting dead socket processes

33. redis-main
client 123.2345
user-jim
socket 123 socket 124 socket 125
donuts
client 123.234
donuts
client 124.23
donuts btc-usd
client 125.1
donutsbtc-usd
user-jim
donuts
btc-usd
donuts
btc-usd
donuts
redis-pubsub
user-jim
donuts
btc-usd
125
123
124
17:35
17:35
17:35
socket:process:all
user-jim
donuts
btc-usd
1
4
2
channels:count:global
user-jim
donuts
btc-usd
1
2
1
channels:count:123
donuts
btc-usd
2
1
channels:count:124
donuts 1
channels:count:125
To identify
dead socket
processes, we
maintain a
“last seen”
time for each
socket
process id in a
Redis hash.
Each socket
process
continually
updates this
hash.

34. redis-main
socket 124 socket 125
client 124.23
donuts btc-usd
client 125.1
donuts
donuts
btc-usd
donuts
redis-pubsub
donuts
btc-usd
125
123
124
17:46
17:35
17:46
socket:process:all
user-jim
donuts
btc-usd
1
4
2
channels:count:global
user-jim
donuts
btc-usd
1
2
1
channels:count:123
donuts
btc-usd
2
1
channels:count:124
donuts 1
channels:count:125
When socket
123 goes
away, its “last
seen” time
gets older.
Then, after ten
minutes, we
consider it
dead.
client 123.2345
user-jim
socket 123
donuts
client 123.234
donuts btc-usd
user-jim
donuts
btc-usd
user-jim

35. redis-main
socket 124 socket 125
client 124.23
donuts btc-usd
client 125.1
donuts
donuts
btc-usd
donuts
redis-pubsub
donuts
btc-usd
125
123
124
17:46
17:35
17:46
socket:process:all
user-jim
donuts
btc-usd
1
4
2
channels:count:global
user-jim
donuts
btc-usd
1
2
1
channels:count:123
donuts
btc-usd
2
1
channels:count:124
donuts 1
channels:count:125
But still
nothing
happens!
Redis-main
doesn’t know
about our
socket
process list.
Instead, we
must check it
ourselves ...

36. redis-main
socket 124 socket 125
client 124.23
donuts btc-usd
client 125.1
donuts
donuts
btc-usd
donuts
redis-pubsub
donuts
btc-usd
cleanup-dead
125
123
124
17:46
17:35
17:46
socket:process:all
user-jim
donuts
btc-usd
1
4
2
channels:count:global
user-jim
donuts
btc-usd
1
2
1
channels:count:123
donuts
btc-usd
2
1
channels:count:124
donuts 1
channels:count:125
We do this with
a separate
process,
“cleanup-dead”.

37. redis-main
socket 124 socket 125
client 124.23
donuts btc-usd
client 125.1
donuts
donuts
btc-usd
donuts
redis-pubsub
donuts
btc-usd
cleanup-dead
125
124
17:46
17:46
socket:process:all
donuts
btc-usd
2
1
channels:count:global
donuts
btc-usd
2
1
channels:count:124
donuts 1
channels:count:125
Cleanup-dead
runs a job every
4 minutes. If a
socket has been
inactive for more
than 10 minutes,
it considers it
dead.
It decrements the
socket’s counts
from the global
counts. Then it
deletes the
socket’s counts.
Then it deletes
the socket from
the “last seen”
times.

39. TESTAMENT SET EVALSHA 123abc 1 socket:123

40. redis
Client C
Local set A
Partial
global index
Application code
Client B
Local set
Partial
global index
Application code
Set C
Set B
Client A
Application
State A
Global
indexes:
A + B + C
Application code
Important
State A
Global
indexes:
A + B + C

41. We’re hiring!
pusher.com/careers
jim@pusher.com