Redis is a rock-solid platform for a variety of real-world use cases, in particular as a poor man’s message queue. At Apple Maps, we built a service to show live I/O from thousands of concurrent SSH sessions in real-time using Redis, Lua scripts, node.js and HTML5 Server-Sent Events. Although our architecture isn’t ideal, and we would do things differently today, our system has performed very well in the real-world over the past couple of years. In particular, after some initial failures, it has scaled very well as usage has grown much faster than we had ever anticipated. I’ll talk about the initial design, implementation, and the evolution of specific features to address real-world memory usage and performance challenges

2. START
job "A"
I/O
stdin,
stdout, …
SSH
session "X"
SEND
queue "A:X"
LISTEN
queue "X:A"
AUTHORIZE
HTTP
SSE
Architecture
for SSH I/O
streaming
SSH I/O streaming via Redis-based
persistent message queue
Mani Tadayon
RedisConf 2016

3. About me
• @bwsr_sr
• Working in software since ’01
• Using Redis since ’13
<shameless-plug>
• Just finished a book on Ruby
testing: RSpec Essentials
• http://amzn.com/1784395900
</shameless-plug>

4. How to build a message
queue with Redis
• Live listener
> subscribe mychannel
• Publisher
> publish mychannel mymessage
And that’s my talk, thanks for listening!
DoDo’s mad!

5. – http://antirez.com/news/88
“Redis apparently is at the same time the best
and the worst system to use like that.”

6. Demo

9. START
job "A"
I/O
stdin,
stdout, …
SSH
host "X"
SEND
queue "A:X"
LISTEN
queue “A:X”
AUTHORIZE
HTTP
SSE
Architecture
for SSH I/O
streaming

10. How to build a persistent
message queue with Redis
• Retrieve persisted messages
> lrange mykey 0 -1
• Publisher
> rpush mykey mymessage
What about a live listener?

11. The best of both worlds
• Live listener
> subscribe mychannel
• Retrieve persisted messages
> lrange mykey 0 -1
• Publisher
> rpush mykey mymessage
> publish mychannel mymessage
That’s pretty much it. But the SSH I/O feature needs a few more features.

12. • Lookup by session ID (“A:X”)
• Use a simple list of key names
> rpush byjob mykey
> expire byjob 604800
• Lookup by hostname (“A:*”)
• Use a zset with timestamp as score
> zadd byhost 1463012431 mykey
> zremrangebyscore byhost 0 1462407631

13. • Wrap each message in a transaction
• Protect against excessive memory usage (150GB in 2 hours on the 1st day…)
• Limit number of persisted messages per job
• Expire persisted messages (more aggressively for verbose jobs)
• Stop sending messages above a threshold (handled outside Redis)
> multi
> rpush mykey mymessage
> expire mykey 604800
> ltrim mykey -100 -1
> publish mychannel mymessage
> exec

14. • Set up indexes for lookup (only once per job)
> rpush byjob mykey
> expire byjob 604800
> zadd byhost 1463012431 mykey
> zremrangebyscore byhost 0 1462407631

15. Key and channel names
ssh-io:session:event:persisted:$ID:$FQDN
ssh-io:session:event:live:$ID:$FQDN
ssh-io:session:event_lookup:by_job:$ID
ssh-io:session:event_lookup:by_hostname:$FQDN

16. > multi
> rpush ssh-io:session:event:persisted:b8042948:fakehost.example.com
{"session_started":"About to run 4 commands on fakehost.example.com for
CustomScript-
TestStreamingSimple:b8042948","io_type":"session_started","timestamp":1462407631}
> expire ssh-io:session:event:persisted:b8042948:fakehost.example.com 604800
> ltrim ssh-io:session:event:persisted:b8042948:fakehost.example.com -100 -1
> publish ssh-io:session:event:live:b8042948:fakehost.example.com
{"session_started":"About to run 4 commands on fakehost.example.com for
CustomScript-
TestStreamingSimple:b8042948","io_type":"session_started","timestamp":1462407631}
> exec
> rpush ssh-io:session:event_lookup:by_job:b8042948 ssh-
io:session:event:persisted:b8042948:fakehost.example.com
> expire ssh-io:session:event_lookup:by_job:b8042948 604800
> zadd ssh-io:session:event_lookup:by_hostname:fakehost.example.com 1463012431 ssh-
io:session:event:persisted:b8042948:fakehost.example.com
> zremrangebyscore ssh-io:session:event_lookup:by_hostname:fakehost.example.com 0
1462407631

17. > multi
> rpush ssh-io:session:event:persisted:b8042948:fakehost.example.com
{"username":"deploy","stdin":"echo "ping number
1"","hostname":"fakehost.example.com","io_type":"stdin","timestamp":1462407631}
> expire ssh-io:session:event:persisted:b8042948:fakehost.example.com 604800
> ltrim ssh-io:session:event:persisted:b8042948:fakehost.example.com -100 -1
> publish ssh-io:session:event:live:b8042948:fakehost.example.com
{"username":"deploy","stdin":"echo "ping number
1"","hostname":"fakehost.example.com","io_type":"stdin","timestamp":1462407631}
> exec
# … multi,rpush,expire,ltrim,publish,exec …
> multi
> rpush ssh-io:session:event:persisted:b8042948:fakehost.example.com {"session_finished":"Ran 4
of 4 commands on fakehost.example.com for CustomScript-
TestStreamingSimple:b8042948","success":false,"io_type":"session_finished","timestamp":
1462407631}
> expire ssh-io:session:event:persisted:b8042948:fakehost.example.com 604800
> ltrim ssh-io:session:event:persisted:b8042948:fakehost.example.com -100 -1
> publish ssh-io:session:event:live:b8042948:fakehost.example.com {"session_finished":"Ran 4 of
4 commands on fakehost.example.com for CustomScript-
TestStreamingSimple:b8042948","success":false,"io_type":"session_finished","timestamp":
1462407631}

18. Lookup by hostname
• Lua script that uses the zset index
• Returns all events, in order, per hostname

19. $ redis-cli zrange ssh-io:session:event_lookup:by_hostname:fakehost.example.com 0 -1
1) "ssh-io:session:event:persisted:0bd6005b-1999-42ab-9b54-2585e0383bcb:fakehost.example.com"
2) "ssh-io:session:event:persisted:84ae935f-55a2-4210-a356-87a0da7c3b58:fakehost.example.com"
3) "ssh-io:session:event:persisted:35696242-a59d-4a1f-b36f-4ee988b558c5:fakehost.example.com"
4) "ssh-io:session:event:persisted:b72488ec-816c-48b9-a48d-ab31cbc41802:fakehost.example.com"
5) "ssh-io:session:event:persisted:5b13b4bd-2800-47fe-a821-a548c90054b6:fakehost.example.com"
6) "ssh-io:session:event:persisted:26078a92-e349-43a7-9375-d0ce511b1dbd:fakehost.example.com"
7) "ssh-io:session:event:persisted:4c3f31f0-dd2d-411f-bfbe-d6591698bf3a:fakehost.example.com"
8) "ssh-io:session:event:persisted:60983166-f580-44d6-9058-3b8bb04c1441:fakehost.example.com"
9) "ssh-io:session:event:persisted:572de4a5-3d15-4160-ab42-006407662ca8:fakehost.example.com"
10) "ssh-io:session:event:persisted:3b405df9-3618-4cc2-b245-4dac4b7a203b:fakehost.example.com"
11) "ssh-io:session:event:persisted:5968f887-9228-4963-9515-b92eda944063:fakehost.example.com"
12) "ssh-io:session:event:persisted:ebcd6068-4a22-48f1-a333-36a312bb78f0:fakehost.example.com"
13) "ssh-io:session:event:persisted:d22ae42d-4343-46b8-a477-8fa057bc34b5:fakehost.example.com"
14) "ssh-io:session:event:persisted:49c994db-9d3d-4453-ac83-8dbf038a655d:fakehost.example.com"
15) "ssh-io:session:event:persisted:2fc5ea10-c8f8-42e7-93d4-073b68826ffa:fakehost.example.com"
16) "ssh-io:session:event:persisted:b8042948-63f1-4c6f-afc8-8d2384b3b155:fakehost.example.com"
17) "ssh-io:session:event:persisted:5fee9662-af66-4027-9963-ff59716dc7e0:fakehost.example.com"
18) "ssh-io:session:event:persisted:c919f801-ab8b-4f31-92f8-3f51d6b7b7dc:fakehost.example.com"
19) "ssh-io:session:event:persisted:22f480d9-31cb-4054-9861-51d0275c9468:fakehost.example.com"
20) "ssh-io:session:event:persisted:7710e61f-921c-4d6d-bbf4-e9c35d932f1a:fakehost.example.com"

20. -- finds the keys for all sessions for...
-- ...a given hostname using lookup lists,
-- then retrieve all events in all keys
-- use nested numeric lua tables (i.e. arrays)...
-- ...since redis will wipe out string keys
-- see: http://redis.io/commands/eval
-- (Conversion between Lua and Redis data types)
local keys = redis.call("zrange", KEYS[1], 0, -1)
local returner = {}
for i, key in ipairs(keys) do
returner[i] = {
key,
redis.call("lrange", key, 0, -1)
}
end
return returner

21. $ redis-cli eval "$(cat values-from-lookup-set.lua)"
1
ssh-io:session:event_lookup:by_hostname:fakehost.example.com
1) 1) "ssh-io:session:event:persisted:
0bd6005b-1999-42ab-9b54-2585e0383bcb:fakehost.example.com"
2) 1) "{"session_started":"About to run 4 commands on
fakehost.example.com for :0bd6005b-1999-42ab-9b54-2585e0383bcb","io_type":
"session_started","timestamp":1462405267}"
# … 18 more sessions …
20) 1) "ssh-io:session:event:persisted:7710e61f-921c-4d6d-bbf4-
e9c35d932f1a:fakehost.example.com"
2) 1) "{"session_started":"About to run 6 commands on
fakehost.example.com for CustomScript-TestStreamingSimple:7710e61f-921c-4d6d-
bbf4-e9c35d932f1a","io_type":"session_started","timestamp":1462524784}"

22. Authentication
• From browser directly to node.js web service
• Web app creates an auth token
• Token written to Redis…
• …and stored in browser session
• 1 day expiry
• Simple Lua script to create or retrieve token

23. local token = ''
if redis.call('exists', KEYS[1]) == 1
then
token = redis.call('get', KEYS[1])
else
redis.call('setex', KEYS[1], ARGV[1],
ARGV[2])
token = ARGV[2]
end
return token

24. $ redis-cli eval "$(cat stream_auth_token.lua)"
1
ssh-io:session:stream:auth:myuser
86400
"7292ed99-da44-42c3-897a-745b6e43ac33"
# => "7292ed99-da44-42c3-897a-745b6e43ac33"
$ redis-cli eval "$(cat stream_auth_token.lua)"
1
ssh-io:session:stream:auth:myuser
86400
"some-new-random-token"
# => "7292ed99-da44-42c3-897a-745b6e43ac33"

25. START
job "A"
I/O
stdin,
stdout, …
SSH
host "X"
SEND
queue "A:X"
LISTEN
queue “A:X”
AUTHORIZE
HTTP
SSE
Architecture
for SSH I/O
streaming

26. Thanks for listening
No Shiba Inus were harmed in the making of this presentation