Noah Davis & Luke Melia of Weplay share a series of examples of Redis in the real world. In doing so, they cover a survey of Redis' features, approach, history and philosophy. Most examples are drawn from the Weplay team's experience using Redis to power features on Weplay.com, a social site for youth sports.

1. Redis in Practice
NoSQL NYC • December 2nd, 2010
Noah Davis & Luke Melia

2. About Noah & Luke
Weplay.com
Rubyists
Running Redis in production since mid-2009
@noahd1 and @lukemelia on Twitter

3. Pronouncing Redis

4. Tonight’s gameplan
Fly-by intro to Redis
Redis in Practice
View counts Q&A
throughout, please.
Global locks We love being
Presence interrupted.
Social activity feeds
Friend suggestions
Caching

5. Salvatore
Sanfilippo
Original author of Redis.
@antirez on Twitter.
Lives in Italy.
Recently hired by VMWare.
Great project leader.

6. Describing Redis
Remote dictionary server
“advanced, fast, persistent key- value database”
“Data structures server”
“memcached on steroids”

7. In Salvatore’s words
“I see Redis definitely more as a flexible tool
than as a solution specialized to solve a specific
problem: his mixed soul of cache, store, and
messaging server shows this very well.”
- Salvatore Sanfilippo

8. Qualities
Written in C
Few dependencies
Fast
Single-threaded
Lots of clients available
Initial release was March 2009

9. Features
Key-value store where a value is one of:
scalar/string, list, set, sorted sets, hash
Persistence: in-memory, snapshots, append-only log, VM
Replication: master-slave, configureable
Pub-Sub
Expiry
“Transaction”-y
In development: Redis Cluster

10. What Redis ain’t (...yet)
Big data
Seamless scaling
Ad-hoc query tool
The only data store you’ll ever need

11. Who’s using Redis?
VMWare
Grooveshark
Github
Superfeedr
Craigslist
Ravelry
EngineYard
mediaFAIL
The Guardian
Weplay
Forrst
More...
PostRank

12. Our infrastructure
App server App server Utility server
MySQL master Redis master
MySQL slave Redis slave

13. Redis in Practice #1
View Counts

14. View counts
potential for massive amounts of simple writes/reads
valuable data, but not mission critical
lots of row contention in SQL

15. Redis incrementing
$ redis-cli
redis> incr "medium:301:views"
(integer) 1
redis> incr "medium:301:views"
(integer) 2
redis> incrby "medium:301:views" 5
(integer) 7
redis> decr "medium:301:views"
(integer) 6

16. Redis in Practice #2
Distributed Locks

17. Subscribe to a Weplay Calendar
Subscribe to “ics”

18. Subscription challenges
Generally static content for long periods of time, but with
short sessions of frequent updates
Expensive to compute on the fly

19. Without Redis/Locking
CANCEL EVENT BACKGROUND Pubisher
QUEUE
Queue: Publish ICS
Publish
Contention/Redundancy
ADD EVENT BACKGROUND Pubisher
QUEUE
Queue: Publish ICS
Publish

20. Redis Locking: SetNX
redis = Redis.new
redis.setnx "locking.key", Time.now + 2.hours
=> true
redis.setnx "locking.key", Time.now + 2.hours
=> false
redis.del "locking.key"
=> true¨
redis.setnx "locking.key", Time.now + 2.hours
=> true

21. Redis: Obtained Lock
ADD EVENT REDIS
SETNX "group_34_publish_ics"
Returns: 1
BACKGROUND PUBLISHER REDIS
QUEUE
5 minutes later: PUBLISH....
Queue Job: Publish.publish_ics("34")
DEL "group_34_publish_ics"

22. Redis: Locked Out
CANCEL EVENT REDIS
SETNX "group_34_publish_ics"
Returns: 0
ADD EVENT REDIS
SETNX "group_34_publish_ics"
Returns: 0

23. Redis in Practice #3
Presence

24. “Who’s online?”
Weplay members told us
they wanted to be able
to see which of their
friends were online...

25. About sets
0 to N elements Adding a value to a set
does not require you
Unordered
to check if the value
No repeated members exists in the set first

26. Working with Redis sets 1/3
# SADD key, member
# Adds the specified member to the set stored at key
redis = Redis.new
redis.sadd 'my_set', 'foo' # => true
redis.sadd 'my_set', 'bar' # => true
redis.sadd 'my_set', 'bar' # => false
redis.smembers 'my_set' # => ["foo", "bar"]

27. Working with Redis sets 2/3
# SUNION key1 key2 ... keyN
# Returns the members of a set resulting from the union of all
# the sets stored at the specified keys.
# SUNIONSTORE <i>dstkey key1 key2 ... keyN</i></b>
# Works like SUNION but instead of being returned the resulting
# set is stored as dstkey.
redis = Redis.new
redis.sadd 'set_a', 'foo'
redis.sadd 'set_a', 'bar'
redis.sadd 'set_b', 'bar'
redis.sadd 'set_b', 'baz'
redis.sunion 'set_a', 'set_b' # => ["foo", "baz", "bar"]
redis.sunionstore 'set_ab', 'set_a', 'set_b'
redis.smembers 'set_ab' # => ["foo", "baz", "bar"]

28. Working with Redis sets 3/3
# SINTER key1 key2 ... keyN
# Returns the members that are present in all
# sets stored at the specified keys.
redis = Redis.new
redis.sadd 'set_a', 'foo'
redis.sadd 'set_a', 'bar'
redis.sadd 'set_b', 'bar'
redis.sadd 'set_b', 'baz'
redis.sinter 'set_a', 'set_b' # => ["bar"]

29. Approach 1/2

30. Approach 2/2

31. Implementation 1/2
# Defining the keys
def current_key
key(Time.now.strftime("%M"))
end
def keys_in_last_5_minutes
now = Time.now
times = (0..5).collect {|n| now - n.minutes }
times.collect{ |t| key(t.strftime("%M")) }
end
def key(minute)
"online_users_minute_#{minute}"
end

32. Implementation 2/2
# Tracking an Active User, and calculating who’s online
def track_user_id(id)
key = current_key
redis.sadd(key, id)
end
def online_user_ids
redis.sunion(*keys_in_last_5_minutes)
end
def online_friend_ids(interested_user_id)
redis.sunionstore("online_users", *keys_in_last_5_minutes)
redis.sinter("online_users",
"user:#{interested_user_id}:friend_ids")
end

33. Redis in Practice #4
Social Activity Feeds

34. Social Activity Feeds

35. Via SQL, Approach 1/2
Doesn’t scale to more
complex social graphs
e.g. friends who are SELECT activities.*
FROM activities
JOIN friendships f1 ON f1.from_id =
‘hidden’ from your feed activities.actor_id
JOIN friendships f2 ON f2.to_id =
activities.actor_id
May still require multiple WHERE f1.to_id = ? OR f2.from_id = ?
ORDER BY activities.id DESC
queries to grab LIMIT 15
unindexed data

36. Via SQL, Approach 2/2 user_id
11
activity_id
96
22 96
11 97
22 97
33 97
11 98
22 98
Friend: 11 11 99
Activity: 100
11 100
Actor 1 Friend: 22
INSERTS
22 100
Teammate: 33
33 100

37. Large SQL table
Grew quickly
Difficult to maintain
Difficult to prune

38. Redis Lists 1/2
redis> lpush "teams" "yankees"
(integer) 1
redis> lpush "teams" "redsox"
(integer) 2
redis> llen "teams"
(integer) 2
redis> lrange "teams" 0 1
1. "redsox"
2. "yankees"
redis> lrange "teams" 0 -1
1. "redsox"
2. "yankees"
LTRIM, LLEN, LRANGE

39. Redis Lists 2/2
redis> ltrim "teams" 0 0
OK
redis> lrange "teams" 0 -1
1. "redsox"
LTRIM

40. via Redis, 1/2
LPUSH “user:11:feed”, “100”
LTRIM “user:11:feed”, 0, 50
LPUSH 'user:11:feed', '100' Key Value
Friend: 11
LTRIM 'user:11:feed', 0, 50
Activity: 100
user:11:feed [100,99,97,96]
LPUSH 'user:22:feed', '100'
Actor 1 Friend: 22
LTRIM 'user:22:feed', 0, 50
user:22:feed [100,99,98]
LPUSH 'user:33:feed', '100'
Teammate: 33
LTRIM 'user:33:feed', 0, 100 user:33:feed [100,99]

41. via Redis, 2/2
Key Value
Friend: 11
user:11:feed [100,99,97,96]
Friend: 22
user:22:feed [100,99,98]
Activity: 100
Actor 1
user:33:feed [100,99]
Teammate: 33
Key Value
New York
new_york:feed [100,67]
Boston boston:feed [100,99]

42. Rendering the Feed

43. Redis in Practice #5
Friend Suggestions

44. Friend Suggestions
suggest new connections based on
existing connections
expanding the social graph and
mirroring real world connections is key

45. The Concept
Paul
Me George John
Ringo

46. The Yoko Factor
Paul
Me George John Yoko
Ringo

47. The Approach 1/2
Sarah
2
Frank Donny
Me 2
Ted
Erika

48. The Approach 2/2
Sarah
2
Frank Donny
Me
Ted
3
Erika
Sue

49. ZSETS in Redis
“Sorted Sets”
Each member of the set has a score
Ordered by the score at all times

50. Friend Suggestions in Redis
zincrby “user:1:suggestions” 1 “donny”
zincrby “user:1:suggestions” 1 “donny”
Sarah
Frank Donny
Me
zincrby “user:1:suggestions” 1 “erika”
Ted
Erika
zincrby “user:1:suggestions” 1 “erika”
[ Donny 2 , Erika 2 ]

51. Friend Suggestions in Redis
zincrby “user:1:suggestions” 1 “erika”
Sarah
Frank Donny
Me
Ted
zrevrange “user:1:suggestions” 0 1 Erika
Sue
[ Erika 3 , Donny 2 ]

52. Redis in Practice #6
Caching

53. Suitability for caching
Excellent match for managed denormalization
ex. friendships, teams, teammates
Excellent match where you would benefit from persistence
and/or replication
Historically, not a good match for a “generational cache,” in
which you want to optimize memory use by evicting least-
recently used (LRU) keys

54. As an LRU cache
TTL-support since inception, but with unintuitive behavior
Writing to volatile key replaced it and cleared the TTL
Redis 2.2 changes this behavior and adds key features:
ability to write to, and update expiry of volatile keys
maxmemory [bytes], maxmemory-policy [policy]
policies: volatile-lru, volatile-ttl, volatile-random,
allkeys-lru, allkeys-random

55. Easy to adapt
Namespaced Rack::Session, Rack::Cache, I18n and cache
Redis cache stores for Ruby web frameworks
implementation is under 1,000 LOC

56. Contentious benchmarks

57. Any
questions?
Thanks!
Follow us on Twitter: @noahd1 and @lukemelia
Tell your friends in youth sports about Weplay.com