This is a practice report showing how Redis, an in-memory data structure store, can be used in distributed environments to communicate, coordinate and to share state. The talk contains a small introduction to Redis and shows how it can be used in Perl by highlighting key implementation details of a productively used job scheduling system.

1. Swiss Perl Werkshop, 25 August 2017
Coordination of Distributed Software with
by Konrad Bucheli
Redis

2. Topology & Task
Ready for Redis
Illustrated Implementation

3. Topology & Task
Ready for Redis
Illustrated Implementation

5. Alice

6. Alice
new firewall rule

7. Alice
bastion

8. Alice
bastion
Bob

9. Alice
bastion
Bob patch all hosts

10. Alice
bastion
Bob

11. Alice
bastion
Bob

12. Job Scheduling System

13. Job Scheduling System
●
run jobs now or in the future

several job types

job grouping

several worker hosts

command line and web portal UI

14. user
jobgroup
places
job jobjob

15. user
jobgroup
places
job jobjob
jobqueue
schedules

16. user
jobgroup
places
job jobjob
log_event
log_event
log_event
jobqueue
schedules

17. Topology & Task
Ready for Redis
Illustrated Implementation

18. https://redis.io

26. redis> GET nonexisting
(nil)
redis> SET mykey "Hello"
"OK"
redis> GET mykey
"Hello"
redis>

27. redis> GET nonexisting
(nil)
redis> SET mykey "Hello"
"OK"
redis> GET mykey
"Hello"
redis> expire mykey 5

28. redis> GET nonexisting
(nil)
redis> SET mykey "Hello"
"OK"
redis> GET mykey
"Hello"
redis> expire mykey 5
(integer) 1
redis>

29. redis> GET nonexisting
(nil)
redis> SET mykey "Hello"
"OK"
redis> GET mykey
"Hello"
redis> expire mykey 5
(integer) 1
redis> get mykey
"Hello"
redis>

30. redis> GET nonexisting
(nil)
redis> SET mykey "Hello"
"OK"
redis> GET mykey
"Hello"
redis> expire mykey 5
(integer) 1
redis> get mykey
"Hello"
redis> get mykey
(nil)
redis>

32. 01 use Redis;
02
03 my $r = Redis->new(server => '192.168.0.1:6379');
04
05 $r->set('mykey', 'Hello');
06
07 my $value = $r->get('mykey');

33. Topology & Task
Ready for Redis
Illustrated Implementation

34. user
jobgroup
places
job jobjob
log_event
log_event
log_event
jobqueue
schedules

35. Object ID

36. Object ID
redis> INCR jobgroup:last_id
(integer) 1
redis> INCR jobgroup:last_id
(integer) 2
redis>

37. Object ID
object ID counter
jobgroup jobgroup:last_id
job job:last_id
event jobgroup:last_id

38. Object Data

39. Object Data
redis> HMSET jobgroup:2 id 2 user konrad
"OK"
redis> HGET jobgroup:2 user
"konrad"
redis> HGETALL jobgroup:2
1) "id"
2) "2"
3) "user"
4) "konrad"
redis>

40. Object Data
object ID counter object data
(hash)
jobgroup jobgroup:last_id jobgroup:$ID
job job:last_id job:$ID
event jobgroup:last_id event:$ID

41. Object Collections

42. Object Collections
Use

Sets if uniqueness is imporant

Lists if order is important

Sorted Sets if uniqueness and order is important

43. Object Collections
object ID counter object data
(hash)
object collection
(set or list)
jobgroup jobgroup:last_id jobgroup:$ID jobgroups
job job:last_id job:$ID jobgroup:$ID:jobs
event jobgroup:last_id event:$ID job:$ID:events

44. Data Expiration
all keys with $ID expire
object ID counter object data
(hash)
object collection
(set or list)
jobgroup jobgroup:last_id jobgroup:$ID jobgroups
job job:last_id job:$ID jobgroup:$ID:jobs
event jobgroup:last_id event:$ID job:$ID:events

45. Job Queue

46. Job Queue
redis> ZADD jobqueue 120 job:1
(integer) 1
redis> ZADD jobqueue 60 job:2
(integer) 1
redis> ZRANGEBYSCORE jobqueue -INF 100
1) "job:2"
redis>

47. Job Lock

48. Job Lock (simple)
redis> SET job:2:lock whatever NX
"OK"
redis> SET job:2:lock whatever NX
(Nil)
redis> DEL job:2:lock
(integer) 1
redis> SET job:2:lock whatever NX
"OK"
redis>

49. Job Lock (timeout)
redis> SET job:2:lock unique_nounce_ao45u89 EX 3600 NX
"OK"
redis> SET job:2:lock sl8ts90eu EX 3600 NX
(Nil)
redis>
Unlock:
run serverside Lua code to delete the key if the nouce is still correct
(https://redis.io/commands/set)

50. 01 sub pick_pending_job {
02 my ($r) = @_;
03 # get jobs ready to run
04 my @jobqueue = $r->zrangebyscore('jobqueue', '-INF', time());
05 for my $job (@jobqueue) {
06 # lock job
07 if($r->set("$job:lock", "$HOSTNAME:$$", 'EX', $TIMEOUT, 'NX')) {
08 # did anyone change it in between? (race condition)
09 my $current_schedule = $r->zscore('jobqueue', $job);
10 if (defined $current_schedule && $current_schedule <= time()) {
11 # all good, we move it now in the job queue to the timeout
12 $r->zadd('jobqueue', time() + $TIMEOUT, $job);
13 return $job;
14 }
15 # unlock job (simple)
16 $r->del("$job:lock");
17 }
18 }
19 return;
20 }

51. t=0
Bob
Emergency patch for all hosts

52. t=1

53. t=2
Alice
another firewall rule

54. t=3

55. t=4

56. t=5

57. t=6

58. t=7
Alice

59. fair scheduling

60. t=7
lag

61. 01 sub pick_pending_job {
02 my ($r) = @_;
03 # get jobs ready to run
04 my @jobqueue = $r->zrangebyscore('jobqueue', '-INF', time());
05 for my $job (@jobqueue) {
06 # lock job
07 if($r->set("$job:lock", "$HOSTNAME:$$", 'EX', $TIMEOUT, 'NX')) {
08 # did anyone change it in between? (race condition)
09 my $current_schedule = $r->zscore('jobqueue', $job);
10 if (defined $current_schedule && $current_schedule <= time()) {
11 # all good, we move it now in the job queue to the timeout
12 $r->zadd('jobqueue', time() + $TIMEOUT, $job);
13 # fair scheduling on overload
14 if (time() - $current_schedule > $OVERLOAD_LIMIT ){
15 handle_overload($r);
16 }
17 return $job;
18 }
19 # unlock job (simple)
20 $r->del("$job:lock");
21 }
22 }
23 return;
24 }

62. t=8

63. Alice
t=8

64. Redis rocks
for parts of a distributed system which

rarely or never do bulk operations

has data with limited lifetime

can bootstrap itself

65. Topology & Task
Ready for Redis
Illustrated Implementation
Questions?

66. Thanks