Describe how to create, manage and traverse links in Riak KV.

1. Link Walking with Riak (KV)
Susan Potter / Finsignia
June 25, 2011

2. Riak Ecosystem
Riak
Search
Riak
KV
Riak
Core

3. Riak Ecosystem
Riak
Search
Riak
KV
Riak
Core

4. Riak Ecosystem
Riak
Search
Riak
KV
Riak
Core

5. Riak Ecosystem
Riak
Search
Riak
KV
Riak
Core

6. Riak Ecosystem
Riak
Search
Riak
KV
Riak
Core

7. % whoami
# curl http://susanpotter.net/whoami/windycitydb2011.json
• 15 years RDBMS
• 3-4 years KV databases (CouchDB and now Riak)
• 1 year Redis
• Using graph database in next part of platform

8. % whoami
# curl http://susanpotter.net/whoami/windycitydb2011.json
• 15 years RDBMS
• 3-4 years KV databases (CouchDB and now Riak)
• 1 year Redis
• Using graph database in next part of platform

9. % whoami
# curl http://susanpotter.net/whoami/windycitydb2011.json
• 15 years RDBMS
• 3-4 years KV databases (CouchDB and now Riak)
• 1 year Redis
• Using graph database in next part of platform

10. % whoami
# curl http://susanpotter.net/whoami/windycitydb2011.json
• 15 years RDBMS
• 3-4 years KV databases (CouchDB and now Riak)
• 1 year Redis
• Using graph database in next part of platform

11. Scope of Talk
• Problem Domain • not Consistent Hashing or DHTs
• Linking Objects • not Vector Clocks
• Traversing Links • not Gossip Protocol
• Link Maintainence • not Hinted Handoff
• not Read Repair
• not Eventual Consistency

12. Scope of Talk
• Problem Domain • not Consistent Hashing or DHTs
• Linking Objects • not Vector Clocks
• Traversing Links • not Gossip Protocol
• Link Maintainence • not Hinted Handoff
• not Read Repair
• not Eventual Consistency

13. Scope of Talk
• Problem Domain • not Consistent Hashing or DHTs
• Linking Objects • not Vector Clocks
• Traversing Links • not Gossip Protocol
• Link Maintainence • not Hinted Handoff
• not Read Repair
• not Eventual Consistency

14. Scope of Talk
• Problem Domain • not Consistent Hashing or DHTs
• Linking Objects • not Vector Clocks
• Traversing Links • not Gossip Protocol
• Link Maintainence • not Hinted Handoff
• not Read Repair
• not Eventual Consistency

15. Scope of Talk
• Problem Domain • not Consistent Hashing or DHTs
• Linking Objects • not Vector Clocks
• Traversing Links • not Gossip Protocol
• Link Maintainence • not Hinted Handoff
• not Read Repair
• not Eventual Consistency

16. Scope of Talk
• Problem Domain • not Consistent Hashing or DHTs
• Linking Objects • not Vector Clocks
• Traversing Links • not Gossip Protocol
• Link Maintainence • not Hinted Handoff
• not Read Repair
• not Eventual Consistency

17. Scope of Talk
• Problem Domain • not Consistent Hashing or DHTs
• Linking Objects • not Vector Clocks
• Traversing Links • not Gossip Protocol
• Link Maintainence • not Hinted Handoff
• not Read Repair
• not Eventual Consistency

18. Scope of Talk
• Problem Domain • not Consistent Hashing or DHTs
• Linking Objects • not Vector Clocks
• Traversing Links • not Gossip Protocol
• Link Maintainence • not Hinted Handoff
• not Read Repair
• not Eventual Consistency

19. Scope of Talk
• Problem Domain • not Consistent Hashing or DHTs
• Linking Objects • not Vector Clocks
• Traversing Links • not Gossip Protocol
• Link Maintainence • not Hinted Handoff
• not Read Repair
• not Eventual Consistency

20. Scope of Talk
• Problem Domain • not Consistent Hashing or DHTs
• Linking Objects • not Vector Clocks
• Traversing Links • not Gossip Protocol
• Link Maintainence • not Hinted Handoff
• not Read Repair
• not Eventual Consistency

21. Problem Domain

22. Problem Domain

23. Linking[1/2]
curl -X PUT -H ’Content-type: text/json’
-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’
-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’
-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’
...
-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’
http://host:8098/riak/album/Born-This-Way
data ...

24. Linking[1/2]
curl -X PUT -H ’Content-type: text/json’
-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’
-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’
-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’
...
-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’
http://host:8098/riak/album/Born-This-Way
data ...

25. Linking[1/2]
curl -X PUT -H ’Content-type: text/json’
-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’
-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’
-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’
...
-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’
http://host:8098/riak/album/Born-This-Way
data ...

26. Linking[1/2]
curl -X PUT -H ’Content-type: text/json’
-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’
-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’
-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’
...
-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’
http://host:8098/riak/album/Born-This-Way
data ...

27. Linking[1/2]
curl -X PUT -H ’Content-type: text/json’
-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’
-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’
-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’
...
-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’
http://host:8098/riak/album/Born-This-Way
data ...

28. Linking[2/2]
• Two Dimensional
both direction and type/tag
• 0..* links per object
of any type/tag
• Metadata traversing
not load, parse, then lookup cycle
• Model sufficiency
otherwise graph database might be more useful
• Link to other buckets
(in same Riak cluster)

29. Linking[2/2]
• Two Dimensional
both direction and type/tag
• 0..* links per object
of any type/tag
• Metadata traversing
not load, parse, then lookup cycle
• Model sufficiency
otherwise graph database might be more useful
• Link to other buckets
(in same Riak cluster)

30. Linking[2/2]
• Two Dimensional
both direction and type/tag
• 0..* links per object
of any type/tag
• Metadata traversing
not load, parse, then lookup cycle
• Model sufficiency
otherwise graph database might be more useful
• Link to other buckets
(in same Riak cluster)

31. Linking[2/2]
• Two Dimensional
both direction and type/tag
• 0..* links per object
of any type/tag
• Metadata traversing
not load, parse, then lookup cycle
• Model sufficiency
otherwise graph database might be more useful
• Link to other buckets
(in same Riak cluster)

32. Linking[2/2]
• Two Dimensional
both direction and type/tag
• 0..* links per object
of any type/tag
• Metadata traversing
not load, parse, then lookup cycle
• Model sufficiency
otherwise graph database might be more useful
• Link to other buckets
(in same Riak cluster)

33. Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1
• GET /riak/album/Born-This-Way/artist,performer,1
• GET /riak/album/Born-This-Way/artist,producer,1
• GET /riak/album/Born-This-Way/_,performer,1
• GET /riak/album/Born-This-Way/_,producer,1

34. Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1
=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1
• GET /riak/album/Born-This-Way/artist,producer,1
• GET /riak/album/Born-This-Way/_,performer,1
• GET /riak/album/Born-This-Way/_,producer,1

35. Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1
=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1
• GET /riak/album/Born-This-Way/artist,producer,1
• GET /riak/album/Born-This-Way/_,performer,1
• GET /riak/album/Born-This-Way/_,producer,1

36. Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1
=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1
=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1
• GET /riak/album/Born-This-Way/_,performer,1
• GET /riak/album/Born-This-Way/_,producer,1

37. Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1
=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1
=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1
• GET /riak/album/Born-This-Way/_,performer,1
• GET /riak/album/Born-This-Way/_,producer,1

38. Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1
=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1
=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1
=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1
• GET /riak/album/Born-This-Way/_,producer,1

39. Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1
=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1
=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1
=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1
• GET /riak/album/Born-This-Way/_,producer,1

40. Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1
=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1
=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1
=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1
=> object linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/_,producer,1

41. Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1
=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1
=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1
=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1
=> object linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/_,producer,1

42. Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1
=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1
=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1
=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1
=> object linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/_,producer,1
=> object linked to album with tag/type “producer”

43. Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
• GET /riak/album/A/artist,producer,0/album,release,1

44. Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
• GET /riak/album/A/artist,producer,0/album,release,1

45. Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1
=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
• GET /riak/album/A/artist,producer,0/album,release,1

46. Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1
=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
• GET /riak/album/A/artist,producer,0/album,release,1

47. Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1
=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
• GET /riak/album/A/artist,producer,0/album,release,1

48. Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1
=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
=> all “collaborators” of “performers” on album A excluding “performers”
• GET /riak/album/A/artist,producer,0/album,release,1

49. Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1
=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
=> all “collaborators” of “performers” on album A excluding “performers”
• GET /riak/album/A/artist,producer,0/album,release,1

50. Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1
=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
=> all “collaborators” of “performers” on album A excluding “performers”
• GET /riak/album/A/artist,producer,0/album,release,1

51. Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1
=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
=> all “collaborators” of “performers” on album A excluding “performers”
• GET /riak/album/A/artist,producer,0/album,release,1
=> all albums that “producers” on album A have released

52. Traversing Links[3/4]
artist,performer,1
• Bucket name
e.g. artist, album
• Tag (or type) name
e.g. performer, producer
• Return (or “keep”)
useful in nested traversals
• Wildcarding
’_’ for bucket and tag means “any”
• Nesting
traverse on multiple levels
• Multiple Sources?
how to walk links from multiple objects?

53. Traversing Links[3/4]
artist,performer,1
• Bucket name
e.g. artist, album
• Tag (or type) name
e.g. performer, producer
• Return (or “keep”)
useful in nested traversals
• Wildcarding
’_’ for bucket and tag means “any”
• Nesting
traverse on multiple levels
• Multiple Sources?
how to walk links from multiple objects?

54. Traversing Links[3/4]
artist,performer,1
• Bucket name
e.g. artist, album
• Tag (or type) name
e.g. performer, producer
• Return (or “keep”)
useful in nested traversals
• Wildcarding
’_’ for bucket and tag means “any”
• Nesting
traverse on multiple levels
• Multiple Sources?
how to walk links from multiple objects?

55. Traversing Links[3/4]
artist,performer,1
• Bucket name
e.g. artist, album
• Tag (or type) name
e.g. performer, producer
• Return (or “keep”)
useful in nested traversals
• Wildcarding
’_’ for bucket and tag means “any”
• Nesting
traverse on multiple levels
• Multiple Sources?
how to walk links from multiple objects?

56. Traversing Links[3/4]
_,_,0
• Bucket name
e.g. artist, album
• Tag (or type) name
e.g. performer, producer
• Return (or “keep”)
useful in nested traversals
• Wildcarding
’_’ for bucket and tag means “any”
• Nesting
traverse on multiple levels
• Multiple Sources?
how to walk links from multiple objects?

57. Traversing Links[3/4]
_,_,0/album,release,1
• Bucket name
e.g. artist, album
• Tag (or type) name
e.g. performer, producer
• Return (or “keep”)
useful in nested traversals
• Wildcarding
’_’ for bucket and tag means “any”
• Nesting
traverse on multiple levels
• Multiple Sources?
how to walk links from multiple objects?

58. Traversing Links[3/4]
• Bucket name
e.g. artist, album
• Tag (or type) name
e.g. performer, producer
• Return (or “keep”)
useful in nested traversals
• Wildcarding
’_’ for bucket and tag means “any”
• Nesting
traverse on multiple levels
• Multiple Sources?
how to walk links from multiple objects?

59. curl -X POST -H “Content-type: application/json”
http://host:8098/mapred data @-
{"inputs":[
["artist","Ke$ha"],
["artist", "Lady+Gaga"]
],
"query":[
{"link": { "tag": "release", "bucket": "album" }},
{"map":{ "language":"javascript",
"source":"function(v) return [v]; "}
}
]
}
^D

60. curl -X POST -H “Content-type: application/json”
http://host:8098/mapred data @-
{"inputs":[
["artist","Ke$ha"],
["artist", "Lady+Gaga"]
],
"query":[
{"link": { "tag": "release", "bucket": "album" }},
{"map":{ "language":"javascript",
"source":"function(v) return [v]; "}
}
]
}
^D

61. curl -X POST -H “Content-type: application/json”
http://host:8098/mapred data @-
{"inputs":[
["artist","Ke$ha"],
["artist", "Lady+Gaga"]
],
"query":[
{"link": { "tag": "release", "bucket": "album" }},
{"map":{ "language":"javascript",
"source":"function(v) return [v]; "}
}
]
}
^D

62. Link Maintenance
• Commit Hooks
can use either pre- or post-commit hooks
• Automatic Backlinks
can use commit hooks to generate backlinks automatically
• Trigger Messages
links could only be used as triggers
• Delete Cleanup
post-commit hook with backlinks can clean up links to new orphans automatically

63. Link Maintenance
• Commit Hooks
can use either pre- or post-commit hooks
• Automatic Backlinks
can use commit hooks to generate backlinks automatically
• Trigger Messages
links could only be used as triggers
• Delete Cleanup
post-commit hook with backlinks can clean up links to new orphans automatically

64. Link Maintenance
• Commit Hooks
can use either pre- or post-commit hooks
• Automatic Backlinks
can use commit hooks to generate backlinks automatically
• Trigger Messages
links could only be used as triggers
• Delete Cleanup
post-commit hook with backlinks can clean up links to new orphans automatically

65. Link Maintenance
• Commit Hooks
can use either pre- or post-commit hooks
• Automatic Backlinks
can use commit hooks to generate backlinks automatically
• Trigger Messages
links could only be used as triggers
• Delete Cleanup
post-commit hook with backlinks can clean up links to new orphans automatically

66. Possibilities
Object metadata
extensions?

67. Limitations
Not a graph database!

68. References[1/2]
• Riak Wiki
http://wiki.basho.com/
• Riak Mailing List
riak-users@lists.basho.com
• Riak Contrib
http://contrib.basho.com/
• Riak Code Repos
http://github.com/basho

69. References[1/2]
• Riak Wiki
http://wiki.basho.com/
• Riak Mailing List
riak-users@lists.basho.com
• Riak Contrib
http://contrib.basho.com/
• Riak Code Repos
http://github.com/basho

70. References[1/2]
• Riak Wiki
http://wiki.basho.com/
• Riak Mailing List
riak-users@lists.basho.com
• Riak Contrib
http://contrib.basho.com/
• Riak Code Repos
http://github.com/basho

71. References[1/2]
• Riak Wiki
http://wiki.basho.com/
• Riak Mailing List
riak-users@lists.basho.com
• Riak Contrib
http://contrib.basho.com/
• Riak Code Repos
http://github.com/basho

72. References[2/2]
• Secondary index support in Riak?
Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806
• riak_core applications
e.g. Riak Search, riak_zab, riak_pipe
• Comet
KV database supports Active Storage Objects (ASO), handlers
http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf

73. References[2/2]
• Secondary index support in Riak?
Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806
• riak_core applications
e.g. Riak Search, riak_zab, riak_pipe
• Comet
KV database supports Active Storage Objects (ASO), handlers
http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf

74. References[2/2]
• Secondary index support in Riak?
Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806
• riak_core applications
e.g. Riak Search, riak_zab, riak_pipe
• Comet
KV database supports Active Storage Objects (ASO), handlers
http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf

75. Credits
• Basho & Riak logos
Courtesy of Basho
• Icy chained fence photo
http://www.flickr.com/photos/existentist/342177746/
• Frozen chain photo
http://www.flickr.com/photos/skedonk/4197921511/
• Hook and chain photo
http://www.flickr.com/photos/37491467@N07/3871760093/

76. Questions?
Figure: http://www.flickr.com/photos/42682395@N04/
@SusanPotter

77. Questions?
Figure: http://www.flickr.com/photos/42682395@N04/
@SusanPotter