Ricon|West talk about Yokozuna

1. Eric Redmond
@coderoshi
Tuesday, November 12, 13

2. [Slide Intentionally Left Blank]
Tuesday, November 12, 13

3. A million values isn’t cool.
You know what’s cool?...
Tuesday, November 12, 13

4. A trillion values spread across
lots of nodes that auto-index and you can
also search in cool ways like geographically
or maybe you could extract from a pdf on the server side if you used the extractor API
Tuesday, November 12, 13

5. ┬─┬ ノ ^_^ノ)
(
Setup
Tuesday, November 12, 13

6. ## The enabled Yokozuna set this 'on'.
yokozuna = off
riak.conf
## The port number which Solr binds to.
yokozuna.solr_port = 10014
## The port number which Solr JMX binds to.
yokozuna.solr_jmx_port = 10013
## The arguments to pass to the Solr JVM. Non-standard
## arguments, i.e. -XX, may not be portable across JVM
## implementations. E.g. -XX:+UseCompressedStrings.
yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX:
+UseStringCache -XX:+UseCompressedOops
## The data under which to store all Yokozuna related data.
## Including the Solr index data.
yokozuna.data_dir = ./data/yz
Tuesday, November 12, 13

7. ## The enabled Yokozuna set this 'on'.
yokozuna = on
riak.conf
## The port number which Solr binds to.
yokozuna.solr_port = 10014
## The port number which Solr JMX binds to.
yokozuna.solr_jmx_port = 10013
## The arguments to pass to the Solr JVM. Non-standard
## arguments, i.e. -XX, may not be portable across JVM
## implementations. E.g. -XX:+UseCompressedStrings.
yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX:
+UseStringCache -XX:+UseCompressedOops
## The data under which to store all Yokozuna related data.
## Including the Solr index data.
yokozuna.data_dir = ./data/yz
Tuesday, November 12, 13

8. riak-admin cluster join riak@10.0.0.2
... and so on...
riak-admin cluster plan
riak-admin cluster commit
Tuesday, November 12, 13

9. riak-admin cluster join riak@10.0.0.2
... and so on...
riak-admin cluster plan
riak-admin cluster commit
riak-admin security add-user eric 12345
riak-admin security add-user admin 123456
riak-admin security grant yokozuna.search
ON index simple TO eric
riak-admin security grant yokozuna.admin
ON index TO admin
Tuesday, November 12, 13

10. require 'riak'
client = Riak::Client.new
# create index
client.create_search_index('simple')
# tie the index to bucket 'cats'
bucket = Riak::Bucket.new(client, 'cats')
bucket.props = { yz_index: 'simple' }
Tuesday, November 12, 13

11. Indexing Datatypes
Tuesday, November 12, 13

12. UTF8中搜索
UTF8‫חיפוש ב‬
Αναζήτηση σε UTF8
Искать в UTF8
Search in UTF8
Tuesday, November 12, 13

13. Advanced Search
aka. Next Level Search
Tuesday, November 12, 13

14. solr = RSolr.connect(url: 'http://yokozuna01.bos1:8098/solr/docs')
resp = solr.get('select', params: {q: '*:*'})
Tuesday, November 12, 13

15. Tuesday, November 12, 13

16. Facets, Stats, and stuff
Tuesday, November 12, 13

17. eDisMax
defType=edismax
Tuesday, November 12, 13

18. Highlighting
Tuesday, November 12, 13

19. Languages
Tuesday, November 12, 13

20. Map/Reduce
Tuesday, November 12, 13

21. {"inputs": {
"module":"yokozuna",
"function":"mapred_search",
"arg":["docs","title_s:Key* AND language_s:en"] },
"query":[
{"map":{
"language":"javascript",
"keep":false,
"source":"function(v) { return [1]; }"}},
{"reduce"{
"language":"javascript",
"keep":true,
"name":"Riak.reduceSum"
}} ]
}
Tuesday, November 12, 13

22. Tuesday, November 12, 13

23. client.create_search_index('simple')
bucket = Riak::Bucket.new(client, 'people')
bucket.props = { yz_index: 'simple' }
bucket = Riak::Bucket.new(client, 'cats')
bucket.props = { yz_index: 'simple' }
bucket = Riak::Bucket.new(client, 'mountains')
bucket.props = { yz_index: 'simple' }
Tuesday, November 12, 13

24. Schemas
Tuesday, November 12, 13

25. <schema name="default" version="1.5">
<fields>
<field name="kinds" type="string" indexed="true" stored="false" multiValued="true" />
<field name="name" type="string" indexed="true" stored="true" />
<dynamicField name="*_ss" type="string" indexed="true" stored="true" multiValued="true"/>
<field name="_yz_ed" type="_yz_str" indexed="true" stored="false"/>
<field name="_yz_pn" type="_yz_str" indexed="true" stored="false"/>
<field name="_yz_fpn" type="_yz_str" indexed="true" stored="false"/>
<field name="_yz_vtag" type="_yz_str" indexed="true" stored="false"/>
<field name="_yz_node" type="_yz_str" indexed="true" stored="false"/>
<field name="_yz_rk" type="_yz_str" indexed="true" stored="true"/>
<field name="_yz_rb" type="_yz_str" indexed="true" stored="true"/>
</fields>
</schema>
Tuesday, November 12, 13

26. Fallen Nodes
Tuesday, November 12, 13

27. Why?
Tuesday, November 12, 13

28. Thermocline
Tuesday, November 12, 13

29. Thermocline of Replication
Tuesday, November 12, 13

30. Goals
Tuesday, November 12, 13

31. But... why?
Tuesday, November 12, 13

32. Tuesday, November 12, 13

33. How
Tuesday, November 12, 13

34. Yokozuna = Glue
Tuesday, November 12, 13

35. yz_extractor.erl
yz_json_extractor.erl
yz_noop_extractor.erl
yz_text_extractor.erl
yz_xml_extractor.erl
yz_doc.erl
Tuesday, November 12, 13
Extractors

36. yz_pb_admin.erl
yz_pb_search.erl
yz_schema.erl
yz_wm_extract.erl
yz_wm_index.erl
yz_wm_schema.erl
yz_wm_search.erl
Tuesday, November 12, 13
Interfaces

37. Security
Tuesday, November 12, 13

38. Permissions
riak-admin security grant yokozuna.admin ON schema TO admin
riak-admin security grant yokozuna.admin ON index TO admin
riak-admin security grant yokozuna.search ON index TO user
riak-admin security grant yokozuna.search ON index wiki TO user
Tuesday, November 12, 13

39. yz_pb_admin.erl
yz_schema.erl
yz_wm_index.erl
yz_wm_schema.erl
Tuesday, November 12, 13
Administration

40. yz_pb_search.erl
yz_wm_search.erl
Tuesday, November 12, 13
Searches

41. yokozuna.erl
yz_app.erl
yz_general_sup.erl
yz_kv.erl
yz_misc.erl
yz_sup.erl
Tuesday, November 12, 13
Backend Magic

42. Distribution
yz_cover.erl
yz_events.erl
Tuesday, November 12, 13

43. yz_entropy.erl
yz_entropy_mgr.erl
yz_exchange_fsm.erl
yz_index.erl
yz_index_hashtree.erl
yz_index_hashtree_sup.erl
Tuesday, November 12, 13
AAE

44. Solr code
EntropyData.java
Monitor.java
yz_solr.erl
yz_solr_proc.erl
yz_solr_sup.erl
Tuesday, November 12, 13

45. Stats
yz_stat.erl
yz_stat_worker.erl
Tuesday, November 12, 13

46. aae_test.erl
yokozuna_essential.erl
yz_errors.erl
yz_fallback.erl
yz_flag_transitions.erl
yz_index_admin.erl
yz_languages.erl
yz_mapreduce.erl
yz_monitor_solr.erl
yz_pb.erl
yz_rs_migration.erl
yz_rt.erl
yz_schema_admin.erl
yz_siblings.erl
yz_solr_start_timeout.erl
yz_stat_test.erl
yz_wm_extract_test.erl
yz_component_tests.erl
yz_json_extractor_tests.erl
yz_kv_tests.erl
yz_misc_tests.erl
yz_text_extractor_tests.erl
yz_xml_extractor_tests.erl
yz_driver.erl
yz_file_terms.erl
Tuesday, November 12, 13
“Testing only shows
the presence, not
the absence,
of bugs”
Dijkstra

47. We Want More
•
•
•
•
Tuesday, November 12, 13
CRDT Support
Simplify custom extractors
Bucket Types (?)
Users

48. Thanks
Eric Redmond
@coderoshi
Tuesday, November 12, 13