Eric Redmond – Distributed Search on Riak 2.0 Riak excels at one type of query: key puts and key gets. But the world demands more from a database. Since Basho isn't primarily a search company, we decided to leverage the power of Solr for Riak 2.0. This is a walkthrough of what new features we added, how works, and why you'd want to use it. Also, of course, live demos.

1. Riak Search 2.0
Eric Redmond
@coderoshi

2. After two years,
Riak 2.0 is out

3. ┬─┬ ( ^_^ノ)
Setup
ノ

4. riak.conf
## The enabled Search set this 'on'.
search = off
!
## The port number which Solr binds to.
search.solr_port = 10014
!
## The port number which Solr JMX binds to.
search.solr_jmx_port = 10013
!
## The arguments to pass to the Solr JVM. Non-standard
## arguments, i.e. -XX, may not be portable across JVM
## implementations. E.g. -XX:+UseCompressedStrings.
search.solr_jvm_args = -Xms1g -Xmx1g -XX:+UseStringCache -
XX:+UseCompressedOops

5. riak.conf
## The enabled Search set this 'on'.
search = on
!
## The port number which Solr binds to.
search.solr_port = 10014
!
## The port number which Solr JMX binds to.
search.solr_jmx_port = 10013
!
## The arguments to pass to the Solr JVM. Non-standard
## arguments, i.e. -XX, may not be portable across JVM
## implementations. E.g. -XX:+UseCompressedStrings.
search.solr_jvm_args = -Xms1g -Xmx1g -XX:+UseStringCache -
XX:+UseCompressedOops

6. riak-admin cluster join riak@10.0.0.2
... and so on...
riak-admin cluster plan
riak-admin cluster commit

7. riak-admin cluster join riak@10.0.0.2
... and so on...
riak-admin cluster plan
riak-admin cluster commit
riak-admin security add-user eric 12345
riak-admin security add-user admin 123456
riak-admin security grant search.query
ON index simple TO any
riak-admin security grant search.admin
ON schema TO admin

8. require 'riak'
client = Riak::Client.new
!
# create index
client.create_search_index('simple')
!
# tie the index to bucket 'cats'
bucket = Riak::Bucket.new(client, 'cats')
bucket.props = { search_index: 'simple' }

9. Indexing Datatypes

10. UTF8中搜索
חיפוש ב UTF8
Αναζήτηση σε UTF8
Искать в UTF8
Search in UTF8

11. Advanced Search
aka. Next Level Search

12. solr = RSolr.connect(url: 'http://yokozuna01.bos1:8098/solr/docs')
resp = solr.get('select', params: {q: '*:*'})

14. Facets, Stats, and stuff

15. eDisMax
defType=edismax

16. Highlighting

17. Languages

18. Map/Reduce

19. {"inputs": {
"module":"yokozuna",
"function":"mapred_search",
"arg":["docs","title_s:Key* AND language_s:en"] },
"query":[
{"map":{
"language":"javascript",
"keep":false,
"source":"function(v) { return [1]; }"}},
{"reduce"{
"language":"javascript",
"keep":true,
"name":"Riak.reduceSum"
}} ]
}

21. bucket = Riak::Bucket.new(client, 'people')
bucket.props = { search_index: 'faces' }
bucket.get('stuart').store
bucket = Riak::Bucket.new(client, 'cats')
bucket.props = { search_index: 'faces' }
bucket.get('stuart').store
bucket = Riak::Bucket.new(client, 'mountains')
bucket.props = { search_index: 'faces' }
bucket.get('hood').store

22. riak-admin bucket-type create faces '{"props":
{"search_index":"faces"}}'
!
bucket = Riak::Bucket.new(client, ‘people')
bucket.get('stuart', type:'faces').store
bucket = Riak::Bucket.new(client, ‘cats')
bucket.get('stuart', type:'faces').store
bucket = Riak::Bucket.new(client, ‘mountains')
bucket.get('stuart', type:'faces').store

23. Schemas

24. <schema name="default" version="1.5">
<fields>
<field name="kinds" type="string" indexed="true" stored="false" multiValued="true" />
<field name="name" type="string" indexed="true" stored="true" />
!
<dynamicField name="*_ss" type="string" indexed="true" stored="true" multiValued="true"/>
!
<field name="_yz_ed" type="_yz_str" indexed="true" stored="false"/>
<field name="_yz_pn" type="_yz_str" indexed="true" stored="false"/>
<field name="_yz_fpn" type="_yz_str" indexed="true" stored="false"/>
<field name="_yz_vtag" type="_yz_str" indexed="true" stored="false"/>
<field name="_yz_node" type="_yz_str" indexed="true" stored="false"/>
<field name="_yz_rk" type="_yz_str" indexed="true" stored="true"/>
<field name="_yz_rb" type="_yz_str" indexed="true" stored="true"/>
</fields>
</schema>

25. Fallen Nodes

26. Why?

27. Thermocline

28. Thermocline of Replication

29. Goals

30. But... why?

32. How

33. Yokozuna = Glue

34. Extractors
yz_extractor.erl
yz_json_extractor.erl
yz_noop_extractor.erl
yz_text_extractor.erl
yz_xml_extractor.erl
yz_doc.erl

35. Interfaces
yz_pb_admin.erl
yz_pb_search.erl
yz_schema.erl
yz_wm_extract.erl
yz_wm_index.erl
yz_wm_schema.erl
yz_wm_search.erl

36. Security

37. Permissions
riak-admin security grant search.admin ON schema TO admin
riak-admin security grant search.admin ON index TO admin
riak-admin security grant search.query ON index TO user
riak-admin security grant search.query ON index wiki TO user

38. yz_pb_admin.erl
yz_schema.erl
yz_wm_index.erl
Administration yz_wm_schema.erl

39. yz_pb_search.erl
Searches
yz_wm_search.erl

40. yokozuna.erl
Backend Magic
yz_app.erl
yz_general_sup.erl
yz_kv.erl
yz_misc.erl
yz_sup.erl

41. Distribution
yz_cover.erl
yz_events.erl

42. yz_entropy.erl
AAE
yz_entropy_mgr.erl
yz_exchange_fsm.erl
yz_index.erl
yz_index_hashtree.erl
yz_index_hashtree_sup.erl

43. Solr code
EntropyData.java
Monitor.java
yz_solr.erl
yz_solr_proc.erl
yz_solr_sup.erl

44. Stats
yz_stat.erl
yz_stat_worker.erl

45. “Testing only shows
the presence, not
the absence,
of bugs”
aae_test.erl
yokozuna_essential.erl
yz_errors.erl
yz_fallback.erl
yz_flag_transitions.erl
yz_index_admin.erl
yz_languages.erl
yz_mapreduce.erl
yz_monitor_solr.erl
yz_pb.erl
yz_rs_migration.erl
yz_rt.erl
yz_schema_admin.erl
yz_siblings.erl
yz_solr_start_timeout.erl
yz_stat_test.erl
yz_wm_extract_test.erl
yz_component_tests.erl
yz_json_extractor_tests.erl
yz_kv_tests.erl
yz_misc_tests.erl
yz_text_extractor_tests.erl
yz_xml_extractor_tests.erl
yz_driver.erl
yz_file_terms.erl
Dijkstra

46. Thanks
Eric Redmond
@coderoshi