Gisting is an implementation of Google\'s MapReduce framework for processing and extracting useful information from very large data sets. At the time of this writing, the code is available for PREVIEW at http://github.com/mchung/gisting. I am currently working to release this framework for general usage.

1. Hacking with ruby2ruby
Marc Chung
OpenRain
blog.marcchung.com
blog.openrain.com
@heisenthought

2. Hacking with ruby2ruby
Marc Chung
OpenRain
blog.marcchung.com
blog.openrain.com
@heisenthought

4. ruby2ruby
Ruby2Ruby
Walks the Ruby AST
To emit Ruby

5. sudo gem install ruby2ruby

6. Limitations
• Only Ruby 1.8
• Not 1.9, yet
• Built into Rubinius

7. function rhino_rocks() {
return “Rhino, rocks!”;
}

8. js> rhino_rocks();
Rhino, rocks!

9. js> rhino_rocks.toString();
function rhino_rocks() {
return “Rhino, rocks!”;
}

10. js> eval(rhino_rocks.toString()).call();
Rhino, rocks!

11. ruby_rocks = proc {
return “Ruby, rocks”;
}

12. irb> ruby_rocks.call;
Ruby, rocks!

13. irb> require “ruby2ruby”
irb> require “parse_tree_extensions”
irb> ruby_rocks.to_ruby
proc { return “Ruby, rocks” }

14. irb> eval(ruby_rocks.to_ruby).call
Ruby, rocks!

16. define_method
class Voice
define_method(:shout) do |word|
word.upcase!
end
end
puts Ruby2Ruby.translate(Voice)

17. class Voice < Object
def yell(word)
word.upcase!
end
end

18. alias
class AliasClass
def old_busted
quot;42quot;
end
alias_method :new_hotness, :old_busted
end
puts Ruby2Ruby.translate(AliasClass)

19. class AliasClass < Object
def new_hotness
quot;42quot;
end
def old_busted
quot;42quot;
end
end

20. #module_function
module HelperModule
def help
puts quot;Help mequot;
end
module_function : help
end
puts Ruby2Ruby.translate(HelperModule)

21. module HelperModule
def help
puts(quot;Help mequot;)
end
def self. help
puts(quot;Help mequot;)
end
end

22. ActiveRecord Models
class Doctor < ActiveRecord::Base
has_many :patients
end

23. See, Ruby, Run!
• Drmap
• git clone git://github.com/mchung/drmap.git
• Optionally, install Journeta

24. Under the hood
• Abstract syntax trees
• S-expressions
• ParseTree

25. class Book
def title
“Ruby programming”
end
end

26. case NODE_BREAK:
case NODE_NEXT:
if (node->nd_stts)
add_to_parse_tree(self, current, node->nd_stts, locals);
break;
case NODE_YIELD:
if (node->nd_stts)
add_to_parse_tree(self, current, node->nd_stts, locals);
if (node->nd_stts
&& (nd_type(node->nd_stts) == NODE_ARRAY
|| nd_type(node->nd_stts) == NODE_ZARRAY)
&& !node->nd_state)
rb_ary_push(current, Qtrue);
break;

27. [:class, :Book,
[:const, :Object],
[:defn, :title, [:scope, [:block,
[:args], [:str, quot;Ruby programmingquot;]]]]]

28. class Book < Object
def title
“Ruby programming”
end
end

29. /opt/local/usr/bin
• parse_tree_show
• r2r_show

30. class Book
def title
“Ruby programming”
end
end

31. $ parse_tree_show book.rb

32. s(:class, :Book,
nil,
s(:scope,
s(:defn, :title,
s(:args),
s(:scope, s(:block,
s(:str, quot;Ruby programmingquot;))))))

33. $ r2r_show book.rb

34. class Book
def title()
“Ruby programming”
end
end

35. Operator Precedence
with Guy Decoux: a b c, d
[[:fcall, :a,
[:array,
[:fcall, :b,
[:array,
[:vcall, :c],
[:vcall, :d]]]]]]
ParseTree.new.parse_tree_for_string(“a b c, d”)

36. ruby2java

37. class JavaClass
def self.main
puts “Java, rocks!”
end
end

38. [:class, :JavaClass,
[:const, :Object],
[:defs, [:self], :main,
[:scope,
[:block, [:args],
[:fcall, :puts, [:array, [:str, quot;Hello, Javaquot;]]]]]]]

39. class Ruby2Java < SexpProcessor
def process_class(exp)
“public class #{exp.shift} #{next_token(exp, true)}”
end
def process_const(exp)
“extends #{exp.shift}”
end
....
end

40. public class JavaClass {
public static void main(String argv[]) {
return “Hello, Java”;
}
}

42. Distributing Ruby
• drmap - multi-machine worker queue
• gisting - map/reduce framework

43. drmap
irb> fn = proc {|x| x**10}
irb> YAML::dump(fn)
=> quot;--- !ruby/object:Proc {}nnquot;
irb> Marshal.dump(fn)
TypeError: no marshal_dump is defined for class Proc
irb> YAML::dump(fn.to_ruby)
=> quot;--- proc { |x| (x ** 10) }nquot;
irb> Marshal.dump(fn.to_ruby)
=> quot;004bquot;eproc { |x| (x ** 10) }quot;

44. Enumerable#drmap

45. module Enumerable
def drmap(&block)
pool = Drmap::BeanstalkPool.new
jid = rand(100)
each_with_index do |element, idx|
pool.put_job(jid, block.to_ruby, element)
end
results = []
while results.size < length
results << pool.next_result(jid)
end
results
end
end

46. def process
loop do
begin
job_payload = next_job
fn = job_payload[:proc]
data = job_payload[:data]
result = eval(fn).call(data)
save(job_payload[:jid], result)
success!
rescue => e
fail!(e)
end
end
end

47. drmap
• Trivial
• Distributed
• Powerful

48. Demo time!

49. Gisting
• MapReduce in Ruby
• http://github.com/mchung/gisting

50. map / reduce
• Functional programming
• Iterating over collections
• (5..10).map {|x| x+1} #=> [6, 7, 8, 9, 10, 11]
• (5..10).reduce(:+) #=> 45

51. Google MapReduce
• Modeled after map() and reduce()
• Programming model
• Programs are trivially parallelizable

52. Demo time!

53. Gisting::Spec

54. input.file_pattern = quot;file.txtquot;
input.map do |map_input|
words = map_input.strip.split(quot;tquot;)
Emit(words[1], quot;1quot;)
end

55. output.reduce do |reduce_input|
count = 0
reduce_input.each do |value|
count += value.to_i
end
Emit(count)
end

56. inputs = args
spec = Gisting::Spec.new
inputs.each do |file_input|
input = spec.add_input
input.file_pattern = file_input
input.map do |map_input|
# 2722
mailbox
2006-05-23 00:08:39
words = map_input.strip.split(quot;tquot;)
Emit(words[1], quot;1quot;)
end
end
output = spec.output
output.filebase = quot;/Users/mchung/Public/datasets/outputquot;
output.num_tasks = 2
output.reduce do |reduce_input|
count = 0
reduce_input.each do |value|
count += value.to_i
end
Emit(count)
end
result = MapReduce(spec)
pp result

57. Gisting
• MapReduce programming model in Ruby
• Uses ruby2ruby to serialize procs
• Uses EventMachine for Map/ReduceServer

58. • -: 94978
• mailbox: 20872
• google: 6758
• ebay: 2832
• yahoo: 2674
• yahoo.com: 2198
• myspace.com: 1916
• google.com: 1882

59. self.give(:thanks)
OpenRain
openrain.com
My Git Repo
github.com/mchung
Marc Chung
blog.marcchung.com
Ruby::AZ
rubyaz.org

60. Resources
Code Generation: The Safety Scissors Of Metaprogramming
mwrc2008.confreaks.com/03bowkett.html
Forbidden Fruit: A Taste of Ruby's Parse Tree
goruco2008.confreaks.com/03_wanstrath.html
Ruby2Ruby
seattlerb.rubyforge.org/ruby2ruby/
ParseTree
parsetree.rubyforge.org