Presentation in RubyKaigi 2015

1. Ruby meets Go
Dec. 12, 2015
Masaki Matsushita

2. About Me
● Masaki Matsushita
● CRuby Committer
○ 138 Commits
■ Mainly for performance improvement
■ Marshal.load, Hash#flatten, etc.
● Software Engineer at NTT Communications
○ Contribution to OpenStack
○ Slide at OpenStack Summit Tokyo
http://goo.gl/OXTYor
● Twitter: @_mmasaki Github: mmasaki

3. Today’s Topic
● Go 1.5 Feature: buildmode “c-shared”
○ Cgo Basics
● Using Go Function from Ruby
○ FFI and Fiddle without ruby.h
● Writing Extension Library with Go (and C)
○ Define Functions Equivalent to C Macros
○ Avoid Copy of Strings
○ Propagate Reference from Ruby to Go
○ Creating Gem including Go code

4. Buildmode “c-shared”
● Go 1.5 relased in August 2015
● Buildmode “c-shared” was introduced
○ go build -buildmode c-shared
○ Build C shared library with cgo
● cgo enables:
○ Refer to C functions, types and variables
○ Export Go functions for use by C

5. Cgo Example: Say hello with puts() in C
package main
/*
#include <stdlib.h>
#include <stdio.h>
*/
import "C"
import "unsafe"
func main() {
cstr := C.CString("Hello, world!")
defer C.free(unsafe.Pointer(cstr))
C.puts(cstr)
}
Include C header file
Convert Go string into C String

6. Cgo Example: define and use C function
package main
/*
char *hello(void) {
return "Hello, world!";
}
*/
import "C"
import "fmt"
func main() {
cstr := C.hello()
fmt.Println(C.GoString(cstr))
}
Define C Function
Convert into Go String
Call C Function from Go

7. Try c-shared: add.go
package main
import "C"
//export add
func add(a C.int, b C.int) C.int {
return a + b
}
func main() {}
● go build -buildmode c-shared -o add.so add.go
Export Go Function for use by C

8. Load c-shared Libraries
● ruby-ffi
○ https://github.com/ffi/ffi
○ gem install ffi
● fiddle
○ Standard ext library
● useful to call Go functions simply
(without ruby.h)

9. Call Go Function from Ruby: ruby-ffi
require "ffi"
module Int
extend FFI::Library
ffi_lib "int.so"
attach_function :add, [:int, :int], :int
end
p Int.add(15, 27) #=> 42
Load c-shared library
Add Go Function to Module

10. Call Go Function from Ruby: fiddle
require "fiddle/import"
module Int
extend Fiddle::Importer
dlload "int.so"
extern "int add(int, int)"
end
p Int.add(15, 27) #=> 42

11. Go String and C String: str.so
package main
import "C"
import "fmt"
//export hello
func hello(cstr *C.char) {
str := C.GoString(cstr)
fmt.Println("Hello, " + str)
}
func main() {}
Receive C String
Convert to Go String

12. Returning String: ruby-ffi
require "ffi"
module Hello
extend FFI::Library
ffi_lib "str.so"
attach_function :hello, [:string], :void
end
Hello.hello("world") #=> "Hello, world"
Ruby String can be passed

13. Returning String: fiddle
require "fiddle/import"
module Hello
extend Fiddle::Importer
dlload "str.so"
extern "void hello(char *str)"
end
Hello.hello("world") #=> "Hello, world"

14. Cgo Functions to Convert String
● C.CString(goString string) *C.char
○ copy Go String to C String
○ Users are responsible to free C String
● C.GoString(cString *C.char) string
● C.GoStringN(cString *C.char, length C.int) string
○ copy C String to Go String

15. Writing Extension Library with Go
● Naruse-san’s Amazing Talk:
“Writing extension libraries in Go”
at OedoRubyKaigi 05
https://speakerdeck.com/naruse/writing-extension-libraries-in-go
● gohttp: https://github.com/nurse/gohttp
○ Implementation of extension library in Go

16. C Extension Library Basics
static VALUE
rb_magic_number(VALUE self)
{
return INT2NUM(42);
}
void
Init_test(void)
{
rb_cFoo = rb_define_class("Foo");
rb_define_method(rb_cFoo, "magic_number", rb_magic_number, 0);
}

17. C Extension Library Basics
static VALUE
rb_magic_number(VALUE self)
{
return INT2NUM(42);
}
void
Init_test(void)
{
rb_cFoo = rb_define_class("Foo");
rb_define_method(rb_cFoo, "magic_number", rb_magic_number, 0);
}
Method Implementation

18. C Extension Library Basics
static VALUE
rb_magic_number(VALUE self)
{
return INT2NUM(42);
}
void
Init_test(void)
{
rb_cFoo = rb_define_class("Foo");
rb_define_method(rb_cFoo, "magic_number", rb_magic_number, 0);
}
Using C Macro

19. C Extension Library Basics
static VALUE
rb_magic_number(VALUE self)
{
return INT2NUM(42);
}
void
Init_test(void)
{
rb_cFoo = rb_define_class("Foo");
rb_define_method(rb_cFoo, "magic_number", rb_magic_number, 0);
}
Function Pointer

20. Minimal Go Ext Example?
//export rb_magic_num
func rb_magic_num(self C.VALUE) C.VALUE {
return INT2NUM(42)
}
//export Init_foo
func Init_foo() {
rb_cFoo = rb_define_class("Foo", C.rb_cObject)
rb_define_method(rb_cFoo, "magic_num", C.rb_magic_num, 0)
}

21. Writing Extension Library with Go
● Wrapper Function equivalent to C Macro
○ C macros can’t be used by Cgo
● Convert Go String into Ruby without Copy
● Propagate Ruby Reference to Go
● Create gem including Go code
○ Modify Rakefile and extconf.rb

22. C Macros for Ruby Extention Libraries
● Useful C macros are defined in ruby.h
○ INT2NUM: C int to Ruby Numeric
○ NIL_P: true if obj is nil
○ RSTRING_PTR: pointer to buffer of String
○ RSTRING_LEN: lengh of String
● These macros can’t be used from Cgo…
● Define Go functions equivalent to C macros
○ Use equivalent C function
○ Wrap C macros with C function

23. Use Equivalent C Function
func LONG2NUM(n C.long) C.VALUE {
return C.rb_long2num_inline(n)
}
func NUM2LONG(n C.VALUE) C.long {
return C.rb_num2long(n)
}

24. Wrap C macros with C function
package main
/*
long rstring_len(VALUE str) {
return RSTRING_LEN(str);
}
*/
import "C"
func RSTRING_LEN(str C.VALUE) C.long {
return C.rstring_len(str)
}

25. Convert Go String into Ruby without Copy
● Go String -> C String -> Ruby String
● C.CString(goString string) *C.char
○ copy Go String to C String
○ Users are responsible to free C String
● VALUE rb_str_new(const char *ptr, long len)
○ copy C String to Ruby String

26. Basic Usage of C.CString()
// go/doc/progs/cgo4.go
func Print(s string) {
cs := C.CString(s)
defer C.free(unsafe.Pointer(cs))
C.fputs(cs, (*C.FILE)(C.stdout))
}
● Call C func and discard C str soon

27. Basic Usage of C.CString()
str := "Hello, world!"
// Copy #1
cstr := C.CString(str) // will be discarded soon
// Copy #2
rbstr := C.rb_str_new(cstr, C.long(len(str)))
● Need to copy twice!

28. Avoid Copy of Strings
● Get *C.char from Go String without Copy
func GOSTRING_PTR(str string) *C.char {
bytes := *(*[]byte)(unsafe.Pointer(&str))
return (*C.char)(unsafe.Pointer(&bytes[0]))
}
// example of use
cstr := GOSTRING_PTR(str) C.rb_utf8_str_new
(cstr, C.long(len(str)))

29. Avoid Copy of Strings
● Technique to Get []byte from Go w/o Copy
http://qiita.com/mattn/items/176459728ff4f854b165
func GOSTRING_PTR(str string) *C.char {
bytes := *(*[]byte)(unsafe.Pointer(&str))
return (*C.char)(unsafe.Pointer(&bytes[0]))
}

30. Avoid Copy of Strings
● Get *C.char from []byte
func GOSTRING_PTR(str string) *C.char {
bytes := *(*[]byte)(unsafe.Pointer(&str))
return (*C.char)(unsafe.Pointer(&bytes[0]))
}
Cast to char

31. Example Usage of GOSTRING_PTR()
func RbString(str string) C.VALUE {
if len(str) == 0 { return C.rb_utf8_str_new(nil, C.long(0)) }
return C.rb_utf8_str_new(GOSTRING_PTR(str), GOSTRING_LEN(str))
}
func rb_define_class(name string, parent C.VALUE) C.VALUE {
return C.rb_define_class(GOSTRING_PTR(name), parent)
}
func rb_define_method(klass C.VALUE, name string, fun unsafe.Pointer, args int) {
cname := GOSTRING_PTR(name)
C.rb_define_method(klass, cname, (*[0]byte)(fun), C.int(args))
}

32. Propagate Ruby Reference to Go
● Go’s GC doesn’t know refs from Ruby
● Go obj referenced from Ruby can be collected
● We have to propagate Ruby Refs to Go
● Use Map to keep reference to Go Objects

33. Propagate Ruby Reference to Go
var objects = make(map[interface{}]int)
//export goobj_retain
func goobj_retain(obj unsafe.Pointer) {
objects[obj]++ // increment reference count
}
//export goobj_free
func goobj_free(obj unsafe.Pointer) {
objects[obj]-- // decrement reference count
if objects[obj] <= 0 { delete(objects, obj) }
}

34. Propagate Ruby Reference to Go
static const rb_data_type_t go_type = {
"GoStruct",
{NULL, goobj_free, NULL},
0, 0, RUBY_TYPED_FREE_IMMEDIATELY|RUBY_TYPED_WB_PROTECTED
};
VALUE
NewGoStruct(VALUE klass, void *p)
{
goobj_retain(p);
return TypedData_Wrap_Struct((klass), &go_type, p);
}
Increment Reference Count
Decrement Reference Count

35. Create Gem including Go code
● Directory Structure
● Rakefile
● extconf.rb

36. Directory Structure
● Use “bundle gem --ext foo”
├── ext
│ └── foo
│ ├── extconf.rb // configured to use go build
│ ├── foo.c // helper functions for use by Go
│ └── foo.h // export helper functions
│ ├── foo.go // created by hand
│ └── wrapper.go // created by hand
└── lib

37. Rakefile
require 'bundler'
Bundler::GemHelper.install_tasks
require 'rake/extensiontask'
task :default => [:compile]
spec = eval File.read('foo.gemspec')
Rake::ExtensionTask.new('foo', spec) do |ext|
ext.lib_dir = File.join(*['lib', 'foo', ENV['FAT_DIR']].compact)
ext.ext_dir = 'ext/foo'
ext.source_pattern = "*.{c,cpp,go}"
end
● Add .go into source_pattern

38. extconf.rb
require 'mkmf'
find_executable('go')
$objs = []
def $objs.empty?; false ;end
create_makefile("memberlist/memberlist")
case `#{CONFIG['CC']} --version`
when /Free Software Foundation/
ldflags = '-Wl,--unresolved-symbols=ignore-all'
when /clang/
ldflags = '-undefined dynamic_lookup'
end
● Some techniques to build successful

39. extconf.rb
File.open('Makefile', 'a') do |f|
f.write <<-EOS.gsub(/^ {8}/, "t")
$(DLLIB): Makefile $(srcdir)/memberlist.go $(srcdir)/wrapper.go
CGO_CFLAGS='$(INCFLAGS)' CGO_LDFLAGS='#{ldflags}'
go build -p 4 -buildmode=c-shared -o $(DLLIB)
EOS
end
● Modify Makefile to use go build

40. Ruby meets Go
● Buildmode “c-shared” and Cgo Basics
● Using Go Function from Ruby
○ FFI and Fiddle without ruby.h
● Writing Extension Library with Go
○ Define Functions Equivalent to C Macros
○ Avoid Copy of Strings
○ Propagate Reference from Ruby to Go
○ Creating Gem including Go code
● Let’s Hack Go for Ruby together!