This document discusses Hadoop Record Compiler (Jute) and its extension JuteRC. JuteRC automatically generates serialization/deserialization code for data types to support the RC storage format. It extends Jute by adding RC-specific code generation. The execution path involves running rcc against a DDL file, which uses the JRcCodeGenerator to set up members, functions, and get RC read/write methods. These methods write to/read from the RC format using utilities like RcUtil.writeBuffer. JuteRC is packaged with Maven and its usage involves generating the JAR then running rcc against a DDL file.

1. Hadoop Record RC Compiler
Tanping Wang

2. Motivation
 Automatically generate serialization/de-
serialization code for any give primitive or
composite data type.
 Directly plug serialization/de-serialization code to
generate MapReduce output file that supports RC
storage format.
 An extension to the existing Hadoop Record
Compiler (Jute) package – So is named JuteRc.

3. Haoop Record Compiler
 Two important packages:
 Hadoop.record.compiler.generated – rcc.jj
 Hadoop.record.compiler
 CodeGenerator
 JType : Jboolean, Jstring, JCompType (JMap, Jvector..)
 Execution path:
bin/rcc ->
rcc#JFile.genCode ->
CodeGenerator#genCode

4. Haoop Record Compiler cont.
 Inside CondeGenerator
JRecord .codeGen :
 set up members,
 set up member functions:
 setter/getter,
 serialization/de-serialization (JType.getRead/WriteMethod)
 abastract JType (JavaType, CppType)
 JBoolean, JString, JMap, Jvector
 JType.getRead/WriteMethod ( important)

5. Haoop Record Compiler cont.
 JType.getReadMethod
_rio_a.startRecord(this,_rio_tag);
_rio_a.writeBuffer(bcookie,"bcookie");
_rio_a.writeLong(timestamp,"timestamp");
_rio_a.endRecord(this,_rio_tag);

6. Hadoop Record RC Compiler
• Execution Path
bin/rcc => bin/rc-rcc
rcc#JFile.genCode => JRcFile.genCode
CodeGenerator#genCode => JRcCodeGenerator

7. Hadoop Record RC Compiler cont.
 Inside CondeGenerator
JRcRecord .codeGen :
 set up members,
 set up memberfunctions:
 setter/getter,
 serialization/de-serialization
(JRcType.getRcRead/WriteRcMethod
 abastract JRcType (JavaType only)
 JRcBoolean, JRcString, JRcMap, JRcVector .. etc.
 JType.getRcReadMethod

8. Hadoop Record RC Compiler cont.
 JType.getRcReadMethod
com.yahoo.ccdi.fetl.RcUtil.writeBuffer(this,
bcookie, writeIndx++);
com.yahoo.ccdi.fetl.RcUtil.writeLong(this,
timestamp, writeIndx++);

9. How to use the JuteRC
 Maven packaged
 README
1) generate JuteRC.jar
% mvn install
2) run rcc against DDL file
% ./rcc --language javarc something.jr