K Young, CEO of Mortar, gave a presentation on using MongoDB and Hadoop/Pig together. He began with a brief introduction to Hadoop and Pig, explaining their uses for processing large datasets. He then demonstrated how to load data from MongoDB into Pig using a connector, and store data from Pig back into MongoDB. The rest of the presentation focused on use cases for combining MongoDB and Pig, such as being able to separately manage data storage and processing. Young also showed some Mortar utilities for working with MongoDB data in Pig.

1. K Young - CEO, Mortar
MongoDB + Hadoop
sittin’ in a tree

2. OF THIS SESSION
Overview
Super-fast intro to Hadoop, Pig
Why MongoDB + Pig?
Demo: Move data MongoDB <=> Pig
Demo: processing data with Pig

3. SUPER-FAST INTRO
Hadoop
From Google research
Built for massive parallelization
Batch (for now)
Widely applicable

4. SUPER-FAST INTRO
Hadoop

5. Social Graph

6. Predict

7. Detect

8. Genetics

9. SUPER-FAST INTRO
Hadoop

10. ON HADOOP
Pig
Less code
Expressive code
Compiles to MR
Insulates from API
Popular
(LinkedIn, Twitter,
Salesforce, Yahoo,
Stanford University...)

11. BRIEF, EXPRESSIVE
LIKE PROCEDURAL SQL
Pig
(thanks: twitter hadoop world presentation)

12. FOR SERIOUS
The Same Script, In MapReduce

13. Alternatives to Hadoop
Write MapReduce in Javascript
• Javascript is not fast
• Has limited data types
• Hard to use complex analytic libs
Adds load to data store
MONGODB NATIVE MAPREDUCE

14. Alternatives to Hadoop
MONGODB AGGREGATION FRAMEWORK
Great when
• Doing SQL-style aggregation
• Do not require external data libs
• Extra load is ok

15. MOTIVATIONS
MongoDB + Pig
Data storage and data processing are often
separate concerns
Hadoop is built for scalable processing of large
datasets

16. SIMILAR PHILOSOPHY
MongoDB, Pig
Poly-structured data
• MongoDB: stores data, regardless of structure
• Pig: reads data, regardless of structure (got its
name because Pigs are omnivorous)

17. Mortar
FAST INTRO
Open-source code-based dev framework for data,
built on Hadoop and Pig
Inspired by Rails
Self-contained, organized, executable projects

18. > gem install mortar
> git clone https://github.com/
mortardata/mongo-pig-examples.git

19. LOAD
MONGO => PIG
Mongo-Hadoop connector
LOAD 'mongodb://
<username>:<password>@<host>:<port>/
<database>.<collection>'
USING com.mongodb.hadoop.pig.MongoLoader();

20. STORE
PIG => MONGO
STORE result
INTO 'mongodb://
<username>:<password>@<host>:<port>/
<database>.<collection>'
USING com.mongodb.hadoop.pig.MongoStorage(
'update [key1, key2, key3]',
'{key1: 1, key2: 1, key3: 1},
{unique:false, dropDups: false}'
);

21. What’s my schema?
GENERATE IT
Pig is schema-optional.
No schema: document#'user'#'name'
With schema: user.name

22. What’s in the collection?
CHARACTERIZE IT
Hadoop-based utility describes your collection
• Field name
• Unique value count
• Example value
• Data type
• Example value count

23. Appendix
LINKS
Reference:
http://help.mortardata.com/reference/
loading_and_storing_data/MongoDB
Mongo-Hadoop connector
https://github.com/mortardata/mongo-hadoop

24. @kky
@mortardata
help.mortardata.com

25. Lunch 1:20 – 2:05
Next Sessions at 2:05
5th Floor:
West Side Ballroom 3&4: How to Keep Your Data Safe in MongoDB
West Side Ballroom 1&2: Geospatial Enhancements in MongoDB 2.4
Juilliard Complex: Business Track: Business Track: How MongoDB Helps
Telefonica Digital Accelerate Time to Market
Lyceum Complex: Ask the Experts: MongoDB Monitoring and Backup
Service Session
7th Floor:
Empire Complex: Real-Time Integration Between MongoDB and SQL
Databases
SoHo Complex: High Performance,Scalable MongoDB in a Bare Metal
Cloud