Spark is a powerful framework for distributed processing of massive datasets. With an interactive shell, machine learning libraries, and in-memory data structures, Spark provides a tool set for high performance advanced analytics. Connecting Spark with MongoDB enables you to achieve sophisticated back-end analytics in combination with the performance of MongoDB. We'll take a look at how these two systems integrate with one another through sample code and demonstrations. Presentation from Bryan Reneiro, Developer Advocate at MongoDB.

1. MongoDB + Spark
@blimpyacht

2. Level Setting

6. TROUGH OF
DISILLUSIONMENT

7. HDFS
Distributed Data

8. HDFS
YARN
Distributed Resources

9. HDFS
YARN
MapReduce
Distributed Processing

10. HDFS
YARN
Hive
Pig
Domain Specific Languages
MapReduce

11. Interactive Shell
Easy (-er)
Caching

12. HDFS
Distributed Data

13. HDFS
YARN
Distributed Resources

14. HDFS
YARN
SparkHadoop
Distributed Processing

15. HDFS
YARN
SparkHadoop
Mesos

16. HDFS
Stand
Alone
YARN
SparkHadoop
Mesos

17. HDFS
Stand
Alone
YARN
SparkHadoop
Mesos
Hive
Pig

18. HDFS
Stand
Alone
YARN
SparkHadoop
Mesos
Hive
Pig
Spark
Shell

19. HDFS
Stand
Alone
YARN
SparkHadoop
Mesos
Hive
Pig
Spark
Shell
Spark
Streaming

20. HDFS
Stand
Alone
YARN
SparkHadoop
Mesos
Hive
Pig
Spark
SQL
Spark
Shell
Spark
Streaming

21. HDFS
Stand
Alone
YARN
SparkHadoop
Mesos
Hive
Pig
Spark
SQL
Spark
Shell
Spark
Streaming

22. HDFS
Stand
Alone
YARN
SparkHadoop
Mesos
Hive
Pig
Spark
SQL
Spark
Shell
Spark
Streaming

23. Stand
Alone
YARN
SparkHadoop
Mesos
Hive
Pig
Spark
SQL
Spark
Shell
Spark
Streaming

24. Spark
Streaming
Hive
Spark
Shell
Mesos
Hadoop
Pig
Spark
SQL
Spark
Stand
Alone
YARN

25. Stand
Alone
YARN
Spark
Mesos
Spark
SQL
Spark
Shell
Spark
Streaming

26. Stand
Alone
YARN
Spark
Mesos
Spark
SQL
Spark
Shell
Spark
Streaming

27. executor
Worker
Node
executor
Worker
Node
Driver
Resilient Distributed Datasets

28. Parallelization
Parellelize = x

29. Transformation
s
Parellelize = x t(x) = x’ t(x’) = x’’

30. Transformations
filter( func )
union( func )
intersection( set )
distinct( n )
map( function )

31. Action
f(x’’) = yParellelize = x t(x) = x’ t(x’) = x’’

32. Actions
collect()
count()
first()
take( n )
reduce( function )

33. Lineage
f(x’’) = yParellelize = x t(x) = x’ t(x’) = x’’

34. Transform Transform ActionParallelize
Lineage

35. Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Lineage

36. Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Lineage

37. Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Lineage

38. https://github.com/mongodb/mongo-hadoop

39. {
"_id" : ObjectId("4f16fc97d1e2d32371003e27"),
"body" : "the scrimmage is still up in the air.
"subFolder" : "notes_inbox",
"mailbox" : "bass-e",
"filename" : "450.",
"headers" : {
"X-cc" : "",
"From" : "michael.simmons@enron.com",
"Subject" : "Re: Plays and other information",
"X-Folder" : "Eric_Bass_Dec2000Notes
FoldersNotes inbox",
"Content-Transfer-Encoding" : "7bit",
"X-bcc" : "",
"To" : "eric.bass@enron.com",
"X-Origin" : "Bass-E",
"X-FileName" : "ebass.nsf",
"X-From" : "Michael Simmons",
"Date" : "Tue, 14 Nov 2000 08:22:00 -0800 (PST)",
"X-To" : "Eric Bass",
"Message-ID" :
"<6884142.1075854677416.JavaMail.evans@thyme>",
"Content-Type" : "text/plain; charset=us-ascii",
"Mime-Version" : "1.0"
}
}

40. {
"_id" : ObjectId("4f16fc97d1e2d32371003e27"),
"body" : "the scrimmage is still up in the air.
"subFolder" : "notes_inbox",
"lfpwoojjf0wig=-i1qf=q0qif0=i38 -00 1-8" : "bass-e",
"filename" : "450.",
"headers" : {
"X-cc" : "",
"From" : "michael.simmons@enron.com",
"Subject" : "Re: Plays and other information",
"X-Folder" : "Eric_Bass_Dec2000Notes
FoldersNotes inbox",
"Content-Transfer-Encoding" : "7bit",
"X-bcc" : "",
"To" : "eric.bass@enron.com",
"X-Origin" : "Bass-E",
"X-FileName" : "ebass.nsf",
"X-From" : "Michael Simmons",
"Date" : "Tue, 14 Nov 2000 08:22:00 -0800 (PST)",
"X-To" : "Eric Bass",
"Message-ID" :
"<6884142.1075854677416.JavaMail.evans@thyme>",
"Content-Type" : "text/plain; charset=us-ascii",
"Mime-Version" : "1.0"
}

41. {
_id : "gretchen.hardeway@enron.com|shirley.crenshaw@enron.com",
value : 2
}
{
_id : "kmccomb@austin-mccomb.com|brian@enron.com",
value : 2
}
{
_id : "sally.beck@enron.com|sandy.stone@enron.com",
value : 2
}

42. Eratosthenes
Democritus
Hypatia
Shemp
Euripides

43. Spark Configuration
Configuration conf = new Configuration();
conf.set(
"mongo.job.input.format",
"com.mongodb.hadoop.MongoInputFormat”
);
conf.set(
"mongo.input.uri",
"mongodb://localhost:27017/db.collection”
);

44. Spark Context
JavaPairRDD<Object, BSONObject> documents =
context.newAPIHadoopRDD(
conf,
MongoInputFormat.class,
Object.class,
BSONObject.class
);

45. Spark Context
JavaPairRDD<Object, BSONObject> documents =
context.newAPIHadoopRDD(
conf,
MongoInputFormat.class,
Object.class,
BSONObject.class
);

46. Spark Context
JavaPairRDD<Object, BSONObject> documents =
context.newAPIHadoopRDD(
conf,
MongoInputFormat.class,
Object.class,
BSONObject.class
);

47. Spark Context
JavaPairRDD<Object, BSONObject> documents =
context.newAPIHadoopRDD(
conf,
MongoInputFormat.class,
Object.class,
BSONObject.class
);

48. Spark Context
JavaPairRDD<Object, BSONObject> documents =
context.newAPIHadoopRDD(
conf,
MongoInputFormat.class,
Object.class,
BSONObject.class
);

49. mongos mongos
Data
Services

50. Deployment Artifacts
Hadoop
Connector Jar
Fat Jar
Java Driver Jar

51. Spark Submit
/usr/local/spark-1.5.1/bin/spark-submit
--class com.mongodb.spark.examples.DataframeExample
--master local Examples-1.0-SNAPSHOT.jar

52. Stand
Alone
YARN
Spark
Mesos
Spark
SQL
Spark
Shell
Spark
Streaming

53. JavaRDD<Message> messages = documents.map (
new Function<Tuple2<Object, BSONObject>, Message>() {
public Message call(Tuple2<Object, BSONObject> tuple)
{
BSONObject header =
(BSONObject)tuple._2.get("headers");
Message m = new Message();
m.setTo( (String) header.get("To") );
m.setX_From( (String) header.get("From") );
m.setMessage_ID( (String) header.get( "Message-ID" ) );
m.setBody( (String) tuple._2.get( "body" ) );
return m;
}
}
);

54. MognoDB & Spack
code demo

55. THE FUTURE
AND
BEYOND THE INFINITE

56. Stand
Alone
YAR
N
Spark
Meso
s
Spark
SQL
Spark
Shell
Spark
Streaming

60. MongoDB + Spark

61. THANKS!
{
name: ‘Bryan Reinero’,
role: ‘Developer Advocate’,
twitter: ‘@blimpyacht’,
email: ‘bryan@mongodb.com’
}