The talk will compare Cascalog, fully-featured data processing and querying library on top of Hadoop, and Sparkling – A Clojure API for Apache Spark. How both of these compare in terms of performance and code complexity for Big Data processing and why you shouldn’t be writing MapReduce jobs in plain Hadoop API.

1. ETL in ClojureETL in Clojure
Dmitriy Morozov / JEEConf 2015

2. Dmitriy MorozovDmitriy Morozov
Software engineer at
Functional programming junky
Occasional cyclist
Zoomdata.com
@argc

3. Plan of attackPlan of attack
ETL at ZoomdataETL at Zoomdata
CascalogCascalog
SparkSpark
DemoDemo
ConclusionConclusion

4. Is a modern BI application focused onIs a modern BI application focused on
allowing everyday business users toallowing everyday business users to
be able to visually interact andbe able to visually interact and
explore their data and discoverexplore their data and discover
insight out of that data.insight out of that data.

5. What we do at ZoomdataWhat we do at Zoomdata

6. What we do at ZoomdataWhat we do at Zoomdata

8. We did ETL inWe did ETL in
Hive/ImpalaHive/Impala

9. Using SQL for ETLUsing SQL for ETL
Hive is slow, and so is Hive on Tez
SQL is horrible for doing anything complicated
Code is hard to maintain, reuse and test
Lessons learnedLessons learned

10. Why Clojure?Why Clojure?
Functional!
Runs on JVM
Interactive development
Zero delta between prototyp code and
production code

11. CascalogCascalog
Datalog DSL in CLojure
Built on top of Hadoop and Cascading
Query compiles to Hadoop MapReduce jobs
Supports local execution for prototyping
Great testing story

12. DatalogDatalog
language
Syntactically is a subset of Prolog
It is often used as a for
deductive databases.
Query statements can be stated in any order
Logic programming
query language

13. DatalogDatalog

14. Word Count using Hadoop API

15. Word count in CascalogWord count in Cascalog

16. Cascalog Query StructureCascalog Query Structure

17. Cascalog / GeneratorsCascalog / Generators

18. Cascalog / OperationsCascalog / Operations

19. Cascalog / OperationsCascalog / Operations

20. Cascalog / JoinsCascalog / Joins

21. Cascalog / OperationsCascalog / Operations

22. Cascalog / AggregatorsCascalog / Aggregators

23. Cascalog / AggregatorsCascalog / Aggregators

24. Cascalog / TroubleshootingCascalog / Troubleshooting

25. Cascalog / TestingCascalog / Testing

26. Cascalog / TroubleshootingCascalog / Troubleshooting

27. Flow Visualisation /Flow Visualisation / DOTDOT

28. Flow Visualisation /Flow Visualisation / DrivenDriven

29. DEMODEMO

30. Cascalog DownsidesCascalog Downsides
Hadoop < SparkHadoop < Spark **

31. Cascalog DownsidesCascalog Downsides
No supportNo support
for streamingfor streaming
datadata

32. Cascalog DownsidesCascalog Downsides

33. What are the alternatives?What are the alternatives?
Java API forJava API for
FlamboFlambo
SparklingSparkling
SparkSpark

34. Customer XCustomer X

35. Customer X wants to do DataCustomer X wants to do Data
Science!Science!

36. Drug PersistenceDrug Persistence
Determining whether a patient isDetermining whether a patient is
persistent or not based on whether shepersistent or not based on whether she
refilled the prescription in time.refilled the prescription in time.

37. Drug PersistenceDrug Persistence

38. Drug PersistenceDrug Persistence

39. Drug PersistenceDrug Persistence

40. Drug PersistenceDrug Persistence

46. Example: Drug PersistenceExample: Drug Persistence

48. Things to check outThings to check out
How Yieldbot does Data science in Clojure
Cascalog for the Impatient
Streaming MapReduce in Clojure
Sparkling
Flambo

49. Thank you!Thank you!