In the Hadoop world, writing Map Reduce is always a painful task especially when you want to create complex workflows with multiple Map Reduce jobs and having complex dependencies between them. There are high level languages like Pig latin or Hive which make writing Map Reduce jobs easy. But if you want to write complex logic, you need to write custom functions in Java, which makes testing and debugging difficult. This is where Cascading makes a developer's life easy. Everything is written in Java with ease of writing Map Reduce in high level language, similar to Pig or Hive. Once the logic is written, it can be easily be tested by running in stand-alone mode since everything is in Java. Not only that, Cascading provides Hadoop (or any other framework) independent APIs, which means workflows written in Cascading can be executed on multiple frameworks without any code change as long as Cascading connector is available

1. Building Complex Data Workflows with
Cascading on Hadoop
Gagan Agrawal, Sr. Principal Engineer,
Snapdeal

2. Agenda
•
What is Cascading
•
Why use Cascading
•
Cascading Building Blocks
•
Example Workflows
•
Testing
•
Advantages / Disadvantages
•
Cascading @ Snapdeal

3. What is Cascading ?

4. What is Cascading ?
•
Abstraction over Map Reduce
•
API for data-processing workflows
•
JVM framework and SDK for creating abstracted
data flows
•
Translates data flows into actual Hadoop/Local
jobs

5. Why use Cascading ?

6. Why use Cascading ?
•
Options in Hadoop
– Map Reduce API
– Pig
– Hive

7. Why use Cascading ?
•
Map Reduce is
– Low Level API
– Requires lot of boilerplate code
– Can be difficult to chain jobs

8. Why use Cascading ?
•
Pig
– Good for scripting and ad-hoc queries
– Can be difficult to manage multi-module
workflows
– Can be difficult to debug and unit test
– UDF written in different language (Java,
Python (via streaming) etc.)

9. Why use Cascading ?
•
Hive
– Good for simple queries
– Complex workflows can be very difficult to
implement

10. Why use Cascading ?
•
Cascading
– Helps create higher level data processing
abstractions
– Sophisticated data pipelines
– Re-usable components

11. TF-IDF Example

12. Cascading Building Blocks

13. Cascading Building Blocks
•
Tap
– Source
– Sink
•
Pipes
– Each
– Every
– Merge
– GroupBy
– CoGroup
– HashJoin

14. Cascading Building Blocks
•
Tuple
•
Fields
•
Functions
•
Filter
•
Aggregator
•
Buffer
•
Sub Assemblies

15. Cascading Terminologies
•
Flow
– A path for data with some number of inputs,
some operations, and some outputs
•
Cascade
– A series of connected flows
•
Operation
– A function applied to data, yielding new data
•

16. Cascading Terminologies
•
Pipe
– Moves data from some place to some other
place
•
Tap
– Feeds data from outside the flow into it and
writes data from inside the flow out of it.

17. Pipe Assemblies
•
Define work against a Tuple Stream
•
Read from tap sources and written to tap sinks
•
Work includes actions
– Filtering
– Transforming
– Calculating
•
May use multiple sources and sinks
•
May define splits, merges and joins to
manipulate tuple streams
•

18. Pipe Assemblies

19. Pipe Assemblies

20. Each and Every Pipe
•
new Each( previousPipe, argumentSelector,
operation, outputSelector )
•
new Every( previousPipe, argumentSelector,
operation, outputSelector )

21. Each Pipe

22. Every Pipe

23. Example Workflows

24. Distributed Copy

25. Word Count

26. Custom Function
•
A function expects a stream of individual tuples
•
Returns zero or more tuples
•
Used with Each pipe
•
To create custom function
– Sublcass cascading.operation.BaseOperation
– Implement cascading.operation.Function

27. Custom Function

28. Joins
•
CoGroup
•
HashJoin

29. Joins

30. TF-IDF Example

31. Cascade

32. Cascade
●
CascadeConnector connector = new
CascadeConnector();
●
Cascade cascade = connector.connect( flowFirst,
flowSecond, flowThird );

33. Testing

34. Testing
•
Create flows entirely in code on a local machine
•
Write tests for controlled sample data sets
•
Run tests as a regular old Java without needing
access to actual Hadoop or databases
•
Local machine and CI testing are easy

35. Testing

36. Advantages / Disadvantages

37. Advantages
•
Re-usability
– Pipe assemblies are designed for reuse
– Once created and tested, use them in other
flows
– Write logic to do something only once
•
Simple Stack
– Cascading creates DAG of dependent jobs
for us
– Removes most of the need for oozie

38. Advantages
•
Simpler Stack
– Keeps track of where a flow fails and can
rerun from that point on failure
•
Common Code Base
– Since everything is written in java, everybody
can use same terms and same tech

39. Disadvantages
•
JVM Based
– Java / Scala / Clojure
•
Doesn't have job scheduler
– Can figure out dependency graph for jobs,
but nothing to run them on regular interval
– We still need scheduler like quartz
•
No real built-in monitoring

40. Cascading @ Snapdeal