1. Pig on Tez
Daniel Dai
@daijy
Rohini Palaniswamy
@rohini_pswamy
H a d o o p S u m m i t 2 0 1 4 , S a n J o s e

2. Agenda
 Team Introduction
 Apache Pig
 Why Pig on Tez?
 Pig on Tez
- Design
- Tez features in Pig
- Performance
- Current status
- Future Plan
2

3. 3
Apache Pig on Tez Team
Daniel Dai
Pig PMC
Hortonworks
Rohini Palaniswamy
Pig PMC
Yahoo!
Olga Natkovich
Pig PMC
Yahoo!
Cheolsoo Park
VP Pig, Pig PMC
Netflix
Mark Wagner
Pig Committer
LinkedIn
Alex Bain
Pig Contributor
LinkedIn

4. Pig Latin
 Procedural scripting language
 Closer to relational algebra
 Heavily used for ETL
 Schema / No schema data, Pig eats everything
 More than SQL and Feature rich
4
Multiquery Nested Foreach Illustrate
Algebraic and Accumulator java
UDFs
Script Embedding Scalars
Macros
non-java UDFs (jython, python,
javascript, groovy, jruby)
Distributed Orderby, Skewed
Join

5. Pig users
 Heavily used for ETL at Web Scale by Major Internet Companies
 At Yahoo!
- 60% of total hadoop jobs run daily
- 12 million monthly pig jobs
 Other heavy users
- Twitter
- Netflix
- LinkedIn
- Ebay
- Salesforce
 Standard data science tool, in university textbook
5

6. Why Pig on Tez?
 DAG execution framework
 Low level DAG framework
- Build DAG by defining vertices and edges
- Customize scheduling of DAG and routing of data
 Highly customizable with pluggable implementations
 Resource efficient
 Performance
- Without having to increase memory
 Natively built on top of YARN
- Multi-tenancy, resource allocation come for free
 Scale
 Security
 Excellent support from Tez community
- Bikas Saha, Siddharth Seth, Hitesh Shah
6

7. PIG on TEZ

8. Design
8
Logical Plan
Tez Plan MR Plan
Physical Plan
Tez Execution Engine MR Execution Engine
LogToPhyTranslationVisitor
MRCompilerTezCompiler

9. DAG Plan – Split Group by + Join
9
f = LOAD ‘foo’ AS (x, y, z);
g1 = GROUP f BY y;
g2 = GROUP f BY z;
j = JOIN g1 BY group,
g2 BY group;
Group by y Group by z
Load foo
Join
Load g1 and Load g2
Group by y Group by z
Load foo
Join
Multiple outputs
Reduce follows
reduce
HDFS HDFS
Split multiplex de-multiplex

10. DAG Execution - Visualization
10
Vertex 1
(Load)
Vertex 2
(Group)
Vertex 3
(Group)
Vertex 4
(Join)
MROutput
MRInput

11. DAG Plan – Distributed Orderby
11
Aggregate
Sample
Sort
Partition
A = LOAD ‘foo’ AS (x, y);
B = FILTER A by $0 is not
null;
C = ORDER f BY x;
Stage sample map
on distributed cache
Load/Filter
& Sample
Aggregate
Partition
Sort
Broadcast sample map
HDFS
HDFS
Load/FilterHDFS
HDFS
Map
Reduce
Map
Reduce
Map
1-1 Unsorted
Edge
Cache sample map

12. Session Reuse
 Feature
- Session reuse
 Submit more than one DAG to same AM
 Usage
- Each Pig script uses a single session
- Grunt shell uses one session for all commands till timeout
- More than one DAG submitted for merge join, ‘exec’
 Benefits
- A pig script with 5 MR jobs has 5 AM containers launched. Single AM for one
pig script in Tez saves capacity.
- Eliminates issue of queue and resource contention faced in MR by every new
MR job in the pipeline of a multi-stage pig script.
12

13. Container Reuse
 Features
- Container reuse
 Rerun new tasks on already launched containers (jvm)
 Usage
- Turned on by default for all pig scripts and grunt shell
 Benefits
- Reduced launch overhead
 Container request and release overhead
 Resource localization overhead
 JVM launch time overhead
- Reduced network IO
 1-1 edge tasks are launched on same node
- Object caching
 User impact
- Have to review/profile and fix custom LoadFunc/StoreFunc/UDFs for static variables
and memory leaks due to jvm reuse.
13

14. Custom Vertex Input/Output/Processor/Manager
 Features
- Custom Vertex Processor
- Custom Input and Output between vertices
- Custom Vertex Manager
 Usage
- PigProcessor instead of MapProcessor and ReduceProcessor
- Unsorted input/output
 with Partitioner – Union
 without Partitioner – Broadcast Edge (Replicate join, Orderby and Skewed join), 1-1
Edge (Order by, Skewed join and Multiquery off)
- Custom Vertex Manager – Automatic Parallelism Estimation
 Benefits
- No framework restrictions like MR
- More efficient processing and algorithms
14

15. Broadcast Edge and Object Caching
 Feature
- Broadcast Edge
 Broadcast same data to all tasks in successor vertices
- Object Caching
 Ability to cache objects in memory for scope of Vertex, DAG and Session
- Input fetch on choice
 Usage
- Replicate join small table
- Orderby and Skewed join partitioning samples
 Benefits
- Replace use of Distributed cache and avoid NodeManager bottleneck of localization
- Avoid input fetching if in cache on container reuse
- Performance gains of upto 3x in tests for replicated join on smaller clusters with
higher container reuse
15

16. Vertex Groups
 Feature
- Vertex Grouping
 Ability to group multiple vertices into one vertex group and produce a combined output
 Usage
- Union operator
 Benefits
- Better performance due to elimination
of an additional vertex
- Performance gains of 1.2x to 2x over MR
16
A = LOAD ‘a’;
B = LOAD ‘b’;
C = UNION A, B;
D = GROUP C by $0;
Load A Load B
GROUP

17. Dynamic Parallelism
 Determine parallelism beforehand is hard
 Dynamic adjust parallelism at runtime
 Tez VertexManagerPlugin
- Custom policy to determine parallelism at runtime
- Library of common policy: ShuffleVertexManager
17

18. Dynamic Parallelism - ShuffleVertexManager
18
Load A
JOIN
Load A
JOIN 4 2
Load B
Load B
 Stock VertexManagerPlugin from Tez
 Used by Group, Hash Join, etc
 Dynamic reduce parallelism of vertex based on estimated input size

19. Dynamic Parallelism – PartitionerDefinedVertexManager
 Custom VertexManagerPlugin Used by Order by / Skewed Join
 Dynamic increase / decrease parallelism based on input size
19
Load/Filter
& Sample
Sample
Aggregate
Partition
Sort
Calculate
#Parallelism

20. PERFORMANCE

21. Performance numbers –
21
0
10
20
30
40
50
60
70
80
Prod script 1
1.5x
1 MR Job
3172 vs 3172 Tasks
Prod script 2
2.1x
12 MR jobs
966 vs 941 Tasks
Prod script 3
1.5x
4 MR jobs on 8.4
TB input
21397 vs 21382
Tasks
Prod script 4
2%
4 MR Jobs on 25.2
TB input
101864 vs 101856
tasks
Timeinmins
MR
Tez
28 vs 18m
11 vs 5m
50 vs 35m
74 vs 72m

22. Performance numbers –
22
0
20
40
60
80
100
120
140
160
Prod script 1
2.52x
5 MR Jobs
Prod script 2
2.02x
5 MR Jobs
Prod script 3
2.22x
12 MR Jobs
Prod script 4
1.75x
15 MR jobs
Timeinmins
MR
Tez
25 vs 10m
34 vs 16m
2h 22m vs 1h 21m
1h 46m vs 48m

23. Lipstick from
23

24. Performance Numbers – Interactive Query
24
0
100
200
300
400
500
600
700
10G 5G 1G 500M
Timeinsecs
Input Size
TPC-H Q10
MR
Tez
2.49X
3.41X
4.89X 6X
 When the input data is small, latency dominates
 Tez significantly reduce latency through session/container reuse

25. Performance Numbers – Iterative Algorithm
25
 Pig can be used to implement iterative algorithm using embedding
 Iterative algorithm is ideal for container reuse
 Example: k-means Algorithm
- Each iteration takes an average 1.48s after the first iteration (vs 27s for MR)
0
1000
2000
3000
10 50 100
Timeinsecs
Iteration
k-means
MR
Tez
14.84X
13.12X
5.37X
* Source code can be downloaded at http://hortonworks.com/blog/new-apache-pig-features-part-2-embedding

26. Performance is proportional to …
 Number of stages in the DAG
- Higher the number of stages in the DAG, performance of Tez over MR will be
better due to elimination of map read stages.
 Size of intermediate output
- More the size of intermediate output, the performance of Tez over MR will be
better due to reduced HDFS usage.
 Cluster/queue capacity
- More congested a queue is, the performance of Tez over MR will be better due
to container reuse.
 Size of data in the job
- For smaller data and more stages, the performance of Tez over MR will be
better as percentage of launch overhead in the total time is high for smaller
jobs.
26

27. CURRENT & FUTURE

28. Where are we?
 90% feature parity with Pig on MR
- No Local mode (TEZ-235)
- Rarely used operators not implemented
 MAPREDUCE (native mapreduce jobs)
 Collected CoGroup
 98% of ~1300 e2e tests pass.
 35% of ~2850 unit tests pass. Porting of rest pending on Tez Local mode.
 Tez branch merged into trunk and will be part of Pig 0.14 release
 Netflix has Lipstick working with Pig on Tez
- Credits: Jacob Perkins, Cheolsoo Park
28

29. User Impact
 Tez
- Zero pain deployment
- Tez library installation on local disk and copy to HDFS
 Pig
- No pain migration from Pig on MR to Pig on Tez
 Existing scripts work as is without any modification
 Only two additional steps to execute in Tez mode
– export TEZ_HOME=/tez-install-location
– pig -x tez myscript.pig
- Users to review/profile and fix custom LoadFunc/StoreFunc/UDFs for static
variables and memory leaks due to jvm reuse.
29

30. What next?
 Support for Tez Local mode
 All unit tests ported
 Improve
- Stability
- Usability
- Debuggability
 Apache Release
- Pig 0.14 with Tez released by Sep 2014
 Deployment
- In research in Yahoo! by early Q3
- In production in Yahoo and Netflix by Q3/Q4
 Performance
- From 1.2-3x to 1.5x-5x by Q4
30

31. Tez Features - WIP
 Tez UI
- Application Master UI and Job history UI is in the works by integrating via
Application Timeline server.
- Currently only AM logs are easily viewable. Task logs are available but have to
grep the AM log to find the URL.
 Tez Local mode
 Tez AM Recovery
- Tez checkpointing and resuming on AM failure is functional but needs more
work. With single DAG execution of whole script, AM retries can be very costly.
 Input fetch optimizations
- Custom ShuffleHandler on NodeManager
- Local input fetch on container reuse
31

32. What next - Performance?
 Shared Edges
- Same output to multiple downstream vertices
 Multiple Vertex Caching
 Unsorted shuffle for skewed join and order by
 Custom edge manager and data routing for skewed join
 Groupby and join using hashing and avoid sorting
 Better memory management
 Dynamic reconfiguration of DAG
- Automatically determine type of join - replicate, skewed or hash join
32

33. We are hiring!!!
Hortonworks
Stop by Kiosk D5
Yahoo!
Stop by Kiosk P9
or reach out to us at
bigdata@yahoo-inc.com.
Thank You