Building a reliable pipeline of data ingress, batch computation, and data egress with Hadoop can be a major challenge. Most folks start out with cron to manage workflows, but soon discover that doesn't scale past a handful of jobs. There are a number of open-source workflow engines with support for Hadoop, including Azkaban (from LinkedIn), Luigi (from Spotify), and Apache Oozie. Having deployed all three of these systems in production, Joe will talk about what features and qualities are important for a workflow system.
3. Background
⢠Devops/Infra for Hadoop
⢠~4 years with Hadoop
⢠Have done two migrations from EMR to the colo.
⢠Formerly Data/Analytics Infrastructure @
⢠worked with Apache Oozie and Luigi
⢠Before that, Hadoop @
⢠worked with Azkaban 1.0
Disclosure: Iâve contributed to Luigi and Azkaban 1.0
3
13. Hadoop-Driven Features
⢠What happens if thereâs a failure?
⢠possibly OK to skip a day.
⢠WorkďŹow tends to be self-contained, so
you donât need to rerun downstream.
⢠Sanity check your data before pushing to
production.
13
14. WorkďŹow Engine Evolution
⢠Usually start with cron
⢠at 01:00 import data
⢠at 02:00 run really expensive query A
⢠at 03:00 run query B, C, D
⢠...
⢠This goes on until you have ~10 jobs or so.
⢠Itâs hard to debug and rerun.
⢠Doesnât scale to many people.
14
15. WorkďŹow Engine Evolution
⢠Two possibilities:
1. âa workďŹow engine canât be too hard,
letâs write our ownâ
2. spend weeks evaluating all the options
out there.Try to shoehorn your
workďŹow into each one.
15
16. WorkďŹow Engine
Considerations
How do I...
⢠Deploy and Upgrade
⢠workďŹows and the workďŹow engine
⢠Test
⢠Detect Failure
⢠Debug/ďŹnd logs
⢠Rebuild/backďŹll datasets
⢠Load data to/from a RDBMS
⢠Manage a set of similar tasks
16
19. Oozie - the good
⢠Great community support
⢠Integrated with HUE, Cloudera Manager,Apache
Ambari
⢠HCatalog integration
⢠SLA alerts (new in Oozie 4)
⢠Ecosystem support: Pig, Hive, Sqoop, etc.
⢠Very detailed documentation
⢠Launcher jobs as map tasks
19
20. Oozie - the bad
⢠Launcher jobs as map tasks.
⢠UI - but HUE, oozie-web (and
good API)
⢠Confusing object model (bundles,
coordinators, workďŹows) - high
barrier to entry.
⢠Setup - extjs, hadoop proxy user,
RDBMS.
⢠XML!
20
24. Azkaban - the good
⢠Great UI
⢠DAG visualization
⢠Task history
⢠Easy access to log ďŹles
⢠Plugin architecture
⢠Pig, Hive, etc. Also, voldemort âbuild and pushâ integration
⢠SLA Alerting
⢠HDFS Browser
⢠User Authentication/Authorization and auditing.
⢠Reportal: https://github.com/azkaban/azkaban-plugins/pull/6
24
26. Azkaban - the bad
⢠Representing data dependencies
⢠i.e. run job X when datasetY is available.
⢠Executors run on separate workers, can be
under-utilized (YARN anyone?).
⢠Community - mostly just LinkedIn, and they
rewrote it in isolation.
⢠mailing list responsiveness is good.
26
27. Azkaban - good and
bad
⢠Job deďŹnitions as java properties
⢠Web uploads/deploy
⢠Running jobs, scheduling jobs.
⢠nearly impossible to integrate with
conďŹguration management
27
30. Luigi - the good
⢠Task deďŹnitions are code.
⢠Tasks are idempotent.
⢠WorkďŹow deďŹnes data (and task) dependencies.
⢠Growing community.
⢠Easy to hack on the codebase (<6k LoC).
⢠Postgres integration
⢠Foursquare got this working with Redshift and
Vertica.
30
31. Luigi - the bad
⢠Missing some key features, e.g. Pig support
⢠but this is easy to add
⢠Deploy situation is confusing (but easy to
automate)
⢠visualizer scaling
⢠no persistent backing
⢠JVM overhead
31
32. Comparison matrix -
part 1
Lang
Code
Complexity
Frameworks Logs Community Docs
oozie java high - 105k
pig, hive, sqoop,
mapreduce
decentralized,
map tasks
Good - ASF in
many distros
excelle
nt
azkaban java moderate - 26k
pig, hive,
mapreduce
UI-accessible
few users,
responsive on
MLs
good
luigi python simple - 5.9k
hive, postgres,
scalding, python
streaming
decentral-ized
on workers
few users,
responsive on
github and MLs
good
32
33. Comparison matrix -
part 2
property
conďŹguration
Reruns
Customizat
ion (new
job type)
Testing User Auth
oozie
command-line,
properties ďŹle, xml
defaults
oozie job -
rerun
difďŹcult MiniOozie
Kerberos, simple,
custom
azkaban
bundled inside
workďŹow zip, system
defaults
partial
reruns in UI
plugin
architecture
?
xml-based,
custom
luigi
command-line,
python ini ďŹle
remove
output,
idempotency
subclass
luigi.Task
python
unittests
linux-based
33
35. Qualities I like in a
workďŹow engine
⢠scripting language
⢠you end up writing scripts to run your job anyway
⢠custom logic, e.g. representing a dep on 7-days of data or run
only every week
⢠Less property propagation
⢠Idempotency
⢠WYSIWYG
⢠It shouldn't be hard to take my existing job and move it to the
workďŹow engine (it should just work).
⢠Easy to hack on
35
36. Less important
⢠High availability (cold failover with manual
intervention is OK)
⢠Multiple cluster support
⢠Security
36
37. Best Practices
⢠Version datasets
⢠BackďŹlling datasets
⢠Monitor the absence of a job running
⢠Continuous deploy?
37
38. Resources
⢠Azkaban talk at Hadoop User Group:
http://www.youtube.com/watch?
v=rIUlh33uKMU
⢠PyData talk on Luigi: http://vimeo.com/
63435580
⢠Oozie talk at Hadoop user Group: http://
www.slideshare.net/mislam77/oozie-hug-
may12
38