Building applications on YARN involves considering many factors including architecture, operations, deployment, logging, metrics, configuration, security, languages, fault tolerance, isolation, and state management. The anatomy of a YARN application includes a client, application master, container code, and resource manager. The application master communicates with the resource manager and manages containers.
3. What I want to Talk About
Anatomy of a YARN Application
Things to consider when building your application
Architecture
Operations
4. Anatomy of a YARN App
Client
Application Master
Container Code
Resource Manager
Node Manager
5. Anatomy of a YARN App
Client
Client
Client RM
RM
Application Master
Container Code
Resource Manager
NM
NM NM
NM
Node Manager
AM
AM CC
CC
* simplified
6. A lot to consider
Deployment Logging
Metrics Fault Tolerance
Configuration Isolation
Security Dashboard
Language State
10. Configuration
YARN config (yarn-site.xml, core-site.xml, etc)
Application Configuration
Transporting Configuration
What we do: Config is fully resolved at client execution time.
No admin-override/locked config protection yet. Config is
passed from client to AM to containers via environment
variables.
11. Security
Kerberos?
Firewalls are your friend
Gateway machine
Dashboard
What we do: Firewall all YARN machines so they can only
talk to each-other. All users go through LDAP controlled
dashboard.
12. Language
Favor complexity in Application Master, and make
container-logic thin
Talk to RM via REST
Potential to talk to RM via Protobuf RPC
What we do: Application AM is Java. Tasks-side of
application has Python and Java implementations.
13. Logging
Local storage (application is running)
HDFS storage (application has stopped for a while)
Be careful with STDOUT/STDERR (rollover)
What we do: No HDFS. Logs sit for 7 days, then disappear.
Not ideal.
14. Fault Tolerance
Failure matrix
HA RM/NM
Orphaned processes
Pay attention to process trees
What we do: No HA. Manual fail over when RM dies.
Orphaned process monitor (proc start time < RM start time).
20. Takeaways
There’s a lot more than just the YARN API
Look for examples (Spark, Storm, Map-Reduce)
Decide your level of Hadoop integration
Metrics2
HDFS
Config
Kerberos and doAs