YARN: Hadoop Beyond MapReduce - Andreas Neumann

YARN: HADOOP
BEYOND MAPREDUCE
Andreas Neumann @anew68
TerenceYim @chtyim

Andreas Neumann, Terence Yim
IN THIS TALK, YOU WILL
• Learn about Hadoop
• Learn about other distributed compute paradigms
• Learn about YARN and how it works
• Learn how to get more value out of your Hadoop cluster
• Learn about Weave

A WEB-APP
Database
Presentation
(stateless)
Business Logic
(stateless)

A SCALABLE WEB-APP
Database
Presentation
(stateless)
Business Logic
(stateless)

A LOGGING WEB-APP
Presentation
(stateless)
Business Logic
(stateless) log log log
Database

WHAT TO DO WITH MY LOGS?
Analyzing web logs can create real business value
• Optimize for observed user behavior
• Personalize each user’s experience
• ...
Many web logs are simply being discarded
• Too expensive?
• Too Hard to analyze?
• Too much heavy lifting?

WHAT TO DO WITH MY LOGS?

APACHE
• Open source and free
• Runs on commodity hardware - cheap to store
• Programming Paradigm: Map/Reduce (Google 2004)
• Batch processing of very large ﬁles in a distributed ﬁle system
• Simple yet powerful
• Turn your web app into a data-driven app

A MAP/REDUCE APP
Reducers
Mappers
(shufﬂe)
split split split
part part part

MAP/REDUCE CLUSTER
split split split
part part part
split split split
part part part
split split
part part
split split
part part

I HAVE HADOOP - WHAT NEXT?
What if:
• Map/Reduce is not the only thing I want to run?
• I would like to do real-time analysis?
• Or run a message-passing algorithm over my data?
• My cluster has more compute capacity than my big data needs?
Can I utilize its idle resources for other, compute-intensive tasks?

A MESSAGE PASSING (MPI) APP
data
data data
data data
data

A STREAMING APP
Database
events

A DISTRIBUTED LOAD TEST
test testtest
test test test
test test test
test
test
test
Web
Service

HADOOP AT CONTINUUITY
Continuuity AppFabric
• Developer centric big data app platform
• Runs many different types of jobs in Hadoop cluster
• Real-time stream processing
• Ad-hoc OLAP queries
• Map/Reduce

A MULTI-PURPOSE CLUSTER
log log log
Database
spli
t spli spli
par par par
data
data data
data data
data
test testtest
test test test
test test test
test
test
test
Web
Service
Database
events

THE ANSWER: YARN
• The Resource Manager of Hadoop 2.0
• Separates resource management from programming paradigm
• Allows (almost) all kinds of distributed application in Hadoop cluster
• Multi-tenant, secure
• Scales to many thousands of nodes
• Pluggable scheduling algorithms

YARN APPLICATION MASTER
log log log
Database
spli
t spli spli
par par par
data
data data
data data
data
test testtest
test test test
test test test
test
test
test
Web
Service
Database
events
AM AM AM
AM
AM
YARN
Resource
Manager

YARN - HOW IT WORKS
• Every node (machine) in the cluster has several slots.
• A node manager runs on each node to manages the node’s slots.
• The resource manager assigns the slots of the cluster to
applications.
• Applications must have a master.
• The master requests slots from the resource manager and starts
tasks.

YARN - HOW IT WORKS
Protocols
1) Client-RM: Submit the app master
2) RM-NM: Start the app master
3) AM-RM: Request + release containers
4) RM-NM: Start tasks in containers
YARN
Resource
Manager
Node Mgr
AM
Node Mgr
Node MgrNode Mgr
Task Task
Task
Task
TaskTask
YARN
Client
1)
2)
3)
4)

BACKGROUND: HADOOP+HDFS
Local
file system
Local
file system
. . .
Name
Node
HDFS distributed file system
Data
Node
• Every node contributes part of its
local file system to HDFS.
• Tasks can only depend on the
local file system
(JVM class path does not
understand HDFS protocol)
Data
Node

YARN
Resource
Manager AM
YARN
Client
Local
file system
Local
file system
. . .
Name
Node
Local
file system
AM.jar
Node
Mgr
Node
Mgr
Data
Node
Data
Node
X

YARN
Resource
Manager AM
YARN
Client
Local
ﬁle system
. . .
Name
Node
Local
ﬁle system
AM.jar
1) copy
to HDFS
AM.jar
2) copy
to local
AM.jar
3)
Node
Mgr
Node
Mgr
Data
Node
Data
Node

WRITING THE YARN CLIENT
1) Connect to the Resource Manager.
2) Request a new application ID.
3) Create a submission context and a container launch context.
4) Define the local resources for the AM.
5) Define the environment for the AM.
6) Define the command to run for the AM.
7) Define the resource limits for the AM.
8) Submit the request to start the app master.

Connect to the Resource Manager:
YarnConfiguration yarnConf = new YarnConfiguration(conf);
InetSocketAddress rmAddress =
NetUtils.createSocketAddr(yarnConf.get(
YarnConfiguration.RM_ADDRESS,
YarnConfiguration.DEFAULT_RM_ADDRESS));
LOG.info("Connecting to ResourceManager at " + rmAddress);
configuration rmServerConf = new Configuration(conf);
rmServerConf.setClass(
YarnConfiguration.YARN_SECURITY_INFO,
ClientRMSecurityInfo.class, SecurityInfo.class);
ClientRMProtocol resourceManager = ((ClientRMProtocol) rpc.getProxy(
ClientRMProtocol.class, rmAddress, appsManagerServerConf));

Request an application ID, create a submission context and a launch
context:
GetNewApplicationRequest request =
Records.newRecord(GetNewApplicationRequest.class);
GetNewApplicationResponse response =
resourceManager.getNewApplication(request);
LOG.info("Got new ApplicationId=" + response.getApplicationId());
ApplicationSubmissionContext appContext =
Records.newRecord(ApplicationSubmissionContext.class);
appContext.setApplicationId(appId);
appContext.setApplicationName(appName);
ContainerLaunchContext amContainer =

Deﬁne the local resources:
Map<String, LocalResource> localResources = Maps.newHashMap();
// assume the AM jar is here:
Path jarPath; // <- known path to jar file
// Create a resource with location, time stamp and file length
LocalResource amJarRsrc = Records.newRecord(LocalResource.class);
amJarRsrc.setType(LocalResourceType.FILE);
amJarRsrc.setResource(ConverterUtils.getYarnUrlFromPath(jarPath));
FileStatus jarStatus = fs.getFileStatus(jarPath);
amJarRsrc.setTimestamp(jarStatus.getModificationTime());
amJarRsrc.setSize(jarStatus.getLen());
localResources.put("AppMaster.jar", amJarRsrc);
amContainer.setLocalResources(localResources);

Deﬁne the environment:
// Set up the environment needed for the launch context
Map<String, String> env = new HashMap<String, String>();
// Setup the classpath needed.
// Assuming our classes are available as local resources in the
// working directory, we need to append "." to the path.
String classPathEnv = "$CLASSPATH:./*:";
env.put("CLASSPATH", classPathEnv);
// setup more environment
env.put(...);
amContainer.setEnvironment(env);

Deﬁne the command to run for the AM:
// Construct the command to be executed on the launched container
String command =
"${JAVA_HOME}" + /bin/java" +
" MyAppMaster" +
" arg1 arg2 arg3" +
" 1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout" +
" 2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr";
List<String> commands = new ArrayList<String>();
commands.add(command);
// Set the commands into the container spec
amContainer.setCommands(commands);

Deﬁne the resource limits for the AM:
// Define the resource requirements for the container.
// For now, YARN only supports memory constraints.
// If the process takes more memory, it is killed by the framework.
Resource capability = Records.newRecord(Resource.class);
capability.setMemory(amMemory);
amContainer.setResource(capability);
// Set the container launch content into the submission context
appContext.setAMContainerSpec(amContainer);

Submit the request to start the app master:
// Create the request to send to the Resource Manager
SubmitApplicationRequest appRequest =
Records.newRecord(SubmitApplicationRequest.class);
appRequest.setApplicationSubmissionContext(appContext);
// Submit the application to the ApplicationsManager
resourceManager.submitApplication(appRequest);

YARN: APPLICATION MASTER
YARN
Resource
Manager AM
YARN
Client
Local
ﬁle system
Local
ﬁle system
. . .
Name
Node
Node
Mgr
Node
Mgr
Data
Node
Data
Node

YARN - APPLICATION MASTER
• Now we have an Application Master running in a container.
• Next, the master starts the application’s tasks in containers
• For each task, similar procedure as starting the master

YARN: APPLICATION MASTER
YARN
Resource
Manager AM
YARN
Client
Local
ﬁle system
Local
ﬁle system
. . .
Name
Node
Node
Mgr
Node
Mgr
Data
Node
Data
Node
Task
TaskTask

1) Connect with Resource Manager and register itself.
2) Loop:
• Send request to Resource Manager, to:
a) Send heartbeat
b) Request containers
c) Release containers
• Receive Response from Resource Manager
a) Receive containers
b) Receive notiﬁcation of containers that terminated
• For each container received, send request to Node Manager to start task.

• Master should terminate after all containers have terminated.
• If master crashes (or fails to send heart beat), all containers are killed.
• In the future, YARN will support restart of the master.
• Starting a task in a container
• Very similar to YARN client starting the master.
• Node manager starts and monitors task.
• If task crashes, Node Manager informs Resource Manager
• Resource Manager informs master in next response
• Master must still release the container.

• Resource Manager assigns containers asynchronously:
• Requested containers are returned at the earliest in the next
call.
• Master must send empty requests until all it receives all
containers.
• Subsequent requests are incremental and can ask for additional
containers.
• Master must keep track of what it has requested and received.

YARN - APPLICATION LOGS
YARN
Resource
Manager AM
Task
TaskTask
YARN
Client
Local
ﬁle system
. . .
Name
Node
2) copy
to HDFS
logs
4) log access through Name Node
logs
1) log4j,
stdout,
stderr
3)
query
RM for
logs
location
Node
Mgr
Node
Mgr
Data
Node
Data
Node

EXAMPLE - DISTRIBUTED SHELL
• Example application in the YARN/Hadoop distribution
• Runs a shell command in a number of containers
• hadoop-2.0.0-alpha: 1681 lines of code
• some simpliﬁcation in 2.0.3-alpa: 1543 lines
• Conclusion: YARN is complex!

YARN IS NOT SIMPLE!

YARN - SHORTCOMINGS
• Complexity
• Protocols are at very low level and very verbose.
• Client must prepare all dependent jars on HDFS.
• Client must set up environment, class path etc.
• Logs are only collected after application terminates
• What about long-running apps?
• Applications don’t survive master crash: SPOF!
• No built-in communication between containers and master.
• Hard to debug

TYPICAL YARN APP
• Many distributed applications are simple.
• For example: Distributed load test for a queuing system:
• “Run n producers and m consumers for k seconds”
• Neither producer nor consumer know that they run in YARN
• This would be easy if we run them as threads in a single JVM
• Can we make YARN as simple as Java Threads?

TYPICAL NEEDS ...
... of a distributed application:
• Monitor and restart containers
• Collect logs and metrics
• Elastic scale (dynamic adding/removing of instances)
• Long-running jobs

WEAVE IT!

HERE COMES WEAVE
• Open-Source, Apache License
• Does not replace YARN, but complements it
• Generic App Master
+ a DSL for specifying applications
+ an API for writing container tasks
+ built-in service discovery, logging, metrics, monitoring

SIMPLICITY
140. public class ApplicationMaster {
141.
.
150. private AMRMClientAsync resourceManager;
.
435. public boolean run() throws YarnRemoteException {
436. LOG.info(“Starting ApplicationMaster”);
.
498. }
.
500. public void finish() {
504. for(Thread launchThread: launchThreads) {
. ...
511. }
517. FinalApplicationStatus appStatus;
518. String appMessage = null;
538. }
.
790. public ContainerRequest setupContainerAskForRM(int numContainers) {
.
800. Resource capability = Records.newRecord(Resource.class);
801. capability.setMemory(containerMemory);
.
803. ContainerRequest request = new ContainerRequest(capability, ..);
807. }
808. }
.
.
1102. public class Client extends YarnClientImpl {
.
1336. public boolean run() throws IOException {
.
1354. GetNewApplicationResponse newApp = super.getNewApplication();
1355. ApplicationId appId = newApp.getApplicationId();
.
1583. return monitorApplication(appId);
1585. }
.
1594. private boolean monitorApplication(ApplicationId appId) throws ... {
.
1596. while(true) {
.
1606. ApplicationReport report = super.getApplicationReport(appId);
.
1647. }
1648. }
.
1657. private void forceKillApplication(ApplicationId appId) throws ... {
.
1664. super.killApplication(appId);
.
1665. }
.
1667. }
001. public class DistributedShell extends AbstractWeaveRunnable {
002. static Logger LOG = LoggerFactory.getLogger(DistributedShell.class);
003. private String command;
004.
005. public DistributedShell(String command) {
006. this.command = command;
007. }
008.
009. @Override
010. public WeaveRunnableSpecification configure() {
011. return WeaveRunnableSpecification.builder().with()
012. .setName("Executor " + command)
013. .withArguments(ImmutableMap.of("cmd", this.command))
014. .build();
015. }
016.
017. @Override
018. public void initialize(WeaveContext context) {
019. this.command = context.getSpecification().getArguments("cmd");
020. }
021.
022. @Override
023. public void run() {
024. try {
025. Process process = Runtime.getRuntime().exec(this.command);
026. InputStream stdin = proc.getInputStream();
027. BufferedReader br = new BufferedReader(new InputStreamReader(stdin));
028.
029. String line;
030. while( (line = br.readline()) != null) {
031. LOG.info(line);
032. }
033. } catch (Throwable t) {
034. LOG.error(t.getMessage());
035. }
036. }
037. }
038.
039. public static main(String[] args) {
040. String zkStr = args[0];
041. String command = args[1];
042. yarnConfig = ...
043. WeaveRunnerService weaveService = new YarnWeaveRunnerService(yarnConfig, zkStr);
044. weaveService.startAndWait();
045. WeaveController controller = weaveService.prepare(new DistributedShell(command))
046. .addLogHandler(new PrinterLogHandler(new PrintWriter(System.out)))
047. .start();
048. }
Distributed Shell - Vanilla YARN Distributed Shell - WEAVE
~ 500 lines ~ 50 lines

WEAVE
YARN
Resource
Manager
Node Mgr Node Mgr
Node MgrNode Mgr
Task
Weave
Client
...
AM+
Kafka
Task
Task
Task
Task Task
Weave
Runnable
Weave
Runner
Weave
Runnable
log
ZooKeeper
ZooKeeper
ZooKeeper
state +
messages
log stream
This is the only
programming
interface you need

RUNNABLE
Deﬁne a task:
public class EchoServer implements WeaveRunnable
{
static Logger LOG = ...
private WeaveContext context;
//....
@Override
public void run() {
ServerSocket serverSocket = new ServerSocket(0);
context.announce("echo", serverSocket.getLocalPort());
while (isRunning()) {
Socket socket = serverSocket.accept();
//...
}
}
}
WeaveRunnable = single task
SLF4J can be used for logging
- collected via Kafka
- available to client
WeaveRunnable can be run in
- Thread
- YARN container
Service endpoint announcement

RUNNER
Start a task:
WeaveRunner runner = ...;
WeaveController controller =
runner.prepare(new EchoServer())
.addLogHandler(new PrinterLogHandler(
new PrintWriter(System.out)
)
).start();
//...
controller.sendCommand(
Commands.create("stat", "incoming"));
//...
controller.stop();
WeaveRunner
- packages and its dependencies
- plus speciﬁed resources
- attaches log collector to all containers
WeaveController is used to manage
the lifecycle of a runnable or application
Controller also provides ability to
send commands to running containers

APPLICATION
Deﬁne an application:
public class WebServer implements WeaveApplication {
@Override
public WeaveSpecification configure() {
return WeaveSpecification.Builder.with()
.setName(“Jetty WebServer”)
.withRunnables()
.add(new JettyWebServer())
.withLocalFiles()
.add(“www-html”, new File(“/var/html”))
.apply()
.add(new LogsCollector())
.anyOrder()
.build();
}
}
WeaveController controller =
runner.prepare(new WebServer()).start();
WeaveApplication = collection of tasks
Application is given by
a WeaveSpeciﬁcation
Application can specify additional local
directories to be made available.
Weave starts containers in no order.
WeaveRunner can also start
an application

APPLICATION
Deﬁne an application with order:
public class DataApplication implements WeaveApplication
{
@Override
public WeaveSpecification configure() {
return WeaveSpecification.Builder.with()
.setName(“Cool Data Application”)
.withRunnables()
.add(“reader”, new InputReader())
.add(“processor”, new Processor())
.add(“writer”, new OutputWriter())
.order()
.first(“reader”)
.next(“processor”)
.next(“writer”)
.build();
}
}
starts containers in given order.

SUMMARY
• Web apps generate data.
• Data driven apps with Hadoop.
• More value out of Hadoop with YARN.
• Easier with Weave.
• Now every Java developer can write distributed apps.

THANK YOU
https://github.com/continuuity/weave
http://hadoop.apache.org/docs/current/hadoop-yarn/
hadoop-yarn-site/YARN.html
Yes, we are hiring:
careers@continuuity.com

YARN: Hadoop Beyond MapReduce - Andreas Neumann

Recomendados

Recomendados

Mais conteúdo relacionado

Último

Último (20)

Destaque

Destaque (20)

YARN: Hadoop Beyond MapReduce - Andreas Neumann