Контроль зверей: инструменты для управления и мониторинга распределенных систем от Cloudera. Александр Козлов, Cloudera Inc.

Managing a Zoo: Tools for Managing and
Monitoring Distributed Systems from
Cloudera
Alex Kozlov, Cloudera Inc.

YaC, Москва, 19 сентября 2011 года

Agenda
• About Cloudera and myself
• Background info
– Data, data everywhere
– Corporate data management, distributed systems, functional languages
– Hadoop ecosystem
• Distributed system maintenance
– Installation/Updating/Monitoring
• Fixed images
• Standard configuration management tools
– Our solution
• Partial failures
• Node cast
• …
• Implementation
• What’s next

2
©2011 Cloudera, Inc. All Rights Reserved.

About Cloudera

Founded in the summer 2008

Cloudera’s mission is to help organizations profit from all of their data.

Cloudera helps organizations profit from all of their data. We deliver the industry-
standard platform which consolidates, stores and processes any kind of data, from
any source, at scale. We make it possible to do more powerful analysis of more
kinds of data, at scale, than ever before. With Cloudera, you get better insight into
their customers, partners, vendors and businesses.

Мы поставляем стандартные платформы для объединения, хранения и
обрабатывания большого количества данных любого типа, от
любого источника. Мы делаем это в масштабе большем чем когда-
либо прежде. С Cloudera, вы получите лучшее понимание своих клиентов,
партнеров, поставщиков и предприятий.

3

Introduction

# whoami # whoru
alexvk
– Sysadmin
– Закончил ФизФак МГУ, – IT Manager
Stanford University – TechOps
– Работал в SGI, HP, Turn – Data Scientist
– Senior Solutions Architect, – Researcher
Cloudera, Inc. – Developer
– CTO?
…
– Just curious?

4

Data, data everywhere

We are storing a lot more data:
– 1 click on an average web-site generates
about 100 lines of logs (somewhere)
– 1 additional attribute/integer (8 bytes) means
1TB/day of data (from an ex-Google
employee)

40-80PB stores are becoming common

5

Corporate data management

• Traditional • Future
– Data from
– EDW, centralized (SPoF) • Any source
• Any kind
• At scale
– Fixed set of queries – Flexible insights
(sales/revenue by quarter, etc.) • The value is not known
beforehand
• Multiple facets of deep,
– ETL pipeline taking up to 24- exhaustive analysis
hours to run – Interactive
• 5 min delay from click to
insights

6

The Origins of Hadoop

Open Source
Open source web MapReduce and Releases Cloudera
Hadoop tops Terabyte Enterprise 3.5 & SCM
crawler project created HDFS project created sort benchmark
by Doug Cutting by Doug Cutting Express

2002 2004 2008 2010 2011 2012

Publishes MapReduce Runs 4,000-node Created Hive, adding Releases CDH3 and
and GFS Paper Hadoop cluster SQL support Cloudera Enterprise

7

What is Hadoop?

HDFS + MapReduce = Hadoop

Hadoop is an ecosystem (HBase
+ friends)
Distributed storage
Moving computations to the data
A new model for fault tolerance

8

CDH Overview
The #1 commercial and non-commercial
Apache Hadoop distribution.

File System Mount UI Framework SDK
FUSE-DFS HUE HUE SDK

Workflow Scheduling Metadata
APACHE OOZIE APACHE OOZIE APACHE HIVE

Languages / Compilers
APACHE PIG, APACHE HIVE

Data Integration Fast Read/Write Access

APACHE FLUME, APACHE SQOOP
APACHE HBASE

Coordination
APACHE ZOOKEEPER

9

Landscape

In 2008 (Cloudera founded) In 2011

– 3-5 companies, mostly in social – 100s of paying clients
networking space, using – 2-3x growth in Hadoop
Hadoop in production conference attendance year-
– A lot of interest, but mostly for over-year
the wrong reason – HBase, Oozie, Mahout
– Biggest applications just smart – Lots of research (Spark,
log processing Mesos, Low latency DFS/MR,
– Largest installation in 10s of PB Graph algorithms)
– Largest installations in 100s of
PB

10

Problem

• Handling large data in distributed systems is
uniquely challenging from an operational
perspective.

• Traditional approaches are valuable, but
insufficient. Domain knowledge is vital.

• Need for support within the frameworks
themselves for operational concerns.

11

Datacenter(s) as a computer
• Existing tools do not generalize well
– Partial failure (how many machines might fail before
the datacenter becomes non-operational? … about
50%)
– Hadoop like metrics (data locality, # of slots, heartbeat
delays)
– Installation and lifecycle management
– Heterogenious nodes

• The ultimate user wants to USE the system, not
CONFIGURE it
– let insight = [ for i in my_smart_algos -> data |> i ]

12

Why not machine images

• Machines have complex state (config,
local data)
– Hard unless the state is trivial
– Images need (rolling) upgrades
– Machines can change (multiple) roles

13

Why not config management tools

• Make assumption of running M services
on N machines, not X services running in
the “cloud”
– Very bad with “partial failures”
– Don’t understand Hadoop specific “state”
– Don’t understand Hadoop specific metrics

14

Our solution
• Managing partial failure
– Cluster is still usable if x% fail (but might have a data loss
if 3 nodes fail at the same time)
– “Running with concerning health”
• Node cast
– Every node can be multiple things (think zoo: it can be a
tiger or a lion or a monkey)
• Finding nodes like one or jobs like one
– Nodes are grouped according to functionality (datanode,
tasktracker, regionserver, namenode, jobtracker)
– Find jobs that are similar to a given one and track outliers
• Drill down for Hadoop-specific diagnostic
– Workflow -> Jobs -> Tasks -> Attempts

15

Details

• Written in Python and Django
• Each node runs “SCM agent”
• Dial-in mode
• Agent does the best effort to make the
prescribed service(s) run
• All state managed by the “server”
• Diagnostic is passed via heartbeats
• Centralized configuration management

16

Services

17

Services (partial failure)

18

New service

19

SCM node selection

20

Visualising drill-down

21

Job matching
• Requires that we build up a rich model of job
performance over time

• Surprisingly subtle problem - how do we know
when two jobs are the same?

• Periodic jobs offer more clues - time of day,
submitting user, map class, reduce class.

• Query jobs are more difficult
– For e.g. Hive, query string analysis can tell us
something

22

Job matching

23

Diagnosing job performance

• Ok, your job really is slow. What now?

• Major cause of slowness, as seen by customers, is skew

• Two predominant types of skew
– Environmental skew, when identical tasks run differently depending
on where they run. Breaks MR notion of homogeneity, causes
severe slowdown.

– Workload skew, when supposedly identical tasks have vastly
differing amounts of work to do,

24

Visualising skew

25

What’s next

• Cloudera Enterprise 3.5 & Hadoop
express (June 2011, SCM & SCM
Express)
• Cloudera Enterprise 3.7 on the way

26

Questions?

Do not hesitate to email me alexvk at cloudera dot com

27

Контроль зверей: инструменты для управления и мониторинга распределенных систем от Cloudera. Александр Козлов, Cloudera Inc.

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (9)

Semelhante a Контроль зверей: инструменты для управления и мониторинга распределенных систем от Cloudera. Александр Козлов, Cloudera Inc.

Semelhante a Контроль зверей: инструменты для управления и мониторинга распределенных систем от Cloudera. Александр Козлов, Cloudera Inc. (20)

Mais de yaevents

Mais de yaevents (20)

Último

Último (20)

Контроль зверей: инструменты для управления и мониторинга распределенных систем от Cloudera. Александр Козлов, Cloudera Inc.