Hadoop has become a backbone of many enterprises. While it can do wonders for businesses, it sometimes can be overwhelming for its operators and users. Amateurs as well as seasoned operators of Hadoop are caught unaware by common pitfalls of deploying, tuning and operating a Hadoop cluster. Having spent 5+ years working with 100s of Hadoop users, running clusters with 1000s of nodes, managing 10s of petabytes of data and running 100s of 1000s of tasks per day, we have seen people's unintentional acts, suboptimal configurations and common mistakes have resulted into downtimes, SLA violations, many hours of recovery operations and in some cases even data loss! Most of these traumas could have been easily avoided by applying easy to follow best practices that would protect data and optimize performance. In this talk we present real life stories, common pitfalls and most importantly, strategies on how to correctly deploy and manage Hadoop clusters. The talk will empower users and help make their Hadoop journey more fulfilling and rewarding. We will also discuss SmartSense. SmartSense can identify latent problems in a cluster and provide recommendations so that an operator can fix them before they manifest as a service degradation or outage.

1. 1
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Keep your Hadoop
cluster at its best!
Chris Nauroth
Sheetal Dolas
Hadoop Summit, San Jose, 2016

2. 2
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
About Us
⬢ Principal Engineer @ Hortonworks
⬢ Committer and PMC, Apache Hadoop
– Key
contributor
to
HDFS
ACLs,
Windows
compaJbility,
and
operability
improvements
⬢ Hadoop user since 2010
– Experience
deploying,
maintaining
and
using
Hadoop
clusters
cnauroth@hortonworks.com
cnauroth
Chris Nauroth

3. 3
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
About Us
⬢ SmartSense Engineering Lead @ Hortonworks
⬢ Most of the career has been in the field, solving real life business problems
⬢ Last 6+ years in Big Data
⬢ Committer and PMC, Apache Metron
sheetal@hortonworks.com
sheetal_dolas
Sheetal Dolas

4. 4
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Agenda
⬢ Days in a life of Hadoop users – Real war stories!
⬢ Hadoop Operational Challenges
⬢ Winning and avoiding the wars
⬢ Q & A

5. 5
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Days in a life of
Hadoop users
Real war stories!

6. 6
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Story I: Unstable NameNode, Frequent Fail Overs
⬢ NameNode periodically becomes unresponsive
⬢ In HA scenario, fails over to standby
⬢ In short time, falls back again
⬢ Very frequent fail overs and fail backs
It was the garbage collection!

7. 7
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Story II: Very high CPU usage but low throughput
⬢ Unusually high system CPU usage
⬢ Jobs slowed down
⬢ Reduced data IO
System CPU
User CPU N/W IO
Transparent Huge Pages (THP) was turned on!

8. 8
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Job
Perfor
mance
Cluster
Stability
Story III: Cascading impact and cluster melt down
⬢ HDFS upgraded
⬢ HDFS utilization kept on increasing even after large data deletion
⬢ Rebalancing made the situation worse
⬢ Eventually HDFS became unresponsive
un-finalized HDFS had cascading
impact on cluster!

9. 9
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Story IV: Overloaded cluster
⬢ Jobs run slower
⬢ Always waiting containers and jobs, all YARN queues are fully utilized
⬢ Some jobs had to wait for hours to get the container slots
Sub optimally configured container sizes!
Requested Memory
Used Memory

10. 10
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Story V: Accidental deletion of critical datasets
⬢ User accidentally executed hdfs dfs -rm -R on a root directory
⬢ Delete is issued in parallel, control + c did not help
⬢ In panic, user shuts down HDFS immediately (fortunately)
⬢ Restarts later to check trash, loses all data
⬢ It’s nearly impossible to recover blocks from local file system
This is a more common mistake than one may think!

11. 11
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Story VI: Hive query returning random results
⬢ A hive query returns different results every time
⬢ Results are usually accurate during office hours
⬢ After office hours, results keep changing randomly on every execution
-- QUERY: WHAT IS TODAY’S TOTAL SALE AS OF NOW ?
SELECT SUM(amount)
FROM sales
WHERE sale_date = TO_DATE (UNIX_TIMESTAMP())
One of the host had a different time zone!

12. 12
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
and the stories
continue…

13. Hadoop
operational
challenges

14. 14
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Hadoop has lots of configurations
⬢ So many configurations! Overwhelming for many users
⬢ Best practices are evolving and change across versions

15. 15
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Many configurations are cluster and workload specific
⬢ A configuration good for one cluster may not be suitable for another cluster
⬢ Optimally configured clusters may become sub optimal tomorrow as they grow

16. 16
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Large clusters add to the complexities
⬢ Managing, updating and keeping nodes in sync becomes challenging
⬢ Nodes going down miss the maintenance cycles and get out of sync
⬢ Newly added nodes may have different standards (java version, os, user
configurations etc.)
⬢ Clusters start having heterogeneous hardware over period of time

17. 17
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Winning
and
avoiding
the wars with
SmartSense

18. 18
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
⬢ Proactive support personalized cluster insights by
– Enabling
faster
case
resoluJon
– Applying
industry
best
pracJces
– Providing
proacJve
analysis
⬢ SmartSense is a collection of tools and services
– Evaluates
cluster’s
current
configuraJon
and
runJme
environment
against
rich
set
of
rules
– Rules
are
dynamic,
reacJng
to
thresholds
tailored
to
the
specific
cluster
and
its
workloads
– ConJnuously
evolving
and
improving
rule
sets,
developed
by
or
in
close
consultaJon
with
acJve
commiWers,
support
engineers,
field
engineers.
SmartSense

19. 19 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
AGENT
AGENT
AGENT
AGENT
AGENT
AGENT
LANDING
ZONE
SERVER
AMBARI
AGENT
AGENT
AGENT
AGENT
AGENT
AGENT
BUNDLE
WORKER
NODE
WORKER
NODE
WORKER
NODE
WORKER
NODE
WORKER
NODE
WORKER
NODE
SmartSense
AnalyJcs
SmartSense Architecture
GATEWAY

20. 20 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
Addressing: Unstable NameNode, Frequent Fail Overs
Daunting Questions
⬢ What is right Heap size for my
NN ?
⬢ What should be the new gen
size ?
⬢ Which GC should I use ?
⬢ What GC options to be
configured?
⬢ What if my cluster grows ?
SmartSense Answer
⬢ Rule: hdfs_nn_jvm_opts
⬢ Calculates Heap size based on
– Current
heap
usage
– Total
number
of
objects
in
file
system
– Best
pracJces
⬢ Recalculates dependent JVM
options based on Heap size
⬢ Validates existing JVM opts
⬢ Provides continuous validations
and proactive recommendations

21. 21 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
Ã Heap Size
– 200 bytes per HDFS object (files, directories, blocks)
– 25 % buffer
Ã -Xms should be same as –Xmx
Ã New generation size should be 1/8th of –Xmx (capped at 8G)
Ã Use Concurrent Mark Sweep (CMS) Garbage Collection
– -XX:+UseConcMarkSweepGC
– -XX:CMSInitiatingOccupancyFraction=70
– -XX:+UseCMSInitiatingOccupancyOnly
– -XX:ParallelGCThreads=8
NameNode JVM Opts

22. 22 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
Addressing: Very high CPU usage but low throughput
Daunting Questions
⬢ Is THP applicable to my OS
version ?
⬢ Is it disabled ? Completely
disabled ?
⬢ How do I make sure it is disabled
on newly added nodes too ?
⬢ How do I make these
configurations person
independent ?
SmartSense Answer
⬢ Rule: os_thp
⬢ Checks if thp is completely
disabled
⬢ Provides OS specific disabling
instructions
⬢ Continuous evaluation that
validates newly added nodes and
re-commissioned nodes

23. 23 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
Disable THP
⬢ For RedHat CentOS
echo never /sys/kernel/mm/redhat_transparent_hugepage/enabled
⬢ For Debian, Ubuntu SUSE
echo never /sys/kernel/mm/transparent_hugepage/enabled
System CPU
User CPU
N/W IO

24. 24 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
Addressing: Cascading impact and cluster melt down
Daunting Questions
⬢ Should I finalize upgrade ?
⬢ What is right time to finalize ?
⬢ How do I make sure it does not
fall through cracks ?
SmartSense Answer
⬢ Rule: hdfs_nn_finalize_upgrade
⬢ Checks HDFS health after
upgrade
⬢ Evaluates how long HDFS is
running in un-finalized state
⬢ Reminds until it is finalized

25. 25 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
Ã Check NN UI / JMX for upgrade status
Ã Do not finalize HDFS upgrade until
– All files and blocks have been verified after upgrade
– Critical jobs have been executed at least once after upgrade
Ã Finalize between 2 - 7 days after upgrade
hdfs dfsadmin -finalizeUpgrade
HDFS Upgrade finalization

26. 26 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
Addressing : Overloaded cluster
Daunting Questions
⬢ What is right container size for
my cluster ?
⬢ If I add additional components
(HBase, Storm), how does the
container size change ?
⬢ How does container sizes change
when I add new types of nodes in
the cluster ?
⬢ What’s impact on container sizes
if I add SSDs to the nodes?
SmartSense Answer
⬢ Rules: yarn_container_size,
mr_container_size,
tez_container_size
⬢ Evaluates resources available on
individual host (CPU, Memory,
Disks, Running Services etc.)
⬢ Calculates technology specific
container sizes (MR,Tez, Hive)
⬢ Continuously evaluates as the
cluster dynamics change

27. 27 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
Container sizing
Ã Identify resources (CPU, Memory, Disks) available on each node
Ã Keep aside resources required for other processes (OS, DN, NM, HBase RS)
Ã Calculate max possible containers for each resource (CPU, Memory, Disks)
– CPU Containers: 4x cores
– Disk Containers: ( 3x HDD + 10x SSD )
– Memory Containers: (Available RAM / 2 )
Ã Number of containers = Min (CPU Containers, Disk Containers, Memory
Containers)

28. 28 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
Addressing: Accidental deletion of critical datasets
Daunting Questions
⬢ Is HDFS trash enabled ?
⬢ What is safe trash interval ?
⬢ How to prevent accidental
deletion of critical data ?
SmartSense Answer
⬢ Rule: hdfs_trash_interval
– Checks
if
trash
is
enabled
– Validates
if
trash
interval
is
within
reasonable
limits
⬢ Rule: hdfs_nn_protect_imp_dirs
– New
feature
available
in
Hadoop
2.8
– Helps
you
mark
criJcal
directories
such
as
“/”,
“/user”,
“/user/apps/hive”,
“/
user/apps/hbase”
etc.
are
delete
protected.

29. 29 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
HDFS Trash interval and directory protection
Ã fs.trash.interval detects number of minutes after which the trashed data
gets deleted
– 0 means trash disabled (data gets deleted immediately)
– Keep it the range 1440 (1 day) – 10080 (7 days)
– Recommended 4320 (3 days)
Ã fs.protected.directories specifies directories that will be delete
protected
– Available from Hadoop 2.8
– List all key directories there (/, /user,/user/apps, /user/apps/hive,
/user/apps/hbase, /user/apps/hbase/data, /mapred, /mapred/
system, /tmp etc. )

30. 30 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
Addressing : Hive query returning random results
Daunting Questions
⬢ Is my cluster configured
consistently ?
⬢ How do I prevent such hard to
analyze issues ?
⬢ How do I make sure newly added
do not bring these types of
issues ?
⬢ How do I make these set ups
person independent ?
SmartSense Answer
⬢ Rule: os_time_zone
⬢ Checks if all hosts have same time
zone
⬢ Rule os_service_ntpd_on make
sure all host times are in sync
⬢ Continuous evaluation that
validates newly added nodes and
re-commissioned nodes

31. 31 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
There are 250+ more such rules
Operations
Ã hdfs_dn_volume_tolerance
Ã hdfs_dn_xceivers
Ã hdfs_nn_handler_count
Ã …
Ã yarn_zk_quorum
Ã yarn_nm_recovery
Ã …
Ã os_hostname_reverse_lookup
Ã os_ssd_tuning
Ã …
Ã hive_mr_strict_mode
Ã hive_datanucleus_cache
Ã …
Ã tez_am_heap
Ã tez_shuffle_buffer
Ã …
Performance
Ã ams_mc_distributed_configs
Ã ams_mc_write_path
Ã ...
Ã hbase_jvm_opts
Ã hbase_rs_open_region_threads
Ã hbase_tcp_nodelay
Ã ...
Ã hdfs_dn_jvm_opts
Ã hdfs_mount_options
Ã hdfs_nn_dn_staleness_interval
Ã ...
Ã hive_auto_convert_join
Ã hive_disable_caching
Ã hive_enable_cbo
Ã ...
Security
Ã hdfs_dn_volume_tolerance
Ã hdfs_audit_log
Ã hdfs_block_access_token
Ã hdfs_enable_security_check
Ã hdfs_nn_super_user_group
Ã hdfs_zkfc_ha_acl
Ã ...
Ã ranger_policy_refresh_interval
Ã smartsense_2_way_ssl_enabled
Ã ...
Ã yarn_ats_security
Ã yarn_enable_acl
Ã ...

32. 32 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
There is more than just configurations
How do I
show back/
charge back
my tenants ?
Who are the
top users of
my platform ?
What type of
work loads are
running on my
cluster ?
Which jobs
have significant
impact on my
cluster ?
How do I
improve
performance
of key jobs ?
What is good
time for
maintenance?

33. 33 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
Activity Analysis

34. 34 © Hortonworks Inc. 2011 – 2016.All Rights Reserved
Summary
Ã There are many things involved in managing Hadoop cluster
Ã Best practices evolve and change across versions
Ã What is optimal today may not be optimal for tomorrow
Ã Changing cluster dynamics, workload characteristic need continuous re-evaluation
and configuration adjustments
Ã SmartSense can significantly help avoid common mistakes, issues, pitfalls and
simplify Hadoop operations

35. 35
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Lets keep your
Hadoop cluster at
its best!
Thank You!

36. 36
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Appendix

37. 37
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
More Resources
⬢ https://docs.hortonworks.com/index.html
⬢ http://hortonworks.com/products/subscriptions/smartsense/
⬢ http://hortonworks.com/info/smartsense/
⬢ http://hortonworks.com/blog/introducing-hortonworks-smartsense/
⬢ https://www.youtube.com/watch?v=IKulo9c8PjE
⬢ https://community.hortonworks.com/topics/smartsense.html

38. 38
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
SmartSense Bundle Security
⬢ All
Bundles
are
Anonymized
and
Encrypted
⬢ MulJple
built-­‐in
security
measures
– Ambari clear text passwords are not collected
– Hive and Oozie database properties are not collected
– All IP addresses and host names are anonymized
⬢ Extensible
security
rules
– Exclude properties within specific Hadoop configuration files
– Global REGEX replacements across all configuration, metrics, and logs

39. 39
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
SmartSense Stack Support
HDP 2.4 HDP 2.3 HDP 2.2 HDP 2.1 HDP 2.0
SmartSense 1.x
Ambari 2.2
Built-In!
Ambari 2.1
Plug-In
Ambari 2.0
Plug-In
Ambari 1.7 Ambari 1.6
SmartSense 1.x