Apache Ambari - What's New in 2.4

1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari 2.4.0
What’s New
August 2016

What is Apache Ambari?
A completely open source
management platform for
provisioning, managing,
monitoring and securing
Apache Hadoop clusters.
Apache Ambari takes the
guesswork out of operating
Hadoop.

What Ambari Does
Simplified Installation,
Configuration and Management
Centralized Security Setup
Full Visibility into Cluster Health
Highly Extensible and
Customizable
• Wizard-driven and automated cluster provisioning
• Smart Configurations and Cluster Recommendations
• Automated Rolling and Express cluster upgrades
• Reduce complexity to administer security across the platform
• Automate setup Kerberos
• Simplify the configuration of Apache Ranger
• Predefined alerts based on operational best practices
• Advanced metrics visualization with Grafana
• Seamlessly fit into your enterprise environment
• Bring custom Services under management via Ambari Stacks
• Customize the UI with Ambari Views

What’s New in Ambari 2.4
 Alerts: Customizable SCRIPT Parameters (AMBARI-14898)
 Alerts: Retry Check Counts (AMBARI-15686)
 Alerts: New HDFS Alerts (AMBARI-14800)
 New Host Page Filtering (AMBARI-15210)
 Remove Service (AMBARI-14759)
 Support for SLES 12 Technical Preview (AMBARI-16007)
 Stability: Database Consistency Checking (AMBARI-16258)
 Customizable Ambari Log + PID Dirs (AMBARI-15300)
 New Version Registration Experience (AMBARI-15724)
 Log Search Technical Preview (AMBARI-14927)
 Operational Audit Logging (AMBARI-15241)
 Role-Based Access Control (AMBARI-13977)
 Automated Setup of Ambari Kerberos (AMBARI-15561)
 Automated Setup of Ambari Proxy User (AMBARI-15561)
 Customizable Host Reg. SSH Port (AMBARI-13450)
Core Features Security Features
 View URLs (AMBARI-15821), View Refresh (AMBARI-15682)
 Inherit Cluster Permissions (AMBARI-16177)
 Remote Cluster Registration (AMBARI-16274)
Views Framework Features

Agenda
What’s New in Ambari 2.4.0
Feature Highlights: Alert Retry Check Counts

Alert Check Counts
 Customize the number of times an alert is checked before dispatching a notification
 Avoid dispatching an alert notification (email, snmp) in case of transient issues

Configuring the Check Count
 Set globally for all alerts, or override for a specific alert
Global Setting
Alert Override

State Change Types
 SOFT state changes do not perform a dispatch
 HARD state changes (to non-OK) perform dispatch
 Regardless of change:
– The Ambari Web UI will show the current state (OK/WARN/CRIT)
– The state change is written to ambari-alerts.log
2016-05-31 13:20:52,294 [CRITICAL] [SOFT] [AMBARI_METRICS]
[grafana_webui] (Grafana Web UI) Connection failed to
http://c6401.ambari.apache.org:3000 (<urlopen error [Errno
111] Connection refused>)
2016-05-31 13:22:52,290 [CRITICAL] [HARD] [AMBARI_METRICS]
[grafana_webui] (Grafana Web UI) Connection failed to
http://c6401.ambari.apache.org:3000 (<urlopen error [Errno
111] Connection refused>)
Note: check counts are not configurable for AGGREGATE alert types. All state changes are considered HARD.

Example: Check Count = 3
Check 1/3
State: OK
Change: n/a
Check 1/3
State: OK
Change: n/a
Check 1/3
State: CRIT
Change: SOFT
Check 2/3
State: CRIT
Change: n/a
Check 3/3
State: CRIT
Change: HARD
Check 1/3
State: OK
Change: HARD
DISPATCH
Check Interval Check Interval Check Interval Check Interval Check Interval
no state change
state changes to CRIT
performing multiple checks
back to OK

Agenda
Feature Highlights: Alert Customizable Params

Alert Types and Thresholds
 Ability to customize Thresholds for SCRIPT and SERVER alerts
 Ability to customize Connection Timeout for METRIC alerts
Alert Type Description Thresholds (units)
WEB Connects to a Web URL. Alert status is based on the HTTP response code.
Response Code (n/a)
Connection Timeout (seconds)
PORT Connects to a port. Alert status is based on response time. Response (seconds)
METRIC
Checks the value of a service metric. Units vary, based on the metric being
checked.
Metric Value (units vary)
Connection Timeout (seconds)
AGGREGATE Aggregates the status for another alert. % Affected (percentage)
SCRIPT Executes a script to handle the alert check. Varies
SERVER Executes a server-side runnable class to handle the alert check. Varies

Alerts: Customizable METRIC Connection Timeout
 Ability to set Connection Timeout threshold via Ambari Web UI
NEW!

Alerts: Customizable SCRIPT Thresholds
 Ability to set various thresholds via Ambari Web UI

Alerts: NEW!!! Ambari Server Performance Alert
 Measures the Ambari Server REST API and Backend Database response

Agenda
Feature Highlights: New HDFS Alerts

New HDFS Alerts Watch Trends
 NameNode Client RPC Queue Latency (Hourly/Daily)
 NameNode Client RPC Processing Latency (Hourly/Daily)
 NameNode Service RPC Queue Latency (Hourly/Daily)
 NameNode Service RPC Processing Latency (Hourly/Daily)
 NameNode Heap Usage (Daily/Weekly)
 HDFS Storage Capacity Usage (Daily/Weekly)

Agenda
Feature Highlights: New Host Filtering

New Host Filtering Control in Ambari Web
 Ability to perform complex host filtering from Ambari Web
 Make it easier to find hosts
NEW!

Search by Host Attribute, Service or Component

Host Attribute Filtering
 Host Name
 IP
 Host Status
 Cores
 RAM
 Stack Version + Version State
 Rack

Service Filtering

Component Filtering

Host Filter: Examples

Agenda
Feature Highlights: Remove Service

Remove Service
 Ability to perform Remove Service from Ambari Web
 Eliminates need to use Ambari REST API
 Checks for Service dependencies
 Service must be stopped
 All configuration information and history is also removed
 This operation is not reversible
NEW!

Agenda
Feature Highlights: Other Items

Customizable Ambari Log + PID Dirs (AMBARI-15300)
 Ambari Server and Agents write log activity output to log files and use a PID-file that
contains the process identification number (PID) for their running process.
Log Location PID Location
Ambari Server /var/log/ambari-server/ambari-server.log /var/run/ambari-server/ambari-server.pid
Ambari Agent /var/log/ambari-agent/ambari-agent.log /var/run/ambari-agent/ambari-agent.pid

Customize Ambari Server Log + PID
vi /etc/ambari-server/conf/ambari.properties
pid.dir=/var/run/ambari-server
vi /etc/ambari-server/conf/log4j.properties
ambari.log.dir=${ambari.root.dir}/var/log/ambari-
server
Ambari Server PID Ambari Server Log
1. Stop Ambari Server prior to modifying log or pid directories.
2. You must manually create the new directories and be sure to set the directory
ownership + permissions to allow the Ambari Server process access.

Customize Ambari Agent Log + PID
vi /etc/ambari-agent/conf/ambari-agent.ini
[agent]
logdir=/var/log/ambari-agent
piddir=/var/run/ambari-agent
1. Stop Ambari Agent prior to modifying log or pid directories.
2. You must manually create the new directories and be sure to set the directory
ownership + permissions to allow the Ambari Agent process access.

Customizable Host Registration SSH Port
 Customize SSH Port when performing Host Registration automatically
NEW!

Stability: Database Consistency Checking
 On Ambari Server start, Ambari runs a database consistency check looking for issues.
 If any issues are found, Ambari Server start will abort and a message will be printed to
console “DB configs consistency check failed.”
 Check Ambari Server log file for more details:
/var/log/ambari-server/ambari-server-check-database.log
 Ability to “skip” check and force Ambari Server start
ambari-server start --skip-database-check
Important: if you “skip” the check to force Ambari Server start, do not make any
changes to your cluster topology or perform a cluster upgrade until you correct
the database consistency issues.

Agenda
Feature Highlights: View Framework Enhancements

View URLs (AMBARI-15821)
 Ability to create a “short URL” or “vanity URL” for view instances
 Provide users with a non-version or instance specific URL to a view
/#/main/views/{viewName}/{viewVersion}/{viewInstanceName}/#/main/view/{viewName}/{shortURL}

View Refresh (AMBARI-15682)
 Automatically deploy new views into Ambari Server w/o a restart
1. Copy view archive to: /var/lib/ambari-server/resources/views/
2. Ambari Server detects the new view, automatically extracts + deploys
3. View is available for creating instances
4. Click “Refresh” in Views UI

Remote Cluster
Configuration
AMBARI-16274

Background: View <-> Cluster Communication
 Deployed Views “talk” with cluster using REST APIs (as applicable)
CLUSTER
ATS
RM
Ambari Server
Tez UI
View
Tez UI View talks with cluster
using
REST APIs to
ATS and ResourceManager
Ambari
DB
LDAP
AuthN

Background: Operational vs. Standalone Ambari Server
Ambari Agent
Host
Ambari Agent
Host
Ambari Agent
Host
Standalone Ambari Server
One or More Ambari Server Instances
No Agents, no requirement to operate the cluster
Operational Ambari
One Ambari Server Instance
Talking with Agents, Managing the cluster
Ambari
Server
Ambari
DB
LDAP
AuthN
Ambari
Server
Ambari
DB
LDAP
AuthN

Background: Local Cluster vs. Non-Local
Ambari
Server
Ambari
DB
LDAP
AuthN
Ambari
Server
Ambari
DB
LDAP
AuthN
Standalone Ambari Server
One or More Ambari Server Instances
No Agents, no requirement to operate the cluster
Operational Ambari
One Ambari Server Instance
Talking with Agents, Managing the cluster
LOCAL
CLUSTER
NON-
LOCAL
CLUSTER

Introducing Remote Cluster Configuration (AMBARI-16274)
Option Description
Local Cluster When you select this Local Cluster option, Ambari will automatically determine the
cluster configuration properties needed for the view instance.
Criteria:
• Ambari Server running the views is also managing the cluster
Remote Cluster When you select Remote Cluster option, Ambari will automatically determine the
cluster configuration properties needed for the view instance.
Criteria:
• The cluster is not local to the Ambari Server running the views (i.e. Standalone)
• Cluster is being managed by Ambari
Custom When you select Custom option, you must enter all configuration information, and are
responsible for updating if the cluster configuration changes.
Criteria:
• The cluster running the view is not local to the Ambari Server
• The cluster is not being managed by Ambari
NEW!

Local vs Remote View Configuration
Ambari Server
Views
Cluster
Ambari Server
Views
ClusterAmbari Server
LOCAL CLUSTER
REMOTE CLUSTER
Operational Ambari
Manages cluster
Standalone Ambari
Manages cluster
Talks to cluster
Obtains view config
Obtains view config
Talks to cluster
Operational Ambari

View Configuration: Minimizing Need for Custom
Cluster Config Ambari Server Cluster Mgmt
Ambari 2.2 or
Earlier
Ambari 2.4
No HA, No Kerberos Operational Ambari Local Local
HA or Kerberos Operational Ambari Custom Local
No HA, No Kerberos Standalone Ambari Custom Remote
HA or Kerberos Standalone Ambari Custom Remote
No HA, No Kerberos Standalone Non-Ambari Custom Custom
HA or Kerberos Standalone Non-Ambari Custom Custom

Inherit Cluster
Permissions
AMBARI-16177

Inherit Cluster Permissions (AMBARI-16177)
 Ability to automatically grant View “Use” permission based on Cluster role
 Note: Option is only available when using a Local Cluster Configuration
Explicitly grant users
and groups Use
permission
Automatically grant
users and groups Use
permission based on
Cluster roles

Agenda
Feature Highlights: Log Search
TECH PREVIEW

Log Search
Solr
A M B A R I
Log
Search
Search Cluster Component Logs from within Ambari
Goal: When issues arise, be able to quickly find issues
across all components
⬢ Capabilities
– Rapid Search of all cluster component logs
– Search across time ranges, log levels, and for keywords
⬢ Core Technologies:
– Apache Ambari
– Apache Solr
– Apache Ambari Log Search

Log Search Architecture
A M B A R I
L O G
F E E D E R
L O G
F E E D E R
L O G
F E E D E R
L O G
F E E D E R
L O G
F E E D E R
L O G
F E E D E R
WO R K E R
N O D E
WO R K E R
N O D E
WO R K E R
N O D E
WO R K E R
N O D E
WO R K E R
N O D E
WO R K E R
N O D E
Solr
LO G
S E A R C H
U I

Log Search Details
WO R K E R
N O D E
L O G
F E E D E R
Solr
LO G
S E A R C H
U I
Solr
Solr
A M B A R I
Java Process
Multi-output Support
Grok
Solr Cloud
Local Disk Storage
TTL

Considerations
 Log Feeders are CPU intensive, consider 1 dedicated core
 Solr instances should use dedicated hardware with at least 32GB of RAM dedicated to
the Solr instance
 By default, logs will age out after 7 days

Agenda
Feature Highlights: RBAC

New Role Based Access Control
 Introducing new “roles” for more granular division of control for cluster operations
Old Permission New Role Notable Permissions
Operator Cluster Administrator
Full operational control, including upgrades. Ambari Admins are
implicitly granted this Role.
Cluster Operator Adding and removing hosts.
Service Administrator Manage configurations, move components.
Service Operator
Service stop and start and service-specific operations such as HDFS
Rebalance.
Read-Only Cluster User View cluster service and host information.
Note: Users flagged as “Ambari Administrators / Ambari Admins” are implicitly granted Cluster
Administrator permission.

Managing Cluster Roles
Assign roles to
users or groups
Manage roles in
Block or List View
layouts

Managing Cluster Roles
View users or
groups
Change current
role assignment

Agenda
Feature Highlights: Security Enhancements

Summary of Security Enhancements
 Automatic Setup of Ambari Server as a Proxyuser
 Automatic Setup of Ambari Server for Kerberos

Automatic Setup of Ambari Server as a
Proxyuser

Background: Proxyusers
 HDFS and WebHCat (as part of Hive) support the concept of a Proxyuser
 Proxyuser allows UserA to access the service on behalf of UserB (i.e. the proxyuser is
allowed to impersonate other users)
 Proxyuser is a commonly used capability of Hadoop
HDFS
“UserA” is setup as a proxyuser
UserA can access HDFS as
“UserA” on behalf of “UserB”
HDFS ops performed are as
“UserB”

Background: HDFS Proxyuser Setup
 A proxyuser needs to be configured in core-site.xml configuration:
hadoop.proxyuser.{proxyuser-name}.hosts
hadoop.proxyuser.{proxyuser-name}.groups
 If these settings are not present, impersonation will not be allowed and connection to
the service via proxyuser will fail

Background: Ambari + Proxyuser
 Ambari Views use proxyuser to access the cluster (such as Hive View and Pig View)
 Ambari Server needs to access a service on behalf of an authenticated user
Ambari
Server HDFS
(running as user “ambari”)
“joe” authenticates to Ambari
(setup for proxyuser “ambari”)
hadoop.proxyuser.ambari.hosts=*
hadoop.proxyuser.ambari.groups=*
Ambari Server can talk to HDFS as
“ambari” proxyuser on behalf of “joe”
Configuration of proxyuser is commonly “missed” when setting up Ambari Views

New: Automatic Ambari Server Proxyuser Setup
 Proxyuser configurations are automatically added for HDFS and WebHCat
 For example: Ambari Server as running as “ambari”, the following configurations are
added during HDFS service install
hadoop.proxyuser.ambari.hosts
hadoop.proxyuser.ambari.groups

Automatic Setup of Ambari Server for
Kerberos

Background: Hadoop + Kerberos
 Strongly authenticating and establishing a user’s identity is the basis for secure access in
Hadoop. Users need to be able to reliably “identify” themselves and then have that
identity propagated throughout the Hadoop cluster.
 Once this is done, those users can access resources (such as files or directories) or
interact with the cluster (like running MapReduce jobs).
 Besides users, Hadoop cluster resources themselves (such as Hosts and Services) need
to authenticate with each other to avoid potential malicious systems or daemon’s
“posing as” trusted components of the cluster to gain access to data.

Background: Hadoop + Kerberos
Service
Component
A
Service
Component
B
Hadoop Cluster
KDC
keytabkeytab
Service
Component
C
keytab
Service
Component
D
keytab
Service
Component
X
Service
Component
X
keytabkeytab
Service
Component
X
keytab
Service
Component
X
keytab
Kerberos is used to
secure the
Components in the
cluster. Kerberos
identities are
managed via
“keytabs” on the
Component hosts.
Principals
for the
cluster are
managed in
the KDC.

Background: Automated Kerberos Setup with Ambari

Background: Principal and Keytab Generation & Distribution
1. User provides KDC Admin Account
credentials to Ambari
2. Ambari connects to KDC, creates
principals (Service and Ambari) needed
for cluster
3. Ambari generates keytabs for the
principals
4. Ambari distributes keytabs to Ambari
Server and cluster hosts
5. Ambari discards the KDC Admin
Account credentials (optional)
Ambari
Server KDC
1 2
4
3
5
Cluster

Background: Ambari + Hadoop + Kerberos
 Ambari Server communicates with the cluster to retrieve information (such as metrics)
 Especially important for Ambari Views (e.g. Files, Hive, Pig)
 Therefore: Ambari Server ALSO needs to be “setup for Kerberos”
Ambari
Server Cluster
Kerberos enabled

Background: Manual Setup of Ambari Server for Kerberos
 Manual setup of Ambari Server for Kerberos (outside of “Enable Kerberos” wizard):
1. Create principal for Ambari Server
2. Generate keytab for Ambari Server
3. Place keytab on Ambari Server host
4. Run “ambari-server setup-security” on Ambari Server
5. Restart Ambari Server
Configuration of Ambari Server for Kerberos is commonly “missed” when setting up Ambari Views

New: Automatic Setup of Ambari Server for Kerberos
 When enabling Kerberos and choosing an automated option (MIT or AD), Ambari Server
will be setup for Kerberos automatically:
1. A principal will be created for Ambari Server
2. A keytab will be generated and placed on Ambari Server
3. Ambari Server is setup for Kerberos
Note: you will still need to perform the Ambari Server restart for the Kerberos identity to get picked-up by
Ambari.

What about Proxyuser + Kerberos?

New: Automatic Proxyuser Setup with Kerberos
 When a cluster has Kerberos enabled, the proxyuser needs to be configured based on
the primary part of the Kerberos principal name
hadoop.proxyuser.{principal-name-primary}.hosts
hadoop.proxyuser.{principal-name-primary}.groups
 Ambari will adjust proxyuser configurations during Kerberos setup
Ambari
Server HDFS
(running as user “ambari”)
(setup with principal “ambari-server@EXAMPLE.COM”
“joe” authenticates to Ambari
(setup for proxyuser “ambari-server”)
hadoop.proxyuser.ambari-server.hosts=*
hadoop.proxyuser.ambari-server.groups=*
Ambari Server can talk to HDFS as
“ambari-server” proxyuser on behalf
of “joe”

Agenda
Feature Highlights: Ops Audit Logging

Operational Audit Logging
 Ambari will create entries in an audit log as Ambari + Cluster operations are performed
 Using the audit log, you can determine who performed the operation and when the
operation was performed as well as other operation-specific information
 The Ambari Audit log can be found at: /var/log/ambari-server/ambari-audit.log

List of Operations
 Stop/Start Service
 Stop all Services
 Add Service
 Move Component
 Turn On/Off Maintenance Mode
 Download Client Configurations
 Blueprint Export
 Update Configuration **
 Login (success/failed) / Logout
 Create User, Group
 Delete User, Group
 Change Group Membership
 Change User Status, Admin
 Change User Password
 Grant/Revoke User, Group Cluster Roles
Service Operations User Operations
** Note: When a Service Configuration change is made, an entry is also written to a
specific log file ambari-config-changes.log for configuration changes that
provides even more detail on the change.

List of Operations (continued)
 Add/Remove Host
 Enable/Disable/Edit Alert
 Add/Update/Delete Alert Group
 Add/Upgrade/Delete Notification
 Enable/Disable Kerberos
 Regenerate Kerberos Keytabs
 Rename Cluster
 Add/Remove Remote Clusters
 Register/Deregister Version
 Cluster Upgrade
Cluster Operations Upgrade Operations
 Create/Delete View Instance
 Edit View Instance
 Grant/Revoke View Permissions
 Create/Delete View URLs
View Operations

Example: Change Group Membership
 Add/Remove group members creates a “Membership change” audit entry
2016-06-02T23:12:09.930Z, User(admin), RemoteIp(192.168.64.1), Operation(Membership change), RequestType(PUT),
url(http://c6401.ambari.apache.org:8080/api/v1/groups/customgroup/members), ResultStatus(200 OK), Group(customgroup), Members(joeuser)
2016-06-02T23:12:34.700Z, User(admin), RemoteIp(192.168.64.1), Operation(Membership change), RequestType(PUT),
url(http://c6401.ambari.apache.org:8080/api/v1/groups/customgroup/members), ResultStatus(200 OK), Group(customgroup), Members(joeuser, mike)

Example: Stop ZooKeeper
 A single operation (like “Stop ZooKeeper”) might generate multiple audit entries
 Relate entries via RequestId()
2016-06-02T23:14:35.206Z, User(admin), RemoteIp(192.168.64.1), Operation(INSTALLED: ZOOKEEPER_SERVER/ZOOKEEPER on c6401.ambari.apache.org (MyCluster)),
Host name(c6401.ambari.apache.org), RequestId(7), Status(Successfully queued)
2016-06-02T00:31:56.016Z, User(admin), Operation(Stop ZooKeeper Server), Status(IN_PROGRESS), RequestId(7)
2016-06-02T00:31:56.025Z, User(admin), Operation(STOP ZOOKEEPER_SERVER), Status(QUEUED), RequestId(7), TaskId(52), Hostname(c6401.ambari.apache.org)
2016-06-02T00:31:57.370Z, User(admin), Operation(Stop ZooKeeper Server), Status(COMPLETED), RequestId(7)
2016-06-02T00:31:57.370Z, User(admin), Operation(STOP ZOOKEEPER_SERVER), Status(COMPLETED), RequestId(7), TaskId(52), Hostname(c6401.ambari.apache.org)

Agenda
Feature Highlights: Version Registration Experience

Introducing the Version Definition File (VDF)
 This is a meta file describing which Services are included and at which version

Ambari will “discover” Available Versions
Tabs for list of
available Stacks
List of discovered
Versions
List of Services
w/version #

“Default Version Definition” for Backwards Compat
Ambari provides a
“default” Version
Definition.

Add New Version via File Upload or URL

Changes in Install / Version Registration Flow
Scenario Ambari 2.4 Change
Internet Access / Public Repositories No change.
No Internet Access / Local repositories - Upload a VDF for the Local Repository you created
- Set the Local Repository URLs
OR
- Choose the Default Version Definition
- Set the Local Repository URLs

Other UX Changes: Local vs. Public Repository Radio
Explicit
Choice

Other UX Changes: Local vs. Public Repository Radio
Choose
Local
Must enter
Base URLs

Other UX Changes: OS Add/Remove

Other UX Changes: RedHat Satellite/Spacewalk
Explicit
Choice
- Ambari will not write
the .repo files
- User must register
the repositories
channels via Satellite

Other UX Changes: Viewing, Install and Upgrade

Other UX Changes: Managing Versions

Thank You

Apache Ambari - What's New in 2.4

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (15)

Similar to Apache Ambari - What's New in 2.4

Similar to Apache Ambari - What's New in 2.4 (20)

More from Hortonworks

More from Hortonworks (20)

Recently uploaded

Recently uploaded (20)

Apache Ambari - What's New in 2.4

Editor's Notes