Paris Future of Data meetup - Streams Messaging Manager (SMM) for Kafka

1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Streams Messaging Manager
Paris Future Of Data Meetup
https://bit.ly/2g36oWc

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Next Generation Data Problems
Multiple clusters and data sources
“Enterprises demand self-service data preparation solutions to empower business users like data scientists
to directly engage with big data platforms, reducing the involvement of technology.”
Forrester Report: Big Data Fabric
Business Analysts & Data Scientists
Ability to quickly understand data models and
reliably access and search across data stores
Data Engineers
Ability to consistently create, move and backup
data across multiple locations
Data Steward (Architect/Security)
Ability to consistently apply security and
governance policy across all types of data

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Core Capabilities
Provide basic functions to be used by extensible
data services for for multiple types and tiers
⬢ Data Source Integration
Ability to register and/or create data sources to
allow consolidated access
⬢ Data Services Catalog
Full configuration and management utilities for the
enablement of new services
⬢ Security Controls
Full definition of security access controls including
persona definitions
Hortonworks DataPlane Service

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DataPlane: ‘Data Fabric’ for the Hybrid Cloud
Understand, Secure and Govern across Data Lakes
In the Cloud
On Premises
(Structured)
(Structured)
Cluster 1 Dublin
(Unstructured) (Structured) (Unstructured)
Cluster 2 Las Vegas
(Unstructured)(Structured) (Structured)
Cluster 3 Bangkok
Apache Ranger
(Structured)
Apache Ranger Apache Ranger
(Unstructured)(Structured) (Unstructured) (Structured)
Apache Ranger Apache Ranger
DPS
Cluster
Visibility
Data Services
Delivery
Role-based
Administration

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DPS Console: One place to visualize Virtual Data Lakes
DPS PlatformData Plane Service Console

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks DataPlane Service

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Streams Messaging
Manager (SMM)

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Kafka’s Omnipresence Has Led to the Onset of “Kafka Blindness”
à What is “Kafka Blindness”?
– Customers who use Kafka today struggle with monitoring / “seeing”/troubleshooting what is
happening in their clusters
à Who is Affected?
– Platform Operation Teams
– Developers / DevOps Teams
– Security / Governance Teams
à What are the Symptoms?
– Difficulty seeing who is producing and consuming data
– Difficulty understanding the flow of data from producers -> topics à consumers
– Difficulty troubleshooting/monitoring.

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Cure is Here: Hortonworks Streams Messaging Manager (SMM)
à New Open Source project led by
Hortonworks to Cure the “Kafka Blindness”
à Single Monitoring Dashboard for all your
Kafka Clusters across 4 entities
– Broker
– Topic
– Producer
– Consumer
à Designed for the Enterprise
– Support for Secure/Kerborized Kafka cluster
– Rich Access Control Policies (ACLS)
– Supports multiple HDP and/or HDF Kafka Clusters
à REST as a First Class Citizen
à Delivered as a DataPlane Service

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
SMM features (1/2)
à Intelligent filtering on all entities
à Important KPIs such as lag & throughput
Simplify troubleshooting Kafka
environments
Visualize end-to-end Kafka stream flows
à End to end vision from producers to
consumers passing by topics and brokers

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
SMM features (2/2)
à End-to-end integration with enterprise
services on Hortonworks platforms :
Ambari, Grafana, Ranger & Atlas
Data lineage and governance from edge-to-
enterprise
monitoring / management capabilities
with REST
à Comprehensive REST service for open
integration with APM and ticketing tools

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Data Plane
Core Services
Ambari
Kafka ZK
SMM
Agent
HDP Cluster
…
Data Plane DB
Knox
LDAP / AD
Ambari
Kafka ZK
SMM
Agent
HDF Cluster
Knox
Ambari
Kafka ZK
SMM
Agent
HDP Cluster
Knox
DP
UI
SMM
UI
Single User Store
High Level Architecture of DP/SMM
AMS SRAMS SRAMS SR

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DP Core (Micro) Services and DP Apps are Docker Containers
SMM Agent
Kafka Broker
Zookeeper
AMS
SMM Rest Server

14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DataPlane Components - Access services
à Apache Knox
– Provides authentication for DPS
– Has to be configured with the same LDAP / AD user identity store as the managed HDP clusters
à Consul
– Service registry for DPS components
– Services register with Consul so they can be discovered by other services
– Allows services to be distributed in future to multiple machines
à Zuul
– Zuul enables dynamic routing, monitoring, resiliency and security
– Used to intercept & authenticate all API calls (redirecting to Knox if required)
– Proxy to redirect calls to physical location of components (using Consul to discover their location)

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
SMM REST Admin Server
What is SMM REST Server?
à SMM UI powered by first class REST Services
via the SMM REST Admin Server
à Monitoring Rest Endpoints can be used to
integrate with APM/Alerting/Ticketing
solutions
à Powered by Apache Knox
à Installed via an Ambari Management Pack
on a target HDP/HDF cluster where Kafka
Service is running
à Supports both HDP & HDF Platforms
à Open Source / AGPL licensed projected led
by Hortonworks

16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
SMM Demo

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions