HDF 3.2 - What's New
•
4 likes•1,093 views
We have introduced several new features as well as delivered some significant updates to keep the platform tightly integrated and compatible with HDP 3.0. https://hortonworks.com/webinar/hortonworks-dataflow-hdf-3-2-release-raises-bar-operational-efficiency/
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to HDF 3.2 - What's New
Similar to HDF 3.2 - What's New (20)
More from Hortonworks
More from Hortonworks (17)
Recently uploaded
Recently uploaded (20)
HDF 3.2 - What's New
- 3. 3 © Hortonworks Inc. 2011 – 2018. All Rights Reserved Capture streaming data Deliver perishable insights Combine new & old data Store data forever Access a multi-tenant data lake Model with machine learning DATA AT REST ( Hortonw orks Data Platf orm) DATA IN MOTION ( Hortonw orks DataFlow) ACTIONABLE INTELLIGENCE Perishable Insights Historical Insights A Connected Data Strategy Solves for All Data HORTONWORKS DATAPLANE SERVICE Manage, Secure, Govern MULTIPLE CLUSTERS AND SOURCES MULTIHYBRID
- 5. 5 © Hortonworks Inc. 2011–2018. All rights reserved HDF Use Cases Data Movement Optimize resource utilization by moving data between data centers or between on- premises infrastructure and cloud infrastructure Optimize Log Collection & Analysis Optimize log analytics solutions such as Splunk by using HDF as a single platform to collect and deliver multiple data sources and using HDP for lower cost storage options Gain key insights with Streaming Analytics Accelerate big data ROI by analyzing streaming data for patterns, comparing with ML models and delivering actionable intelligence Single view / 360 view of customer Ingest, transform and combine customer data from multiple sources into a single data view / lake Stream Processing Combine multiple streams of data in real-time, enrich the data and route it to different end points based on rules Capture and Analyze IoT Data Ingest sensor data from IoT devices and stream it for further processing and comprehensive analysis
- 7. 7 © Hortonworks Inc. 2011–2018. All rights reserved HDF 3.2 Highlights EDGE INTELLIGENCE + CURE KAFKA BLINDNESS 100K PROCESSORS PER FLOW HDP 3.0 SUPPORT
- 8. 8 © Hortonworks Inc. 2011–2018. All rights reserved HDF 3.2 Release Themes Cross-platform Integration HDP 3.0 Support Product Enhancements Performance enhancements • Support deployment of single cluster with HDF and HDP services • Single Ambari instance for HDF and HDP – one Ranger, Atlas, Knox etc. • NiFi supports Hive 3 on HDP 3.0 • NiFi processors to support HDP 3.0 • Storm connectors to support HDP 3.0 • SAM components to support HDP 3.0 • NiFi flows can have as much as 100K processors • More stability on large NiFi clusters • Support for Nifi 1.7.0 • Kerberos keytab isolation • Support for Kafka 1.1.1 • Support for TensorFlow in MiNiFi
- 10. 10 © Hortonworks Inc. 2011–2018. All rights reserved Single Shared Ambari Instance • Single Ambari instance • Shared common services • Ranger • Atlas • Knox
- 12. 12 © Hortonworks Inc. 2011–2018. All rights reserved HDF/HDP 3.0 Integration • Hive 3 specific processors • HBase 2 • Updated HDFS processors • New GetHDFSFileInfo processor • Ranger 1.1.0 • Support for group based policies • Knox 1.0.0 – NiFi SSO • Existed previously but now upgraded • Atlas 1.0.0 • SAM compatibility with HDP 3.0 services
- 14. 14 © Hortonworks Inc. 2011–2018. All rights reserved HDF 3.2 – NiFi, Kafka and MiNiFi Updated to 1.7.0 Updated to 1.1.1 Developer enhancements • NiFi-Registry Git Integration • NiFi Registry event hooks Security enhancements • Kerberos keytab isolation • Support for TensorFlow • New Expression Language features / updates • New C-based SDK • SuSE support • Feature to capture producer/topic metrics at partition level • Security and governance enhancements Support for
- 16. 16 © Hortonworks Inc. 2011–2018. All rights reserved NiFi 1.7.0 – Performance Enhancements • NiFi flows can have as much as 100K processors • More stability on large NiFi clusters • Improved multi-tenant concurrent usage
- 19. 19 © Hortonworks Inc. 2011–2018. All rights reserved Streaming Analytics Reference Architecture Data Flow Apps Powered by NiFi Subcribing Streaming Analytics Apps Kafka is Everywhere. Critical Component of Streaming Architectures Kafka Producers Kafka Topics Kafka TopicsKafka Consumers & Producers Kafka Consumers Data Syndication Services Powered by Kafka syndicate- speed Kafka Topic syndicate- geo Kafka Topic syndicate- transmission Kafka Topic syndicate- temp Kafka Topic syndicate- oil Kafka Topic syndicate- breaks Kafka Topic syndicate- battery Kafka Topic syndicate- start/stop Kafka Topic syndicate- acceleration Kafka Topic syndicate- idle Kafka Topic Data Collection at the Edge US West Fleet Truck Sensors C++ Agent US Central Fleet Truck Sensors C++ Agent US East Fleet Truck Sensors C++ Agent Analytics App 1 Analytics App 2 Analytics App 5 Analytics App 3 Analytics App 4 gateway-west- raw-sensors Kafka Topic IOT Ingest Gateway Powered by Kafka gateway-central- raw-sensors Kafka Topic gateway-east- raw-sensors Kafka Topic
- 20. 20 © Hortonworks Inc. 2011–2018. All rights reserved HORTONWORKS DATA PLATFORM (HDP®) DATA-AT-REST PRODUCERS CONSUMERS TOPICS BROKERS SMM REST Server HORTONWORKS DATAFLOW (HDF) DATA-IN-MOTION Optimize your Kafka Clusters Troubleshoot Kafka Streams Trace end-to- end Kafka flows Streams Messaging Manager (SMM)
- 21. 21 © Hortonworks Inc. 2011–2018. All rights reserved Cure is Here: Hortonworks Streams Messaging Manager (SMM) What is SMM? à Kafka Management and Monitoring tool à Cure the “Kafka Blindness” à Single Monitoring Dashboard for all your Kafka Clusters across 4 entities – Broker – Producer – Topic – Consumer à Supports multiple HDP and/or HDF Kafka Clusters à REST as a First Class Citizen à Delivered as a DataPlane Service