Curing Kafka Blindness with Hortonworks Streams Messaging Manager

•Transferir como PPTX, PDF•

3 gostaram•804 visualizações

With the growth of Apache Kafka adoption in all major streaming initiatives across large organizations, the operational and visibility challenges associated with Kafka are on the rise as well. Kafka users want better visibility in understanding what is going on in the clusters as well as within the stream flows across producers, topics, brokers, and consumers. With no tools in the market that readily address the challenges of the Kafka Ops teams, the development teams, and the security/governance teams, Hortonworks Streams Messaging Manager is a game-changer. https://hortonworks.com/webinar/curing-kafka-blindness-hortonworks-streams-messaging-manager/

Tecnologia

1 © Hortonworks Inc. 2011–2018. All rights reserved
Curing Kafka Blindness with
Hortonworks Streams Messaging
Manager (SMM)
George Vetticaden, VP Product Management, @gvetticaden
Dinesh Chandrasekhar, Director, Product Marketing, @AppInt4All

2 © Hortonworks Inc. 2011–2018. All rights reserved
Streaming Analytics Reference Architecture
Data Flow Apps
Powered by NiFi
Kafka is Everywhere. Critical Component of Streaming Architectures
Kafka Producers Kafka Topics Kafka TopicsKafka Consumers & Producers Kafka Consumers
US West Fleet
Truck Sensors C++
Agent
US Central Fleet
Truck Sensors C++
Agent
US East Fleet
Truck Sensors C++
Agent
Analytics App 1
Analytics App 2
Analytics App 5
Analytics App 3
Analytics App 4

3 © Hortonworks Inc. 2011–2018. All rights reserved
Distinct Needs of 3 Personas/Teams using Kafka
Concerned with monitoring
the overall health of the
cluster and the infrastructure
it runs on
Concerned with monitoring
the Kafka entities associated
with their apps
Concerned with audit,
compliance, access control &
chain of custody requirements

4 © Hortonworks Inc. 2011–2018. All rights reserved
HORTONWORKS
DATA PLATFORM (HDP®)
DATA-AT-REST
PRODUCERS CONSUMERS
TOPICS
BROKERS
SMM REST Server
HORTONWORKS
DATAFLOW (HDF)
DATA-IN-MOTION
Optimize your
Kafka Clusters
Troubleshoot
Kafka Streams
Trace end-to-
end Kafka flows
Streams Messaging Manager (SMM)

5 © Hortonworks Inc. 2011–2018. All rights reserved
Cure is Here: Hortonworks Streams Messaging Manager (SMM)
What is SMM?
 Kafka Management and Monitoring tool
 Cure the “Kafka Blindness”
 Single Monitoring Dashboard for all your
Kafka Clusters across 4 entities
– Broker
– Producer
– Topic
– Consumer
 Supports multiple HDP and/or HDF Kafka
Clusters
 REST as a First Class Citizen
 Delivered as a DataPlane Service

6 © Hortonworks Inc. 2011–2018. All rights reserved
Simplify troubleshooting Kafka environments
• Visibility and control to the
Kafka Operations teams
• Intelligent filtering and sorting
• Grafana integration
• Ambari integration

7 © Hortonworks Inc. 2011–2018. All rights reserved
Visualize end-to-end Kafka stream flows
• Interactive visualization
• All streams/flows across all
Kafka clusters
• Analyze the stream dynamics
between producers and
consumers
• Intelligent filtering
• Throughput analysis

8 © Hortonworks Inc. 2011–2018. All rights reserved
Track data lineage and governance from edge-to-enterprise
• Tight integration with Apache
Atlas
• Metadata about Kafka Topic in
Atlas
• Complete data lineage across
multiple Kafka hops

9 © Hortonworks Inc. 2011–2018. All rights reserved
Extend monitoring / management capabilities with REST
• Developer / DevOps friendly
• Search for specific topics
• Drill down for messages
• SMM REST Server
• All capabilities exposed as
REST services

10 © Hortonworks Inc. 2011–2018. All rights reserved
Adopt hybrid cloud deployments for Kafka
• SMM as a DataPlane Platform
app
• Manage multiple Kafka clusters
from a single SMM instance
• Hybrid cloud model

11 © Hortonworks Inc. 2011–2018. All rights reserved
Demo

12 © Hortonworks Inc. 2011–2018. All rights reserved
Recap: What Did We Just Show? Tracking the flow of data across multiple Kafka
Hops with SMM & Atlas Integration-Powerful
Data Flow Apps
Powered by NiFi
US West Fleet
Truck Sensors C++
Agent
US Central Fleet
Truck Sensors C++
Agent
US East Fleet
Truck Sensors C++
Agent
Analytics App 3
Analytics App 2
Analytics App 1

13 © Hortonworks Inc. 2011–2018. All rights reserved
www.hortonworks.com/smm

Mais conteúdo relacionado

Mais procurados

Making Enterprise Big Data Small with Ease

Hortonworks

Data proliferation from 7+ billion humans and 20+ billion devices from every walk of life has been the focus in the last decade. With the velocity, variety and volume of data, every data organization’s goal shifted to protecting and monetizing data from rapidly growing network of IOT embedded objects and sensors. One of the true and tried business continuity methodology of storing and retrieving vast amount of data has been through replication of Hadoop systems on hybrid clouds and in geographically distributed data centers. Replication is similar to Blockchain using autonomous smart contracts instantiated on the metadata and data so that the replicated data follows a single source of truth. Replicas can be maintained across geographically distributed data centers giving greater risk tolerance capabilities to the businesses continuity plan for the data-sets. With intelligent predictive analytics based on usage patterns, dynamic tiering policies can be triggered on the data sets to provide true value-add to the data. The temperature of the data is used to move data between hot/warm/cold/archival storage based on configurable policies leading to greater reduction in total cost of ownership. Users in 2018 and beyond demand absolute availability of data as and when they desire. The dynamic data access management is fundamental concept to satisfy the business continuity plan. Seamless enterprise-grade disaster recovery to support business continuity use case has significant challenges around replicating security and governance on data-sets. In this talk we will discuss how the above challenge can be addressed for supporting seamless replication and disaster recovery for Hadoop-scale data. NIRU ANISETI, Product Manager, Hortonworks

The Implacable advance of the data

DataWorks Summit

TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA

Hortonworks

In recent years, big data has moved from batch processing to stream-based processing since no one wants to wait hours or days to gain insights. Dozens of stream processing frameworks exist today and the same trend that occurred in the batch-based big data processing realm has taken place in the streaming world so that nearly every streaming framework now supports higher level relational operations. On paper, combining Apache NiFi, Kafka, and Spark Streaming provides a compelling architecture option for building your next generation ETL data pipeline in near real time. What does this look like in an enterprise production environment to deploy and operationalized? The newer Spark Structured Streaming provides fast, scalable, fault-tolerant, end-to-end exactly-once stream processing with elegant code samples, but is that the whole story? We discuss the drivers and expected benefits of changing the existing event processing systems. In presenting the integrated solution, we will explore the key components of using NiFi, Kafka, and Spark, then share the good, the bad, and the ugly when trying to adopt these technologies into the enterprise. This session is targeted toward architects and other senior IT staff looking to continue their adoption of open source technology and modernize ingest/ETL processing. Attendees will take away lessons learned and experience in deploying these technologies to make their journey easier.

Using Spark Streaming and NiFi for the next generation of ETL in the enterprise

DataWorks Summit

Accelerating Data Science and Real Time Analytics at Scale

Hortonworks

Using your data smarter and faster than your peers could be the difference between dominating your market and merely surviving. Organizations are investing in IoT, big data, and data science to drive better customer experience and create new products, yet these projects often stall in ideation phase to a lack of global data management processes and technologies. Your new data architecture may be taking shape around you, but your goal of globally managing, governing, and securing your data across a hybrid, multi-cloud landscape can remain elusive. Learn how industry leaders are developing their global data management strategy to drive innovation and ROI. Presented at Gartner Data and Analytics Summit Speaker: Dinesh Chandrasekhar Director of Product Marketing, Hortonworks

Driving Digital Transformation Through Global Data Management

Hortonworks

Hortonworks DataFlow (HDF) is the complete solution that addresses the most complex streaming architectures of today’s enterprises. More than 20 billion IoT devices are active on the planet today and thousands of use cases across IIOT, Healthcare and Manufacturing warrant capturing data-in-motion and delivering actionable intelligence right NOW. “Data decay” happens in a matter of seconds in today’s digital enterprises. To meet all the needs of such fast-moving businesses, we have made significant enhancements and new streaming features in HDF 3.1. https://hortonworks.com/webinar/series-hdf-3-1-technical-deep-dive-new-streaming-features/

HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features

Hortonworks

Historically, insurers used 50-, 100-, and 500-year flood models for risk evaluation and pricing. The extreme weather events we have experienced in 2017 alone prove how dated these methods really are. To better understand their customers and potential current/future liability claims, forward thinking insurers are monitoring, analyzing, and integrating external data sources in real time (weather feeds from USGS.gov, news and stock feeds, and satellite imagery, to name just a few). By integrating and injecting these new data sources into their risk models and underwriting, insurers are better able to identify their risk appetites and effectively price. The session will include real-world case studies, including how a global P and C insurer is now quickly analyzing and monitoring 50,000 customers and targets, gaining new insights into the market. Another example is a global reinsurance and specialty company that now leverages digital news channels to monitor its risk portfolio for early warning claims indicators to help drive down loss costs. CINDY MAIKE, VP Industry Solutions, GM of Insurance, Hortonworks, Inc.

Risk listening: monitoring for profitable growth

DataWorks Summit

With the rise of IoT and the increasing complexity of applications, clouds, networks and infrastructure, the battle to keep your data and your infrastructure safe from attackers is getting harder. As groups of bad actors collaborate, sharing information and offering illegal access, and botnets as a service, terabits of attack can be launched cheaply. Meanwhile, it’s hard to find enough security analysts to catch and prevent these attacks. This is where community collaboration and open source efforts like Apache Metron come in. Metron presents a comprehensive framework for application and network, security built on Apache Hadoop and open source Streaming Analytics(ie Apache Nifi, Apache Kafka) tool’s highly scalable data management and processing stacks. Advanced features like profiling, machine learning, and visualization work with real-time streaming detection to make your SOC analysts more efficient, while the intrinsic extensibility of open source helps your data scientists get security insights out of the lab and into production fast. We will discuss and demonstrate how some real-world businesses and managed service providers are using Apache Metron to identify and solve security threats at scale, and some approaches and ideas for how the platform can fit into your security architecture.

Big Traffic, Big Trouble: Big Data Security Analytics

DataWorks Summit

View the recording of the meet up, including the live demos, here: https://www.youtube.com/watch?v=uaJWB3K8lkg Data science holds tremendous potential for organizations to uncover new insights and drivers of revenue and profitability. Big Data has brought the promise of doing data science at scale to enterprises, however this promise also comes with challenges for data scientists to continuously learn and collaborate. Data Scientists have many tools at their disposal such as notebooks like Juypter and Apache Zeppelin & IDEs such as RStudio with languages like R, Python, Scala and frameworks like Apache Spark. Given all the choices how do you best collaborate to build your model and then work through the development lifecycle to deploy it from test into production? Why Data Science on Big Data? In this meetup you will cover the attributes of a modern data science platform that empowers data scientists to build models using all the data in their data lake and foster continuous learning and collaboration. We will show a demo of Apache Zeppelin, Apache Spark, Apache Livy and Apache Hadoop with the focus on integration, security and model deployment and management. Data Science at Scale DEMO The demo will cover the Data Science life cycle: develop model in team environment, train the model with all the data on a Hadoop cluster, deploy model into production. The model will be a Spark ML model Practical ML with Apache Spark To deliver machine learning solutions data scientists not only need to fit models but also do familiar tasks data collection & wrangling, labelling, feature extraction and transformation, model tuning and evaluation, etc. Apache Spark provide provides a unified solution for all this under the same framework. For example, one can use Spark SQL to generate training data from different sources and then pass it directly to MLlib for feature engineering and model tuning, instead of using Hive/Pig for the first half and then downloading the data to a single machine to train models in R. The latter is actually very common in practice but painful to maintain. Spark MLlib makes life easier for data scientists and machine learning engineers so that they can focus on building better ML models and applications. We will discuss the underlying principles required to develop practical machine learning and data science pipelines and show some hands-on experience using Apache Spark to solve typical machine learning and data science problem. We will also have a short discussion about how Spark MLlib faces challenges from other machine learning libraries such as TensorFlow and XGBoost.

Enterprise Data Science at Scale Meetup - IBM and Hortonworks - Oct 2017

Hortonworks

One of the most fruit aspects of being chosen as a partner bank is that you can have a backend that can communicate directly with the client's system. With this partnership, Banco Santander has led to running a large series of third party applications on their banking system for many years. Banking is the most regulated sector to make day-to-day operations more interesting. Adapting the system to regulation is not optional and is mandatory. For today's banks internal and external audits are an important routine. Furthermore, considering that SCIB is a global player, this pattern is repeated in each country where the group exists. It can be said that it is a really interesting compound! Various kinds of third party systems are installed in many countries, coexisting with our centralized system, transmitting information mutually, being adjusted manually, data is aggregated / integrated at the back office. Spaghetti comes to mind when considering that all data comes and goes without delay. More and more, regulators and auditors are able to perfectly identify the origin of each data. This often means that you need to manually interfere in order to fully locate the data. Javier Nieto, active in Banco Santander's corporate investment banking architecture and innovation department, talks about integration challenges that Santander experienced when building an on-demand Data Lake to move to global big data.

Journey to Big Data: Main Issues, Solutions, Benefits

DataWorks Summit

Connecting Home/Building, Life and Car..The Importance of Insurance Risk Moni...

DataWorks Summit

Every business is looking for a game-changer in data science, machine learning, and AI. Most organizations are also looking for ways to tap into open-source and commercial data science tools such as Python, RStudio, Apache Spark, Jupyter, and Zeppelin notebooks, to accelerate predictive and machine learning model building and deployment while leveraging the scale, security and governance of the Hortonworks Data Platform and other commercial platforms. Ana Maria Echeverri will demonstrate how to accelerate data science, machine learning, and deep learning workflows by using IBM Watson Studio, an integrated environment for data scientists, application developers, and subject matter experts. This suite of tools allows to collaboratively connect to data, wrangle that data and use it to build, train and deploy models at scale while using Open Source skills (i.e.: Python) and expanding into cognitive capabilities through access to Watson APIs to build AI-powered applications. If you love Python and want to tap into the power of IBM Watson, this is the session for you.

Data Science at Speed. At Scale.

DataWorks Summit

Oil & Gas Big Data use cases

elephantscale

Liberty Global is one of the world’s largest international TV and broadband company, operating in multiple European countries, and with tens of millions of TV, broadband internet, telephony and mobile subscribers. The Data Solutions team's journey started last year with a strategic project that aimed to implement a state of the art Hybrid Cloud Big Data platform. In this talk, the Manager and the Platform Architect are presenting the team’s data acquisition journey which begins with implementing NiFi flows with simple Get-Put pattern and, in its the final iteration, produces a solution capable of generating complex flows automatically, leading the path to the DataOps way of working.

Data Acquisition Automation for NiFi in a Hybrid Cloud environment – the Path...

DataWorks Summit

Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. It provides an end-to-end platform that can collect, curate, analyze, and act on data in real-time, on-premises, or in the cloud with a drag-and-drop visual interface. It’s being used across industries on large amounts of data that had stored in isolation which made collaboration and analysis difficult. Join industry experts from Hortonworks and Attunity as they explain how Apache NiFi and streaming CDC technology provides a distributed, resilient platform for unlocking the value of data in new ways.

Unlock Value from Big Data with Apache NiFi and Streaming CDC

Hortonworks

Today’s Big Data teams demand solutions designed for Big Data that are optimized, secure, and adaptable to changing workload requirements. Working together, Hortonworks, IBM, and Attunity have designed an integrated solution that transfers large volumes of data to a platform that can handle rapid ingest, processing and analysis of data of all types from all sources, at scale. https://hortonworks.com/webinar/benefits-transferring-real-time-data-hadoop-scale-ibm-hortonworks-attunity/

Benefits of Transferring Real-Time Data to Hadoop at Scale

Hortonworks

With the explosive growth of IoT, the edge is predicted to grow to 25 billion connected devices by 2020. But, enterprises are still struggling to manage hundreds of devices that they have deployed. Not from a device management standpoint but more from a data management standpoint. Enterprises are unable to capture and process data directly from the edge devices for immediate analysis and gaining real-time actionable intelligence. So, if that is not possible, IoT initiatives are failing to become successful. How can an enterprise gather real-time data from edge devices? How can it change the behavior of such data collection processes? How can it ensure that data will be analyzed immediately? How can it understand the lineage of the data from edge to enterprise? How can it manage edge agents? What is an edge management hub? Attend this session to get a detailed understanding of key edge management challenges and how to address them with the correct solutions.

Addressing Challenges with IoT Edge Management

DataWorks Summit

Data and analytics are at the heart of the digital transformation. Implementing a modern data platform can be challenging; moreover, success requires a shift in culture. Andreas will discuss the ways Munich Re drives cultural and technological change within their company, focusing on three key elements: people, processes, and technology. What does it mean to be a data-driven organization? How can we provide self-service analytics to our internal and external customers in an agile way? How do we get the most value out of our big data lake? How does Munich Re balance technology and culture to meet the data demands of their business? Speaker Andreas Kohlmaier, Head of Data Engineering, Munich Re

Munich Re: Driving a Big Data Transformation

DataWorks Summit

The electric grid has evolved from linear generation and delivery to a complex mix of renewables, prosumer-generated electricity, and electric vehicles (EVs). Smart meters are generating loads of data. As a result, traditional forecasting models and technologies can no longer adequately predict supply and demand. Extreme weather, an aging infrastructure, and the burgeoning worldwide population are also contributing to increased outage frequency. In oil and gas, commodity pricing pressures, resulting workforce reductions, and the need to reduce failures, automate workflows, and increase operational efficiencies are driving operators to shift analytics initiatives to advanced data-driven applications to complement physics-based tools. While sensored equipment and legacy surveillance applications are generating massive amounts of data, just 2% is understood and being leveraged. Operationalizing it along with external datasets enables a shift from time-based to condition-based maintenance, better forecasting and dramatic reductions in unplanned downtime. The session includes plenty of real-world anecdotes. For example, how an electric power holding company reduced the time it took to investigate energy theft from six months to less than one hour, producing theft leads in minutes and an expected multi-million dollar ROI. How a global offshore contract drilling services provider implemented an open source IIoT solution across its fleet of assets in less than a year, enabling remote monitoring, predictive analytics and maintenance. Key takeaways: • How are new processes for data collection, storage and democratization making it accessible and usable at scale? • Beyond time series data, what other data types are important to assess? • What advantage are open source technologies providing to enterprises deploying IIoT? • Why is collaboration important across industrial verticals to increase IIoT open source adoption? Speaker Kenneth Smith, General Manager, Energy, Hortonworks

IIoT + Predictive Analytics: Solving for Disruption in Oil & Gas and Energy &...

DataWorks Summit

Mais procurados (20)

Making Enterprise Big Data Small with Ease

The Implacable advance of the data

TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA

Using Spark Streaming and NiFi for the next generation of ETL in the enterprise

Accelerating Data Science and Real Time Analytics at Scale

Driving Digital Transformation Through Global Data Management

HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features

Risk listening: monitoring for profitable growth

Big Traffic, Big Trouble: Big Data Security Analytics

Enterprise Data Science at Scale Meetup - IBM and Hortonworks - Oct 2017

Journey to Big Data: Main Issues, Solutions, Benefits

Connecting Home/Building, Life and Car..The Importance of Insurance Risk Moni...

Data Science at Speed. At Scale.

Oil & Gas Big Data use cases

Data Acquisition Automation for NiFi in a Hybrid Cloud environment – the Path...

Unlock Value from Big Data with Apache NiFi and Streaming CDC

Benefits of Transferring Real-Time Data to Hadoop at Scale

Addressing Challenges with IoT Edge Management

Munich Re: Driving a Big Data Transformation

IIoT + Predictive Analytics: Solving for Disruption in Oil & Gas and Energy &...

Semelhante a Curing Kafka Blindness with Hortonworks Streams Messaging Manager

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level

Hortonworks

HDF 3.1 : An Introduction to New Features

Timothy Spann

Paris FOD meetup - Streams Messaging Manager

Abdelkrim Hadjidj

Schema Registry & Stream Analytics Manager

Sriharsha Chintalapani

Top Three Big Data Governance Issues and How Apache ATLAS resolves it for the...

DataWorks Summit/Hadoop Summit

Running Apache NiFi with Apache Spark : Integration Options

Timothy Spann

Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies

DataWorks Summit/Hadoop Summit

Is your Enterprise Data lake Metadata Driven AND Secure?

DataWorks Summit/Hadoop Summit

The Atlas/ Ranger integration represents a paradigm shift for big data governance and security. Enterprises can now implement dynamic classification-based security policies, in addition to role-based security. Ranger’s centralized platform empowers data administrators to define security policy based on Atlas metadata tags or attributes and apply this policy in real-time to the entire hierarchy of data assets including databases, tables and columns.

Classification based security in Hadoop

Madhan Neethiraj

Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG

skumpf

Make Streaming IoT Analytics Work for You

Hortonworks

Imagine if you could build and deploy an end to end complex streaming analytics app on a streaming engine like Storm or Flink that did the following: 1. Joining Streams 2. Aggregations over Windows (Time or Count based) 3. Complex Event Processing 4. Pattern Matching 5. Model scoring. Now imagine implementing and deploying this without writing a single line of code in under 10 mins. Imagine no more; it is indeed here. In this talk, we will discuss an exciting open source project led by Hortonworks on building and deploying streaming applications using a drag and drop paradigm.

Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...

DataWorks Summit

Companies who use Kafka today struggle with monitoring and managing Kafka clusters. Kafka is a key backbone of IoT streaming analytics applications. The challenge is understanding what is going on overall in the Kafka cluster including performance, issues and message flows. No open source tool caters to the needs of different users that work with Kafka: DevOps/developers, platform team, and security/governance teams. See how the new Hortonworks Streams Messaging Manager enables users to visualize their entire Kafka environment end-to-end and simplifies Kafka operations. In this session learn how SMM visualizes the intricate details of how Apache Kafka functions in real time while simultaneously surfacing every nuance of tuning, optimizing, and measuring input and output. SMM will assist users to quickly understand and operate Kafka while providing the much-needed transparency that sophisticated and experienced users need to avoid all the pitfalls of running a Kafka cluster.

Curing the Kafka blindness—Streams Messaging Manager

DataWorks Summit

Streaming analytics manager

Sriharsha Chintalapani

Stream processing has become the defacto standard for building real-time ETL and Stream Analytics applications. We see batch workloads move into Stream processing to act on the data and derive insights faster. With the explosion of data with "Perishable Insights" such IoT and machine-generated data, Stream Processing + Predictive Analytics is driving tremendous business value. This is evidenced by the explosion of Stream Processing frameworks like proven and evolving Apache Storm and newer frameworks such as Apache Flink, Apache Apex, and Spark Streaming. Today, users have to choose and try to understand the benefits of each of these frameworks and not only that they have to learn the new APIs and also operationalize their applications. To create value faster, we are introducing new open source tool - Streamline. It is a self-service framework that will ease building streaming application and deploy the streaming application across multiple frameworks/engines that users prefer in a snap. It simplifies integration with Machine Learning models for scoring and classification of data for Predictive Analytics. It provides an elegant way to build Analytics dashboards to derive business insights out of the streaming data and to allow the business users to consume it easily. In this talk, we will outline the fundamentals of real-time stream processing and demonstrate Streamline capabilities to show how it simplifies building real-time streaming analytics applications. Speaker: Priyank Shah, Staff Software Engineer, Hortonworks

SAM - Streaming Analytics Made Easy

DataWorks Summit

Future of Data New Jersey - HDF 3.0 Deep Dive

Aldrin Piri

Presentation from Future of Data Boston Meetup on Oct 24, 2017. Streaming data is rich with insights but these insights can be difficult to find due to the difficulty of developing and deploying streaming applications. During this presentation we will show how to build and deploy a complex streaming application in a few minutes using open source tools. First we will build an application using Streaming Analytics Manager and Schema Registry that ingests data into Apache Druid. Then we will use Apache Superset to build beautiful, informative dashboards.

Unlocking insights in streaming data

Carolyn Duby

Apache Atlas: Tracking dataset lineage across Hadoop components

DataWorks Summit/Hadoop Summit

Curing the Kafka Blindness – Streams Messaging Manager

DataWorks Summit

Enhancing a customer experience has become essential for communication service providers to effectively manage customer churn and build a strong, long lasting relationship with their customers. This has become increasingly challenging as customer interactions occur across multiple channels. Understanding customer behavior and how it applies across channels is the key to ensuring the best level of experience is achieved by each customer. In this webinar Hortonworks and Apigee discuss how service providers can capture and visualize customer behavior across customer interaction points like call center events (IVR and chat) and combine it with network data, to predict customer calls and patterns of digital channel abandonment using Hadoop and predictive analysis and visualization tools.. We will identify ways to develop a 360 degree view across a customer’s household through an HDP Data Lake and visualize customer interaction patterns and predict expected behavior using Apigee Insights to identify and initiate the Next-Best-Action for a customer to ensure a superior level of customer experience.

Predicting Customer Experience through Hadoop and Customer Behavior Graphs

Hortonworks

Semelhante a Curing Kafka Blindness with Hortonworks Streams Messaging Manager (20)

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level

HDF 3.1 : An Introduction to New Features

Paris FOD meetup - Streams Messaging Manager

Schema Registry & Stream Analytics Manager

Top Three Big Data Governance Issues and How Apache ATLAS resolves it for the...

Running Apache NiFi with Apache Spark : Integration Options

Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies

Is your Enterprise Data lake Metadata Driven AND Secure?

Classification based security in Hadoop

Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG

Make Streaming IoT Analytics Work for You

Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...

Curing the Kafka blindness—Streams Messaging Manager

Streaming analytics manager

SAM - Streaming Analytics Made Easy

Future of Data New Jersey - HDF 3.0 Deep Dive

Unlocking insights in streaming data

Apache Atlas: Tracking dataset lineage across Hadoop components

Curing the Kafka Blindness – Streams Messaging Manager

Predicting Customer Experience through Hadoop and Customer Behavior Graphs

Mais de Hortonworks

Forrester forecasts* that direct spending on the Internet of Things (IoT) will exceed $400 Billion by 2023. From manufacturing and utilities, to oil & gas and transportation, IoT improves visibility, reduces downtime, and creates opportunities for entirely new business models. But successful IoT implementations require far more than simply connecting sensors to a network. The data generated by these devices must be collected, aggregated, cleaned, processed, interpreted, understood, and used. Data-driven decisions and actions must be taken, without which an IoT implementation is bound to fail. https://hortonworks.com/webinar/iot-predictions-2019-beyond-data-heart-iot-strategy/

IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy

Hortonworks

Getting the Most Out of Your Data in the Cloud with Cloudbreak

Hortonworks

The healthcare industry—with its huge volumes of big data—is ripe for the application of analytics and machine learning. In this webinar, Hortonworks and Quanam present a tool that uses machine learning and natural language processing in the clinical classification of genomic variants to help identify mutations and determine clinical significance. Watch the webinar: https://hortonworks.com/webinar/interpretation-tool-genomic-sequencing-data-clinical-environments/

Interpretation Tool for Genomic Sequencing Data in Clinical Environments

Hortonworks

Premier Inside-Out: Apache Druid

Hortonworks

Webinewbie to Webinerd in 30 Days - Webinar World Presentation

Hortonworks

Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...

Hortonworks

4 Essential Steps for Managing Sensitive Data

Hortonworks

A business culture that relies on gut checks and feelings for business decisions is a hard hurdle to overcome. Company culture is often the biggest barrier to moving a company toward data-driven decisions. There's a way to get there, when driven by company leaders. Here's how you do that: 1. Get comfortable with softer data sets 2. Must come from top-down 3. A structure where goals are clear 4. Right role for technology 5. Clear stewardship around data

5 Steps to Create a Company Culture that Embraces the Power of Data

Hortonworks

When it comes to the data lakes and data warehouses, there’s no shortage of controversy: Is one better than the other? The real answer is, there’s no need for heated debate—a data lake actually complements the data warehouse. Integrating a data lake with your EDW is really just an evolution of architecture that can provide you with a cross-environment that allows you to explore data creatively to yield great business insights. However, there’s a trick to making it work: EDW optimization. https://hortonworks.com/webinar/exploring-heated-completely-unnecessary-data-lake-debate/

Exploring the Heated-and Completely Unnecessary- Data Lake Debate

Hortonworks

In this webinar, we will hear from Mark McKinney, Director – Enterprise Data Analytics at Sprint about the business drivers, key success factors, and challenges faced while undertaking Sprint’s data modernization journey. You will hear how Sprint set about establishing a Hadoop data lake, ingested data from multiple environments, and overcame key skill shortages. You will also hear from Diyotta and Hortonworks about best practices for modernizing your data architecture to support transformational business initiatives. https://hortonworks.com/webinar/sprints-data-modernization-journey/

Sprint's Data Modernization Journey

Hortonworks

Modernize Your Existing EDW with IBM Big SQL & Hortonworks Data Platform

Hortonworks

Apache Ambari 2.5 helps customers simplify the experience for provisioning, managing, monitoring, securing and troubleshooting Hadoop deployments. Find out how the combination of Ambari and SmartSense delivers a path to success to help IT get Hadoop up and running effectively. The end result – you get the full business impact management and benefits of Big Data for your organization. https://hortonworks.com/webinar/streamline-apache-hadoop-operations-apache-ambari-smartsense/

Streamline Apache Hadoop Operations with Apache Ambari and SmartSense

Hortonworks

Hortonworks and SAS team up to discuss: *What technology is being used in this Global Retailer use case to collect and synchronize customer data across all channels *How the Retailer is able to analyze the data in real-time How they predict and optimize customer interactions to improve sales *The open source and cloud technologies that are on the horizon Webinar Sept. 7, 2017 https://hortonworks.com/webinar/architect-omnichannel-retail-solution-achieve-real-time-customer-insights/

How to Architect and Omnichannel Retail Solution to Achieve Real-Time Custome...

Hortonworks

The Life of a Hadoop Administrator, with and without SmartSense

Hortonworks

Enterprise Data Warehouse Optimization: 7 Keys to Success

Hortonworks

Mais de Hortonworks (15)

IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy

Getting the Most Out of Your Data in the Cloud with Cloudbreak

Interpretation Tool for Genomic Sequencing Data in Clinical Environments

Premier Inside-Out: Apache Druid

Webinewbie to Webinerd in 30 Days - Webinar World Presentation

Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...

4 Essential Steps for Managing Sensitive Data

5 Steps to Create a Company Culture that Embraces the Power of Data

Exploring the Heated-and Completely Unnecessary- Data Lake Debate

Sprint's Data Modernization Journey

Modernize Your Existing EDW with IBM Big SQL & Hortonworks Data Platform

Streamline Apache Hadoop Operations with Apache Ambari and SmartSense

How to Architect and Omnichannel Retail Solution to Achieve Real-Time Custome...

The Life of a Hadoop Administrator, with and without SmartSense

Enterprise Data Warehouse Optimization: 7 Keys to Success

Último

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

Angeliki Cooney has spent over twenty years at the forefront of the life sciences industry, working out of Wynantskill, NY. She is highly regarded for her dedication to advancing the development and accessibility of innovative treatments for chronic diseases, rare disorders, and cancer. Her professional journey has centered on strategic consulting for biopharmaceutical companies, facilitating digital transformation, enhancing omnichannel engagement, and refining strategic commercial practices. Angeliki's innovative contributions include pioneering several software-as-a-service (SaaS) products for the life sciences sector, earning her three patents. As the Senior Vice President of Life Sciences at Avenga, Angeliki orchestrated the firm's strategic entry into the U.S. market. Avenga, a renowned digital engineering and consulting firm, partners with significant entities in the pharmaceutical and biotechnology fields. Her leadership was instrumental in expanding Avenga's client base and establishing its presence in the competitive U.S. market.

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Angeliki Cooney

[BuildWithAI] Introduction to Gemini.pdf

Sandro Moreira

💥 You’re lucky! We’ve found two different (lead) developers that are willing to share their valuable lessons learned about using UiPath Document Understanding! Based on recent implementations in appealing use cases at Partou and SPIE. Don’t expect fancy videos or slide decks, but real and practical experiences that will help you with your own implementations. 📕 Topics that will be addressed: • Training the ML-model by humans: do or don't? • Rule-based versus AI extractors • Tips for finding use cases • How to start 👨‍🏫👨‍💻 Speakers: o Dion Morskieft, RPA Product Owner @Partou o Jack Klein-Schiphorst, Automation Developer @Tacstone Technology

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

UiPathCommunity

Architecting Cloud Native Applications

WSO2

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Manulife - Insurer Transformation Award 2024

The Digital Insurer

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Exploring Multimodal Embeddings with Milvus

Zilliz

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

When you’re building (micro)services, you have lots of framework options. Spring Boot is no doubt a popular choice. But there’s more! Take Quarkus, a framework that’s considered the rising star for Kubernetes-native Java. It always depends on what's best for your situation, but how to choose the best solution if you're comparing 2 frameworks? Both Spring Boot and Quarkus have their positives and negatives. Let us compare the two by live coding a couple of common use cases in Spring Boot and Quarkus. After this talk, you’ll be ready to get started with Quarkus yourself, and know when to select Quarkus or Spring Boot.

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

Jago de Vreede

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

The action of the next cyber saga takes place in the mystical lands of the Asia-Pacific region, where the main characters began their digital activities in the middle of 2021 and qualitatively strengthened it in 2022. Corporate espionage, document theft, audio recordings, and data leaks from messaging platforms were all a matter of one day for Dark Pink. Their geographical focus may have started in the Asia-Pacific region, but their ambitions knew no bounds, targeting a European government ministry in a bold move to expand their portfolio. Their victim profile was as diverse as a UN meeting, targeting military organizations, government agencies, and even a religious organization. Because discrimination is not a fashionable agenda. In the world of cybercrime, they serve as a reminder that sometimes the most serious threats come in the most unassuming packages with a pink bow.

Cyberprint. Dark Pink Apt Group [EN].pdf

Overkill Security

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

Curing Kafka Blindness with Hortonworks Streams Messaging Manager

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved Curing Kafka Blindness with Hortonworks Streams Messaging Manager (SMM) George Vetticaden, VP Product Management, @gvetticaden Dinesh Chandrasekhar, Director, Product Marketing, @AppInt4All

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved Streaming Analytics Reference Architecture Data Flow Apps Powered by NiFi Kafka is Everywhere. Critical Component of Streaming Architectures Kafka Producers Kafka Topics Kafka TopicsKafka Consumers & Producers Kafka Consumers US West Fleet Truck Sensors C++ Agent US Central Fleet Truck Sensors C++ Agent US East Fleet Truck Sensors C++ Agent Analytics App 1 Analytics App 2 Analytics App 5 Analytics App 3 Analytics App 4

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved Distinct Needs of 3 Personas/Teams using Kafka Concerned with monitoring the overall health of the cluster and the infrastructure it runs on Concerned with monitoring the Kafka entities associated with their apps Concerned with audit, compliance, access control & chain of custody requirements

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved HORTONWORKS DATA PLATFORM (HDP®) DATA-AT-REST PRODUCERS CONSUMERS TOPICS BROKERS SMM REST Server HORTONWORKS DATAFLOW (HDF) DATA-IN-MOTION Optimize your Kafka Clusters Troubleshoot Kafka Streams Trace end-to- end Kafka flows Streams Messaging Manager (SMM)

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved Cure is Here: Hortonworks Streams Messaging Manager (SMM) What is SMM?  Kafka Management and Monitoring tool  Cure the “Kafka Blindness”  Single Monitoring Dashboard for all your Kafka Clusters across 4 entities – Broker – Producer – Topic – Consumer  Supports multiple HDP and/or HDF Kafka Clusters  REST as a First Class Citizen  Delivered as a DataPlane Service

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved Simplify troubleshooting Kafka environments • Visibility and control to the Kafka Operations teams • Intelligent filtering and sorting • Grafana integration • Ambari integration

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved Visualize end-to-end Kafka stream flows • Interactive visualization • All streams/flows across all Kafka clusters • Analyze the stream dynamics between producers and consumers • Intelligent filtering • Throughput analysis

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved Track data lineage and governance from edge-to-enterprise • Tight integration with Apache Atlas • Metadata about Kafka Topic in Atlas • Complete data lineage across multiple Kafka hops

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved Extend monitoring / management capabilities with REST • Developer / DevOps friendly • Search for specific topics • Drill down for messages • SMM REST Server • All capabilities exposed as REST services

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved Adopt hybrid cloud deployments for Kafka • SMM as a DataPlane Platform app • Manage multiple Kafka clusters from a single SMM instance • Hybrid cloud model

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved Recap: What Did We Just Show? Tracking the flow of data across multiple Kafka Hops with SMM & Atlas Integration-Powerful Data Flow Apps Powered by NiFi US West Fleet Truck Sensors C++ Agent US Central Fleet Truck Sensors C++ Agent US East Fleet Truck Sensors C++ Agent Analytics App 3 Analytics App 2 Analytics App 1

Notas do Editor

SMM UI powered by first class REST Services via the SMM REST Admin Server Monitoring Rest Endpoints can be used to integrate with APM/Alerting/Ticketing solutions Powered by Apache Knox Installed via an Ambari Management Pack on a target HDP/HDF cluster where Kafka Service is running Supports both HDP & HDF Platforms Open Source / AGPL licensed projected led by Hortonworks

Curing Kafka Blindness with Hortonworks Streams Messaging Manager

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Curing Kafka Blindness with Hortonworks Streams Messaging Manager

Semelhante a Curing Kafka Blindness with Hortonworks Streams Messaging Manager (20)

Mais de Hortonworks

Mais de Hortonworks (15)

Último

Último (20)

Curing Kafka Blindness with Hortonworks Streams Messaging Manager

Notas do Editor