Unlock the value of your big data infrastructure

•Transferir como PPTX, PDF•

0 gostou•196 visualizações

ManageEngine, Zoho Corporation

Learn how to optimize big data apps and leverage graph databases for your business.

Tecnologia

Applications Manager
Unlock the value of your big data infrastructure

Presenter : Selvaraj
Please comment 'YES' in the chat panel if the presenter's voice is audible
and the presentation screen is clear.

Week Module Date Schedule Status
1. Key to Optimal End User
Experience
Nov 13 6:00 AM GMT
11:00 AM EST
Completed
Completed
2. Unlock the value of your big data
infrastructure
Nov 20 6:00 AM GMT
11:00 AM EST
Completed
In Progress
3. Building the right website
monitoring strategy
Dec 4 6:00 AM GMT
11:00 AM EST
Upcoming
Upcoming
4. Monitoring Cloud,Containers and
hyperconverged infrastructure
Dec 11 6:00 AM GMT
11:00 AM EST
Upcoming
Upcoming

Agenda
Big data apps:
Apache Hadoop & Spark
Leverage graph database for your
business
NoSQL databases in Big Data
frameworks
Forecasting utilization trends and
planning capacity

The Hadoop Architecture
Hadoop
Common
HDFS
YARN
MapReduce

The key Hadoop components
• HDFS
• MapReduce
• YARN
• NoSQL Database

Apache Spark architecture
Spark Driver program
Spark Scheduler
Worker node Worker node
HDFS
NoSQ
L
Mesos /YARN

Spark key components
• Drivers and Workers in Spark
• Memory and resource utilization
• Spark cores and applications
• Spark RDDs
• NoSQL database

Leverage graph database for your business

The key metrics
• Nodes & Relationship
• Memory
• Page Cache
• Transactions

MongoDB - Network & connections
• Connections used
• Connections available
• Network traffic received
• Network traffic sent

MongoDB - Lock & Asserted errors
• Read Lock, Write Lock
• Message, User asserts

Cassandra - Keyspace latency
• Read Latency (/min)
• Write Latency (/min)

Redis - Keys
• Number of keys
• Expired keys

Redis - Clients
• Connected clients
• Blocked clients

Discovery & mapping
Disk and Memory issues in NoSQL databases

Cassandra Memory Utilization
• Free Memory
• Used Memory

Cassandra-Cluster
• Node state
• Unreachable nodes

Discovery & mapping
Forecasting utilization trends and planning capacity

Trend analysis report
Pull at a glance report of the DB server to have a quick glance at all monitored attributes

Trend analysis report
Quick glance at the performance of all monitored attributes:
• History Report
• Hour of Day Report
• Day of Week Report
• Statistical Report
• Heat Chart

Compare performance attribute reports of different monitors.
View & compare reports

• Balance hardware costs
with computing needs.
• Plan future utilization
requirements.
Forecast report

Contact : eval-apm@manageengine.com
Online Demo : demo.appmanager.com
Quick Recap
Big Data and the non-relational database
• Big data apps: Apache Hadoop & Spark
• Leverage graph database for your business
• NoSQL databases in Big Data frameworks
• Forecasting utilization trends and planning capacity

Contact : eval-apm@manageengine.com
Online Demo : demo.appmanager.com
Questions?

Contact : eval-apm@manageengine.com
Online Demo : demo.appmanager.com
Can I discover all the nodes in Mongo
DB through Applications Manager?

Contact : eval-apm@manageengine.com
Online Demo : demo.appmanager.com
Am already monitoring my Cassandra
DB with AppManager, However I need
to customize polling for few metrics.
Is it possible?

Contact : eval-apm@manageengine.com
Online Demo : demo.appmanager.com
Can I get a automatic report of
Availability of my databases
everyday?

Upcoming topics : Dec 4, 2019
Contact : eval-apm@manageengine.com
Online Demo : demo.appmanager.com
Building the right website monitoring strategy
• Track key metrics of Apache, IIS, NGINX, NGINX Plus, etc.
• Optimize individual URLs for better end user experience
• Simulate web transactions with synthetic transaction monitoring
• Detect unauthorized content changes in your website

Contact : eval-apm@manageengine.com
Online Demo : demo.appmanager.com

Mais conteúdo relacionado

Mais procurados

Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal. In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.

How a distributed graph analytics platform uses Apache Kafka for data ingesti...

HostedbyConfluent

In this session, learn how Dropbox scales to provide one of the largest cloud storage and file sharing services in the world. Hear how Dropbox leverages Amazon EC2 to run varied workloads including thumbnail generation and document prevent, as well as document indexing to support full-text search. Dropbox presents ''Livefill'' - a generic framework built on top of Amazon SQS. Livefill enables them to trigger customizable data-processing workloads on data stored in Amazon S3 and helps them support more than 200,000 workload requests per second, spread across thousands of machines.

(PFC308) How Dropbox Scales Massive Workloads Using Amazon SQS | AWS re:Inven...

Amazon Web Services

Taboola Road To Scale With Apache Spark

tsliwowicz

Cybersecurity is a broad topic and many commercial products are related to it. We demonstrate a fundamental concept in network analysis: re-construction and visualization of temporal networks. Furthermore, we apply the method to describe operational conditions of a Hadoop cluster. Our experiments provide first results and allow a classification of the cluster state related to current workloads. The temporal networks show significant differences for different operation modes. In reallity we would expect mixed workloads. If such workload parameters are known, we are able to handle a-typical events accordingly - which means, we are able to create alerts based on context information, rather than only the package content. We show an end-to-end example: (1) Data collection is done via python, using the sniffer script; (2) using Apache Hive and Apache Spark we analyze the network traffic data and create the temporary network. Finally, we are able to visualize the results using Gephi in step (3). In a next step, we plan to contribute to the Apache Spot project.

PCAP Graphs for Cybersecurity and System Tuning

Dr. Mirko Kämpf

(Jeff Sharpe + Alex Srisuwan, Capital One) Kafka Summit SF 2018 Using Kafka as a platform messaging bus is common, but bridging communication between real-time and asynchronous components can become complicated, especially when dealing with serverless environments. This has become increasingly common in modern banking where events need to be processed at near-real-time speed. Serverless environments are well-suited to address these needs, and Kafka remains an excellent solution for providing the reliable, resilient communication layer between serverless components and dedicated stream processing services. In this talk, we will examine some of the strengths and weaknesses of using Kafka for real-time communication, some tips for efficient interactions with Kafka and AWS Lambda, and a number of useful patterns for maximizing the strengths of Kafka and serverless components.

Kafka for Real-Time Event Processing in Serverless Environments

confluent

We often need to build applications that analyze Kafka data to unlock the most value from event streams, so how can organizations build these real-time analytics applications? In this talk, we examine an indexing approach that enables fast SQL analytics on data from Kafka, without data flattening or denormalization. Rockset is the real-time indexing database that builds an inverted index, a columnar index and a row index on all fields of your Kafka messages, including nested fields and arrays. This Converged Index accelerates various types of analytic queries–search, aggregations and joins–without the need to denormalize or transform data for performance reasons. With indexing delivering significant gains in query performance, we also need to index new data in a timely manner. We discuss several strategies used for efficient ingestion and indexing from Kafka, including rollups, write optimizations on the underlying RocksDB storage engine, and the disaggregation of ingest and query compute.

Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...

HostedbyConfluent

Today’s products - devices, software and services - are well instrumented to permit users, vendors and service providers to gather maximum insight into how they are used, when they need repair and many other operational insights. Ensuring that products can rapidly adapt to a constantly changing environment and changing customer needs requires that the events they generate are analyzed continuously and in context. Insights can be synthesized from many sources in context - geospatial and proximity, trajectory and even predicted future states.Customers, vendors and service providers need to analyze, learn, and predict directly from streaming events because data volumes are huge and automated responses must often be delivered in milliseconds. To achieve insights quickly, we need to build models on-the-fly whose predictions are accurate and in sync with the real world, often to support automation. Many insights depend on analyzing the joint evolution of data sources whose behavior is correlated in time or space.In this talk we present Swim, an Apache 2.0 licensed platform for continuous intelligence applications. Swim builds a fluid model of data sources and their changing relationships in real-time - Swim applications analyze, learn and predict directly from event data. Swim applications integrate with Apache Kafka for event streaming. Developers need nothing more than Java skills. Swim deploys native or in containers on k8s, with the same code in each instance. Instances link to build an application layer mesh that facilitates distribution and massive scale without sacrificing consistency. We will present several continuous intelligence applications in use today that depend on real-time analysis, learning and prediction to power automation and deliver responses that are in sync with the real-world. We will show how easy it is to build, deploy and run distributed, highly available event streaming applications that analyze data from hundreds of millions of sources - petabytes per day. The architecture is intuitively appealing and blazingly fast.

Continuous Intelligence for Customer Service Using Kafka Event Streams | Simo...

HostedbyConfluent

When the University of California, San Diego launched its largest investment in tech in 2018, they planned to future proof their business processes and systems. Unexpectedly, it also prepared them to handle a global pandemic that changed every norm for the campus. With shelter-in-place orders taking immediate effect, they needed to quickly set up a robust online learning platform - one with powerful analytics to track student success. And, for the times students and staff are on campus, a contact tracing application was essential for their safety. We’d like to offer a conversation with Scott Lee to tell you more about UC San Diego’s rapid transformation from a traditional, on-campus institution to one of the leading examples of remote learning, and the critical role data connectivity played in making this possible.

Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...

HostedbyConfluent

Since its release in 2018, KSQL has grown from interesting curiosity into ksqlDB - a production grade streaming system. What does it look like to run KSQL in the enterprise? How has the promise of the Kafka Streams with an SQL dialect worked in the wild? Let's explore stream processing with ksqlDB in the enterprise. How is it used to rapid prototyping; for taking an idea to production. Using the flexible scripting to help teams with error discover and system introspection. Plus how extended teams can use KSQL as a stepping stone for building and sharing real-time scoring and streaming insights. This session will cover production deployments of ksqlDB in banking, finance, transport and insurance. What can go wrong, and what can go right. See how teams embrace the technology to solve stream processing challenges.

KSQL-ops! Running ksqlDB in the Wild (Simon Aubury, ThoughtWorks) Kafka Summi...

confluent

Stateful Interaction In Serverless Architecture With Redis: Pyounguk Cho

Redis Labs

To remain competitive, organizations need to democratize access to fast analytics, not only to gain real-time insights on their business but also to power smart apps that need to react in the moment. In this session, you will learn how Kafka and SingleStore enable modern, yet simple data architecture to analyze both fast paced incoming data as well as large historical datasets. In particular, you will understand why SingleStore is well suited process data streams coming from Kafka.

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

HostedbyConfluent

Today, many companies that have lots of data are still struggling to derive value from machine learning (ML) and data science investments. Why? Accessing the data may be difficult. Or maybe it’s poorly labeled. Or vital context is missing. Or there are questions around data integrity. Or standing up an ML service can be cumbersome and complex. At Nuuly, we offer an innovative clothing rental subscription model and are continually evolving our ML solutions to gain insight into the behaviors of our unique customer base as well as provide personalized services. In this session, I’ll share how we used event streaming with Apache Kafka® and Confluent Cloud to address many of the challenges that may be keeping your organization from maximizing the business value of machine learning and data science. First, you’ll see how we ensure that every customer interaction and its business context is collected. Next, I’ll explain how we can replay entire interaction histories using Kafka as a transport layer as well as a persistence layer and a business application processing layer. Order management, inventory management, logistics, subscription management – all of it integrates with Kafka as the common backbone. These data streams enable Nuuly to rapidly prototype and deploy dynamic ML models to support various domains, including pricing, recommendations, product similarity, and warehouse optimization. Join us and learn how Kafka can help improve machine learning and data science initiatives that may not be delivered to their full potential.

Maximize the Business Value of Machine Learning and Data Science with Kafka (...

confluent

The Logentries and Hosted Graphite integration allows you to connect two of your favorite Ops tools to easily extract important data from your log files, visualize them as metrics, and share them in Hosted Graphite dashboards. • Integrate your systems to extract the metrics you need, from both your applications and log data. • Set-up log metric dashboards based on common use cases (e.g. error tracking, performance, app usage). • Get off the "complexity elevator" of hosting your own in-house logging or graphite solutions. • Delight your team and organization with valuable metrics and performance insights.

Correlate Log Data with Business Metrics Like a Jedi

Trevor Parsons

RealTime Recommendations @Netflix - Spark

Nitin S

Monitoring Large-Scale Apache Spark Clusters at Databricks

Anyscale

Advanced time series analysis (TSA) requires very special data preparation procedures to convert raw data into useful and compatible formats. In this presentation you will see some typical processing patterns for time series based research, from simple statistics to reconstruction of correlation networks. The first case is relevant for anomaly detection and to protect safety. Reconstruction of graphs from time series data is a very useful technique to better understand complex systems like supply chains, material flows in factories, information flows within organizations, and especially in medical research. With this motivation we will look at typical data aggregation patterns. We investigate how to apply analysis algorithms in the cloud. Finally we discuss a simple reference architecture for TSA on top of the Confluent Platform or Confluent cloud.

Time Series Analysis Using an Event Streaming Platform

Dr. Mirko Kämpf

(Stephen Parente + Jeff Field, Blizzard) Kafka Summit SF 2018 Blizzard’s global data platform has become a driving force in both business and operational analytics. As more internal customers onboard with the system, there is increasing demand for custom applications to access this data in near real time. In order to avoid many independent teams with varying levels of Kafka expertise all accessing the firehose from our critical production Kafkas, we developed our own pub-sub system on top of Kafka to provide specific datasets to customers on their own cloud deployed Kafka clusters.

You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard

confluent

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020

confluent

DataOps challenges us to build data experiences in a repeatable way. For those with Kafka, this means finding a means of deploying flows in an automated and consistent fashion. The challenge is to make the deployment of Kafka flows consistent across different technologies and systems: the topics, the schemas, the monitoring rules, the credentials, the connectors, the stream processing apps. And ideally not coupled to a particular infrastructure stack. In this talk we will discuss the different approaches and benefits/disadvantages to automating the deployment of Kafka flows including Git operators and Kubernetes operators. We will walk through and demo deploying a flow on AWS EKS with MSK and Kafka Connect using GitOps practices: including a stream processing application, S3 connector with credentials held in AWS Secrets Manager.

DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...

HostedbyConfluent

Watch this talk here: https://www.confluent.io/online-talks/siem-modernization-build-a-situationally-aware-organization-with-apache-kafka Of all security breaches, 85% are conducted with compromised credentials, often at the administration level or higher. A lot of IT groups think “security” means authentication, authorization and encryption (AAE), but these are often tick-boxes that rarely stop breaches. The internal threat surfaces of data streams or disk drives in a raidset in a data centre are not the threat surface of interest. Cyber or Threat organizations must conduct internal investigations of IT, subcontractors and supply chains without implicating the innocent. Therefore, they are organizationally air-gapped from IT. Some surveys indicate up to 10% of IT is under investigation at any given time. Deploying a signal processing platform, such as Confluent Platform, allows organizations to evaluate data as soon as it becomes available enabling them to assess and mitigate risk before it arises. In Cyber or Threat Intelligence, events can be considered signals, and when analysts are hunting for threat actors, these don't appear as a single needle in a haystack, but as a series of needles. In this paradigm, streams of signals aggregate into signatures. This session shows how various sub-systems in Apache Kafka can be used to aggregate, integrate and attribute these signals into signatures of interest. In this talk you will learn: -The current threat landscape -The difference between Security and Threat Intelligence -The value of Confluent platform as an ideal complement to hardware endpoint detection systems and batch-based SIEM warehouses

SIEM Modernization: Build a Situationally Aware Organization with Apache Kafka®

confluent

Mais procurados (20)