Apache Pulsar, Supporting the Entire Lifecycle of Streaming Data

•Transferir como PPTX, PDF•

0 gostou•205 visualizações

We have long stressed that there is more and more a need for unified messaging and streaming and that Apache Pulsar is the platform that better supports this vision and makes it possible, at a large scale. In his talk, Matteo Merli will show how we can take this messaging & streaming unified paradigm one step further, to fully take advantage of their integration. The result is a drastically simplified architecture: a single system that is able to support the data throughout its entire lifecycle, from when the event is happening down to the historical archiving. The ramifications of this shift are big, as we can see Pulsar is in the perfect spot to enable tighter integration between online and offline worlds.

Tecnologia

Pulsar Virtual Summit Europe 2021
Supporting the
Entire Lifecycle of
Streaming Data

Pulsar Virtual Summit Europe 2021
Matteo Merli
CTO @ StreamNative
Co-Creator and PMC Chair for Apache Pulsar
PMC Member Apache BookKeeper
Prev: Splunk, Streamlio, Yahoo

Pulsar Virtual Summit Europe 2021
Pulsar and the data in motion
Messaging
Message passing
between components,
application, services
Streaming
Analyze events that just
happened

Pulsar Virtual Summit Europe 2021
Use Cases
Messaging
● OLTP, Integration
○ Main challenges:
○ Latency
○ Availability
○ Data durability
○ High level features
○ Routing, DLQ, delays,
individual acks
Streaming
● Real-time analytics
● Main challenges:
○ Throughput
○ Ordering
○ Stateful processing
○ Batch + Real-Time

Pulsar Virtual Summit Europe 2021
How can Pulsar support both?
Scalable Log Storage
+
Flexible messaging semantics

1.
They get applied in different stages
of handling the same data
Why is messaging + streaming so important?

2.
Integration between different
systems is often:
complex, fragile and inefficient
Why is messaging + streaming so important?

Interactions are complex
1. Who is responsible for getting a feed of events?
2. Where is the data stored?
3. How can we feed updates back to online data stores?
4. What happens when systems are not available?
5. How is the schema of data enforced?
6. What is the security model?

If we could just share a single data platform…

1. Different kind of services can all share the same Pulsar
cluster
2. Decouple services through topics
3. Provide isolation & availability guarantees
How to tackle integration points

1. Single tooling
2. Supports many different APIs
3. Unified AuthN & AuthZ for access to data
4. Supports end-to-end schema validation
Using Pulsar as the single data platform

1. Keep 1 copy of the data
2. Single source of truth
3. No single component is the “owner” of the data
4. Consumer components can get access directly to the
source
5. There is no need for additional ad-hoc integrations
Removing “data ownership”

The data life-cycle
The data can reside in Pulsar for its entire life-
cycle, since its inception and to the long-term
storage.

The data life-cycle
1. Events are happening — (real-time)
2. Streaming Analytics — (< 1 second)
3. Data replay — (1 hour / days)
4. Long term storage and Batch Processing —
(days / months)

Managing the data life-cycle
1. Store the data only once
2. Make it available to all interested parties
3. Able to hold the data for extended time

Managing the data life-cycle
Pulsar provides infinite stream-storage
abstraction
a. Low latency writes
b. Isolation between tail read and catch-up
c. Long term tiered storage

Isolating the different workloads
Every aspect of Pulsar is designed for multi-
tenancy and multi-workloads:
● IO Isolation
● Limits to access to resources: throttling, quotas,
etc..

Conclusion
Pulsar is uniquely suited to be the data
platform that powers all the data in motion
use cases and all their interactions

Mais conteúdo relacionado

Mais procurados

Building event streaming pipelines using Apache Pulsar

StreamNative

MLconf 2022 NYC Event-Driven Machine Learning at Scale https://mlconf.com/agenda/mlconf-2022-NYC/ https://mlconf.com/speakers/timothy-spann/ Bio Tim Spann is a Developer Advocate for StreamNative. He works with StreamNative Cloud, Apache Pulsar, Apache Flink, Flink SQL, Apache NiFi, MiniFi, Apache MXNet, TensorFlow, Apache Spark, Big Data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science. He is currently working on a book about the FLiP Stack. Upcoming Abstract Summary Event-Driven Machine Learning at Scale Utilizing the FLiP stack we can drive NLP and other machine learning classifications in the cloud, on-premise, hybrid and at the edge utilizing the open source Apache Pulsar project. We often utilize the full FLiPN stack which includes Apache Flink, Apache NiFi and Apache Pulsar. Apache Pulsar is a unified messaging platform that can host and execute classifications as events arrive asynchronously to topics. We deploy our functionality in lightweight functions that can run as processes, threads or in K8. We often use the serverless FunctionMesh to run these. This allows us to run machine learning classes built in Go, Python and Java. We will show you utilizing Vader Sentiment, Pytorch, TensorFlow, DJL.AI, MXNet and other machine learning options with ease.

MLconf 2022 NYC Event-Driven Machine Learning at Scale.pdf

Timothy Spann

Apache PulsarFirst Overview

Ricardo Paiva

I Heart Log: Real-time Data and Apache Kafka

Jay Kreps

BIGO currently has two major video products and services, Live and Likee. At present, BIGO Live's live broadcast business has covered more than 150 countries and regions and Likee short video also has more than 100 million users. Being well received and approved among young people, our products are popular all around the world. In the past technical architecture, BIGO adopted open-source Kafka as a basic service to support real-time data processing and analyzing. And based on this, BIGO has built a complete recommendation infrastructure to provide users with high-quality recommendation services. However, as the business continues to develop rapidly, the scale of our processing messages has entered a trillion scale and the past architecture has encountered huge challenges. Regarding these problems, we got in touch with Apache Pulsar. With the gradual deepening of Pulsar, we have benefited a lot from these excellent features brought by Apache Pulsar such as hierarchical persistent storage, low E2E latency and horizontal scalability. These advantages have also helped us overcome many difficulties in the production system. This talk will introduce the development and construction of BIGO's messaging platform based on Pulsar. We will start with message processing, storage, etc., and deeply analyze the performance bottlenecks that may be encountered in the production environment, and share our production processing practice experience.

The Evolution of Trillion-level Real-time Messaging System in BIGO - Puslar ...

StreamNative

In this talk I will present a technique for deploying machine learning models to provide real-time predictions using Apache Pulsar Functions. In order to provide a prediction in real-time, the model usually receives a single data point from the caller, and is expected to provide an accurate prediction within a few milliseconds. Throughout this talk, I will demonstrate the steps required to deploy a fully-trained ML that predicts the delivery time for a food delivery service based upon real-time traffic information, the customer's location, and the restaurant that will be fulfilling the order.

Deploying Machine Learning Models with Pulsar Functions - Pulsar Summit Asia...

StreamNative

How many connects can you run in a single cluster? Disney + Hotstar runs over 10 different connect clusters with over 2000+ connectors. In this talk, we share our experience of running Kafka connect at scale. We will walk through our decisions of using one cluster vs many and how the improvements in the connect ecosystem like incremental rebalancing have allowed us to scale to thousands of connects. We will also discuss challenges with scaling up & down connect workers while keeping the ecosystem stable & present a wishlist of the missing features in this distributed task framework.

How Much Can You Connect? | Bhavesh Raheja, Disney + Hotstar

HostedbyConfluent

What’s easier than building a data pipeline nowadays? You add a few Apache Kafka clusters and a way to ingest data (probably over HTTP), design a way to route your data streams, add a few stream processors and consumers, integrate with a data warehouse… wait, this looks like a lot of things, doesn’t it? And you probably want to make it highly scalable and available too. Join this session to learn best practices for building a data pipeline, drawn from my experience at Activision/Demonware. I’ll share the lessons learned about scaling pipelines, not only in terms of volume, but also in terms of supporting more games and more use cases. You’ll also hear about message schemas and envelopes, Apache Kafka organization, topics naming conventions, routing, reliable and scalable producers and the ingestion layer, as well as stream processing.

Building Scalable and Extendable Data Pipeline for Call of Duty Games (Yarosl...

confluent

High performance messaging with Apache Pulsar

Matteo Merli

Nowadays, real-time computation is heavily used in cases such as online product recommendation, online payment fraud detection and etc.. In the streaming pipeline, Kafka is normally used to store a day/week data, but won't store years-long data, as in looking at the trend historically. So, a batch pipeline is needed for historical data computation. Thus, it's where the Lambda architecture comes in. Lambda has been proved to be effective, and a good balance of speed and reliability. We have been running many systems with Lambda architecture for many years. But the biggest detraction to Lambda architecture has been the need to maintain two distinct (and possibly complex) systems to generate both batch and streaming layers. With that, we have to split our business logic into many segments across different places, which is a challenge to maintain as the business grows and it also increases communication overhead. Secondly, the data are duplicated in two different systems, and we have to move data among different systems for processing. With those challenges, we have been searching for alternatives and found Apache Pulsar a great fit. In this topic, I will show how we solve those problems with Apache Pulsar by making pulsar a unified storage backend for both batch and streaming pipeline, a solution that simplifies the s/w stack, lifts up our work efficiency and lowers the cost at the same time.

Unify Storage Backend for Batch and Streaming Computation with Apache Pulsar_...

StreamNative

Jay Kreps, Open Source Visionary and Co Founder of Confluent and several open source projects will be visiting LA. I have asked him to come present at our group. He will present his vision and will answer questions regarding Kafka and other projects Bio:- Jay is the co-founder and CEO at Confluent a company built around realtime data streams and the open source messaging system Apache Kafka. He is the original author of several of open source projects including Apache Kafka, Apache Samza, Voldemort, and Azkaban.

An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.

Data Con LA

Modern search systems provide incredible feature sets, developer-friendly APIs, and low latency indexing and query response. By some measures, these systems operate "at scale," but rarely is that quantified. Customers of Rocana typically look to push ingest rates in excess of 1 million events per second, retaining years of data online for query, with the expectation of sub-second response times for any reasonably sized subset of data. We quickly found that the tradeoffs made by general purpose search systems, while right for common use cases, were less appropriate for these high cardinality, large scale use cases. This session details the architecture, tradeoffs, and interesting implementation decisions made in building a new time series optimized distributed search system using Apache Lucene, Kafka, and HDFS. Data ingestion and durability, index and metadata organization, storage, query scheduling and optimization, and failure modes will be covered. Finally, a summary of the results achieved will be shown.

High cardinality time series search: A new level of scale - Data Day Texas 2016

Eric Sammer

The business systems of an organization are a continuous source of events. Each system also needs to know about events happening in the other systems. Exchanging these events through direct API calls creates a web of inter-dependencies, is fragile and fails to scale. We examine how this problem can be solved through the use of right integration patterns implemented as a light-weight event hub that leverages the power of Kafka and Confluent to operate at enterprise scale. We demonstrate how JavaScript with its event-driven programming model can be a good fit for implementing an event hub that ensures guaranteed message delivery in the face of failures within the individual subscriber systems. Many organizations having large engineering teams skilled in NodeJS and a multitude of NodeJs applications. We show how these teams can easily leverage the power of Kafka and scale their applications with the right architectural building blocks. We also offer insights from our own experience of building NodeJS based Kafka applications.

Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix

HostedbyConfluent

The Apache Pulsar community is collaborating with the Delta Lake community to add up to both ecosystems: Pulsar - Delta Lake Connector. In this session, we will first provide an overview of Pulsar - Delta Lake connector and Delta Lake Standalone Reader, then introduce the design of Pulsar - Delta Lake CDC source connector including how to capture data change of Delta lake and how to recover from last checkpoint with the help of Pulsar Function state store . We will also discuss the scalability of this Pulsar - Delta lake CDC source connector.

Pulsar in the Lakehouse: Overview of Apache Pulsar and Delta Lake Connector -...

StreamNative

Introduction to Apache BookKeeper Distributed Storage

Streamlio

Pulsar - Distributed pub/sub platform

Matteo Merli

Presented at SF Big Analytics Meetup Online event processing applications often require the ability to ingest, store, dispatch and process events. Until now, supporting all of these needs has required different systems for each task -- stream processing engines, messaging queuing middleware, and pub/sub messaging systems. This has led to the unnecessary complexity for the development of such applications and operations leading to increased barrier to adoption in the enterprises. In this talk, Karthik will outline the need to unify these capabilities in a single system and make it easy to develop and operate at scale. Karthik will delve into how Apache Pulsar was designed to address this need with an elegant architecture. Apache Pulsar is a next generation distributed pub-sub system that was originally developed and deployed at Yahoo and running in production in more than 100+ companies. Karthik will explain how the architecture and design of Pulsar provides the flexibility to support developers and applications needing any combination of queuing, messaging, streaming and lightweight compute for events. Furthermore, he will provide real life use cases how Apache Pulsar is used for event processing ranging from data processing tasks to web processing applications.

Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar

Karthik Ramasamy

Apache Pulsar at Yahoo! Japan

StreamNative

The last few years have seen the emergence of Serverless as a paradigm for event streaming. Its very simple programming model has attracted developers in droves. At the same time, its ability to elastically scale has simplified operations significantly. Combined together with the ubiquity of their presence across all cloud providers, serverless today has become the leading choice to do event processing at scale for a lot of companies. In this talk, Sijie Guo from StreamNative will explore how the serverless paradigm is applied to event streaming in Apache Pulsar, a next-generation event streaming system. Pulsar provides native support for serverless functions where the events are processed as soon as they arrive in a streaming manner and that provides flexible deployment options (thread, process, container). He will describe how these serverless functions make data engineering easier and share the real world usage of Pulsar Functions.

Serverless Event Streaming with Pulsar Functions

StreamNative

Apache kafka

Daan Gerits

Mais procurados (20)

Building event streaming pipelines using Apache Pulsar

MLconf 2022 NYC Event-Driven Machine Learning at Scale.pdf

Apache PulsarFirst Overview

I Heart Log: Real-time Data and Apache Kafka

The Evolution of Trillion-level Real-time Messaging System in BIGO - Puslar ...

Deploying Machine Learning Models with Pulsar Functions - Pulsar Summit Asia...

How Much Can You Connect? | Bhavesh Raheja, Disney + Hotstar

Building Scalable and Extendable Data Pipeline for Call of Duty Games (Yarosl...

High performance messaging with Apache Pulsar

Unify Storage Backend for Batch and Streaming Computation with Apache Pulsar_...

An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.

High cardinality time series search: A new level of scale - Data Day Texas 2016

Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix

Pulsar in the Lakehouse: Overview of Apache Pulsar and Delta Lake Connector -...

Introduction to Apache BookKeeper Distributed Storage

Pulsar - Distributed pub/sub platform

Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar

Apache Pulsar at Yahoo! Japan

Serverless Event Streaming with Pulsar Functions

Apache kafka

Semelhante a Apache Pulsar, Supporting the Entire Lifecycle of Streaming Data

Global Data Stream Network for Internet of Things

Bhavana Srinivas

Real-Time Data Flows with Apache NiFi

Manish Gupta

Open keynote_carolyn&matteo&sijie

StreamNative

UCIAD - quick overview

Mathieu d'Aquin

In this talk, Till Rohrmann and Addison Higham discuss how Flink allows for ambitious stream processing workflows and how Pulsar and Flink enable new capabilities that push forward the state-of-the-art in streaming. They will also share upcoming features and new capabilities in the integrations between Flink and Pulsar and how these two communities are working together to truly advance the power of stream processing.

Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote

StreamNative

Modern IT and application environments are increasingly complex, transitioning to cloud, and large in scale. The managed resources, services and applications in these environments generate tremendous data that needs to be observed, consumed and analyzed in real time (or later) by management tools to create insights and to drive operational actions and decisions. In this talk, Srikanth Natarajan will share Micro Focus’ adoption story of Pulsar, including the experience in consuming from and contributing to Apache Pulsar, the lessons learned, and the help that Micro Focus received from a development support partner in their Pulsar journey.

Why Micro Focus Chose Pulsar for Data Ingestion - Pulsar Summit NA 2021

StreamNative

apidays New York 2022 - Beyond API Regulations for Finance, Insurance, and Healthcare July 27 & 28, 2022 Leveraging Event Streaming to Super-Charge your Business Mary Grygleski, Streaming Developer Advocate at DataStax ------------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/ Deep dive into the API industry with our reports: https://www.apidays.global/industry-reports/ Subscribe to our global newsletter: https://apidays.typeform.com/to/i1MPEW

apidays New York 2022 - Leveraging Event Streaming to Super-Charge your Busin...

apidays

With more traffic than ever being generated through system to system connections, building robust integrations is a must. Keys to any complete integration include (but not limited to) zero data loss, queuing, decoupling application components, synchronous vs. asynchronous executions, throttling, logging and monitoring, and caching. This discussion will dive into many of these key components of integration. The presentation will focus on eliminating data loss with best practice integration patterns, and will demo and show methods to ensure that data is never lost. The discussion will also explore decoupling components of an application into reusable microservices, caching for APIs, and ideas on how you can accomplish synchronous and asynchronous executions, all while maintaining high visibility into your platform with logging and monitoring.

Never Lose Data Again: Robust Integrations With MuleSoft

AaronLieberman5

Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrumgaard | Current 2022 At home, I monitor the temperature, humidity, gas levels, ozone, air quality, and other features around my desk. Let's bring this to the different spots around the conference including lunch tables, vendor booths, hotel rooms, and more. I need to know about these readings now, not when I get back home from the conference. We need to get these sensor readings immediately in case we need to turn on a fan or move to another area. We will also see if my talk produces a lot of hot air!?!?? My setup is pretty simple, a raspberry pi, a breakout garden sensor mount, and as many sensors as I am willing to fly to Austin. The software stack is Python and Java, Apache Pulsar, MQTT, HTML, JQuery, and Apache Kafka. https://dzone.com/articles/five-sensors-real-time-with-pulsar-and-python-on-a https://www.datainmotion.dev/2022/04/flip-py-pi-enviroplus-using-apache.html https://dzone.com/articles/pulsar-in-python-on-pi

Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrum...

HostedbyConfluent

(Current22) Let's Monitor The Conditions at the Conference Let's Monitor The Conditions at the Conference Session Time11:15 am - 12:00 pm Session DateWednesday, 5 October 2022 Session Type:In-Person Location:Ballroom G Session Description: At home, I monitor the temperature, humidity, gas levels, ozone, air quality, and other features around my desk. Let's bring this to the different spots around the conference including lunch tables, vendor booths, hotel rooms, and more. I need to know about these readings now, not when I get back home from the conference. We need to get these sensor readings immediately in case we need to turn on a fan or move to another area. We will also see if my talk produces a lot of hot air!? My setup is pretty simple, a raspberry pi, a breakout garden sensor mount, and as many sensors as I am willing to fly to Austin. The software stack is Python and Java, Apache Pulsar, MQTT, HTML, JQuery, and Apache Kafka. Timothy Spann, StreamNative Developer Advocate Tim Spann is a Developer Advocate @ StreamNative where he works with Apache Pulsar, Apache Flink, Apache NiFi, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC.

(Current22) Let's Monitor The Conditions at the Conference

Timothy Spann

Utopoll Whitepaper.pdf

blondialvarez3520

Distributed Data Processing for Real-time Applications

ScyllaDB

Pulsar Summit San Francisco is the event dedicated to Apache Pulsar. This one-day, action-packed event will include 5 keynotes, 12 breakout sessions, and 1 amazing happy hour. Speakers are from top companies, including Google, AWS, Databricks, Onehouse, StarTree, Intel, ScyllaDB, and more! It’s the perfect opportunity to network with Pulsar thought leaders in person. Join developers, architects, data engineers, DevOps professionals, and anyone who wants to learn about messaging and event streaming for this one-day, in-person event. Pulsar Summit San Francisco brings the Apache Pulsar Community together to share best practices and discuss the future of streaming technologies.

Event-Driven Applications Done Right - Pulsar Summit SF 2022

StreamNative

UTOPOLL白皮書.pdf

aipaypoll

Scenic City Summit (2021): Real-Time Streaming in any and all clouds, hybrid...

Timothy Spann

Streaming and real-time data has high business value, but that value can rapidly decay if not processed quickly. If the value of the data is not realized in a certain window of time, its value is lost and the decision or action that was needed as a result never occurs. Streaming data – whether from sensors, devices, applications, or events – needs special attention because a sudden price change, a critical threshold met, a sensor reading changing rapidly, or a blip in a log file can all be of immense value, but only if the alert is in time.

ADV Slides: Trends in Streaming Analytics and Message-oriented Middleware

DATAVERSITY

Bodleian Library's DAMS system

benosteen

Data Analytics: HDFS with Big Data : Issues and Application

Dr. Chitra Dhawale

Topic # 16 of outline Managing Network Services.pptx

AyeCS11

Pacemaker+DRBD

Dan Frincu

Semelhante a Apache Pulsar, Supporting the Entire Lifecycle of Streaming Data (20)

Global Data Stream Network for Internet of Things

Real-Time Data Flows with Apache NiFi

Open keynote_carolyn&matteo&sijie

UCIAD - quick overview

Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote

Why Micro Focus Chose Pulsar for Data Ingestion - Pulsar Summit NA 2021

apidays New York 2022 - Leveraging Event Streaming to Super-Charge your Busin...

Never Lose Data Again: Robust Integrations With MuleSoft

Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrum...

(Current22) Let's Monitor The Conditions at the Conference

Utopoll Whitepaper.pdf

Distributed Data Processing for Real-time Applications

Event-Driven Applications Done Right - Pulsar Summit SF 2022

UTOPOLL白皮書.pdf

Scenic City Summit (2021): Real-Time Streaming in any and all clouds, hybrid...

ADV Slides: Trends in Streaming Analytics and Message-oriented Middleware

Bodleian Library's DAMS system

Data Analytics: HDFS with Big Data : Issues and Application

Topic # 16 of outline Managing Network Services.pptx

Pacemaker+DRBD

Mais de StreamNative

So, you are a responsible software engineer building microservices for Apache Kafka, and life is good. Eventually, you hear the community talking about the outstanding experience they are having with Apache Pulsar features. They talk about infinite event stream retention, a rebalance-free architecture, native support for event processing, and multi-tenancy. Exciting, right? Most people would want to migrate their code to Pulsar. Especially when you know that Pulsar also supports Kafka clients natively via the protocol handler known as KoP — which enables the Kafka client APIs on Pulsar. But, as said before, you are responsible; and you don't believe in fairy tales, just like you don't believe that migrations like this happen effortlessly. This session will discuss the architecture behind protocol handlers, what it means having one enabled on Pulsar, and how the KoP works. It will detail the effort required to migrate a microservice written for Kafka to Pulsar, and whether the code need to change for this.

Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022

StreamNative

This talk describes Klaviyo’s internal messaging system, an asynchronous application framework built around Pulsar that provides a set of high-quality tools for building business-critical asynchronous data flows in unreliable environments. This framework includes: a pulsar ORM and schema migrator for topic configuration; a retry/replay system; a versioned schema registry; a consumer framework oriented around preventing message loss and in hostile environments while maximizing observability; an experimental “online schema change” for topics; and more. Development of this system was informed by lessons learned during heavy use of datastores like RabbitMQ and Kafka, and frameworks like Celery, Spark, and Flink. In addition to the capabilities of this system, this talk will also cover (sometimes painful) lessons learned about the process of converting a heterogenous async-computing environment onto Pulsar and a unified model.

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

In this talk, learn how Toast leverages our Envoy control-plane to manage blue-green deploys of Pulsar consumers, and how this has helped drive adoption across the engineering organization. Dive into the history of Pulsar at Toast, starting from its introduction in 2019 to provide event-driven architecture across a rapidly scaling restaurant software platform. We will detail some of the hurdles that we encountered gaining buy-in across a diverse set of teams, and dive deep into how we enforce best practices and integrate with our service control plane.

Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...

StreamNative

Event streaming architectures launched a reexamination of applications and systems architectures across the board. We live in a world where answers are needed now in a constant real-time flow. Yet beyond the event streaming system itself, what are the corequisites to ensure our large scale distributed database systems can keep pace with this always-on, always-current real time flow of data? What are the requirements and expectations for this next tech cycle?

Distributed Database Design Decisions to Support High Performance Event Strea...

StreamNative

Pulsar Functions is a succinct framework provided by Apache Pulsar to conduct real-time data processing. Its use cases include ETL pipeline, event-driven applications, and simple data analytics. While Pulsar Functions already provides an extremely simple programming interface, we want to further lower the barrier for users to access real-time data. Since SQL is one of the universal languages in the technology world and well accepted by the vast majority of data engineers, we decided to add a SQL expressing layer on top of Pulsar Functions runtime. In this talk, we will discuss the architecture and implementation of this new service. We will see how SQL syntax, Pulsar Functions, and Function Mesh can work together to deliver a unique user development experience for real-time data jobs in the cloud environment. We will also walk through use cases like filtering, routing, and projecting messages as well as integrating with the Pulsar IO Connectors framework.

Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022

StreamNative

Starting with version 2.10, the Apache ZooKeeper dependency has been eliminated and replaced with a pluggable framework that enables you to reduce the infrastructure footprint of Apache Pulsar by leveraging alternative metadata and coordination systems based on your deployment environment. In this talk, walk through the steps required to utilize the existing etcd service running inside Kubernetes to act as Pulsar's metadata store, thereby eliminating the need to run ZooKeeper entirely, leaving you with a Zookeeper-less Pulsar.

Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022

StreamNative

Apache Pulsar is a highly available, distributed messaging system that provides guarantees of no message loss and strong message ordering with predictable read and write latency. In this talk, learn how this can be validated for Apache Pulsar Kubernetes deployments. Various failures are injected using Chaos Mesh to simulate network and other infrastructure failure conditions. There are many questions that are asked about failure scenarios, but it could be hard to find answers to these important questions. When a failure happens, how long does it take to recover? Does it cause unavailability? How does it impact throughput and latency? Are the guarantees of no message loss and strong message ordering kept, even when components fail? If a complete availability zone fails, is the system configured correctly to handle AZ failures? This talk will help you find answers to these questions and apply the tooling and practices to your own testing and validation.

Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...

StreamNative

Despite what the Ghostbusters said, we’re going to go ahead and cross (or, join) the streams. This session covers getting started with streaming data pipelines, maximizing Pulsar’s messaging system alongside one of the most flexible streaming frameworks available, Apache Flink. Specifically, we’ll demonstrate the use of Flink SQL, which provides various abstractions and allows your pipeline to be language-agnostic. So, if you want to leverage the power of a high-speed, highly customizable stream processing engine without the usual overhead and learning curves of the technologies involved (and their interconnected relationships), then this talk is for you. Watch the step-by-step demo to build a unified batch and streaming pipeline from scratch with Pulsar, via the Flink SQL client. This means you don’t need to be familiar with Flink, (or even a specific programming language). The examples provided are built for highly complex systems, but the talk itself will be accessible to any experience level.

Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...

StreamNative

Apache Pulsar depends upon message acknowledgments to provide at-least-once or exactly-once processing guarantees. With these guarantees, any transmission between the broker and its producers and consumers requires an acknowledgment. But what happens if an acknowledgment is not received? Resending the message introduces the potential of duplicate processing and increases the likelihood of out or order processing. Therefore, it is critical to understand the Pulsar message redelivery semantics in order to prevent either of these conditions. In this talk, we will walk you through the redelivery semantics of Apache Pulsar, and highlight some of the control mechanisms available to application developers to control this behavior. Finally, we will present best practices for configuring message redelivery to suit various use cases.

Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022

StreamNative

Lakehouses are quickly growing in popularity as a new approach to Data Platform Architecture bringing some of the long-established benefits from OLTP world to OLAP, including transactions, record-level updates/deletes, and changes streaming. In this talk, we will discuss Apache Hudi and how it unlocks possibilities of building your own fully open-source Lakehouse featuring a rich set of integrations with existing technologies, including Apache Pulsar. In this session, we will present: - What Lakehouses are, and why they are needed. - What Apache Hudi is and how it works. - Provide a use-case and demo that applies Apache Hudi’s DeltaStreamer tool to ingest data from Apache Pulsar.

Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...

StreamNative

Pulsar is a horizontally scalable messaging system, so the traffic in a logical cluster must be balanced across all the available Pulsar brokers as evenly as possible, in order to ensure full utilization of the broker layer. You can use multiple settings and tools to control the traffic distribution which requires a bit of context to understand how the traffic is managed in Pulsar. In this talk, we will walk you through the load balancing capabilities of Apache Pulsar, and highlight some of the control mechanisms available to control the distribution of load across the Pulsar brokers. Finally, we will discuss the various loading shedding strategies that are available. At the end of the talk, you will have a better understanding of how Pulsar's broker level auto-balancing works, and how to properly configure it to meet your workload demands.

Understanding Broker Load Balancing - Pulsar Summit SF 2022

StreamNative

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

In today’s world, we are seeing a big shift toward the Cloud. With this shift comes a big shift in the expectations we have for a messaging system, especially when the messaging system is presented as managed service in a large-scale, multi-tenant environment. For any large-scale enterprise, it’s very important to evaluate messaging system and be confident before expanding complex distributed data systems like Apache Pulsar from on-premise to elastically scalable, fully managed services on cloud services. We must consider aspects such as: migration from and integration with large-scale on-premise clusters, security, cost efficiency, and the cloud friendliness of the architecture, modeling cost and capacity, tenant isolation, deployment robustness, availability, monitoring, etc. Not every messaging system is built to be cloud-native and run as a managed service with cost efficiency. We have been running large-scale Apache Pulsar at Yahoo for the last 8 years on various platforms and hardware configurations while meeting application SLAs and serving more than 1M topics in a cluster. In this talk, we will talk about Pulsar’s journey in Yahoo! from an on-premise platform to a hybrid cloud and on-premise system. We will talk about Pulsar’s architecture and features that make Pulsar a good cloud-native messaging-system choice for any enterprise.

Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022

StreamNative

Our services team creates, builds, and maintains the as a service offering for base platform services within our organization. Several thousand applications use these custom services daily generating more than 700 million requests per minute. One of these services was our publish / subscriber offering, BQ with custom SDK and custom metrics based on Apache Pulsar. BQ is the core communication service within our organization, having more 200M RPM. All the core processes of the organization depend on this service for operation: the CDC of any of our RDBMS or NoSQL offering, all the eventing efforts of the organization, async communication between apps, notification systems, etc. The backend of the solution was Apache Pulsar running on EC2 on AWS and on top of that we built several components as wrappers of the actual backend, creating our own SDKs and abstractions and in many ways extending the features provided by Pulsar. We had a multi-cluster setup 100% on AWS, with custom Pulsar Docker images running on large ASG setups, along with our own wrapping and admin APIs and DBs. All of this in turn transformed the solution into a volatile solution.

Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022

StreamNative

There is an increasing need to unleash analytical capabilities directly to the end-users to democratize decision-making. User-Facing Analytics is a new frontier that will shape the products of tomorrow and push the limits of existing technology. It demands a solution that will scale to millions of users to provide fast, real-time insights. In this session, Xiang will talk about his journey to build Apache Pinot to tackle the analytics problem space with the architectural changes and technology inventions made over the past decade. He will also talk about how other big data companies such as LinkedIn, Uber, and Stripe power their user-facing analytical applications.

Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022

StreamNative

Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022

StreamNative

Welcome and Opening Remarks - Pulsar Summit SF 2022

StreamNative

MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments. This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.

MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...

StreamNative

KoP(Kafka on Pulsar) is a protocol handler that provides a convenient method for Kafka clients to connect to Pulsar without any change. Since the first GA version (2.8.0) of KoP (Kafka on Pulsar) was released several months ago, there has been a lot of significant bug fixes and enhancements, such as the support for authorization, and lower version Kafka client. This talk will walk through the basics of KoP, then introduce the improvements in the 2.9.0 version.

Improvements Made in KoP 2.9.0 - Pulsar Summit Asia 2021

StreamNative

Recently, the Tencent Cloud MQ team open sourced the 0.1.0 version of RoP(RocketMQ on Pulsar). However, many problems occurred during production practices, such as message ID overflow, incomplete message consumption, unbalanced load on consumption model, and invalid consumption requests. We made a series of optimizations for the above problems in RoP 0.2.0, and improved the overall performance and stability of the current RoP version. In this session, we will introduce the RoP MessageID refactoring, design and implementation of RoP delayed message and routing protocol, RoP ACL design and implementation, RoP performance optimization and the application of RoP in Tencent Cloud. Key takeaways： 1. Common design and implementation ideas for delayed messages in message queues 2. Implementation ideas and practical applications of Apache Pulsar Broker Entry metadata 3. Different ideas for the realization of message routing between Apache RocketMQ and Apache Pulsar 4. The similarities, differences and combination of authentication implementation schemas between Apache Pulsar and Apache RocketMQ.

The Evolution History of RoP(RocketMQ-on-Pulsar) - Pulsar Summit Asia 2021

StreamNative

Mais de StreamNative (20)