Debezium Snapshots Revisited!

•

0 likes•597 views

"Initial snapshots are a core feature of Debezium: when setting up a new CDC connector, existing tables can be scanned in order to export their full state to consumers, before starting to capture changes from the transaction log. While this works great in general, a few questions came up again and again in the Debezium community over time: * How to re-snapshot just a single table? * How to pause and resume long-running snapshots? * How to run snapshots in parallel to reading changes from the log? All this, and more, becomes possible with the notion of incremental snapshots. In this session you'll learn how this innovative scheme of interleaving snapshot queries and log-based change events works under the hood and how it solves common tasks when running CDC pipelines. We'll also discuss advanced topics like parallelising snapshots and customising snapshot contents."

Technology

Image © Nicolas Buffler https://flic.kr/p/jpWcWD (CC BY 2.0)
Debezium Snapshots Revisited!
Gunnar Morling
Senior Staﬀ Software Engineer, Decodable
@gunnarmorling

#DebeziumSnapshotting @gunnarmorling
Agenda

#DebeziumSnapshotting @gunnarmorling
● Software engineer at Decodable
● Former project lead of Debezium
● kcctl 🧸, JfrUnit, ModiTect,
MapStruct
● Spec Lead for Bean Validation 2.0
● Java Champion
Gunnar Morling

#DebeziumSnapshotting @gunnarmorling
Recap – Debezium
Log-Based Change Data Capture

#DebeziumSnapshotting @gunnarmorling
Snapshotting
Why Is It Needed?
● Need to backfill data, but don’t
have all TX logs
● Solution: scan data once before
streaming
● Emit READ event for each record

#DebeziumSnapshotting @gunnarmorling
Snapshotting
Classic Approach – General Idea
● Capture current
position in transaction
log
● Scan all relevant tables
● Start streaming

#DebeziumSnapshotting @gunnarmorling
Snapshotting
Key Conﬁguration Options
● snapshot.mode (initial, never, schema_only_recovery)
● snapshot.select.statement.overrides
● snapshot.max.threads

#DebeziumSnapshotting @gunnarmorling
Snapshotting
Limitations of Classic Approach
● Can’t update filter list

#DebeziumSnapshotting @gunnarmorling
Snapshotting
Limitations of Classic Approach
● Can’t update filter list
● Can’t pause & resume long-running snapshots

Incremental
Snapshots
© Karen Blaha https://flic.kr/p/aeuPys (CC BY-SA 2.0)

#DebeziumSnapshotting @gunnarmorling
Incremental Snapshotting
The Paper
● “DBLog: A Watermark Based
Change-Data-Capture
Framework”, by Andreas Andreakis
and Ioannis Papapanagiotou
● Key idea: interleave snapshot events
and events from TX log
https://arxiv.org/pdf/2010.12597v1.pdf

#DebeziumSnapshotting @gunnarmorling
Incremental Snapshotting
General Idea

#DebeziumSnapshotting @gunnarmorling
Incremental Snapshotting
Windowing via Watermarks

#DebeziumSnapshotting @gunnarmorling
Incremental Snapshotting
Buﬀer Processing

#DebeziumSnapshotting @gunnarmorling
Incremental Snapshotting
Semantics
● No guarantee for snapshot (read) events for all records
● May receive update or delete without prior insert/read
● May receive read and update/delete
● What is guaranteed: complete data set after snapshot

Demo
© Wall Boat https://flic.kr/p/Y6zkmX (Public Domain)

#DebeziumSnapshotting @gunnarmorling
Incremental Snapshotting
Connector Oﬀsets

#DebeziumSnapshotting @gunnarmorling
Incremental Snapshotting
MySQL Read-Only Snapshots
● Write access to DB may be not desirable

#DebeziumSnapshotting @gunnarmorling
Incremental Snapshotting
Signalling Channels
● Database table
● Kafka topic
● JMX
● Custom
id 924e3ff8-2245-43ca-ba77-2af9af02fa07
type log, {execute|pause|resume|stop}-snapshot
value { "data-collections": ["schema1.table1", "schema2.table2"],
"type":"incremental",
"additional-condition":"color=blue" }

#DebeziumSnapshotting @gunnarmorling
Incremental Snapshotting
Notiﬁcations

#Debezium + #ApacheFlink | @gunnarmorling
Comparison

#DebeziumSnapshotting @gunnarmorling
Incremental Snapshotting
Beneﬁts
● Can update filter list ✅

#DebeziumSnapshotting @gunnarmorling
Incremental Snapshotting
Beneﬁts
● Can update filter list ✅
● Long-running snapshots can be paused/resumed ✅

#DebeziumSnapshotting @gunnarmorling
● Incremental Snapshots in Debezium
https://debezium.io/blog/2021/10/07/incremental-snapshots/
● Read-only Incremental Snapshots for MySQL
https://debezium.io/blog/2022/04/07/read-only-incremental-snapshots/
● Flink CDC
https://ververica.github.io/flink-cdc-connectors/
Resources

#DebeziumSnapshotting @gunnarmorling
● Debezium & Kafka Connect – Ask the Experts
With Chris Cranford (Red Hat) and Chris Egerton (Aiven)
Sep 27, 2:30 PM
● Change Stream Processing with Debezium and Apache Flink
With Robert Metzger (Decodable)
Sep 27, 5:30 PM, Dremio Office
https://www.meetup.com/sf-big-analytics/events/294068331/
Upcoming

#DebeziumSnapshotting @gunnarmorling
Q & A
gunnar@decodable.co
@gunnarmorling
📧
Thank You!

Similar to Debezium Snapshots Revisited!

Most people will claim that this never happens, others hope it never happens, but it happened on March 10, 2021, and it was not just the 1 datacenter that got impacted, but the whole campus of the provider that got powered down. This talk will explain how our customers survived this outage, how our culture, opensource tooling and automation saved the da(y,ta). A talk about disaster recovery, business continuity plans and building cloud agnostic stacks that survive disasters.

Help , My Datacenter is on fire

Kris Buytaert

What's coming in Airflow 2.0? - NYC Apache Airflow Meetup

Kaxil Naik

The pathway to Chromium on Wayland (Web Engines Hackfest 2018)

Igalia

How to plan and define your CI-CD pipeline

ElasTest Project

Glusterfs geo-replication

bipin kunal

Spring Boot propose un large panel d'outils pour démarrer et développer rapidement des applications. Après une rapide introduction à l'outil, nous présenterons le code d'une application développé en mode microservices. Ce sera aussi l'occasion d'évoquer les principes de circuit breaker et de monitoring ainsi que de savoir comment Spring Boot permet de les réaliser. Par Nicolas Jozwiak et Thomas Auffredou

Backday Xebia : Découvrez Spring Boot sur un cas pratique

Publicis Sapient Engineering

Swapping Pacemaker Corosync with repmgr

PGConf APAC

pgDay Asia 2016 - Swapping Pacemaker-Corosync for repmgr (1)

Wei Shan Ang

The Open Street Map project provides invaluable data that keeps driving users toward the PostGIS and PostgreSQL stacks. Loading today’s full Planet data set takes a 120GB XML file and unrolls it into over a terabyte of database data. Crunchy’s benchmark labs have followed the expansion of that Planet data over the last six database releases, as the re-ignition of the CPU wars combined with parallel execution features landing in the database. We’ll take a look at that data evolution, which server configurations worked, and which metrics techniques still matter in the all SSD era.

Speedrunning the Open Street Map osm2pgsql Loader

GregSmith458515

Relying on a single content delivery network for your site can impose a number of flexibility limitations. By diversifying your CDN providers you can put the power back in your hands, allowing you to get the best of both worlds in terms of performance, reliability and cost. In this talk Marcus and Laurie will present Etsy’s recent work integrating multiple CDN providers to their site delivery infrastructure. This presentation was delivered at Velocity Europe, November 2013

Integrating multiple CDN providers at Etsy - Velocity Europe (London) 2013

Marcus Barczak

The Accidental DBA

PostgreSQL Experts, Inc.

PGConf.ASIA 2019 - The Future of TDEforPG - Taiki Kondo

Equnix Business Solutions

GPU virtualization is hot in cloud usages including VDI, media processing, etc. While Intel GVT-g (a.k.a XenGT) helps unleash those compelling usages on Intel Processor Graphics, new requirements are emerging such as VM live migration with vGPU. In this session we will introduce the challenges of supporting vGPU live migration on current migration framework, then elaborate techniques to bring vGPU live migration into XenGT.

XPDS16: Live Migration of vGPU - Xiao Zheng, Intel Asia-Pacific Research & De...

The Linux Foundation

Geo-Replication and Disaster Recovery : Glusterfs

bipin kunal

HKG15-409: ARM Hibernation enablement on SoCs - a case study

Linaro

Design choices of golang for high scalability

SeongJae Park

PostgreSQL WAL for DBAs

PGConf APAC

GTC Taiwan 2017 在 Google Cloud 當中使用 GPU 進行效能最佳化

NVIDIA Taiwan

Working as a MySQL Developer as part of the Bugs committee exposes you to a variety of bugs, such as server crashes, memory leaks, wrong query results, internal thread deadlocks, and others. In this talk, I will cover some of the technics we utilize to troubleshoot MySQL when things are not working as expected. Some of the topics covered include: Reproducible test cases Git Bisect Stack Traces GDB Record and Replay By the end of this session, attendees will grasp how to tackle analyses of when software is not working as expected.

Troubleshooting MySQL from a MySQL Developer Perspective

Marcelo Altmann

Unleash sp3d model to enterprise level

Taiheon Choi

Similar to Debezium Snapshots Revisited! (20)

Help , My Datacenter is on fire

What's coming in Airflow 2.0? - NYC Apache Airflow Meetup

The pathway to Chromium on Wayland (Web Engines Hackfest 2018)

How to plan and define your CI-CD pipeline

Glusterfs geo-replication

Backday Xebia : Découvrez Spring Boot sur un cas pratique

Swapping Pacemaker Corosync with repmgr

pgDay Asia 2016 - Swapping Pacemaker-Corosync for repmgr (1)

Speedrunning the Open Street Map osm2pgsql Loader

Integrating multiple CDN providers at Etsy - Velocity Europe (London) 2013

The Accidental DBA

PGConf.ASIA 2019 - The Future of TDEforPG - Taiki Kondo

XPDS16: Live Migration of vGPU - Xiao Zheng, Intel Asia-Pacific Research & De...

Geo-Replication and Disaster Recovery : Glusterfs

HKG15-409: ARM Hibernation enablement on SoCs - a case study

Design choices of golang for high scalability

PostgreSQL WAL for DBAs

GTC Taiwan 2017 在 Google Cloud 當中使用 GPU 進行效能最佳化

Troubleshooting MySQL from a MySQL Developer Perspective

Unleash sp3d model to enterprise level

More from HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

HostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

Renaming a Kafka Topic | Kafka Summit London

HostedbyConfluent

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Evolution of NRT Data Ingestion Pipeline at Trendyol

HostedbyConfluent

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

HostedbyConfluent

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Exactly-once Stream Processing with Arroyo and Kafka

HostedbyConfluent

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Fish Plays Pokemon | Kafka Summit London

HostedbyConfluent

Tiered Storage 101 | Kafla Summit London

HostedbyConfluent

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

Building a Self-Service Stream Processing Portal: How And Why

HostedbyConfluent

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

HostedbyConfluent

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

HostedbyConfluent

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Navigating Private Network Connectivity Options for Kafka Clusters

HostedbyConfluent

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

HostedbyConfluent

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

Explaining How Real-Time GenAI Works in a Noisy Pub

HostedbyConfluent

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

TL;DR Kafka Metrics | Kafka Summit London

HostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

A Window Into Your Kafka Streams Tasks | KSL

HostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

HostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Data Contracts Management: Schema Registry and Beyond

HostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Code-First Approach: Crafting Efficient Flink Apps

HostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Debezium vs. the World: An Overview of the CDC Ecosystem

HostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

Beyond Tiered Storage: Serverless Kafka with No Local Disks

HostedbyConfluent

More from HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Renaming a Kafka Topic | Kafka Summit London

Evolution of NRT Data Ingestion Pipeline at Trendyol

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Exactly-once Stream Processing with Arroyo and Kafka

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

Building a Self-Service Stream Processing Portal: How And Why

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Navigating Private Network Connectivity Options for Kafka Clusters

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Explaining How Real-Time GenAI Works in a Noisy Pub

TL;DR Kafka Metrics | Kafka Summit London

A Window Into Your Kafka Streams Tasks | KSL

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Data Contracts Management: Schema Registry and Beyond

Code-First Approach: Crafting Efficient Flink Apps

Debezium vs. the World: An Overview of the CDC Ecosystem

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Recently uploaded

Google I/O Extended 2024 Warsaw

GDSC PJATK

What's New in Teams Calling, Meetings and Devices April 2024

Stephanie Beckett

Oauth 2.0 Introduction and Flows with MuleSoft

shyamraj55

Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)

Julian Hyde

WebAssembly is Key to Better LLM Performance

Samy Fodil

When you think of a highly secure meeting environment, do you instantly think 'Microsoft Teams'!? Or do you think about some unknown application, troublesome UI and daunting login process...? If you think the latter - let's change that! In this session Femke will show you how using Teams Premium features can create secure, but also good looking meetings! PRETTY. Make sure your company's brand is represented before, during and after the meeting with Customization policies in place. SECURE. Lets utilize Meeting templates and Sensitivity Labels to protect your meeting and data to prevent sensitive information from being leaked. After this session, you will have a clear understanding of the capabilities of Teams Premium features and how to set up the perfect meeting that suits your organizational requirements!

ECS 2024 Teams Premium - Pretty Secure

Femke de Vroome

Extensible Python: Robustness through Addition - PyCon 2024

Patrick Viafore

Overview of Hyperledger Foundation

Hyperleger Tokyo Meetup

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf

FIDO Alliance

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...

FIDO Alliance

Screen flow is a powerful automation tool that is commonly designed for internal and external users. However, what about the guest users? We will dive into various methods of launching screen flows and understand how to make them publicly accessible, extending their usability to a broader audience. The presentation will also cover the implementation of security layers and highlight best practices for a smooth and protected user experience. Discover the potential of screen flows beyond conventional use and learn how to leverage them effectively.

Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade

CzechDreamin

Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf

FIDO Alliance

In today’s fast-paced digital world, harnessing the power of artificial intelligence (AI) can significantly enhance productivity and creativity across various domains. With the advent of advanced language models like ChatGPT, developers, marketers, data analysts, and professionals in numerous other fields can now leverage AI-generated prompts to spark innovative ideas and streamline their workflows.

1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT

iSEO AI

ScyllaDB has the potential to deliver impressive performance and scalability. The better you understand how it works, the more you can squeeze out of it. But before you squeeze, make sure you know what to monitor! Watch our experienced Postgres developer work through monitoring and performance strategies that help him understand what mistakes he’s made moving to NoSQL. And learn with him as our database performance expert offers friendly guidance on how to use monitoring and performance tuning to get his sample Rust application on the right track. This webinar focuses on using monitoring and performance tuning to discover and correct mistakes that commonly occur when developers move from SQL to NoSQL. For example: - Common issues getting up and running with the monitoring stack - Using the CQL optimizations dashboard - Common issues causing high latency in a node - Common issues causing replica imbalance - What a healthy system looks like in terms of memory - Key metrics to keep an eye on This isn’t “Death-by-Powerpoint.” We’ll walk through problems encountered while migrating a real application from Postgres to ScyllaDB – and try to fix them live as well.

Optimizing NoSQL Performance Through Observability

ScyllaDB

AI presentation and introduction - Retrieval Augmented Generation RAG 101

vincent683379

Speed Wins: From Kafka to APIs in Minutes

confluent

IESVE for Early Stage Design and Planning

IES VE

The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf

FIDO Alliance

This our Twelfth semiannual report on the global Cryptocurrency mining industry. Bitcoin is the world’s largest special purpose supercomputer. And it is globally decentralized. Millions of nodes all run the same open-source code to secure the Bitcoin network, create value, and put new transactions onto the distributed ledger. The latest Top500 list has just been announced at the ISC 2024 conference in Hamburg, and once again the Frontier supercomputer with 1.2 Exaflops peak performance is number one on the list. If assigned to SHA-256 hashing, Frontier would provide only the equivalent hash rate of about three cabinets of the latest high-end Bitcoin mining systems, costing less than 0.1% of Frontier’s cost. Michael Saylor, Chairman of MicroStrategy, has pointed out that GPUs are two orders of magnitude slower than the 5-nanometer technology of custom ASICs used for Bitcoin mining today. He makes the point that the Bitcoin network is unassailable by all of the hyperscale computing resources combined in AWS, Google, and Microsoft Azure cloud data centers today.

TopCryptoSupers 12thReport OrionX May2024

Stephen Perrenod

Are you wondering why everyone is talking about the Flutterwave scandal? You’re not alone! It’s been in the news a lot. We’re going to tell you all about the Flutterwave Scandal in a way that’s easy to get. Keep reading, because we’re going to share the whole story with you. The company Flutterwave is headquartered in San Francisco, California, United States. In 2016, Iyinoluwa Aboyeji, Olugbenga Agboola, and Adeleke Adekoya established Flutterwave. It offers a payment infrastructure to international merchants and payment service providers throughout Africa. The company operates in various African countries including Nigeria, Kenya, Uganda, Ghana, South Africa, and others. They focus on digital payments, helping businesses accept and process payments on various channels like the web, mobile, ATM, and POS. Recently, they’ve been in the news due to allegations of misconduct within the company, which has affected their reputation and value. Flutterwave is an essential player in the African fintech landscape, aiming to drive growth for banks and businesses through digital payment solutions. The Flutterwave scandal involves allegations of misconduct and inappropriate behaviour towards female employees by the company’s co-founder and CEO, Olugbenga Agboola. Reports have surfaced from both current and former employees about bullying, intimidation, and sexual harassment at work. The allegations of inappropriate behaviour towards female employees at Flutterwave, specifically involving the CEO Olugbenga Agboola, were brought to public attention in April 2022.This was when Clara Wanjiku Odero, an ex-employee and current CEO of Credrails, published a Medium post and a series of tweets on April 4, 2022, accusing Flutterwave and Agboola of bullying. These allegations were part of a broader range of misconduct claims that surfaced around the same time The reasons behind the scandal are rooted in accusations of unethical behavior within the company. The allegations suggest a workplace culture that allowed for misconduct and failed to protect employees from harassment and intimidation. Specifically, the Flutterwave scandal refers to the series of events where the CEO was accused of engaging in improper conduct with female colleagues. This led to a broader investigation into the company’s practices and raised serious concerns about the fintech’s corporate governance. The scandal had significant repercussions, including a drop in the company’s stock price and a loss of trust among customers and partners. The trust that customers and investors had in Flutterwave was greatly damaged. People started to doubt the company’s integrity and whether their money and personal information were safe. Market Impact The scandal shook the entire fintech market. Flutterwave’s competitors saw this as an opportunity and started to attract customers who were leaving Flutterwave.

Breaking Down the Flutterwave Scandal What You Need to Know.pdf

UK Journal

Recently uploaded (20)

Google I/O Extended 2024 Warsaw

What's New in Teams Calling, Meetings and Devices April 2024

Oauth 2.0 Introduction and Flows with MuleSoft

Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)

WebAssembly is Key to Better LLM Performance

ECS 2024 Teams Premium - Pretty Secure

Extensible Python: Robustness through Addition - PyCon 2024

Overview of Hyperledger Foundation

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...

Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade

Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf

1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT

Optimizing NoSQL Performance Through Observability

AI presentation and introduction - Retrieval Augmented Generation RAG 101

Speed Wins: From Kafka to APIs in Minutes

IESVE for Early Stage Design and Planning

The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf

TopCryptoSupers 12thReport OrionX May2024

Breaking Down the Flutterwave Scandal What You Need to Know.pdf

Debezium Snapshots Revisited!

1. Image © Nicolas Buffler https://flic.kr/p/jpWcWD (CC BY 2.0) Debezium Snapshots Revisited! Gunnar Morling Senior Staﬀ Software Engineer, Decodable @gunnarmorling

3. #DebeziumSnapshotting @gunnarmorling Agenda

4. #DebeziumSnapshotting @gunnarmorling ● Software engineer at Decodable ● Former project lead of Debezium ● kcctl 🧸, JfrUnit, ModiTect, MapStruct ● Spec Lead for Bean Validation 2.0 ● Java Champion Gunnar Morling

5. #DebeziumSnapshotting @gunnarmorling Recap – Debezium Log-Based Change Data Capture

6. #DebeziumSnapshotting @gunnarmorling Snapshotting Why Is It Needed? ● Need to backfill data, but don’t have all TX logs ● Solution: scan data once before streaming ● Emit READ event for each record

7. #DebeziumSnapshotting @gunnarmorling Snapshotting Classic Approach – General Idea ● Capture current position in transaction log ● Scan all relevant tables ● Start streaming

8. #DebeziumSnapshotting @gunnarmorling Snapshotting Key Conﬁguration Options ● snapshot.mode (initial, never, schema_only_recovery) ● snapshot.select.statement.overrides ● snapshot.max.threads

9. #DebeziumSnapshotting @gunnarmorling Snapshotting Limitations of Classic Approach ● Can’t update filter list

10. #DebeziumSnapshotting @gunnarmorling Snapshotting Limitations of Classic Approach ● Can’t update filter list ● Can’t pause & resume long-running snapshots

11. #DebeziumSnapshotting @gunnarmorling Snapshotting Limitations of Classic Approach ● Can’t update filter list ● Can’t pause & resume long-running snapshots ● Can’t stream changes until snapshot completed

12. #DebeziumSnapshotting @gunnarmorling Snapshotting Limitations of Classic Approach ● Can’t update filter list ● Can’t pause & resume long-running snapshots ● Can’t stream changes until snapshot completed ● Can’t re-snapshot selected tables

14. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting The Paper ● “DBLog: A Watermark Based Change-Data-Capture Framework”, by Andreas Andreakis and Ioannis Papapanagiotou ● Key idea: interleave snapshot events and events from TX log https://arxiv.org/pdf/2010.12597v1.pdf

15. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting General Idea

16. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting Windowing via Watermarks

17. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting Buﬀer Processing

18. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting Buﬀer Processing

19. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting Semantics ● No guarantee for snapshot (read) events for all records ● May receive update or delete without prior insert/read ● May receive read and update/delete ● What is guaranteed: complete data set after snapshot

21. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting Connector Oﬀsets

22. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting MySQL Read-Only Snapshots ● Write access to DB may be not desirable

23. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting Signalling Channels ● Database table ● Kafka topic ● JMX ● Custom id 924e3ff8-2245-43ca-ba77-2af9af02fa07 type log, {execute|pause|resume|stop}-snapshot value { "data-collections": ["schema1.table1", "schema2.table2"], "type":"incremental", "additional-condition":"color=blue" }

24. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting Notiﬁcations

25. #Debezium + #ApacheFlink | @gunnarmorling Comparison

26. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting Beneﬁts ● Can update filter list ✅

27. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting Beneﬁts ● Can update filter list ✅ ● Long-running snapshots can be paused/resumed ✅

28. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting Beneﬁts ● Can update filter list ✅ ● Long-running snapshots can be paused/resumed ✅ ● Can stream changes before snapshot completed ✅

29. #DebeziumSnapshotting @gunnarmorling Incremental Snapshotting Beneﬁts ● Can update filter list ✅ ● Long-running snapshots can be paused/resumed ✅ ● Can stream changes before snapshot completed ✅ ● Can re-snapshot selected tables ✅

30.

31. #DebeziumSnapshotting @gunnarmorling ● Incremental Snapshots in Debezium https://debezium.io/blog/2021/10/07/incremental-snapshots/ ● Read-only Incremental Snapshots for MySQL https://debezium.io/blog/2022/04/07/read-only-incremental-snapshots/ ● Flink CDC https://ververica.github.io/flink-cdc-connectors/ Resources

32. #DebeziumSnapshotting @gunnarmorling ● Debezium & Kafka Connect – Ask the Experts With Chris Cranford (Red Hat) and Chris Egerton (Aiven) Sep 27, 2:30 PM ● Change Stream Processing with Debezium and Apache Flink With Robert Metzger (Decodable) Sep 27, 5:30 PM, Dremio Office https://www.meetup.com/sf-big-analytics/events/294068331/ Upcoming

33. #DebeziumSnapshotting @gunnarmorling Q & A gunnar@decodable.co @gunnarmorling 📧 Thank You!

Debezium Snapshots Revisited!

Recommended

Recommended

More Related Content

Similar to Debezium Snapshots Revisited!

Similar to Debezium Snapshots Revisited! (20)

More from HostedbyConfluent

More from HostedbyConfluent (20)

Recently uploaded

Recently uploaded (20)

Debezium Snapshots Revisited!