High performance queues with Cassandra

•

7 gostaram•8,884 visualizações

Everyone knows that Cassandra is a NoSQL solution for data storage. But often for processing of this data message queues are used with some existing messaging provider. Due to this, there is inconsistency of data sometimes and an additional infrastructure level to maintain. Since one of our services stores all the data in Cassandra, we have developed a solution for message queues that automatically gained a lot of useful features: scalability, high availability and flexibility. This solution I will present in the talk.

Tecnologia Negócios

High performance
queues with Cassandra
Mikalai Alimenkou
http://xpinjection.com
@xpinjection

How to
process
data, master
?

Asynchronously!

Some specific requirements
1

Message
Queue

External Service
Provider

Some specific requirements
Message
Queue

1

External Service
Provider

Some specific requirements
2

Message
Queue

External Service
Provider

Some specific requirements
Message
Queue

2
1
Redeliver
after 1 hour

External Service
Provider

Some specific requirements
2

Message
Queue

External Service
Provider

Redeliver
after 1 hour

Redelivery business logic and external service
hourly based usage limits

Message batches “station”
QUEUE
NOW

ALMOST READY

+1 HOUR

WAITING

+6 HOURS

WAITING

+12 HOURS

WAITING

System components: Message
MESSAGE = REAL REQUEST DATA WITH UNIQUE ID
1ST FIELD

3rd FIELD

{ field data}

MESSAGE ID

2nd FIELD
{ field data}

{ field data}

ID FORMAT ALLOWS 4096 MESSAGES
PER MILLISECOND FROM ONE NODE
Timestamp

44 bits

Counter

12 bits

Cluster node ID

8 bits

System components: Batch
• Open for at least 1 second
• Closing if opened for > 10 seconds
• Closing if has > 100 messages
Ascending columns ordering
1ST MESSSAGE ID

2nd MESSAGE ID

3rd MESSAGE ID

BATCH ID { opt message data} { opt message data} { opt message data}

ID FORMAT REQUIRE BATCH TO BE OPENED FOR > 1 SECOND
Timestamp
Rounded to seconds

Cluster node ID + Batch Type
Last 3 digits

System components: Queue
• Similar to batch
• Unlimited
• May have batches with past time

Ascending columns ordering
1ST BATCH ID
QUEUE NAME

2nd BATCH ID

3rd BATCH ID

{ processed at }

{ processed at }

{ processed at }

System components: Broker
batches polling
BROKER
check batch time
process batch
PROCESSOR

QUEUE

lock batch for
processing

ZOOKEEPER

•
•
•
•
•

Natural pre-fetch thanks to batches
Easy to control messages processing
Simple concurrency model
Easy scalable between nodes
No high loading on Cassandra

System components: Processor
PROCESSOR
OK

System components: Processor
PROCESSOR

redeliver
on failure
ANOTHER BATCH

•
•
•
•

Tries to process messages as quickly as possible
On error just redeliver message
Messages are processed concurrently
Any redelivery business logic is easy to implement

Warnings and benefits
• Message and batch must be
checked before processing
• Hard to explain “queue” size
• Separate columns for status
tracking of message
• Perform correct compaction
from time to time

• Expected loading is handled
with single node
• Everything works on
commodity hardware
• Single storage for all data
• System is easily scalable and
reliable (no message was
lost)

@xpinjection
http://xpinjection.com
mikalai.alimenkou@xpinjection.com

Mais conteúdo relacionado

Mais procurados

Ryan Blue explains how Netflix is building on Parquet to enhance its 40+ petabyte warehouse, combining Parquet’s features with Presto and Spark to boost ETL and interactive queries. Information about tuning Parquet is hard to find. Ryan shares what he’s learned, creating the missing guide you need. Topics include: * The tools and techniques Netflix uses to analyze Parquet tables * How to spot common problems * Recommendations for Parquet configuration settings to get the best performance out of your processing platform * The impact of this work in speeding up applications like Netflix’s telemetry service and A/B testing platform

Parquet performance tuning: the missing guide

Ryan Blue

"So, you have built/inherited/discovered one of your many Kafka clusters. How now do you know that it is good enough to sustain and grow your applications? Do you stress test it as a data store, a messaging system, as middleware, or like a REST API? Or are you in production and worried about the next unprecedented surge? Find out from those who have asked and answered before. Repeatable, and recordable stress testing for Kafka is a challenge for novices and some legends. Real supplies like storage, compute, network, threads etc. do not naturally map to demands of messages, bytes, and milliseconds. In the session, we will cover ways to: * Define parameters and variables before beginning * Accommodate for changing conditions - brokers, applications, config, network * Overlap infrastructure, test design, latency, and throughput * Meet cost, service level agreements, and multi-tenancy needs while testing * Do it all without entirely relying on estimation, and extrapolation We will also discuss common and innovative practices observed in the industry to meet this challenge. At the end of the session, you would walk away with the knowledge needed to set up a repeatable stress test suite without stress."

Chill, Distill, No Overkill: Best Practices to Stress Test Kafka with Siva Ku...

HostedbyConfluent

Advanced Postgres Monitoring

Denish Patel

Igor Anishchenko Odessa Java TechTalks Lohika - May, 2012 Let's take a step back and compare data serialization formats, of which there are plenty. What are the key differences between Apache Thrift, Google Protocol Buffers and Apache Avro. Which is "The Best"? Truth of the matter is, they are all very good and each has its own strong points. Hence, the answer is as much of a personal choice, as well as understanding of the historical context for each, and correctly identifying your own, individual requirements.

Thrift vs Protocol Buffers vs Avro - Biased Comparison

Igor Anishchenko

The reality of most large scale data deployments includes storage decoupled from computation, pipelines operating directly on files and metadata services with no locking mechanisms or transaction tracking. For this reason attempts at achieving transactional behavior, snapshot isolation, safe schema evolution or performant support for CRUD operations has always been marred with tradeoffs. This talk will focus on technical aspects, practical capabilities and the potential future of three table formats that have emerged in recent years as solutions to the issues mentioned above – ACID ORC (in Hive 3.x), Iceberg and Delta Lake. To provide a richer context, a comparison between traditional databases and big data tools as well as an overview of the reasons for the current state of affairs will be included. After the talk, the audience is expected to have a clear understanding of the current development trends in large scale table formats, on the conceptual and practical level. This should allow the attendees to make better informed assessments about which approaches to data warehousing, metadata management and data pipelining they should adapt in their organizations.

ACID ORC, Iceberg, and Delta Lake—An Overview of Table Formats for Large Scal...

Databricks

Why I quit Amazon and Build the Next-gen Streaming System

Yingjun Wu

Kafka on Pulsar

StreamNative

How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015

PostgreSQL-Consulting

Dr. Elephant helps improve Spark and Hadoop developer productivity and increase cluster efficiency by making clear recommendations on how to tune workloads and configurations. Originally developed by LinkedIn, Dr. Elephant is now in use at multiple sites. This session will explore how Dr. Elephant works, the data it collects from Spark environments and the customizable heuristics that generate tuning recommendations. Learn how Dr. Elephant can be used to improve production cluster operations, help developers avoid common issues, and green light applications for use on production clusters.

Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop with Carl ...

Databricks

Flink Forward San Francisco 2022. Probably everyone who has written stateful Apache Flink applications has used one of the fault-tolerant keyed state primitives ValueState, ListState, and MapState. With RocksDB, however, retrieving and updating items comes at an increased cost that you should be aware of. Sometimes, these may not be avoidable with the current API, e.g., for efficient event-time stream-sorting or streaming joins where you need to iterate one or two buffered streams in the right order. With FLIP-220, we are introducing a new state primitive: BinarySortedMultiMapState. This new form of state offers you to (a) efficiently store lists of values for a user-provided key, and (b) iterate keyed state in a well-defined sort order. Both features can be backed efficiently by RocksDB with a 2x performance improvement over the current workarounds. This talk will go into the details of the new API and its implementation, present how to use it in your application, and talk about the process of getting it into Flink. by Nico Kruber

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Flink Forward

Data and AI summit: data pipelines observability with open lineage

Julien Le Dem

Structured Streaming has proven to be the best platform for building distributed stream processing applications. Its unified SQL/Dataset/DataFrame APIs and Spark’s built-in functions make it easy for developers to express complex computations. Delta Lake, on the other hand, is the best way to store structured data because it is a open-source storage layer that brings ACID transactions to Apache Spark and big data workloads Together, these can make it very easy to build pipelines in many common scenarios. However, expressing the business logic is only part of the larger problem of building end-to-end streaming pipelines that interact with a complex ecosystem of storage systems and workloads. It is important for the developer to truly understand the business problem that needs to be solved. Apache Spark, being a unified analytics engine doing both batch and stream processing, often provides multiples ways to solve the same problem. So understanding the requirements carefully helps you to architect your pipeline that solves your business needs in the most resource efficient manner. In this talk, I am going examine a number common streaming design patterns in the context of the following questions. WHAT are you trying to consume? What are you trying to produce? What is the final output that the business wants? What are your throughput and latency requirements? WHY do you really have those requirements? Would solving the requirements of the individual pipeline actually solve your end-to-end business requirements? HOW are going to architect the solution? And how much are you willing to pay for it? Clarity in understanding the ‘what and why’ of any problem can automatically much clarity on the ‘how’ to architect it using Structured Streaming and, in many cases, Delta Lake.

Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...

Databricks

[Pgday.Seoul 2021] 2. Porting Oracle UDF and Optimization

PgDay.Seoul

Advanced MySQL Query Tuning - talk at Percona Live and MySQL Meetup tour. Tuning Queries and Schema/Indexes can significantly increase performance of your application and decrease response times. This year I will cover new MySQL 5.6 and 5.7 algorithms that has been designed to improve query performance and simply tuning. Topics: 1. Group by and order by optimizations 2. MySQL temporary tables and filesort 3. Using covered indexes to optimize your queries 4. Loose and tight index scan in MySQL 5. Using summary tables to optimize your reporting queries 6. New MySQL 5.6 and 5.7 Optimizer features and improvements

Advanced MySQL Query Tuning

Alexander Rubin

Apache Spark in Depth: Core Concepts, Architecture & Internals

Anton Kirillov

Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores. With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.

Presto: Optimizing Performance of SQL-on-Anything Engine

DataWorks Summit

New Generation Oracle RAC Performance

Anil Nair

As a general computing engine, Spark can process data from various data management/storage systems, including HDFS, Hive, Cassandra and Kafka. For flexibility and high throughput, Spark defines the Data Source API, which is an abstraction of the storage layer. The Data Source API has two requirements. 1) Generality: support reading/writing most data management/storage systems. 2) Flexibility: customize and optimize the read and write paths for different systems based on their capabilities. Data Source API V2 is one of the most important features coming with Spark 2.3. This talk will dive into the design and implementation of Data Source API V2, with comparison to the Data Source API V1. We also demonstrate how to implement a file-based data source using the Data Source API V2 for showing its generality and flexibility.

Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang

Databricks

Processing Large Data with Apache Spark -- HasGeek

Venkata Naga Ravi

Mongo DB

Tata Consultancy Services

Mais procurados (20)

Parquet performance tuning: the missing guide

Chill, Distill, No Overkill: Best Practices to Stress Test Kafka with Siva Ku...

Advanced Postgres Monitoring

Thrift vs Protocol Buffers vs Avro - Biased Comparison

ACID ORC, Iceberg, and Delta Lake—An Overview of Table Formats for Large Scal...

Why I quit Amazon and Build the Next-gen Streaming System

Kafka on Pulsar

How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015

Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop with Carl ...

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Data and AI summit: data pipelines observability with open lineage

Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...

[Pgday.Seoul 2021] 2. Porting Oracle UDF and Optimization

Advanced MySQL Query Tuning

Apache Spark in Depth: Core Concepts, Architecture & Internals

Presto: Optimizing Performance of SQL-on-Anything Engine

New Generation Oracle RAC Performance

Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang

Processing Large Data with Apache Spark -- HasGeek

Mongo DB

Semelhante a High performance queues with Cassandra

Donatas Mažionis, Building low latency web APIs

Tanya Denisyuk

Network protocols and vulnerabilities

G Prachi

Stream Processing is emerging as a popular paradigm for data processing architectures, because it handles the continuous nature of most data and computation and gets rid of artificial boundaries and delays. In this talk, we are going to look at some of the most common misconceptions about stream processing and debunk them. - Myth 1: Streaming is approximate and exactly-once is not possible. - Myth 2: Streaming is for real-time only. - Myth 4: Streaming is harder to learn than Batch Processing. - Myth 3: You need to choose between latency and throughput. We will look at these and other myths and debunk them at the example of Apache Flink. We will discuss Apache Flink's approach to high performance stream processing with state, strong consistency, low latency, and sophisticated handling of time. With such building blocks, Apache Flink can handle classes of problems previously considered out of reach for stream processing. We also take a sneak preview at the next steps for Flink.

Debunking Common Myths in Stream Processing

DataWorks Summit/Hadoop Summit

This talk shares experiences from deploying and tuning Flink steam processing applications for very large scale. We share lessons learned from users, contributors, and our own experiments about running demanding streaming jobs at scale. The talk will explain what aspects currently render a job as particularly demanding, show how to configure and tune a large scale Flink job, and outline what the Flink community is working on to make the out-of-the-box for experience as smooth as possible. We will, for example, dive into - analyzing and tuning checkpointing - selecting and configuring state backends - understanding common bottlenecks - understanding and configuring network parameters

Stephan Ewen - Experiences running Flink at Very Large Scale

Ververica

In this talk, we'll explain our journey from having near-zero monitoring to having all of our infrastructure monitored with the necessary metrics and alerts. We will share with the audience some of the mistakes we did and what lessons we have learned. We currently have around 200 instances monitored with a comfortable cost-effective in-house monitoring stack based on Prometheus. We want to demonstrate that you don't need to have a big fleet to embrace Prometheus and that it is a non-expensive solution for monitoring. ---------- ShuttleCloud is a small startup specialized in email and contacts migrations. We developed a reliable migration platform in high availability used by clients like Gmail, GContacts and Comcast. For example, Gmail alone has imported data for 3 million users with our API and we process hundreds of terabytes every month. ------------- Follow us on Twitter: @ShuttleCloud: https://twitter.com/ShuttleCloud @ShuttleCloudEng: https://twitter.com/ShuttleCloudEng ShuttleCloud.com

Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016

ShuttleCloud

This presentation was part of EXPERTALKS: Nov 2012 conducted at Equal Experts India on 24th Nov 2012. Visit http://meetup.com/expertalks to know more... ------------------------------------------------------------------------------------- Ever wondered how Facebook is able to support user requests growing at thousands per second without ever crashing ? Any enterprise application that we use today; whether it’s a banking website or an eCommerce app; uses Clustering. Clustering addresses some of the most crucial demands of today’s ever-growing user base; including scalability and 24/7 availability amongst others. This presentation discusses the concept of web application clustering. FOCUS AREAS: * Why clustering * Clustering Types * Load Balancing Algorithms * Fault Tolerance It also has a guide workshop on how to build a Vertical Cluster using Tomcat.

EXPERTALKS: Nov 2012 - Web Application Clustering

EXPERTALKS

Openstack meetup lyon_2017-09-28

Xavier Lucas

Kafka overview v0.1

Mahendran Ponnusamy

Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL

confluent

Amazon Kinesis is a fully managed, cloud-based service for real-time data processing over large, distributed data streams. Customers who use Amazon Kinesis can continuously capture and process real-time data such as website clickstreams, financial transactions, social media feeds, IT logs, location-tracking events, and more. In this session, we first focus on building a scalable, durable streaming data ingest workflow, from data producers like mobile devices, servers, or even a web browser, using the right tool for the right job. Then, we cover code design that minimizes duplicates and achieves exactly-once processing semantics in your elastic stream-processing application, built with the Kinesis Client Library. Attend this session to learn best practices for building a real-time streaming data architecture with Amazon Kinesis, and get answers to technical questions frequently asked by those starting to process streaming events.

(BDT403) Best Practices for Building Real-time Streaming Applications with Am...

Amazon Web Services

3.2 Streaming and Messaging

振东刘

Ingestion and Dimensions Compute and Enrich using Apache Apex

Apache Apex

2.communcation in distributed system

Gd Goenka University

Presto At Treasure Data

Taro L. Saito

Scalable IoT platform

Swapnil Bawaskar

OpenStack High Availability

Jakub Pavlik

OpenStack HA

tcp cloud

Apache Kafka, Apache Cassandra and Kubernetes are open source big data technologies enabling applications and business operations to scale massively and rapidly. While Kafka and Cassandra underpins the data layer of the stack providing capability to stream, disseminate, store and retrieve data at very low latency, Kubernetes is a container orchestration technology that helps in automated application deployment and scaling of application clusters. In this presentation, Paul will reveal how he architected a massive scale deployment of a streaming data pipeline with Kafka and Cassandra to cater to an example Anomaly detection application running on a Kubernetes cluster and generating and processing massive amount of events. Anomaly detection is a method used to detect unusual events in an event stream. It is widely used in a range of applications such as financial fraud detection, security, threat detection, website user analytics, sensors, IoT, system health monitoring, etc. When such applications operate at massive scale generating millions or billions of events, they impose significant computational, performance and scalability challenges to anomaly detection algorithms and data layer technologies. Paul will demonstrate the scalability, performance and cost effectiveness of Apache Kafka, Cassandra and Kubernetes, with results from his experiments allowing the Anomaly detection application to scale to 19 Billion anomaly checks per day. Melbourne Big Data Meetup, March 5 2020 https://www.eventbrite.com/e/melbourne-big-data-meetup-realtime-anomaly-detection-with-cassandra-kafka-tickets-93028445585

Melbourne Big Data Meetup Talk: Scaling a Real-Time Anomaly Detection Applica...

Paul Brebner

Apache Kafka, Apache Cassandra and Kubernetes are open source big data technologies enabling applications and business operations to scale massively and rapidly. While Kafka and Cassandra underpins the data layer of the stack providing capability to stream, disseminate, store and retrieve data at very low latency, Kubernetes is a container orchestration technology that helps in automated application deployment and scaling of application clusters. In this presentation, we will reveal how we architected a massive scale deployment of a streaming data pipeline with Kafka and Cassandra to cater to an example Anomaly detection application running on a Kubernetes cluster and generating and processing massive amount of events. Anomaly detection is a method used to detect unusual events in an event stream. It is widely used in a range of applications such as financial fraud detection, security, threat detection, website user analytics, sensors, IoT, system health monitoring, etc. When such applications operate at massive scale generating millions or billions of events, they impose significant computational, performance and scalability challenges to anomaly detection algorithms and data layer technologies. We will demonstrate the scalability, performance and cost effectiveness of Apache Kafka, Cassandra and Kubernetes, with results from our experiments allowing the Anomaly detection application to scale to 19 Billion anomaly checks per day.

ApacheCon2019 Talk: Kafka, Cassandra and Kubernetesat Scale – Real-time Ano...

Paul Brebner

Performance

Christophe Marchal

Semelhante a High performance queues with Cassandra (20)

Donatas Mažionis, Building low latency web APIs

Network protocols and vulnerabilities

Debunking Common Myths in Stream Processing

Stephan Ewen - Experiences running Flink at Very Large Scale

Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016

EXPERTALKS: Nov 2012 - Web Application Clustering

Openstack meetup lyon_2017-09-28

Kafka overview v0.1

Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL

(BDT403) Best Practices for Building Real-time Streaming Applications with Am...

3.2 Streaming and Messaging

Ingestion and Dimensions Compute and Enrich using Apache Apex

2.communcation in distributed system

Presto At Treasure Data

Scalable IoT platform

OpenStack High Availability

OpenStack HA

Melbourne Big Data Meetup Talk: Scaling a Real-Time Anomaly Detection Applica...

ApacheCon2019 Talk: Kafka, Cassandra and Kubernetesat Scale – Real-time Ano...

Performance

Mais de Mikalai Alimenkou

Люди в мире Agile используют Story Points - для Agile коучей и тренеров это самый простой способ объяснить, как следует проводить оценку и планирование в «новом мире». Но тогда эта простая концепция нарушает реальные практические кейсы. В настоящее время команды состоят из очень специализированных людей, работающих над бэкендом, фронтэндом, тестировании, инфраструктуре и прочим. Для них почти невозможно иметь общий уровень сложности. Это только одна из проблем, которые мы собираемся осветить в этом докладе. Чтобы оставаться конструктивным, а не просто старомодным парнем из XP, Николай поделится своим опытом с более точной и прагматичной техникой оценки/планирования - планированием на основе возможностей.

Rise and fall of Story Points. Capacity based planning from the trenches.

Mikalai Alimenkou

We have spent many years testing our applications and systems manually and with test automation tools. During this time many bug root causes have been classified and could be detected automatically with special static analysis tools. Most of them could be applied at the early stages of development even before code is integrated into the main development branch. In this talk, I will go through available solutions and demonstrate what kinds of issues may be detected automatically reducing the time and effort of traditional testing.

Static analysis tools as the best friend of QA

Mikalai Alimenkou

In this talk, we will go through the design process of modern CI/CD for the microservices-based system with Kubernetes support. We will discuss how to verify consistency between microservices, apply different levels of quality gates and promote artifacts between environments. Thanks to Kubernetes we will review different approaches of environment resources optimization for development needs during CI/CD cycles.

Modern CI/CD in the microservices world with Kubernetes

Mikalai Alimenkou

Most of people nowadays think microservices architecture is a great way to build any system. They visit conference talks, read books and review tutorials, where ‘hello world’ applications are built just in several minutes using microservices approach. But the reality is not so wonderful and one of the biggest pain is hidden inside distributed business transactions. In monolith application this topic is almost completely covered with DB level transactions. In distributed world you have to face many issues trying to implement reliable and consistent business logic. In this talk we will review different types of problems related to distributed business transactions, popular SAGA pattern, frameworks and techniques to simplify your life without compromising quality of the system.

Saga about distributed business transactions in microservices world

Mikalai Alimenkou

Effectiveness tips from Kubernetes trenches by Captain Obvious

Mikalai Alimenkou

For a long time DB related testing in Java world has been a real pain and most developers tried to reduce number of such tests as much as possible. With good in-memory database implementations like H2, schema migration solutions like Liquibase or Flyway, containerization with libraries like TestContainers, database management is now much simpler. But test data management is still a pain. Some developers use SQL dumps, others insert data via JPA/JDBC or rely on prepared data sets. Good old DBUnit may be a good option, but it is not so developer friendly and not adopted well for modern annotations driven development style. Database Rider closes the gap between modern Java development environment and DBUnit, bringing DBUnit closer to your JUnit tests, so database testing will feel like a breeze. In addition to flexible data sets management this library provides other useful features: programmatic data sets definition, leak hunting, data sets export, constraints management, etc. As contributor and loyal user for many years, I would like to share my experience with Database Rider and demonstrate how to make database testing a fun again!

Ride the database in JUnit tests with Database Rider

Mikalai Alimenkou

I think almost everybody experienced cases when things are moving very slowly in IT companies or teams. You have many people, talented engineers, Agile process and development speed is still below expectations. We try to focus on performance and efficiency last 10 years, improving our practices and tools. But we are still there in terms of speed when they are applied to real life cases. How is it possible? In this talk we will review the concept of waste circles and understand what are the main sources of time waste in development process. This concept would help you to check your processess, focus on right things and achieve much better results in your organization or team.

Wastful waste or why everything is so slow in development

Mikalai Alimenkou

Nowadays traditional layered monolithic architecture in Java world is not so popular as 5-10 years ago. I remember how we wrote tons of code for each layer repeating almost the same parts for every application. Add unit and integration testing to understand how much time and efforts has been spent on repeatable work. All cool ideas around DDD (domain driven design) and Hexagonal Architecture was just a nice theory because reality hasn’t allow us to implement it easily. Even Dependency Injection with Spring framework was completely focused on traditional layered approach, not even talking about JavaEE platform. Today we have Spring Boot ecosystem covering most of our needs for integration with almost all possible technologies and microservices architectural trend, enabling completely new approach to build Java applications around domain model. It is so natural to build Java domain-oriented services and connect them with external world using ports and adapters, that Hexagonal Architecture is almost enabled by default. You just need to switch your way of thinking…

Hexagonal architecture with Spring Boot

Mikalai Alimenkou

Wastful waste or why everything is so slow in development

Mikalai Alimenkou

DevOps become a buzzword in a last few years. Several companies, development and product teams have achieved quite impressive results in this area making cultural changes, transforming their processes and practices, introducing new roles, tools and techniques. Do you think is achievable for you team or it’s still a bunch of drama? There is no common approach for measuring achievements and understanding how much DevOps’ich the current team/company is. In this talk I will provide attendees with basic checklist to start with and some reliable tools/techniques to monitor progress of “DevOps transformation”.

DevOps checklist or how to understand where is your team in DevOps landscape ...

Mikalai Alimenkou

DevOps is a hot topic during last several years. Some companies, teams and products have achieved quite impressive results in this area making cultural changes, transforming their processes and practices, introducing new roles, tools and techniques. At the same time there is no common approach for measuring achievements and understanding “how DevOps” the current team/company is. In this talk I will provide attendees with basic checklist to start with and some reliable tools/techniques to monitor progress of “DevOps transformation”.

DevOps checklist or how to understand where is your team in DevOps landscape

Mikalai Alimenkou

Почти год мы в Whirl Software разрабатываем систему Медкарта в масштабе целой страны. За это время мы столкнулись с множеством интересных сложностей и проблем, часть из которых успешно победили, а для некоторых хорошего решения до сих пор не найдено. В этом докладе мы поделимся накопленным практическим опытом и некоторыми техническими решениями, которые могут быть полезны в рамках разработки электронных медицинских систем.

Практические трудности в разработке Медкарты для целой страны

Mikalai Alimenkou

Hexagonal architecture with Spring Boot [EPAM Java online conference]

Mikalai Alimenkou

Almost any application or software system manages data. It is hard to imagine test automation that is not affected by this fact. There are many differenct approaches how to prepare system under test, providing predefined test data: use application UI, invoke API methods, run business logic directly, access DB from test scenarios, etc. In this talk we will review most of existing approaches, starting from the easiest and the most popular ones and finishing with really tricky ways to manage your test data for large distributed systems. There is no ideal solution for every case or silver bullet, but I hope your toolset will become wider after visiting this talk.

Bro, manage test data like a pro! [QA Fest 2018]

Mikalai Alimenkou

Вот уже более 10 лет Agile движение шагает по Украине и стучится практически в каждую компанию. Но приносят ли новые процессы, принципы и практики реальную практическую пользу? Получается ли изменить к лучшему команды, проекты, компании? Я в роли консультанта за эти 10 лет поработал с более чем 100 компаниями, поэтому повидал много хорошего и плохого. В данном докладе хочется пробежаться по основным проблемам, сложностям и анти-паттернам в переходе отечественных компаний на "Agile рельсы". Мы рассмотрим какие практики не очень хорошо приживаются, от чего страдает большинство команд, какие основные препятствия встречают на своем пути и как умудряются их обходить. Я надеюсь, доклад поможет зародиться множеству интересных дискуссий.

Agile antipatterns: review after 10 years of practice

Mikalai Alimenkou

Hexagonal architecture with Spring Boot

Mikalai Alimenkou

Bro, manage test data like a pro!

Mikalai Alimenkou

Тестировщики часто говорят о противостоянии и конфликтах с разработчиками. Но ведь есть команды, где все живут в мире и согласии. Видимо что-то тут не так? Я хочу поговорить о том, как тестировщиков видят сами разработчики. В докладе будет проведена забавная классификация. Кроме известного всем тестировщика-обезьянки будут представлены тестировщик-муха, тестировщик-нацист, тестировщик-панда и многие другие герои. Высможете лишний раз задуматься над тем, как вас видят со стороны и, возможно, изменить ситуацию к лучшему. Доклад будет также полезен менеджерам проектов и лидерам команд. Вы сможете быстрее распознавать те или иные шаблоны поведения тестировщикови принимать меры по повышению уровня командной работы. Приходите, будет интересно!

Бытовая классификация тестировщиков с точки зрения разработчика

Mikalai Alimenkou

Usually it is hard to analyze personal effectiveness and detect wastes in development process because developer’s work decomposition is not transparent and available for analysis. As a good sample of ineffective process imagine developer, who spends 1 day on task implementation and then reimplements it several times according to code review notes during next 2 days. Or another developer, who is waiting for code review during 2 days, switching context to other tasks, finally gets notes and switches back to initial task, trying to refresh all details in his head. And so on and so forth… Code review tool usage helps to aggregate lots of useful information about any code change at any stage (static analysis, code review, rework, acceptance, integration into main branch). In this talk I’m going to demontrate how this information could be used for detailed analysis of development effectiveness and wastes detection. Based on mentioned analysis you could implement many improvements for your development process and then measure their success.

Code Review tool for personal effectiveness and waste analysis

Mikalai Alimenkou

During last several years DevOps became strong buzzword used almost in every project, team and company. But almost everywhere it is used in very funny and strange context. For example, existing ops guys are renamed to DevOps just to sell them to the client for more money. Or DevOps is used as new job title for some magically powerful person who is able to operate cloud environment and modern infrastructure related tools, leading team of old school ops and participating in management meetings. In this talk I’m going to review all different anti-patterns and bad practices in DevOps landscape using stories from my personal experience as Delivery Manager and independent consultant.

Funny stories and anti-patterns from DevOps landscape

Mikalai Alimenkou

Mais de Mikalai Alimenkou (20)

Rise and fall of Story Points. Capacity based planning from the trenches.

Static analysis tools as the best friend of QA

Modern CI/CD in the microservices world with Kubernetes

Saga about distributed business transactions in microservices world

Effectiveness tips from Kubernetes trenches by Captain Obvious

Ride the database in JUnit tests with Database Rider

Wastful waste or why everything is so slow in development

Hexagonal architecture with Spring Boot

Wastful waste or why everything is so slow in development

DevOps checklist or how to understand where is your team in DevOps landscape ...

DevOps checklist or how to understand where is your team in DevOps landscape

Практические трудности в разработке Медкарты для целой страны

Hexagonal architecture with Spring Boot [EPAM Java online conference]

Bro, manage test data like a pro! [QA Fest 2018]

Agile antipatterns: review after 10 years of practice

Hexagonal architecture with Spring Boot

Bro, manage test data like a pro!

Бытовая классификация тестировщиков с точки зрения разработчика

Code Review tool for personal effectiveness and waste analysis

Funny stories and anti-patterns from DevOps landscape

Último

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Architecting Cloud Native Applications

WSO2

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Dubai, known for its towering skyscrapers, luxurious lifestyle, and relentless pursuit of innovation, often finds itself in the global spotlight. However, amidst the glitz and glamour, the emirate faces its own set of challenges, including the occasional threat of flooding. In recent years, Dubai has experienced sporadic but significant floods, disrupting normalcy and posing unique challenges to its infrastructure. Among the critical nodes in this bustling metropolis is the Dubai International Airport, a vital hub connecting the world. This article delves into the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Orbitshub

Retrieval augmented generation (RAG) is the most popular style of large language model application to emerge from 2023. The most basic style of RAG works by vectorizing your data and injecting it into a vector database like Milvus for retrieval to augment the text output generated by an LLM. This is just the beginning. One of the ways that we can extend RAG, and extend AI, is through multilingual use cases. Typical RAG is done in English using embedding models that are trained in English. In this talk, we’ll explore how RAG could work in languages other than English. We’ll explore French, Chinese, and Polish.

Introduction to Multilingual Retrieval Augmented Generation (RAG)

Zilliz

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

Angeliki Cooney has spent over twenty years at the forefront of the life sciences industry, working out of Wynantskill, NY. She is highly regarded for her dedication to advancing the development and accessibility of innovative treatments for chronic diseases, rare disorders, and cancer. Her professional journey has centered on strategic consulting for biopharmaceutical companies, facilitating digital transformation, enhancing omnichannel engagement, and refining strategic commercial practices. Angeliki's innovative contributions include pioneering several software-as-a-service (SaaS) products for the life sciences sector, earning her three patents. As the Senior Vice President of Life Sciences at Avenga, Angeliki orchestrated the firm's strategic entry into the U.S. market. Avenga, a renowned digital engineering and consulting firm, partners with significant entities in the pharmaceutical and biotechnology fields. Her leadership was instrumental in expanding Avenga's client base and establishing its presence in the competitive U.S. market.

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Angeliki Cooney

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

[BuildWithAI] Introduction to Gemini.pdf

Sandro Moreira

Whatsapp Number Escorts Call girls 8617370543 Available 24x7 Mcleodganj Call Girls Service Offer Genuine VIP Model Escorts Call Girls in Your Budget. Mcleodganj Call Girls Service Provide Real Call Girls Number. Make Your Sexual Pleasure Memorable with Our Mcleodganj Call Girls at Affordable Price. Top VIP Escorts Call Girls, High Profile Independent Escorts Call Girls, Housewife Women Escorts Call Girl, College Girls Escorts Call Girls, Russian Escorts Call girls Service in Your Budget.

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Deepika Singh

High performance queues with Cassandra

1. High performance queues with Cassandra Mikalai Alimenkou http://xpinjection.com @xpinjection

2. Kiev, Ukraine #евромайдан

4. How to process data, master ? Asynchronously!

5. Queues usage scenario

6. More realistic scenario

7. More realistic scenario

8. Are you crazy? Cassandra for queues?

9. So many cool MQ providers

10. Initial expected loading

11. Some specific requirements 1 Message Queue External Service Provider

12. Some specific requirements Message Queue 1 External Service Provider

13. Some specific requirements 2 Message Queue External Service Provider

14. Some specific requirements Message Queue 2 1 Redeliver after 1 hour External Service Provider

15. Some specific requirements 2 Message Queue External Service Provider Redeliver after 1 hour Redelivery business logic and external service hourly based usage limits

16. Idea came from railways

17. “Body flow” in regular life

18. Message batches “station” QUEUE NOW ALMOST READY +1 HOUR WAITING +6 HOURS WAITING +12 HOURS WAITING

19. System components: Message MESSAGE = REAL REQUEST DATA WITH UNIQUE ID 1ST FIELD 3rd FIELD { field data} MESSAGE ID 2nd FIELD { field data} { field data} ID FORMAT ALLOWS 4096 MESSAGES PER MILLISECOND FROM ONE NODE Timestamp 44 bits Counter 12 bits Cluster node ID 8 bits

20. System components: Batch • Open for at least 1 second • Closing if opened for > 10 seconds • Closing if has > 100 messages Ascending columns ordering 1ST MESSSAGE ID 2nd MESSAGE ID 3rd MESSAGE ID BATCH ID { opt message data} { opt message data} { opt message data} ID FORMAT REQUIRE BATCH TO BE OPENED FOR > 1 SECOND Timestamp Rounded to seconds Cluster node ID + Batch Type Last 3 digits

21. System components: Queue • Similar to batch • Unlimited • May have batches with past time Ascending columns ordering 1ST BATCH ID QUEUE NAME 2nd BATCH ID 3rd BATCH ID { processed at } { processed at } { processed at }

22. System components: Broker batches polling BROKER check batch time process batch PROCESSOR QUEUE lock batch for processing ZOOKEEPER • • • • • Natural pre-fetch thanks to batches Easy to control messages processing Simple concurrency model Easy scalable between nodes No high loading on Cassandra

23. System components: Processor PROCESSOR

24. System components: Processor PROCESSOR OK

25. System components: Processor PROCESSOR

26. System components: Processor PROCESSOR redeliver on failure ANOTHER BATCH • • • • Tries to process messages as quickly as possible On error just redeliver message Messages are processed concurrently Any redelivery business logic is easy to implement

27. Warnings and benefits • Message and batch must be checked before processing • Hard to explain “queue” size • Separate columns for status tracking of message • Perform correct compaction from time to time • Expected loading is handled with single node • Everything works on commodity hardware • Single storage for all data • System is easily scalable and reliable (no message was lost)

28. Show me the code!

29. @xpinjection http://xpinjection.com mikalai.alimenkou@xpinjection.com

High performance queues with Cassandra

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a High performance queues with Cassandra

Semelhante a High performance queues with Cassandra (20)

Mais de Mikalai Alimenkou

Mais de Mikalai Alimenkou (20)

Último

Último (20)

High performance queues with Cassandra