ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, PostgreSQL, Redpanda, Debezium, and Benthos to master building advanced real-time data pipelines.
•
0 gostou•15 visualizações
The slide that goes with the workshop. - Demonstrate Transitioning to Real-Time Data Systems: This track shows how to transition from traditional batch processing to real-time data handling, emphasizing the critical importance of timely updates in scenarios like air traffic control where delays can be hazardous. - Implement Real-Time Data Solutions: Participants learn to implement and integrate a technology stack featuring Apache Spark, Cassandra, and PostgreSQL to effectively manage and process live data feeds from various sources, suitable for high-stakes environments. - Employ Streaming and Data Capture Technologies: The workshop demonstrates how to utilize streaming platforms and Change Data Capture (CDC) with tools like Redpanda and Debezium to ensure continuous data flow and immediate update capture, streamlining operations and data accuracy. - Apply Data Querying Techniques: It concludes by teaching how to apply advanced querying methods using Flink’s SQL client to incorporate real-time data queries into existing systems, ensuring efficient data manipulation and integration into relational and NoSQL databases.
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, PostgreSQL, Redpanda, Debezium, and Benthos to master building advanced real-time data pipelines.
Semelhante a ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, PostgreSQL, Redpanda, Debezium, and Benthos to master building advanced real-time data pipelines. (20)
Mais de Christina Lin
Mais de Christina Lin (20)
Último
Último (20)
ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, PostgreSQL, Redpanda, Debezium, and Benthos to master building advanced real-time data pipelines.
- 1. B A T C H T O S T R E A M D A T A P I P E L I N E Christina Lin Redpanda Developer Advocate
- 2. Agenda • Quick Intro – 10-15 mins • HANDS-ON – 35-40 mins • Q & A – 5-10 mins © 2024 REDPANDA DATA
- 3. Christina Lin Developer Advocate, Redpanda aka. The Redpanda Lady SOA WebSphere DB2 Sybase Oracle MQ J2EE EJB DevOps Microservice EIP K8s Agile Integration Data Mesh Active MQ Living data stack Resilience - handle failures and scale gracefully Elasticity – infrastructure that can scale dynamically Decentralization - data ownership, empowering individual teams Performance - low latency and high throughput Autonomy – self service, define quality, and access Nimble - efficient data movement Distributed -distributed data processing for cloud native Agility – quickly respond to change in data © 2024 REDPANDA DATA
- 4. © 2024 REDPANDA DATA An ordinary day of Data Engineer
- 5. © 2024 REDPANDA DATA © 2024 REDPANDA DATA Stateless Streaming Pipeline Transform format Change, masking, filtering, validating Dispatch, Wiretap Spilt, multiple destination Control reroute Normalize/ Denormalize Enrich Multiple ingestion Stateful Streaming Pipeline Complex event processing Time-window based processing Enrich Multiple ingestion Micro batch Pipeline Transform for large output (Dataset) Partitioning Split workload Analytics batch Pipeline Analytics large volume (legacy) Transform large output (Dataset, legacy) Transport large unstructured data Better scalability for pipelines
- 6. Batch CSV Every 10 mins CSV Right away! CSV Stream Batch pipeline Batch Processing Batch pipeline TheWorkshop overview © 2024 REDPANDA DATA https://bit.ly/odsc-redpanda
- 7. VM postgres:5432 cassandra:5432 postgresload.ipynb © 2024 REDPANDA DATA https://bit.ly/odsc-redpanda The Batch cassandra:5432
- 8. VM postgres:5432 cassandra:5432 cassandraload.ipynb © 2024 REDPANDA DATA https://bit.ly/odsc-redpanda The Batch
- 9. VM postgres:5432 cassandra:5432 spark.ipynb © 2024 REDPANDA DATA https://bit.ly/odsc-redpanda The Batch CSV CSV CSV
- 10. VM postgres:5432 cassandra:5432 Map.ipynb © 2024 REDPANDA DATA https://bit.ly/odsc-redpanda Dashboard View CSV CSV CSV
- 11. VM postgres:5432cassandra:5432 redpanda-0:9092 © 2024 REDPANDA DATA https://bit.ly/osdc-redpanda Let’s Stream
- 12. VM postgres:5432cassandra:5432 © 2024 REDPANDA DATA https://bit.ly/osdc-redpanda Let’s Stream
- 13. VM postgres:5432cassandra:5432 Config © 2024 REDPANDA DATA https://bit.ly/odsc-redpanda Let’s Stream
- 14. VM postgres:5432cassandra:5432 Job manager Flink Data Stream Java JAR © 2024 REDPANDA DATA https://bit.ly/odsc-redpanda Let’s Stream
- 15. VM postgres:5432cassandra:5432 Job manager CSV © 2024 REDPANDA DATA https://bit.ly/odsc-redpanda Let’s Stream
- 16. VM postgres:5432cassandra:5432 Job manager CSV © 2024 REDPANDA DATA https://bit.ly/odsc-redpanda Let’s Stream CSV CSV
- 17. VM postgres:5432cassandra:5432 SQL Client SQL © 2024 REDPANDA DATA https://bit.ly/odsc-redpanda The Batch CSV CSV CSV
- 18. Batch CSV Batch pipeline Batch pipeline Batch Processing Every 10 mins CSV Right away! CSV Stream © 2024 REDPANDA DATA https://bit.ly/odsc-redpanda TheWorkshop overview
- 19. © 2024 REDPANDA DATA Stateless Streaming Pipeline Transform format Change, masking, filtering, validating Dispatch, Wiretap Spilt, multiple destination Control reroute Normalize/ Denormalize Enrich Multiple ingestion Stateful Streaming Pipeline Complex event processing Time-window based processing Enrich Multiple ingestion Better scalability for pipelines
- 20. © 2024 REDPANDA DATA Keep Learning Streaming - Communication Basics of K8s networking Connectivity Performance Docs Get a peak under the hood. https://docs.redpanda.com/ Blogs Keep up to date with Redpanda. https://redpanda.com/blog Slack Engage with our community. https://redpanda.com/slack Code Check out the source. https://github.com/redpanda-data Redpanda University Free, self-paced online learning https://university.redpanda.com