Apache kafka event_streaming___kai_waehner
•
3 gostaram•286 visualizações
The Rise of Event Streaming- Why Apache Kafka Changes Everything Kai Waehner, Sr. Systems Engineer, Confluent
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Apache kafka event_streaming___kai_waehner
Semelhante a Apache kafka event_streaming___kai_waehner (20)
Mais de confluent
Mais de confluent (20)
Último
Último (20)
Apache kafka event_streaming___kai_waehner
- 1. 1 The Rise Of Event Streaming – Why Apache Kafka Changes Everything Kai Waehner | Technology Evangelist, Confluent contact@kai-waehner.de | LinkedIn | @KaiWaehner | www.confluent.io | www.kai-waehner.de
- 2. 22 The world is changing.
- 5. 5 Banking
- 6. 6 Largest media co - creates no content Largest movie house - owns no cinemas Online bookstore - diversifying across industries Largest taxi company - owns no vehicles Most valuable retailer - has no inventory World’s largest accommodation provider - owns no real estate
- 7. 77 The New Enterprise Reality Innovate or be disrupted ● Create new business models ● Deliver new, real-time customer experiences ● Deliver massive internal efficiencies Every company is a software company ● Capital One 10,000 of 40,000 employees are software engineers ● Goldman Sachs 1.5B (billion!) lines of code across 7,000+ applications ● JP Morgan: 50+% of IT budget for new projects and applications, <50% for maintenance Innovation is about events, not “data” ● Digital-first organizations must pick the right foundational technologies ● Established enterprises must implement new technologies without “rip-and-replace”
- 8. 8 Every Company is Becoming a Software Company
- 9. 9 Every Company is Becoming a Software Company
- 11. 11 The World has Changed Business Digitalization Trends are Driving the Need to Process Events at a whole new Scale, Speed and Efficiency
- 13. 1313 A company is built on DATA FLOWS but all we have is DATA STORES
- 14. 14 A company built on data stores looks like this
- 15. 15 ETL/Data Integration Messaging Highly Scalable Durable Persistent Ordered Real-time Difficult to Scale No Persistence After Consumption No Replay Batch Expensive Time Consuming
- 16. 1616 How is it done with an Event Streaming Platform
- 17. 1717 Highly Scalable Persistent ETL/Data Integration MessagingETL/Data Integration MessagingMessaging Batch Expensive Time Consuming Difficult to Scale No Persistence After Consumption No Replay Real-timeHighly Scalable Durable Persistent Ordered Real-time Event Streaming
- 18. 18 Events A Sale An Invoice A Trade A Customer Experience
- 19. 19 An Event Streaming Platform is the Underpinning of an Event-driven Architecture Microservices DBs SaaS apps Mobile Customer 360 Real-time fraud detection Data warehouse Producers Consumers Database change Microservices events SaaS data Customer experience s Streams of real time events Streaming AppsStreaming Apps Streaming Apps
- 20. 20
- 21. 21 The beginning of a new Era https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying The first use case: Log Analytics. This is why Kafka was created!
- 22. 22 ● Global Scale ● Real-time ● Persistent Storage ● Stream Processing Apache Kafka: The De-facto Standard for Real-Time Event Streaming Edge Cloud Data LakeDatabases Datacenter IoT SaaS AppsMobile Microservices Machine Learning Apache Kafka
- 23. 23 Apache Kafka at Scale at Tech Giants > 7 trillion messages / day > 6 Petabytes / day “...you name it” * Kafka Is not just used by tech giants ** Kafka is not just used for big data
- 25. 25 Consuming from Kafka P Time C2 Near Real Time C3 Real Time C1 Batch
- 26. 26 Distributed System with Replication and High Availability Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
- 27. 27 Distributed System with Replication and High Availability read, write App (Kafka Streams) Kafka (data) More Apps (KSQL, Connect, Python, REST, “You-name-it”) BookingsTeam FraudTeam … MobileTeam …
- 28. 28 Apache Kafka (kafka.apache.org) includes Kafka Connect and Kafka Streams Kafka Streams Your app sinksource KafkaConnect KafkaConnect
- 29. 2929 KSQL Continuous queries on infinite streams
- 30. 3030 STREAM PROCESSING Create and store materialized views Filter Analyze in-flight Time C CC
- 31. 31 TRADITIONAL DATABASE EVENT STREAM PROCESSING SELECT * FROM DB_TABLE CREATE TABLE T AS SELECT * FROM EVENT_STREAM Active Query: Passive Data: DB Table Active Data: Passive Query: Event Stream
- 33. 33 PUSH PULL APP Jay’s credit score is 670 Jay’s credit score is 710 Jay’s credit score is 695 What is Jay’s credit score now? 695 APP
- 34. 37 The stream is materialized into a table, which is emitted as a stream Payments Stream Credit Score Stream CREATE TABLE credit_scores AS SELECT user, updateScore(p.amount)…
- 35. 3838 What is the result
- 36. 4040 Event-Driven App (Location Tracking) Only Real-Time Events Messaging Queues can do this Contextual Event-Driven App (ETA) Real-Time combined with stored data Only Event Streaming Platforms can do this Where is my driver? When will my driver get here? Where is my driver? When will my driver get here? Why add the word Contextual?
- 37. 41 Front, rear and top view cameras Parking assistant Environment pointer Ultrasonic Sensors Parking assistant with front and rear camera plus environment indicator Crash Sensors Front protection adaptivity Side protection Tail impact protection Infrared Camera Rearview assistance with Pedestrian recognition Front and Rear Radar Sensors ACC with stop and go function Side assist Front Camera Audi Active lane assistant Speed limit indicator Adaptive light The Future of the Automotive Industry is a Real Time Data Cluster
- 38. 42 The Future of the Automotive Industry is a Real Time Data Cluster Front, rear and top view cameras Ultrasonic SensorsCrash Sensors Front Camera Infrared Camera Front and Rear Radar Sensors Traffic Alerts Hazard Alerts Personalization Anomaly Detection MQTT MQTT MQTT MQTT MQTTMQTT
- 39. 43
- 40. 44
- 41. 454545
- 42. 464646
- 43. 474747
- 44. 484848
- 45. 494949
- 46. 52 Real Time Machine Learning with Kafka and TensorFlow
- 47. 53 Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $ Example Case Studies (of many)
- 48. 5454 How does Confluent help
- 49. $$€€££ - Making Money - Saving Money - Protecting Money Every Company is Becoming a Software Company
- 50. 5656 I N V E S T M E N T & T I M E VALUE 3 4 5 1 2 Event Streaming Maturity Model 56 Initial Awareness / Pilot Start to Build Pipeline / Deliver 1 New Outcome Leverage Stream Processing Build Contextual Event-Driven Apps Central Nervous System Product, Support, Training, Partners, Technical Account Management...
- 51. 5757 CONFLUENT PLATFORM EFFICIENT OPERATIONS AT SCALE PRODUCTION-STAGE PREREQUISITES UNRESTRICTED DEVELOPER PRODUCTIVITY Multi-language Development Rich Pre-built Ecosystem SQL-based Stream Processing GUI-driven Mgmt & Monitoring Flexible DevOps Automation Dynamic Performance & Elasticity Enterprise-grade Security Data Compatibility Global Availability APACHE KAFKA Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE COMMITTER-LED EXPERTISE PartnersTraining Professional Services Enterprise Support DEVELOPER OPERATOR ARCHITECT Hybrid Infrastructure
- 52. 58 Questions? Let’s connect... Kai Waehner Technology Evangelist kai.waehner@confluent.io @KaiWaehner www.confluent.io www.kai-waehner.de LinkedIn