(Sönke LIebau, OpenCore GmbH & Co.KG) Kafka Summit SF 2018
Airports are complex networks consisting of an immense number of systems that are necessary to keep the daily stream of passengers in constant motion. Connecting these systems in order to make the big picture transparent to the people running the show, authorities and last but not least the passengers is no simple endeavor.
In this talk I will describe a fictional airport and its effort to restructure the IT infrastructure around Kafka Streams to serve the real-time data needs of a busy airport. I will start by giving a brief overview of Kafka Streams, KSQL and the opportunities they offer for real-time stream processing. Following that we will explore the the target architecture, which relies heavily on manifested views to serve up-to-date data, while also persisting to a traditional data lake for larger analytics workflows. Additionally we will take a look at the generic data transformation framework that was created to minimize integration effort of the data receiving systems. To illustrate these ideas I will describe some examples of possible integrations: joining flight data with radar and weather data to predict arrival time at the gate down to the second, constantly updated processing data from the luggage conveyor belts as well as results from prediction models for passenger flow, and many more.
Human Factors of XR: Using Human Factors to Design XR Systems
Serving the Real-Time Data Needs of an Airport with Kafka Streams and KSQL
1. Stream Processing Airport Data
Sönke Liebau – Co-Founder and Partner @ OpenCore
October 17th 2018
Serving the Real-Time Data Needs of an Airport with Kafka
Streams and KSQL
2. Who Am I?
• Partner & Co-Founder at
• Small consulting company with a Big Data & Open Source focus
• First production Kafka deployment in 2014
Website: www.opencore.com
soenke.liebau@opencore.com
https://www.linkedin.com/in/soenkeliebau/
@soenkeliebau
13. Source: https://kafka.apache.org/20/documentation/streams/ 13
What Is Kafka Streams?
“The easiest way to write mission-critical real-time applications and
microservices”
“Kafka Streams is a client library for building applications and microservices, where the input and output
data are stored in Kafka clusters. “
14. What Is KSQL?
Confluent KSQL is the open source,
streaming SQL engine that enables
real-time data processing against
Apache Kafka®
Source: https://www.confluent.io/product/ksql/ 14