Apache Kafka in the Airline, Aviation and Travel Industry

@KaiWaehner - www.kai-waehner.de
Event Streaming with Apache Kafka
in the Aviation Industry
Airlines, Airports, Aerospace, Manufacturing, GDS, Flight Safety, and Retail
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.confluent.io
www.kai-waehner.de
linkedin.com/in/kaiwaehner

The Aviation Industry is Changing…
Aviation and travel
are notoriously vulnerable to social,
economic, and political events.
Digital modernization is coming
not just due to the Corona Virus,
but because of the ever-changing
expectations of consumers.

The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for
growth
“Good enough” to run on
yesterday’s data

Stores become Software

Banks become Software

Taxis become Software
2
min

Cars become Software

Car Production becomes Software

Aviation becomes an Experience

This transformation is
happening everywhere

What enables this
transformation?

Cloud Machine
Learning
Mobile Event
Streaming
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers

Real-time Data beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Banking
Fraud detection
Trading, risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases

This is a fundamental paradigm shift...
18
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

What is Event Streaming?

Event Streaming is the
Central Nervous System
for today’s enterprises.
Apache Kafka®
is the technology.

‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Banking
Alice sent $250 to Bob on Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka

Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.

The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka

Kafka & Confluent

Confluent pioneered Event Streaming
Hall of Innovation
CTO Innovation
Award Winner
2019
Enterprise Technology
Innovation
AWARDS
Confluent founders are
original creators of Kafka
Confluent team wrote
80% of Kafka software
commits, has over 1M hours
technical experience with
Kafka, and operates 5,000+
clusters
Confluent Cloud
is the only multi-cloud, fully
managed, pay-as-you-go
event streaming service in
the world
Confluent Platform
completes Apache Kafka
and turns it into a secure,
enterprise-ready platform

Lufthansa’s KUSCO
(Kafka Unified Streaming Cloud Operations)
Confluent Streaming Event 2020:
https://videos.confluent.io/watch/X4j6nh3e8Kr8EvsRCeSEhJ

Singapore Airlines
Streaming Predictive Maintenance with Kafka and Machine Learning
https://speakerdeck.com/devacto/predictive-maintenance-pipeline-using-kafka-connect-streams-and-ksql

28
https://www.confluent.io/kafka-summit-sf18/hop-airlines-jets-to-real-time/

Amadeus IT Group
29
https://kafka.apache.org/powered-by
Major European IT provider for the global travel and tourism industry
Global Distribution System (GDS) provides search, pricing, booking,
ticketing and other processing services in real-time
Customers include airlines, hotels, tour operators, insurers, car
rental and railway companies

What is Kafka?

Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams

Kafka Loves Your Existing Systems.
...many more
Other
Systems
Other
Systems
Kafka
Connect
Kafka Cluster
Kafka
Connect

Kafka Loves Your Existing Systems.
✓ Hundreds of ready-to-use
connectors available
✓ Supported by Confluent
or by partners (certified)
✓ Managed Connectors in
Confluent Cloud
$ confluent-hub install mongodb/kafka-connect-mongodb:1.3.0

Confluent Cloud
✓ Infinite Storage
Confluent Platform
✓ Tiered Storage (e.g., AWS S3)
Apache Kafka
• Tiered Storage upcoming
Kafka Stores Your Data Durably.
Kafka is the source of truth.
Powers NYTimes.com, and stores
all articles ever published since 1851.
September 30, 1851, Page 1

Kafka Makes Your Business Real-time.
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB

Kafka Streams — #1 popular choice
✓ Client library for Java, Scala
✓ Elastic, highly scalable, fault-tolerant
✓ Filter, transform, join, aggregate, window, etc.
✓ Exactly-once processing
✓ Full Kafka security integration
Build Apps, not Processing Clusters
✓ Standard JVM application
✓ Run in container, VMs, K8S, bare-metal, etc.
✓ No separate ‘processing cluster’ needed
✓ For S, M, L, XXL use cases
Kafka Makes Your Business Real-time.
Kafka Cluster
REAL-TIME APPLICATIONS

ksqlDB, the Event Streaming Database
DB
APP
APP
PULL
PUSH
CONNECTORS
STREAM PROCESSING
TABLES & STATE
ksqlDB
DB One Solution for:
✓ capturing events
✓ stream processing
✓ serving both push
and pull queries
✓ also available in
Confluent Cloud
CREATE SOURCE CONNECTOR postgresSource WITH (...);
CREATE STREAM customer_changes …;
CREATE TABLE customers AS SELECT ... FROM customer_changes;
APP
“The easiest way to get started with Kafka.”

Events at an Airport
https://www.confluent.io/kafka-summit-sf18/serving-the-real-time-data-needs-of-an-airport/

Event Streaming with Kafka and ksqlDB at an Airport
https://www.confluent.io/kafka-summit-sf18/serving-the-real-time-data-needs-of-an-airport/

Event Topics Storage Partitions
Events / sec Kafka Servers
10,000,000 25,000 1,000,000 1,500
250,000 500 25,000 25
100 5 300 3
Kafka Scales with Your Business.

Distributed System with Replication and High Availability
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4

Distributed System with Replication and High Availability
read,
write
App
(Kafka Streams)
Kafka
(Data)
More Apps
(KSQL, Connect, Python,
REST, “You-name-it”)
Bookings
Team
Fraud
Team
…
Mobile
Team
…

Multi-Region Cluster
Kafka Clusters
can Stretch over Regions
No Downtime + Zero Data loss
e.g. Stretched over US East + Mid + West
Automated Disaster Recovery
Sync or Async Replication per Topic
Offset Preserving
Automated Client Failover without Custom Code

45
Cluster Linking requires no additional
infrastructure and preserves offsets:
Global Central
Nervous System
Cluster Linking
simplifies hybrid-cloud
and multi-cloud
deployments for Kafka

Connected Car Infrastructure
https://www.youtube.com/watch?v=yGLKi3TMJv8
• Real Time Data Analysis
• Swarm Intelligence
• Collaboration with Partners
• Predictive AI
• …

Streaming Aviation Data Exchange
MirrorMaker 2 /
Confluent Replicator
Airport
Server: Latest version
Clients: 0.11, 2.0
Streaming replication
between stakeholders
Version compatibility
between different
clients and servers
Airline
Server: AK 2.3
Clients: 2.0, 2.5
GDS
Server: CP 54 / AK 2.4
Clients: 0.11, 2.4
Confluent
Cluster Linking

ETL/Data Integration Messaging
Highly Scalable
Durable
Persistent
Ordered
Real-time Difficult to Scale
No Persistence After
Consumption
No Replay
Batch
Expensive
Time Consuming

Highly Scalable
Persistent
Messaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence After
Consumption
No Replay
Real-time
Highly Scalable
Durable
Persistent
Ordered
Real-time
Event Streaming

Kafka Connect
Kafka Cluster
CRM
Integration
Domain-Driven Design for your Integration Layer
Legacy
Integration
Custom
Application
ESB Connector
Java / Python /
ksqlDB / etc.
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
è Independent and loosely coupled, but scalable, highly available and reliable!

Kafka Cluster
Domain-Driven Design for your Event Steaming Platform
Schema
Registry
Event Streaming Platform
Big Data Cluster Big Data Cluster
KSQL
App
KSQL
App
KSQL
App

Use Cases for Reprocessing Historical Events
Give me all events from time A to time B
Real-time Producer
Time
• New consumer application
• Error-handling
• Compliance / regulatory processing
• Query and analyze existing events
• Schema changes in analytics platform
• Model training
Real-time Consumer
Consumer of
Historical Data

Confluent Tiered Storage for Kafka
53

Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model B
Model A
Producer
Distributed
Commit Log
Streaming Ingestion and Model Training
with TensorFlow IO
https://github.com/tensorflow/io
54
Model X
(at a later time)

Event Streaming Is The Future Of Data
55
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

Why Confluent?

I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
57

The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
58

Confluent Platform
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER EXECUTIVE
Apache Kafka
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-Region Clusters | Replicator
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient Operations
at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for
Business Success

Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Apache Kafka in the Airline, Aviation and Travel Industry

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Apache Kafka in the Airline, Aviation and Travel Industry

Semelhante a Apache Kafka in the Airline, Aviation and Travel Industry (20)

Mais de Kai Wähner

Mais de Kai Wähner (17)

Último

Último (20)

Apache Kafka in the Airline, Aviation and Travel Industry