This document provides an introduction to Apache Flink, a framework for distributed stream and batch data processing. It discusses the differences between batch and stream processing, with batch processing operating on static data periodically and stream processing operating immediately on event streams. The document then describes Flink's programming model including data sources, transformations, and sinks. It explains Flink's time classification of event time, ingestion time, and processing time. It also covers windows, watermarks, and compares Flink to other frameworks like Spark and Hadoop. Key features of Flink highlighted are its streaming capabilities, high speed, fault tolerance, and flexible windowing.
8. Differences between Batch and Real-Time Processing
Batch Processing Real-Time Processing
Data Static Files Event Streams
Speed
Processed Periodically in minute,
hour, day etc.
Processed immediately
nanoseconds
Storage Past data on disk storage In Memory Storage
Example Bill Generation ATM Transaction Alert
11. FLINK program
Data source
Source is responsible for reading data from data
sources such as HDFS, KAFKA …
Transformation
Responsible for data transformation operations
Reduce(), sum(), max(), min() …
Data Sink
Responsible for final data outputs ()
14. FLINK time & window
EVENT TIME CLASSIFICATION TYPES
Event Time:
Time when an
event occurs
Ingestion time:
Time when an
event arrives at the
stream processing
system
Processing Time:
Time when an
event is processed
by the stream.
16. FLINK time & window
DEFINITION
Window is a
method for splitting
infinite data sets
into finites blocks
for processing.
Windows split the
stream into buckets
of infinite size,
which we can apply
computation.
TYPES
20. Flink vs Spark vs Hadoop
Apache Hadoop Apache Spark Apache Flink
Data Processing Engine Batch Batch Stream
Processing Speed
Slower than Spark and
Flink
100x Faster than
Hadoop
Faster than spark
Throughput Medium High High
Optimization Manual Manual Automatic
Streaming Support NA Spark Streaming Flink Streaming
Graph Support NA GraphX Gelly
Machine Learning
Support
NA SparkML FlinkML
SQL Support Hive, Impala SparkSQL Table API and SQL
Data Transfer Batch Batch Pipelined and Batch
21. Features of Apache Flink
1) Has a streaming processor, which can run both batch and stream programs.
2) Can process data at lightning-fast speed.
3) APIs available in Java, Scala and Python.
4) Processes data in low latency (nanoseconds) and high throughput.
5) Its fault tolerant. If a node, application or a hardware fails, it does not affect the
cluster.
6) In-memory management can be customized for better computation.
7) Windowing is very flexible in Apache Flink.
