Now that we know how to create simple streaming analytics pipelines, it is time to learn something more interesting. In this session I will show you how to add Complex Event Processing to your Apache Flink (or Kinesis Data Analytics) application using JAVA. For those of you that prefer SQL, I will show you how to run streaming analytics using only SQL.

1. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Part 3 of 3: Deep Dive
Getting started with streaming analytics
Javier Ramirez
AWS Developer Advocate
@supercoco9

2. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Agenda
Working with Avro and schemas
Time windows, session windows, and accumulators
Complex Event Processing
Streaming analytics with SQL

3. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Ingestion/in-stream storage: Apache Kafka
A distributed streaming platform
Concepts:
Producers
Topics
Brokers
Consumers

4. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Apache Avro
Apache Avro™ is a data serialization system.
Avro provides:
• Rich data structures.
• A compact, fast, binary data format.
• A container file, to store persistent data.
• Remote procedure call (RPC).
• Simple integration with dynamic languages. Code
generation is not required

5. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Confluent Schema Registry
https://docs.confluent.io/current/schema-registry/index.html

6. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Stream Processing: Apache Flink
Stateful computation over Data Streams
Concepts:
Job Manager/Workers
Source
DataStream
Transforms/Operators
TableAPI/SQL
Sinks

7. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Different types of streaming windows
13:00 14:008:00 9:00 10:00 11:00 12:00
Processing
Time
Event Time
Processing
Time
11:0010:00 15:0014:0013:0012:00
11:0010:00 15:0014:0013:0012:00
Input
Output
Event Time
Processing
Time 11:0
0
10:0
0
15:0
0
14:0
0
13:0
0
12:0
0
11:0
0
10:0
0
15:0
0
14:0
0
13:0
0
12:0
0
Input
Output

8. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Apache Flink Windows
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html

9. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
FlinkCEP
Complex Event Processing for Flink
FlinkCEP is the Complex Event Processing (CEP) library
implemented on top of Flink.
It allows you to detect event patterns in an endless stream
of events, giving you the opportunity to get hold of what’s
important in your data.

10. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Kinesis Data Analytics for SQL
• Sub-second end to end processing latencies
• SQL steps can be chained together in serial or parallel
steps
• Build applications with one or hundreds of queries
• Pre-built functions include everything from sum and
count distinct to machine learning algorithms
• Aggregations run continuously using window operators

11. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Automatic schema discovery which
works for CSV and JSON data
Supports multiple event types,
arbitrary object nesting, single level
of array nesting
Connect to Streaming Data Sources
Easily connect to Kinesis Data streams and
Kinesis Data Firehose delivery streams
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Firehose

12. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Built-in AWS Lambda integration provides flexible pre-
processing ahead of SQL code for:
• Normalizing different event types
• Converting other data formats (AVRO, Protobuf, ZIP) to JSON and CSV
• Custom enrichment from database tables or API calls
Pre-process Data Streams Using AWS Lambda
AWS Lambda function
raw data
Amazon Kinesis Data Analytics application
transformed
data
SQL
code
source destination

13. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Fast, iterative development with SQL templates in console to get started
Interactive SQL Editor

14. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Window Types
• Sliding, tumbling, and custom windows
• Tumbling windows are fixed size and grouped keys do not overlap

15. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
• Add a SQL table to your streaming application from Amazon S3
• Periodically update the table by calling the update application API
Enrich your Data Stream using Amazon S3 Data
In-application stream
Amazon Kinesis Data Analytics application
SQL code joining
table and stream
streaming source destination
Amazon
S3
In-application table

16. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
Getting Started
https://ci.apache.org/projects/flink/flink-docs-stable/
Apache Flink official docs
https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html
Getting started with Apache Kafka/Amazon MSK
https://aws.amazon.com/kinesis/
Amazon Kinesis Services for streaming data
https://aws.amazon.com/elasticsearch-service/
Amazon ElasticSearch Service
https://kafka.apache.org/documentation/
Apache Kafka official docs

17. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
ThanksJavier Ramirez
AWS Developer Advocate
@supercoco9