AWS Summit Milano 2019 - Architetture per l'analisi di flussi di dati in tempo reale - Frank Munz, Technical Evangelist, AWS | Cliente: Cerved

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Antonello Mantuano
Head of Software Engineering
Cerved
Dr Frank Munz
Senior Technical Evangelist
Amazon Web Services
Architecting for Real-Time Insights
with Streaming Data

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Introductory - 200
“These sessions provide an overview of AWS services and
features, and they assume that attendees are new to the
topic. These sessions highlight basic use cases, features,
functions, and benefits."

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
- Streaming Architectures
- Amazon Kinesis
- Serverless Stream Processing
- Amazon Managed Streaming for Kafka (MSK)
- Customer Success Story:
Antonello Mantuano from Cerved
Agenda

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Streaming Data
Web Clickstream Application LogsIoT Sensors
[Wed Oct 11 14:32:52
2018] [error] [client
127.0.0.1] client
denied by server
configuration:
/export/home/live/ap/ht
docs/test
Continuously generated, small size events,
low latency requirements

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Transform and Process Continuously
Streaming
Ingest video
& data as it’s
generated
Process data
on the fly
Real-time
analytics/ML,
alerts, actions

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
From Batch to Streaming Analytics
https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

7. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Kinesis
Real-time data streaming and analytics
Easily collect, process, and analyze streams in real time
Kinesis
Video Streams
Kinesis
Data Streams
Kinesis
Data Firehose
Kinesis
Data Analytics
Capture, process,
and store video
streams for
analytics
Load data streams
into AWS data
stores
Analyze data streams
with SQL or Java
Build custom
applications that
analyze data
streams
NEW!

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Kinesis Data Streams Overview

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Data Ingestion from a Variety of Sources
Kinesis Data
Streams
Transactions
ERP
Web logs/
cookies
Connected
devices
AWS SDKs
• Publish directly from application code via APIs
• AWS Mobile SDK
• Managed AWS sources: CloudWatch Logs, AWS IoT, Kinesis Data
Analytics and more
• RDS Aurora via Lambda
Kinesis Agent
• Monitors log files and forwards lines as messages to Kinesis Data Streams
3rd party and open source
• Log4j appender
• Apache Kafka
• Flume, fluentd, and more …
Kinesis Producer Library (KPL)
• Background process aggregates and batches messages

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Kinesis Data Streams: Standard consumers

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Kinesis Data Streams: Standard consumers
Shard 1
Shard 2
Shard 3
Shard n
Consumer
Application A
GetRecords()
Data
GetRecords():
5 transactions or
2MB per second, per shard
Data
Producer
up to 1 MB or
1000 records
per second, per shard With one consumer application,
records can be retrieved every 200 ms.
Stream

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Kinesis Data Streams: Enhanced fan-out consumers
Every consumer gets dedicated 2MB per second, per
shard. Latency is typically less than 70 msec.
Shard 1
Data
Producer
Consumer
Application B
Consumer
Application A
RegisterStreamConsumer()
EFO Pipe
SubscribeToShard()
Data: up to 2MB per second
EFO Pipe
HTTP/2: Consumers do not poll. Messages are pushed to the consumer as they arrive.
RegisterStreamConsumer()
SubscribeToShard()
Data: up to 2MB per second
Stream

15. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The Serverless Operational Model
No provisioning,
no management
Pay for value
Automatic
scaling
Highly available
and secure

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Processing a Data Stream with Lambda
data
producer
Kinesis Data
Streams
Amazon
SNS
Continuously stream data
Lambda
service
Lambda
functionA
Lambda
function B
Continuously polls for new data,
1 poll per second
Automatically invokes your
function(s) when data found
Lambda polls each shard once per second
Lambda’s maximum execution time is 15 minutes

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Kinesis Streaming Data Analytics:
SQL or Apache Flink (Java)

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Kinesis Streaming Data Analytics / SQL

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Kinesis Streaming Data Analytics / Apache Flink
Framework and engine for stateful processing of data streams.
Simple
programming
High performance
Stateful
Processing
Strong data
integrity
Easy to use and
flexible APIs make
building apps fast
In-memory
computing provides
low latency & high
throughput
Durable
application state
saves
Exactly-once
processing and
consistent state

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Kinesis Data Firehose:
Ingest Transform Load (ITL)

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Kinesis Data Firehose—How it Works
Ingest Transform Deliver
Amazon S3
Amazon Redshift
Amazon Elasticsearch Service
AWS IoT
Amazon Kinesis Agent
Amazon Kinesis Streams
Amazon CloudWatch Logs
Amazon CloudWatch Events
Apache Kafka

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Kinesis Data Firehose: Record format Conversion
Kinesis Data
Firehose
Amazon S3
Glue Data
Catalog
Data
Producer
schema
convert to
columnar format
JSON data
/failed

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Kinesis Data – Streams vs. Firehose
Scalable and durable real-time data streaming service with
provisioned throughput and sub-second latency that can
continuously capture gigabytes of data per second from hundreds
of thousands of sources.
Kinesis Data
Streams
Kinesis Data
Firehose
Capture, transform, convert and load data streams into AWS data
stores for near real-time analytics. Data latency 60 seconds.

25. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Demo Architecture

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Live Demo!
Use your phone & connect to:
XXX
2. modo !
3. modo "
1. preparazione

28. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Comparing Amazon Kinesis Data Streams to MSK
Amazon Kinesis Data Streams Amazon MSK
Newest dataOldest data
50 1 2 3 4
0 1 2 3
0 1 2 3 4
Shard 2
Shard 1
Shard 3
Writes
from
Producers
Stream with 3 shards
Newest dataOldest data
50 1 2 3 4
0 1 2 3
0 1 2 3 4
Partition 2
Partition 1
Partition 3
Writes
from
Producers
Topic with 3 partitions

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
TopicA Partition1
TopicA Partition3
Partition
Replica
Replica
Producer
Zoo-
keeper
Zoo-
keeper
Zoo-
keeper
State
& Config
TopicA Partition2 Replica
Cluster
Apache Kafka: Partitioned, Replicated Commit Log

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Challenges operating Apache Kafka
Difficult to setup,
configure and operate
Hard to achieve high
availability
Tricky to scale
AWS integrations
= development
No console, no visible
metrics

32. Getting started with Amazon MSK Preview is easy

33. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

34. Fast Data from Legacy to Cloud
The battle to overcome the gravity
Marzo ’19
Antonello Mantuano
Head of Software Engineering
antonellomantuano
@manant74

35. Cerved – the Data Driven Company
42
Credit
Information
Credit
Management
Marketing
Solutions
LEAD GENERATION
CREDIT COLLECTION
DATA PROVIDING
& MARKETING
ANALYSIS
CREDIT INFORMATION
CREDIT SCORING
BAD CREDITS EVALUATION
We are deeply passionate about data. Our data enables
various financial services, from credit risk analysis to marketing
solutions to managing non-performing loans and bad debt.
•1M companies sitesWeb Data
•4M info from open data setOpen Data
•70M payment
•60M scoring
Cerved Data
•70M Real EstateProperty
•20M companies
•16M shareholders
Company Data
2.600
Persons
34.000
Customers
40M€
In Data &
Technologies
30 M
Decisions
1.400 TB
Of Data

36. Why Cerved in Cloud?
43
TIME-TO-MARKET
Rapid implementation for
basic services.
Benefits of
Cloud for Cerved
PRIVACY & SEGREGATION
Manage customers data in secure
mode
AVAILABILITY
Services available 7x24
SCALABILITY
Infrastructure quickly
adaptable to the load

37. The Data Gravity
44
As data accumulates, it begins to have gravity. This Data Gravity pulls
services and applications closer to the data.
- Dave McCrory, 2010
DATA
Services Apps
Latency
Throughpu
t
This attraction (gravitational force) is caused by the need for
services and applications to have higher bandwidth and/or lower
latency access to the data.

38. 45
Data Ecosystem in Cerved
Sourcing
Liv.2
Sourcing
Liv. 1
REPOS
SYNTH
Mondo Dati Lince
Dati
clienti
NCA
ERG
EBS
HUB
DWH
MBD
Teradata
Tabula
Mongo4
DW
DB4You
XPCH 2
MATCHNEMO
Quaes
tio
LUDO
Tabula
(su
AWS)
Aracne
G4U
MBD1
R3
Pragma
Splunk CDR
Mambo
CAS
Dedalo
ELK
CSS
CR-RIBA
(Payline)

39. How to overcome Gravity?
How to lift to the cloud with Data Gravity?

40. Cerved Data in Cloud Architecture
Cerved
DBs
CDC
DBs Operational
Online Data
OLTP Processes
Batch
Hadoop DataLakeNoSql
Tabula
Cloud DB
DynamoDB RDS S3
Streaming is the new ETL
CDC
Producer
Raw Events
Aggregator
Basic Events
Aggregator
HL Events
Aggregator
NoSql
Ingestion
Synk Connector
For Cloud
Hadoop
Ingestion
Stream Processing
Streaming is the Anti-Gravity

41. Cerved API: a Data In Cloud use case
Cerved
DBs
CDC
DBs Operational
Online Data
OLTP Processes
Batch
Hadoop DataLakeNoSql
Tabula
Cloud DB
DynamoDB RDS S3
CDC
Producer
Raw Events
Aggregator
Basic Events
Aggregator
HL Events
Aggregator
NoSql
Ingestion
Sync
Connector to
CloudHadoop
Ingestion
Stream Processing
Back End
AWS Lambda Spring Boot API Gateway
Front End
ReactJs Redux Swagger

42. The Results of API in Cloud
49
SLA
API available 7x24x365
99,998% in January 2019
PERFORMANCE
High scalability with quickly
adaptability to load
COSTS
With AWS Lambda, DynamoDB,
S3, ecc… the cost of infrastructure
grows with the load
DATA SYNC
Data are continuously updated in
near real time mode

43. Future use case of Data in Cloud
Cerved
DBs
CDC
DBs Operational
Online Data
OLTP Processes
Batch
Hadoop DataLakeNoSql
Tabula
Cloud DB
DynamoDB RDS S3
CDC
Producer
Raw Events
Aggregator
Basic Events
Aggregator
HL Events
Aggregator
NoSql
Ingestion
Sync
Connector
To CloudHadoop
Ingestion
Stream Processing
Back End
AWS Lambda
API Gateway
EMR
SageMaker
AWS Kinesis or
Managed
Streaming for
Kafka
Data
Scientist in
Cloud
Real Time
Apps
DR &
Backup
Use Cases

44. THANK YOU
Moving Fast Data in cloud creates a new gravity for
new and innovative apps and services
Antonello Mantuano
Head of Software Engineering
antonellomantuano
@manant74

45. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

46. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
antonellomantuano
@manant74
frankmunz
@frankmunz