Lightbend Fast Data Platform - A Technical Overview Dean Wampler, O’Reilly author and Big Data Strategist in the office of the CTO at Lightbend discusses practical tips for architecting stream-processing applications and explains how you can tame some of the complexity in moving from data at rest to data in motion.

1. Lightbend Fast Data Platform
A Technical Overview

2. A Bit of History:
IT Services vs. Big Data

3. Big DataServices
Some Overlap: Concerns, Architecture
The Recent Past…

4. Microservices
& Fast Data
Much More Overlap
The Future?

5. Drilling Down:
Monoliths to Microservices

6. lightbend.com/reactive-microservices-architecture

7. • Tangled responsibilities, lead to
infrequent, “big-bang” deployments
• App lifetimes months to forever!
Monoliths

9. • Each does one thing, so must be
message driven & asynchronous
• Updates easier, deployments frequent
• App lifetimes: minutes! to forever
Microservices

10. • But be careful:
• Message overhead is much better than
function calls!
Microservices

11. Reactive Systems
reactivemanifesto.org

12. Open Source Core
For Development in Scala / Java
Produc8on Suite
For Opera8ons / DevOps
Lightbend Reactive Platform

13. Drilling Down:
Big Data

14. YARN
HDFS
MR job #1
MR job #2
Flume Sqoop
DBs
Slave Node
DiskDiskDiskDiskDisk
Node Mgr
Data Node
Master
Resource
Manager
Name Node
Hadoop

15. YARN
HDFS
MR job #1
MR job #2
Flume Sqoop
DBs
Slave Node
DiskDiskDiskDiskDisk
Node Mgr
Data Node
Master
Resource
Manager
Name Node
Hadoop 2013:
Embrace Spark
Spark job #1
Spark job #2

16. YARN
HDFS
MR job #1
MR job #2
Flume Sqoop
DBs
Slave Node
DiskDiskDiskDiskDisk
Node Mgr
Data Node
Master
Resource
Manager
Name Node
Go beyond batch?
Spark job #1
Spark job #2

17. The Emergence of Fast Data:
(Time Is Money)

18. Respond to change

19. IoT

20. 20
Logs
Sockets
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Streaming
Streams
SQL
Batch
…
Streaming
Flink

21. Fast Data and Microservices;
Are they Converging?

22. •Each [stream app or μservice]:
•does one responsibility
•ingests unending [data or messages]
Synergies
22
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Streaming
Streams
SQL
Batch
…
Streaming
Flink

23. •Each [stream app or μservice] must:
•operate asynchronously
•offer never-ending service
Synergies
23
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Streaming
Streams
SQL
Batch
…
Streaming
Flink

24. •These architectures are converging:
1.Similar design problems
2.Data becomes dominant problem
Thesis
24
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Streaming
Streams
SQL
Batch
…
Streaming
Flink

25. Lightbend
Fast Data Platform

26. 1. An accelerated on ramp for building
streaming data systems, data
applications, and other microservices.
Value Three Ways

27. 2. Best practices guidance for solving
specific design problems:
- Sample apps
- Documentation
- Enablement services
Value Three Ways

28. 3. Machine learning-based monitoring
and management:
-Keep your systems resilient, scalable,
and responsive with minimal user
intervention.
Value Three Ways

29. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

30. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

31. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

32. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

33. Service 1
Log &
Other Files
Internet
Services
Service 2
Service 3
Services
Services
N * M links ConsumersProducers

34. Service 1
Log &
Other Files
Internet
Services
Service 2
Service 3
Services
Services
N + M links ConsumersProducers

35. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

36. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

37. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

38. •Low latency? How low?
•High volume? How high?
Streaming Tradeoffs (1/3)
38

39. •Which kinds of data processing &
analytics are required?
•How will this processing be done?
•Individual processing of events?
•Bulk processing of records?
Streaming Tradeoffs (2/3)
39

40. •Which tools and data sources/sinks
must interoperate with your
streaming tool?
Streaming Tradeoffs (3/3)
40

41. 41

42. 42
•Low latency
•Low volume
•Complex flows
•Complex Event
Processing

43. 43
•Med. latency
•High volume
•Data flows, SQL
•En masse
processing

44. 44
•Low latency
•High volume
•Data flows,
correctness
•En masse
processing

45. 45
•Low latency
•Med. volume
•ETL, “tables”
•Data flow or
per event

46. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

47. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

48. Kafka & Spark Metrics

49. Correlated Troubleshooting

50. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

51. • Only metrics are uploaded, not sensitive
domain data.
• One less service for you to manage.
• …
Why Only Hosted Services?

52. • …
• We can rapidly evolve this service.
without impacting your environment.
• You benefit from aggregated knowledge
from all FDP clusters.
Why Only Hosted Services?

53. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

54. •Alpha: Jan 9
•Beta 1: April
•Beta 2: July
•V1.0: September
Timeline
54

55. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS, Marathon
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
Graviton
Cluster
Analysis
Machine
LearningMicroservices
ProducCon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles
For More Information:
lightbend.com/fast-data-platform

56. For More
Information
lightbend.com/
reactive-microservices-architecture

57. bit.ly/lightbend-fast-data
For More
Information

58. • FDP: lightbend.com/fast-data-platform
• Book: bit.ly/lightbend-fast-data