This slide deck discusses distributed data processing using Java 8 Streams and Hazelcast Jet. It provides an overview of Java Streams and their limitations for large datasets. It then introduces Hazelcast IMDG for distributed caching and collections. Hazelcast Jet is presented as a distributed data processing framework built on Hazelcast IMDG that can be used to distribute and parallelize stream processing. Examples of distributed streams and the directed acyclic graph (DAG) model used by Jet are shown.

1. This slide blank on purpose

3. > whoami
• Solutions Architect @Hazelcast
• Hang out with awesome people
• @gamussa in internetz
Please, follow me in Twitter
I’m very interesting ©

4. Agenda
• Refreshing knowledge on Java 8 Streams
• Distribute and Conquer
• Distributed Data
• Distributed Streams
• How we did all this

5. Java 8 Streams

6. Java 8 Streams…
• An abstraction represents a sequence of elements
• Is not a data structure
• Convey elements from a source through a pipeline
of operations
• Operation doesn’t modify a source

7. Why I should care about
Stream API?
• You’re Java developer

8. What does regular Java developer think about Scala?
advanced

9. Why I should care about
Stream API?
• You’re Java developer
• Many Java developers know Java
• It’s all about data processing

10. java.util.stream operations
• map(), flatMap(), filter()
• reduce(), collect()
• sorted()

14. Problem
• One does not simply put all Big Data in one
machine

15. Problem
• Data doesn’t fit just one machine

16. Problem
• One does not simply put all Big Data in one
machine
• Data is too important to have it only one machine

18. CACHES

19. Replication on Sharding?
http://book.mixu.net/distsys/single-page.html

20. Solution
• Use Distributed Map aka IMap

21. What’s Hazelcast IMDG?
• In-memory Data Grid
• Apache v2 Licensed
• Distributed
• Caches (IMap, JCache)
• Java Collections (IList, ISet, IQueue)
• Messaging (Topic, RingBuffer)
• Computation (ExecutorService, M-R)

25. Green
Primary
Green
Backup
Green
Shard

27. Problem
• Lambda serialization
27

28. 28

29. Solution
• serializable version of the interfaces
• Introducing DistributedStream
29

30. 30

32. 32
Jet Streams

34. What’s Hazelcast Jet?
• General purpose distributed data processing
framework
• Based on Direct Acyclic Graph to model data flow
• Built on top of Hazelcast IMDG
• Comparable to Apache Spark or Apache Flink
34

36. DAG
36

37. Job Execution
37

39. Future (It’s bright!)
• Memory module for processing big data
• Higher level streaming and batching APIs
• Reactive Streams
• Distributed Classloading
• Integrations (HDFS/Yarn/Mesos)

40. Your fuel, our Jet Engine
• Public release – Feb 7th.
• Developer Preview today - yay!
• http://hazelcast.org/jet-signup
• Send me a note viktor@hazelcast.com
• Follow @hazelcast and @gamussa (duh!!)
• Your questions #hazelcast #hazelcastjet

41. Conclusion
• Java Stream API provides very white range of data
processing tools
• War And Piece – is a Big (a lot of data) Book!
• Now we’re pretty sure that Andrew and Pierre are
the main characters