1. Spark-Kafka message
streaming
-vritika godara

2. ❖ Scope
❖ Zookeeper server start
❖ Kafka server start
❖ Create kafka topic and send message
❖ Consume kafka message
❖ List kafka topics
❖ Kafka producer in java
❖ Kafka consumer in java
❖ Spark CLI
❖ Spark word count java app
❖ Kafka-Spark integration
❖ Dataframes
❖ Create table schema in spark
❖ SQL queries
❖ Front-end design
Contents

3. Scope
❖ Create an application to capture user’s location and send
it to database for real-time data analysis
❖ Demonstrate the use of kafka-spark data streaming for
creation of meaningful analytical reports

4. Zookeeper server start

5. Zookeeper server start for kafka

6. Kafka Server Start

7. Create Kafka Topic and Send Message

8. Consume Kafka Message

9. List Kafka Topics

10. Kafka Producer
❖ Message is
sent from
kafka in java
app and
consumed
through
command
line

11. Kafka Consumer
❖ Message produced and
consumed by kafka
using java app

12. Spark CLI

13. ❖ Following slides contain code
snippets of messages produced
through kafka and consumed
through spark-kafka streaming

14. User Class

15. Kafka Producer
❖ Create KafkaProducer object and initialise it by setting its
required properties.

16. Kafka Producer
❖ Create user object, specify the topic and send the
message using KafkaProducer object

17. Kafka-Spark Integration
❖ Create JavaStreamingContext with interval of 5 seconds
❖ Set kafka topic from which messages are to be
consumed
❖ Create kafka connector for spark

18. PostgreSQL Driver
❖ Register PostgreSQL driver for database integration

19. Dataframe Read
❖ Create dataframe and show tbl_location data
❖ Register tbl_location as data frame table

20. Spark Table Schema
❖ Create table schema to create a new table in postgresql
using spark

21. Consume kafka data
❖ Consume the data sent from kafka to create data frame
and register it as table

22. SQL Queries
❖ Using sqlContext, query the dataframes
❖ Create table “user-location” in postgresql and save
meaningful consumed data for further processing

23. SQL Queries
❖ Using sqlContext, query the dataframes
❖ Create table “user-institute” in postgresql and save
meaningful consumed data for further processing

24. SQL Queries
❖ Using sqlContext, query the dataframes
❖ Get list of users logged in from India along with state
and city
❖ Create table “user-country” in postgresql and save
meaningful consumed data for further processing

25. PostgreSQL Table
❖ Table created by
spark in postgresql

26. Front End

27. THANK YOU