Real Time search using Spark and Elasticsearch - Sushmitha Ch

1. “Streaming Simplified”
Real Time search using Spark
and Elasticsearch
Sushmitha Ch
sushmitha@sigmoidanalytics.com

2. • Usecase
• Spark Streaming
• Spark
• Elasticsearch
• Elasticsearch with spark streaming
• elasticsearch-hadoop connector
Contents

4. Usecase
Stream of
Organization’s Data
{ Ip : “1.1.1.1”,
Response : “HTTP/1.1
404 not found”,
Port : 80,
Timestamp :1409076765
}
Document format
Process and Enrich
the data
Store in DB
Real time
search,analyze and
generate alerts

5. Input Stream Processing
Output
Storage
Low latency
Fault tolerance
Fast computing
Consistency
Latency
Usecase(Contd..)

6. Spark Streaming
• Runs on top of Spark
• Applications will be scalable,fault tolerant
• Can achieve second scale latencies.
• Can get data from live data streams like Kafka, flume etc
Spark
Streaming
Spark
Live Data Stream
Processed Results

7. Dstreams Transformations Output

8. Spark

9. Elasticsearch
• Powerful open source, distributed real-time search
engine
Document Mapping Index
Type ShardNode

10. Percolate API

11. Elasticsearch with Spark Streaming
Spark
Streaming
Elasticsearch

12. (elasticsearch-hadoop)
connector
• Spark integrates with Elasticsearch through its dedicated
InputFormat, OutputFormat
• Operations:
• Enable “Kryo” serialization for efficient handling of
conversions
• Integrates 2 distributed systems
–Hadoop ( Batch oriented computing platform)
–Elasticsearch
Reading Writing

13. Query
elasticsearch-
hadoop
Elasticsearch
Shard 1
Elasticsearch
Shard 2
Elasticsearch
Shard 3
Elasticsearch
Shard 4
Elasticsearch
Shard 5
HadoopTask
Scalability
HadoopTask
HadoopTask
HadoopTask
HadoopTask

14. Architecture

15. Thank You