This slides are used to present the following Twitter pipeline using the ELK stack (Elasticsearch, Logstash, Kibana): https://github.com/melvynator/ELK_twitter It shows how to integrate Machine Learning into your Twitter pipeline.

1. Machine Learning in a Twitter
ETL using ELK
(ELASTICSEARCH, LOGSTASH, KIBANA)
MELVYN PEIGNON
melvynpeignon@gmail.com

2. What is an ETL?
• ETL: Extract Load Transform
Source Transformation
Data
warehouse/
Data store
Raw data
Processed
data
Extract Transform Load

3. Our use case: An ETL for Twitter
https://github.com/melvynator/ELK_twitter
Goals:
•Simplify a recurrent task for several members of the lab
•Normalize the data collection (Have a “universal” format)
•Have tweets analyzed the way we want (Emoji, punctuation)
•Include some machine learning model in our ETL

4. Our tools: ELK
•E : Elasticsearch
Elasticsearch is a distributed, RESTful search and analytics engine
•L : Logstash
Logstash is an open source, server-side data processing pipeline that
simultaneously ingests data from multiple sources transforms it, and then
sends it to your favorite “stash”.
•K: Kibana
Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack
Source: Elastic website

5. Our model
Twitter Logstash Elasticsearch Kibana
Sentiment
API
Extract Transform Load

6. Logstash
Logstash allows you to ingest data from multiple sources, transform
the data and then store your processed data.
Notions you need to know:
•Event
•Input
•Filter
•Output

7. Petrol Pipeline
Storage
Truck
Refinery
Pipeline
Petrol Chemical

8. Logstash Pipeline
Elasticsearch
Filters
Twitter
API
Twitter
Input
Elasticsearch
outputData
Logstash

9. Logstash: Event
An event can be described as one raw data traveling across the
pipeline:
•A Log
•A Line
•A JSON
•…

10. Logstash: Input
The input plugins consume data from a source.
•File
•Elasticsearch
•Twitter API
•Github API
•…

11. Logstash: Filter
A filter plugin performs intermediary processing on an event. Filters are
often applied conditionally depending on the characteristics of the
event.
•Clone
•Mutate
•Ruby
•…

12. Logstash: Output
An output plugin sends event data to a particular destination.
•Elasticsearch
•MongoDB
•File
•…

13. Logstash: Flow
Input Filters Output
Events
Processed
events

14. Example: tutorial_input.txt
{
"last_name":"John",
"first_name":"Doe",
"age": 25,
"degree":"Master",
"school":"Stanford",
"comment": "A guy from somewhere"
}
{
"name":"John Doe",
"age": 25,
"comment": "A guy from somewhere",
"created_at" : 2017-09-04T08:39:54.847Z
}
{
"school": "Stanford",
"degree": "Master",
"created_at" : 2017-09-04T08:39:54.877Z
}
What we have: What we want:

15. Example: Building the configuration file
input {
file {
path => "/usr/local/Cellar/logstash/5.5.2/tutorial_input.txt"
start_position => beginning
sincedb_path => "/dev/null"
codec => "json"
} # the input will be a file containing JSON (takes only absolute path)
}
filter {} # Empty for the moment (We will only fill this)
output { stdout {codec => rubydebug}} # Print the result to the console

16. Example: Output
{
"path" => "/usr/local/Cellar/logstash/5.5.2/tutorial_input.txt",
"@timestamp" => 2017-09-04T07:03:26.388Z,
"degree" => "Master",
"@version" => "1",
"host" => "Melvyns-MacBook-Pro.local",
"last_name" => "Doe",
"school" => "Stanford",
"comment" => "A guy from somewhere",
"first_name" => "John",
"age" => 25
}

17. Example: Building “name” key
ruby { # Allows you to insert ruby code
code =>
‘ event.set("name", event.get("[first_name]") + " " +
event.get("[last_name]")) ’
} #event.get() allows retrieving a value given a key, event.set() to create
one
mutate {
remove_field => ["first_name", "last_name", "host", "path"]
} # mutate allows basic transformation on events; here we are removing 4 keys
The order used to modify the
document matters! Here
“Ruby” will be applied first
then “Mutate”.
1. Merging “first_name” and “last_name” into ”name”
2. Removing “first_name” and “last_name”

18. Example: Output
{
"@timestamp" => 2017-09-04T08:33:26.599Z,
"school" => "Stanford",
"degree" => "Master",
"@version" => "1",
"name" => "John Doe",
"comment" => "A guy from somewhere",
"age" => 25
}

19. Example: Building two events out of one
clone {
clones => ["education"]
} # we are duplicating our event and add in the replica a key "type" with "education" as a value.
if ([type] == "education") {
mutate {
remove_field => ["name", "age", "comment", "type"]
}
}
else {
mutate {
remove_field => ["school", "degree"]
}
}

20. Example: Output
{
"@timestamp" => 2017-09-04T08:39:54.847Z,
"name" => "John Doe",
"comment" => "A guy from somewhere",
"@version" => "1",
"age" => 25
}
{
"@timestamp" => 2017-09-04T08:39:54.897Z,
"school" => "Stanford",
"@version" => "1",
"degree" => "Master"
}

21. Example: Changing the name of a field
mutate {
remove_field => ["@version"] # @version, @timestamp are created
for every events
rename => { "@timestamp" => "created_at" } # change the name of a
field
}
The order inside mutate may
vary. You cannot know if
“remove_field” will be applied
before or after “rename”

22. Example: Output
{
"created_at" => 2017-09-04T08:39:54.847Z,
"name" => "John Doe",
"comment" => "A guy from somewhere",
"age" => 25
}
{
"created_at" => 2017-09-04T08:39:54.897Z,
"school" => "Stanford",
"degree" => "Master"
}

23. Logstash: Twitter input
input {
twitter {
consumer_key => "<YOUR-KEY>"
consumer_secret => "<YOUR-KEY>"
oauth_token => "<YOUR-KEY>"
oauth_token_secret => "<YOUR-KEY>"
keywords => [ "random", "word"]
full_tweet => true
type => "tweet"
}
}
Only one input:

24. Logstash: Our filters
There are many filters, but the overall goals of the filters are to:
•Remove depreciated fields
•Divide the tweet into two or three events (users and tweet)
•Remove the nesting of the JSON
•Remove the fields not used

25. Our model
Twitter Logstash Elasticsearch Kibana
Sentiment
API

26. Building the API
•Python
•Flask
•Building your model based on labeled data
•Design an endpoint that will receive the data you want to predict

27. Logstash REST filter
•https://github.com/lucashenning/logstash-filter-rest
•Allows RESTful resources inside Logstash
•Will call your Machine learning API
•Will add information to your events

28. Logstash REST filter: Example
rest {
request => {
url => http://localhost:5000/predict
method = "post"
params => {
"submit" => "%{tweet_content}"
}
headers => {
"Content-Type" => "application/json"
}
}
target => 'rest_result'
}

29. Time to do it yourself
•https://github.com/melvynator/Logstash_tutorial
•https://docs.google.com/forms/d/e/1FAIpQLSdDdFxmT5ZCXInaSohJB
Bo6fKKmCg3KLegeOOrxl1l4_sc-7g/viewform

30. Thank you!
Questions?