data search analysis visualization

1. ELK
Data Search / Analysis / Visualization
Caleb Wang

2. What is ELK ?
ElasticSearch + Logstash + Kibana
{ELK}

3. ElasticSearch
{ELK}
Elasticsearch is a highly scalable open-source full-text search
and analytics engine. It allows you to store, search, and
analyze big volumes of data quickly and in near real time. It
is generally used as the underlying engine/technology that
powers applications that have complex search features and
requirements.
https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html

4. Logstash
{ELK}
Logstash is an open source data collection engine with real-
time pipelining capabilities. Logstash can dynamically unify
data from disparate sources and normalize the data into
destinations of your choice. Cleanse and democratize all your
data for diverse advanced downstream analytics and
visualization use cases.
https://www.elastic.co/guide/en/lo
gstash/current/introduction.html

5. Kibana
{ELK}
Kibana is an open source analytics and visualization platform
designed to work with Elasticsearch. You use Kibana to
search, view, and interact with data stored in Elasticsearch
indices. You can easily perform advanced data analysis and
visualize your data in a variety of charts, tables, and maps.
https://www.elastic.co/guide/
en/kibana/current/introductio
n.html

6. Real Case
{ELK}

7. Concept
{ELK}
Shipper
(logstash)
Indexer
(logstash)
Broker
(redis)
Engine
(ElasticSearch)
GUI
(Kibana)

8. Architecture
{ELK}
web1 web2 ap1 ap2
Ap4Deploy
x4

9. Plan
{ELK}
Machine web1 web2 ap1 ap2
Log File Sys.log Sys.log
Sys.log
Wcpapi.log
Sys.log
Wcpapi.log
Broker Key web1 web2 ap1 ap2
Field “type” web1-sys web2-sys
ap1-sys
ap1-wcpapi
ap2-sys
ap2-wcpapi

10. Setup
{ELK}
Step 1 (Linux)
 Build Redis & Run (C)
 Configure & Run ElasticSearch (java)
 Configure & Run Kibana (Node.js)
 Configure & Run Logstash as Indexer (java)
Step 2 (Linux or Windows)
 Configure & Run Logstash as Shipper (java)

11. Setup Redis
{ELK}
1.Unpack tar ball source file.
2.make
3.I didn’t configure anything… XDDD
4.Just run… > redis-server redis.conf &
5.Default port: 6379
http://redis.io/

12. Setup ElasticSearch
{ELK}
1.Unpack tar ball file.
2.I didn’t configure anything… XDDD
3.Just run… > elasticsearch -d -p es.pid
4.Default port: 9200
elasticsearch.yml

13. Setup Kibana
{ELK}
1.Unpack tar ball file.
2.I didn’t configure anything… XDDD
3.Just run… > sh kibana > kibana.log &
4.Default port: 5601
kibana.yml

14. Setup Indexer
{ELK}
1.Unpack tar ball file.
2.Configure
3.Run… >logstash -f idx.ap1.conf -l idx.ap1.log &
idx.ap1.conf

15. Setup Shipper
{ELK}
Regex !!!

16. Clear Data
{ELK}
>curator delete indices
--older-than 30
--time-unit days
--timestring '%Y.%m.%d'
>pip install curator

17. We’ve done
&
Start to have fun !
{ELK}

18. Kibana’s 3 Steps
{ELK}
Discover
Visualize
Dashboard

19. Kibana’s Discover
{ELK}
Search Syntax
Exact String “ActiveMQ max” , ActiveMQ
Wildcard *INQ00S4 , *INQ??S7
AND / OR “ActiveMQ max” AND ( *INQ00S4 OR *INQ00S7)
String OR ActiveMQ-max-whatever
Field
type:”ap1-wcpapi” , type:ap1-wcpapi ?
mod_date:[20020101 TO 20030101]
NOT *INQ00S4 AND NOT *INQ00S7
Escaping + - && || ! ( ) { } [ ] ^ " ~ * ? :
https://lucene.apache.org/core/2_9_4/queryparsersyntax.html

20. Kibana’s Visualize
{ELK}

21. DEMO Dashborad
{ELK}