使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化－曾書庭

1. 使⽤用 Elasticsearch 及 Kibana 進⾏行
巨量資料搜尋及視覺化
Suiting @ DSC 2015

2. Who
Am
I
曾書庭
(@suitingtseng)
Data
Engineer
Gogolook

4. Jeff, CEO

9. As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？

10. As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
我算⼀一下...

11. As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
資料好了嗎?
我算⼀一下...

12. As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
資料好了嗎?
還在跑...
我算⼀一下...

13. As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？

14. As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
可以分國家嗎？

15. As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
可以分版本嗎？
可以分國家嗎？

16. As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
可以分版本嗎？
可以看⼀一年嗎?
可以分國家嗎？

17. As
a
data
engineer
in
Gogolook…
書庭，請問我們的
DAU
是多少？
可以分版本嗎？
可以看⼀一年嗎?
可以嗎? 可以嗎? 可以嗎?
可以分國家嗎？

18. ⼀一句話激怒⼯工程師⼤大賽
• 可以分XX嗎

19. ⼀一句話激怒⼯工程師⼤大賽
• 可以分XX嗎
• 可以畫成圖嗎
• 可以給我
raw
data
嗎

20. ⼀一句話激怒⼯工程師⼤大賽
• 可以分XX嗎
• 可以畫成圖嗎
• 可以給我
raw
data
嗎
• 有沒有辦法知道
user
住哪裡
• 可以知道哪些
user
⽐比較有錢嗎
• 下⾬雨天
user
會睡⽐比較晚嗎

21. Table
of
Contents
• Problems
• Solution
Requirements
• Elasticsearch
&
Kibana
• In
Gogolook
• Future

22. Problems
• Request-­‐response
model
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83

23. Problems
• Request-­‐response
model
• Long
cycle
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83

24. Problems
• Request-­‐response
model
• Long
cycle
• EAAB
(engineer
as
a
bottleneck)
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83

25. Problems
• Request-­‐response
model
• Long
cycle
• EAAB
(engineer
as
a
bottleneck)
• HDC
(Hippo-­‐driven
company)
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83

26. Problems
• Request-­‐response
model
• Long
cycle
• EAAB
(engineer
as
a
bottleneck)
• HDC
(Hippo-­‐driven
company)
• Lack
of
speed
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83

27. Problems
• Request-­‐response
model
• Long
innovation
cycle
• EAAB
(engineer
as
a
bottleneck)
• HDC
(Hippo-­‐driven
company)
• Lack
of
speed
• =>
We
are
not
alone
(500px)
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83

28. Table
of
Contents
• Problems
• Solution
Requirements
• Elasticsearch
&
Kibana
• In
Gogolook
• Future

29. Possible
solutions
• Approach
1:
SQL
monkey*
zoo
http://www.slideshare.net/GloriaLau1/keynote-at-spark-summit/5

30. Possible
solutions
• Approach
1:
SQL
monkey
zoo
• Approach
2:
Provide
limited
yet
easy
visualization
http://www.slideshare.net/GloriaLau1/keynote-at-spark-summit/5

31. Requirement
• Easy:
Even
CEO
can
use
it
• Fast:
Must
be
interactive
• Export:
Provide
the
csv
file
• Big:
Must
be
scalable
• 80-­‐20:
Solves
80%
problems

32. Table
of
Contents
• Problems
• Solution
Requirements
• Elasticsearch
&
Kibana
• In
Gogolook
• Future

33. Elasticsearch
• Lucene-­‐based
search
engine
• Document
storage
(JSON)
• Distributed,
scalable
• Serve
search
request
in
ms
• Build
index
for
every
field

34. Kibana
• ES
visualization
tool
• No
code
required

35. ES
+
Kibana
• Fast:
index
every
field
• Fast:
columnar
storage*
• Big:
born
distributed/scalable
• Easy:
GUI
without
code
• Export:
csv

36. Kibana
• Discover
• Visualization
• Dashboard

37. Discover
• Raw
data
• Check
data
• Find
dirty
data
• Try
query

38. Discover

39. Discover

40. Visualization
• 8
visualization
types
• 9
group
methods
• 9
aggregation
values

41. Visualization

42. Visualization
types

43. Grouping
methods
• Date
histogram
• Histogram
• Range
of
a
value
• Top
N
• Filter

44. Aggregation
values
• Count
• Avg,
Sum,
Min,
Max,
S.D.
• Unique
count*
(Hyperloglog)
• Percentile*
(T-­‐digest)

45. Visualization
• Same
concept,
different
graph
• FILTER
• GROUP
• AGGREGATE

46. DAU
書庭，請問我們的
DAU
是多少？

47. DAU
by
region
可以分國家嗎？

48. DAU
by
version
可以分版本嗎？

49. server
request
log

50. Request_total
per
minute
GROUP
BY
DATE
HISTOGRAM(minute)
COUNT(*)

51. Request_total
by
path
GROUP
BY
TOP(path,
5),
DATE
HISTOGRAM(minute)
COUNT(*)

52. Dashboard
• Collection
of
visualizations

53. Community
tag
in
MongoDB

54. Dashboard

55. Dashboard
-­‐
1st
peak

56. Dashboard
-­‐
2nd
peak

57. Table
of
Contents
• Problems
• Solution
Requirements
• Elasticsearch
&
Kibana
• In
Gogolook
• Future

58. In
Gogolook
(Aug.
2015)
• 200M+
data
point
daily
• 150GB+
data
size
daily
• 24
dashboards,
160
visualizations
• Service
status
e.g.
requests_total
• Application
data
e.g.
tag_total
• Log
data
e.g.
button_ctr

59. In
Gogolook
(currently)
• Log
user
behavior
on
features
• ⾃自⼰己的
log
⾃自⼰己記
(Planner/PM)
• ⾃自⼰己的
board
⾃自⼰己拉
(every
one)
• Monitor
performance
from
day
1

60. In
Gogolook

61. In
Gogolook
• Tracking
Kibana
usage
by
Google
Analytics

62. Table
of
Contents
• Problems
• Solution
Requirements
• Elasticsearch
&
Kibana
• In
Gogolook
• Future

63. In
Gogolook
(future)
• Log
all
user-­‐event,
not
feature-­‐based

64. In
Gogolook
(future)
• Log
all
user-­‐event,
not
feature-­‐based
• {
"userid":
"suiting",
"@timestamp":
"2015-­‐08-­‐23T11:48:00",
"page":
"login",
"button":
"register",
"period":
3500
}

65. In
Gogolook
(future)
• Answer
questions
A
B
40%
60%

66. In
Gogolook
(future)
• Answer
questions
A
B
40%, 7000ms
60%, 1500ms

67. Limit
• No
SQL
JOIN
• Subquery

68. How
about
20%
• Powerful
engine/tool
required
• Compute
engines:
• Google
BigQuery
• AWS
Redshift
• Visualization
tools:
• Tableau
• Periscope

69. Thank you

70. Questions ?