Enviar pesquisa
Carregar
Let's reconsider about collecting logs. Plus, visiting elastic@Moutain View!
•
1 gostou
•
1,684 visualizações
心 谷本
Seguir
第13回elasticsearch勉強会 「ログ収集の仕組みを再考しよう! あとマウンテンビューに行ってきました。」
Leia menos
Leia mais
Tecnologia
Denunciar
Compartilhar
Denunciar
Compartilhar
1 de 59
Baixar agora
Baixar para ler offline
Recomendados
Saltstack tutorial
Saltstack tutorial
HarikaReddy115
Challenge for GlassFish Builpack
Challenge for GlassFish Builpack
Kenji Kazumura
Tika tutorial
Tika tutorial
Ashoka Vanjare
JJUG CCC 2015 Fall keynote
JJUG CCC 2015 Fall keynote
心 谷本
Spring Boot + Netflix Eureka
Spring Boot + Netflix Eureka
心 谷本
Spring Bootをはじめる時にやるべき10のこと
Spring Bootをはじめる時にやるべき10のこと
心 谷本
The Role of Standards in IoT Security
The Role of Standards in IoT Security
Hannes Tschofenig
Hyderabad Mar 2019 - Autonomous Database
Hyderabad Mar 2019 - Autonomous Database
Connor McDonald
Recomendados
Saltstack tutorial
Saltstack tutorial
HarikaReddy115
Challenge for GlassFish Builpack
Challenge for GlassFish Builpack
Kenji Kazumura
Tika tutorial
Tika tutorial
Ashoka Vanjare
JJUG CCC 2015 Fall keynote
JJUG CCC 2015 Fall keynote
心 谷本
Spring Boot + Netflix Eureka
Spring Boot + Netflix Eureka
心 谷本
Spring Bootをはじめる時にやるべき10のこと
Spring Bootをはじめる時にやるべき10のこと
心 谷本
The Role of Standards in IoT Security
The Role of Standards in IoT Security
Hannes Tschofenig
Hyderabad Mar 2019 - Autonomous Database
Hyderabad Mar 2019 - Autonomous Database
Connor McDonald
NTTドコモ様 導入事例 OpenStack Summit 2015 Tokyo 講演「After One year of OpenStack Cloud...
NTTドコモ様 導入事例 OpenStack Summit 2015 Tokyo 講演「After One year of OpenStack Cloud...
VirtualTech Japan Inc.
Breaking Extreme Networks WingOS: How to own millions of devices running on A...
Breaking Extreme Networks WingOS: How to own millions of devices running on A...
Priyanka Aash
" Breaking Extreme Networks WingOS: How to own millions of devices running on...
" Breaking Extreme Networks WingOS: How to own millions of devices running on...
PROIDEA
Porting_uClinux_CELF2008_Griffin
Porting_uClinux_CELF2008_Griffin
Peter Griffin
Overpowered Kubernetes: CI/CD for K8s on Enterprise IaaS
Overpowered Kubernetes: CI/CD for K8s on Enterprise IaaS
J On The Beach
Latin America Tour 2019 - slow data and sql processing
Latin America Tour 2019 - slow data and sql processing
Connor McDonald
Perth APAC Groundbreakers tour - The Autonomous Database
Perth APAC Groundbreakers tour - The Autonomous Database
Connor McDonald
“Quantum” Performance Effects: beyond the Core
“Quantum” Performance Effects: beyond the Core
C4Media
Cooking security sans@night
Cooking security sans@night
jtimberman
PostgreSQL major version upgrade using built in Logical Replication
PostgreSQL major version upgrade using built in Logical Replication
Atsushi Torikoshi
Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28
Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28
Amazon Web Services
Arm developement
Arm developement
hirokiht
OpenWorld 2018 - 20 years of hints and tips
OpenWorld 2018 - 20 years of hints and tips
Connor McDonald
Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17
Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17
Muga Nishizawa
ILOUG 2019 - Autonomous, what does it mean for DBAs
ILOUG 2019 - Autonomous, what does it mean for DBAs
Connor McDonald
APEX Connect 2019 - SQL Tuning 101
APEX Connect 2019 - SQL Tuning 101
Connor McDonald
Architecture & Pitfalls of Logical Replication
Architecture & Pitfalls of Logical Replication
Atsushi Torikoshi
[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
CODE BLUE
APEX Connect 2019 - successful application development
APEX Connect 2019 - successful application development
Connor McDonald
Diagnose Your Microservices
Diagnose Your Microservices
Marcus Hirt
プレゼンの技術 2 実践編
プレゼンの技術 2 実践編
心 谷本
プレゼンの技術 1 考え方
プレゼンの技術 1 考え方
心 谷本
Mais conteúdo relacionado
Semelhante a Let's reconsider about collecting logs. Plus, visiting elastic@Moutain View!
NTTドコモ様 導入事例 OpenStack Summit 2015 Tokyo 講演「After One year of OpenStack Cloud...
NTTドコモ様 導入事例 OpenStack Summit 2015 Tokyo 講演「After One year of OpenStack Cloud...
VirtualTech Japan Inc.
Breaking Extreme Networks WingOS: How to own millions of devices running on A...
Breaking Extreme Networks WingOS: How to own millions of devices running on A...
Priyanka Aash
" Breaking Extreme Networks WingOS: How to own millions of devices running on...
" Breaking Extreme Networks WingOS: How to own millions of devices running on...
PROIDEA
Porting_uClinux_CELF2008_Griffin
Porting_uClinux_CELF2008_Griffin
Peter Griffin
Overpowered Kubernetes: CI/CD for K8s on Enterprise IaaS
Overpowered Kubernetes: CI/CD for K8s on Enterprise IaaS
J On The Beach
Latin America Tour 2019 - slow data and sql processing
Latin America Tour 2019 - slow data and sql processing
Connor McDonald
Perth APAC Groundbreakers tour - The Autonomous Database
Perth APAC Groundbreakers tour - The Autonomous Database
Connor McDonald
“Quantum” Performance Effects: beyond the Core
“Quantum” Performance Effects: beyond the Core
C4Media
Cooking security sans@night
Cooking security sans@night
jtimberman
PostgreSQL major version upgrade using built in Logical Replication
PostgreSQL major version upgrade using built in Logical Replication
Atsushi Torikoshi
Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28
Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28
Amazon Web Services
Arm developement
Arm developement
hirokiht
OpenWorld 2018 - 20 years of hints and tips
OpenWorld 2018 - 20 years of hints and tips
Connor McDonald
Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17
Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17
Muga Nishizawa
ILOUG 2019 - Autonomous, what does it mean for DBAs
ILOUG 2019 - Autonomous, what does it mean for DBAs
Connor McDonald
APEX Connect 2019 - SQL Tuning 101
APEX Connect 2019 - SQL Tuning 101
Connor McDonald
Architecture & Pitfalls of Logical Replication
Architecture & Pitfalls of Logical Replication
Atsushi Torikoshi
[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
CODE BLUE
APEX Connect 2019 - successful application development
APEX Connect 2019 - successful application development
Connor McDonald
Diagnose Your Microservices
Diagnose Your Microservices
Marcus Hirt
Semelhante a Let's reconsider about collecting logs. Plus, visiting elastic@Moutain View!
(20)
NTTドコモ様 導入事例 OpenStack Summit 2015 Tokyo 講演「After One year of OpenStack Cloud...
NTTドコモ様 導入事例 OpenStack Summit 2015 Tokyo 講演「After One year of OpenStack Cloud...
Breaking Extreme Networks WingOS: How to own millions of devices running on A...
Breaking Extreme Networks WingOS: How to own millions of devices running on A...
" Breaking Extreme Networks WingOS: How to own millions of devices running on...
" Breaking Extreme Networks WingOS: How to own millions of devices running on...
Porting_uClinux_CELF2008_Griffin
Porting_uClinux_CELF2008_Griffin
Overpowered Kubernetes: CI/CD for K8s on Enterprise IaaS
Overpowered Kubernetes: CI/CD for K8s on Enterprise IaaS
Latin America Tour 2019 - slow data and sql processing
Latin America Tour 2019 - slow data and sql processing
Perth APAC Groundbreakers tour - The Autonomous Database
Perth APAC Groundbreakers tour - The Autonomous Database
“Quantum” Performance Effects: beyond the Core
“Quantum” Performance Effects: beyond the Core
Cooking security sans@night
Cooking security sans@night
PostgreSQL major version upgrade using built in Logical Replication
PostgreSQL major version upgrade using built in Logical Replication
Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28
Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28
Arm developement
Arm developement
OpenWorld 2018 - 20 years of hints and tips
OpenWorld 2018 - 20 years of hints and tips
Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17
Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17
ILOUG 2019 - Autonomous, what does it mean for DBAs
ILOUG 2019 - Autonomous, what does it mean for DBAs
APEX Connect 2019 - SQL Tuning 101
APEX Connect 2019 - SQL Tuning 101
Architecture & Pitfalls of Logical Replication
Architecture & Pitfalls of Logical Replication
[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
[CB19] MalConfScan with Cuckoo: Automatic Malware Configuration Extraction Sy...
APEX Connect 2019 - successful application development
APEX Connect 2019 - successful application development
Diagnose Your Microservices
Diagnose Your Microservices
Mais de 心 谷本
プレゼンの技術 2 実践編
プレゼンの技術 2 実践編
心 谷本
プレゼンの技術 1 考え方
プレゼンの技術 1 考え方
心 谷本
プレゼンの技術
プレゼンの技術
心 谷本
人気番組との戦い! Javaシステムのパフォーマンスチューニング奮闘記
人気番組との戦い! Javaシステムのパフォーマンスチューニング奮闘記
心 谷本
from old java to java8 - KanJava Edition
from old java to java8 - KanJava Edition
心 谷本
実例Javaトラブルシューティング! 〜稼働中のシステムを立て直した半年間の軌跡
実例Javaトラブルシューティング! 〜稼働中のシステムを立て直した半年間の軌跡
心 谷本
from old Java to modern Java
from old Java to modern Java
心 谷本
日本一細かいJavaOne2011報告
日本一細かいJavaOne2011報告
心 谷本
勝敗は常に見積もりで決まる〜Redmineを使った時間記録の話
勝敗は常に見積もりで決まる〜Redmineを使った時間記録の話
心 谷本
日本で二番目に「細かい」JavaOne2011報告
日本で二番目に「細かい」JavaOne2011報告
心 谷本
[関ジャバ]Java News 2011 Mar
[関ジャバ]Java News 2011 Mar
心 谷本
Mais de 心 谷本
(11)
プレゼンの技術 2 実践編
プレゼンの技術 2 実践編
プレゼンの技術 1 考え方
プレゼンの技術 1 考え方
プレゼンの技術
プレゼンの技術
人気番組との戦い! Javaシステムのパフォーマンスチューニング奮闘記
人気番組との戦い! Javaシステムのパフォーマンスチューニング奮闘記
from old java to java8 - KanJava Edition
from old java to java8 - KanJava Edition
実例Javaトラブルシューティング! 〜稼働中のシステムを立て直した半年間の軌跡
実例Javaトラブルシューティング! 〜稼働中のシステムを立て直した半年間の軌跡
from old Java to modern Java
from old Java to modern Java
日本一細かいJavaOne2011報告
日本一細かいJavaOne2011報告
勝敗は常に見積もりで決まる〜Redmineを使った時間記録の話
勝敗は常に見積もりで決まる〜Redmineを使った時間記録の話
日本で二番目に「細かい」JavaOne2011報告
日本で二番目に「細かい」JavaOne2011報告
[関ジャバ]Java News 2011 Mar
[関ジャバ]Java News 2011 Mar
Último
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
Martijn de Jong
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
wesley chun
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Juan lago vázquez
presentation ICT roal in 21st century education
presentation ICT roal in 21st century education
jfdjdjcjdnsjd
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
sammart93
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Neo4j
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
The Digital Insurer
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
apidays
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
debabhi2
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Edi Saputra
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
Rafal Los
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
sudhanshuwaghmare1
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Safe Software
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
The Digital Insurer
Top 10 Most Downloaded Games on Play Store in 2024
Top 10 Most Downloaded Games on Play Store in 2024
SynarionITSolutions
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
Andrey Devyatkin
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
DianaGray10
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
apidays
Último
(20)
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
presentation ICT roal in 21st century education
presentation ICT roal in 21st century education
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Top 10 Most Downloaded Games on Play Store in 2024
Top 10 Most Downloaded Games on Play Store in 2024
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Let's reconsider about collecting logs. Plus, visiting elastic@Moutain View!
1.
Let’s reconsider about
collecting logs. Plus, visiting elastic@Mountain View! Shin Tanimoto Acroquest Technology Co., LTD.
2.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved. Who am I? 2 • 谷本 心 (Shin Tanimoto) - Acroquest Technology Co., LTD. (Sales partner of elastic) - Java Troubleshooter - Board member of JJUG (Japan Java User Group) - Twitter : @cero_t - Facebook : shin.tainmoto
3.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. Quiz🙋 3
4.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. What is the origin of the word “log”? 4
5.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 5
6.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ancient Greece people record the “date” using branches of the tree. 6
7.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 2. In medieval Europe, people measured “speed” of ship with log (round wood). 7
8.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 3. In the early 20th century United States, engineers used a logarithm table for “usage history” of computers. 8
9.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ancient Greece people’s “date” record. 2. Medieval Europe sailors’ “speed” record. 3. American engineers’ “usage” record. 9
10.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. 2. Medieval Europe sailors’ “speed” record. 3. 10
11.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. Common sense: Log is important 11
12.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. True common sense: Watching log is painful! 12
13.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. Then log should be watched and processed by machine (ordinary) 13
14.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 14 #1 Ordinal Log Processing
15.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing ELK stack 15 send logs search
16.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Access counts (upper) / response time (lower) 16
17.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Access counts (upper) / response time (lower) 17 10/sec 100/sec 30sec 20sec 10sec
18.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Huge performance issue 18
19.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Huge performance issue 19 3000sec 2000sec 1000sec
20.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Slow query log of MySQL 20 same shape!
21.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Slow query log of MySQL 21 2000sec 1000sec same shape! same scale!
22.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing But where do these shapes come? 22
23.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing 23 But where do these shapes come? 1. Lock tables? 2. Up to maximum size of connection pool? 3. CPU bottle neck? 4. Disk I/O bottle neck?
24.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing 24 Confirm the stored procedure in detail 1. Found 100,000 times of insert into “temporary table” query – (even in the search function … ) – causing high CPU and Disk I/O usage 2. Optimized the stored procedure removing wasting process – Only a drop in the bucket 😩 3. Modify the create temporary table state in the stored procedure to create that temporary table “on memory” – with memory tunings ( tmp_table_size etc. ) – resulted in …
25.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Performance issue was resolved! 25
26.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Performance issue was resolved! 26 500sec 100sec Never mind, some heavy batch
27.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing Disk I/O improved!!! 27 Disk I/O on MySQL server before ← →after
28.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 1. Ordinal Log Processing I/O wait had gone! 28 before ← →after CPU usage on MySQL server
29.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 29 #2 Reconsider Log Processing
30.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 30 Watching logs to detect errors is a responsibility of developers, isn’t it?
31.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 31 Watching logs is important but painful
32.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 32 Let’s think about painless log processing system
33.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Logs can be used in various ways Visualizing - as chart Watching - and notifying by e-mail Viewing - by human’s eyes Keeping - backup just in case 33
34.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Logs can be used in various purpose Visualizing - To find “unknown” issues Watching - To find “known” issues Viewing - To find the cause of issues Keeping - To use as necessary 34
35.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Logs retention period are also various Visualizing - last 2 or 4 weeks Watching - last 24 hours Viewing - last 2 or 4 weeks Keeping - entire period 35
36.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Tools for processing logs are different Visualizing - Elasticsearch Watching - Zabbix or some custom batch Viewing - Text editor Keeping - File server 36
37.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 37 Log is not necessarily files.
38.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 38 Log can be regarded as events.
39.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 39 Log streaming hub
40.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Log Streaming Hub 40 Application Agent Streaming Hub Viewer Watcher Visualizer Storage Application Agent
41.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Ordinal case 41 Application fluentd Text Editor Zabbix Elasticsearch + Kibana NAS Application fluentd
42.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing Using fluentd 42 Application fluentd fluentd Zabbix Elasticsearch + Kibana Application fluentd Text Editor NAS
43.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Processing Logs Using Redis? 43 Application Logstash Redis Zabbix Elasticsearch + Kibana Application Logstash Text Editor NAS
44.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Reconsider Log Processing An example on AWS 44 Application Kinesis Agent Kinesis Cloudwatch Logs Elasticsearch + Kibana Application Kinesis Agent S3 Lambda
45.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Processing Logs An example on AWS 45 Application awslogs Cloudwatch Logs Elasticsearch + Kibana Application awslogs S3 Lambda
46.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Processing Logs An example on AWS 46 Application logstash S3 Elasticsearch + Kibana Application logstash Cloudwatch Logs Lambda
47.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #2 Processing Logs Anyway… 47 Application Agent Streaming Hub Viewer Watcher Visualizer Storage Application Agent
48.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 48 すんなり入る話 ですよね?
49.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 49 By the way
50.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 50 I had visited elastic @Mountain View!!
51.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 51 写真はブログで http://acro-engineer.hatenablog.com/ entry/2015/11/08/150942
52.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 52 elastic stack .Next
53.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #3 elastic stack .Next Elasticsearch Task management API Reindex API https://www.elastic.co/elasticon/2015/sf/whats-next- for-elasticsearch-2x-and-beyond Logstash Clustering Persistent https://www.elastic.co/guide/en/logstash-roadmap/ current/index.html 53
54.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #3 elastic stack .Next Kibana Custom Apps / plugins https://www.elastic.co/elasticon/2015/sf/whats- cookin-in-kibana-4 Beats Packetbeat Filebeat Topbeat https://www.elastic.co/products/beats 54
55.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #3 elastic stack .Next Commercial plugin Cross-stack monitoring / management Cross-stack security PDF reporting Orchestration / Automation 55
56.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 56 Using elastic stack .Next
57.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. #3 elastic stack .Next Using next ELK stack + AWS 57 Application Filebeat Topbeat Logstash Elasticsearch + Kibana Application Filebeat Topbeat S3 Cloudwatch Logs
58.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved. 58 Beatsシリーズ 調べなくちゃって気に なりました
59.
Copyright © Acroquest
Technology Co., Ltd. All rights reserved. 59 Enjoy processing logs using ELK!
Baixar agora