O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
Motivate for Dev and Ops by mackerel 
@koudaiii 
at 2014/12/15
Profile 
id: koudaiii 
fullname: Kodai Sakabe
Attention 
• 小規模なWebサイト
Agenda 
1. 今までの監視と行動 
2. mackerel 
3. fluentd 
4. mackerel + fluentd 
5. chef 
6. LTSV 
7. fluent-plugin 
8. グラフの追加 
9. Mo...
今までの監視と行動 
• Metrics(メトリクス設定) 
• Graf(メトリクス毎にWebで閲覧) 
• Notification(閾値設定してアラート通知) 
• Job(バッチを流したり)
今までの監視と行動 
• Service Check(サービスのチェック) 
• Situation Analysis(状況判断切り分け) 
• Maintenance(メンテナンスページ切替) 
• Announce(Webサイト、Blog、...
監視っていつ使われる? 
普段からガンガン分析してと言うよりは、 
問題があった時に触れることが多い。 
または、定例的にチェックする程度。
mackerel
what’s mackerel? 
• 貴重なエンジニアをもっと開発に集中! 
• 管理/監視サーバー不要/パフォーマンス管理ツール!! 
• API等のツール連携 
• グラフで時系列にデータを可視化。(差分・重ねあわせ) 
• サービス(H...
mackerel 
https://mackerel.io
fluentd
Build Your Unified Logging Layer 
fluentd
what’s Fluentd? 
• ログ収集システム 
• ログをJSONとして扱う 
• 豊富なプラグイン 
• 5000台以上のサーバー から一日あたり5TB以 
上、ピークタイムには秒間50,000メッセージに 
ものぼるログを収集
mackerel + fluentd
User Ops 
AWS cloud 
fluentd-plugin-mackerel 
fluentd-plugin-datacounter
How to~ 
• Chef 
• Logのformat変更 
• fluentdのplugin 
• 画面を見る 
• ちょっと修正
Chef
chef-td-agent 
cookbook-mackerel-agent
ハマったポイント 
• td-agentのサポートはUbuntuの場合、LTS 
• Vagrantでテストする際にOpscodeのBENTO使って 
も動かなかったのでubuntu/trusyを利用 
• mackerel-agentのapi...
LTSV
What’s LTSV ? 
• Labeled Tab-Separated Values 
• テキストデータのフォーマット名 
• タブ区切りで値に名前を付けただけ 
• やっておくとparse しやすいのとfluentdが対応 
• ng...
nginxのaccess_log 
log_format ltsv "time:$time_local" 
"thost:$remote_addr" 
"tforwardedfor:$http_x_forwarded_for" 
"treq:$...
before after LTSV. 
before 
$ 222.186.31.196 - - [14/Dec/2014:16:19:49 +0900] "POST / HTTP/ 
1.1" 403 162 "-" "Mozilla/5.0...
fluentd in_tailの設定 
<source> 
type tail 
format ltsv 
time_format %d/%b/%Y:%H:%M:%S %z 
path /var/log/nginx/access.log 
po...
fluentdに渡すとこうなる 
$ 2014-12-05T15:37:44+09:00 access.nginx 
{"host":"66.249.75.80","forwardedfor":"-","req":"GET / HTTP/ 
1...
fluent-plugin
fluent-plugin-datacounter 
<match access.nginx> 
type datacounter 
count_interval 1m #間隔 
count_key status #statusをcount 
...
fluent-plugin-mackerel 
<match ...> 
type mackerel 
api_key <投稿先オーガニゼーションのAPIキー> 
service <投稿先のサービス名> 
metrics_name http_s...
あれ? 
極端に2XX_countが増えてるのはなんでだろう?
グラフの追加
fluent-plugin-datacounter 
!修正 
<match access.nginx> 
type copy # 追加。これがないと正しく動かない 
# 既存のmatchをstoreに変更、下記追加 
<store> 
typ...
fluent-plugin-mackerel 
!も修正 
# 新しく追加 
<match nginx.referer.**> 
type mackerel 
api_key <投稿先オーガニゼーションのAPIキー> 
service <投稿先...
requestの数とか知りたい!! 
サービスメトリック 
accessのステータスの数とrefererでどこからのリンクから来て 
いるか把握
カスタムメトリックの設定 
nginx 
$/etc/nginx/conf.d/default.conf 
server { 
listen 80; 
server_name localhost; 
・・・・ 
location /nginx_...
カスタムメトリックの設定 
mackerel-agent 
$ apt-get install mackerel-agent-plugins 
! 
$ vim /etc/mackerel-agent/mackerel-agent.conf 
...
カスタムメトリック 
Connection / Connetcion Status / Requests
Motivate(まとめ)
監視から将来の展望を 
• LoadAverageがやべえええ!!!より、アクセスがや 
べええ!の方がテンション上がる 
• 仮説を立てて、fluentdとサーバーでグラフを見比べて 
検証できる 
• fluent-pluginで自分が書い...
よりOffensiveな監視へ 
• どちらかと言うと問題があった時に見ること 
が多い印象(あとは定例作業?) 
• Systemの監視を行いつつ、Service状態を把握 
• よりパフォーマンス管理を! 
• 円グラフでServiceの特...
Reference 
• https://github.com/mackerelio/cookbook-mackerel-agent 
• https://github.com/mackerelio/cookbook-mackerel-agen...
Próximos SlideShares
Carregando em…5
×

Motivate for dev and ops by mackerel

1.168 visualizações

Publicada em

mackerelを使うことで、今まで使っていた監視システムの考え方を変える!というテーマでお話いたしました。

Publicada em: Tecnologia
  • accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ...................................ALL FOR EBOOKS................................................. Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes ,Download or read Ebooks here ... ......................................................................................................................... Download FULL PDF EBOOK here { http://bit.ly/2m6jJ5M }
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THI5 BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THI5 BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Motivate for dev and ops by mackerel

  1. 1. Motivate for Dev and Ops by mackerel @koudaiii at 2014/12/15
  2. 2. Profile id: koudaiii fullname: Kodai Sakabe
  3. 3. Attention • 小規模なWebサイト
  4. 4. Agenda 1. 今までの監視と行動 2. mackerel 3. fluentd 4. mackerel + fluentd 5. chef 6. LTSV 7. fluent-plugin 8. グラフの追加 9. Motivate
  5. 5. 今までの監視と行動 • Metrics(メトリクス設定) • Graf(メトリクス毎にWebで閲覧) • Notification(閾値設定してアラート通知) • Job(バッチを流したり)
  6. 6. 今までの監視と行動 • Service Check(サービスのチェック) • Situation Analysis(状況判断切り分け) • Maintenance(メンテナンスページ切替) • Announce(Webサイト、Blog、SNS、Mail、Telephone) • Take the action(人力で対応する!) • Patch Release(緊急リリース)
  7. 7. 監視っていつ使われる? 普段からガンガン分析してと言うよりは、 問題があった時に触れることが多い。 または、定例的にチェックする程度。
  8. 8. mackerel
  9. 9. what’s mackerel? • 貴重なエンジニアをもっと開発に集中! • 管理/監視サーバー不要/パフォーマンス管理ツール!! • API等のツール連携 • グラフで時系列にデータを可視化。(差分・重ねあわせ) • サービス(Hatena-Blog)/ロール(Web/App/DB)でサー バーを整理
  10. 10. mackerel https://mackerel.io
  11. 11. fluentd
  12. 12. Build Your Unified Logging Layer fluentd
  13. 13. what’s Fluentd? • ログ収集システム • ログをJSONとして扱う • 豊富なプラグイン • 5000台以上のサーバー から一日あたり5TB以 上、ピークタイムには秒間50,000メッセージに ものぼるログを収集
  14. 14. mackerel + fluentd
  15. 15. User Ops AWS cloud fluentd-plugin-mackerel fluentd-plugin-datacounter
  16. 16. How to~ • Chef • Logのformat変更 • fluentdのplugin • 画面を見る • ちょっと修正
  17. 17. Chef
  18. 18. chef-td-agent cookbook-mackerel-agent
  19. 19. ハマったポイント • td-agentのサポートはUbuntuの場合、LTS • Vagrantでテストする際にOpscodeのBENTO使って も動かなかったのでubuntu/trusyを利用 • mackerel-agentのapiがconfファイルに適応されない • https://github.com/mackerelio/cookbook-mackerel-agent/ pull/19 Mergeして欲しい…
  20. 20. LTSV
  21. 21. What’s LTSV ? • Labeled Tab-Separated Values • テキストデータのフォーマット名 • タブ区切りで値に名前を付けただけ • やっておくとparse しやすいのとfluentdが対応 • nginx やapacheのlogとか
  22. 22. nginxのaccess_log log_format ltsv "time:$time_local" "thost:$remote_addr" "tforwardedfor:$http_x_forwarded_for" "treq:$request" "tstatus:$status" "tsize:$body_bytes_sent" "treferer:$http_referer" "tua:$http_user_agent" "treqtime:$request_time" "tcache:$upstream_http_x_cache" "truntime:$upstream_http_x_runtime" “tvhost:$host"; access_log /var/log/nginx/access.log ltsv;
  23. 23. before after LTSV. before $ 222.186.31.196 - - [14/Dec/2014:16:19:49 +0900] "POST / HTTP/ 1.1" 403 162 "-" "Mozilla/5.0" “-" ! after $ time:14/Dec/2014:21:41:11 +0900 host:116.80.20.96 forwardedfor:- req:GET / HTTP/1.1 status:304 size:0 referer:- ua:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/ 537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 reqtime:0.000 cache:- runtime:- vhost:matsuri4.doyomeki.com
  24. 24. fluentd in_tailの設定 <source> type tail format ltsv time_format %d/%b/%Y:%H:%M:%S %z path /var/log/nginx/access.log pos_file /var/log/nginx/access_log.pos tag access.nginx </source>
  25. 25. fluentdに渡すとこうなる $ 2014-12-05T15:37:44+09:00 access.nginx {"host":"66.249.75.80","forwardedfor":"-","req":"GET / HTTP/ 1.1","status":"200","size":"6686","referer":"-","ua":"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html)","reqtime":"0.000","cache":"-","runtime":"-","vhost":"ma tsuri4.doyomeki.com"}
  26. 26. fluent-plugin
  27. 27. fluent-plugin-datacounter <match access.nginx> type datacounter count_interval 1m #間隔 count_key status #statusをcount aggregate all tag nginx.status pattern1 2xx ^2dd$ pattern2 3xx ^3dd$ pattern3 4xx ^4dd$ pattern4 5xx ^5dd$ </match>
  28. 28. fluent-plugin-mackerel <match ...> type mackerel api_key <投稿先オーガニゼーションのAPIキー> service <投稿先のサービス名> metrics_name http_status.${out_key} out_keys <投稿するメトリックのキー。例: 2xx_count,3xx_count,4xx_count,5xx_count> </match>
  29. 29. あれ? 極端に2XX_countが増えてるのはなんでだろう?
  30. 30. グラフの追加
  31. 31. fluent-plugin-datacounter !修正 <match access.nginx> type copy # 追加。これがないと正しく動かない # 既存のmatchをstoreに変更、下記追加 <store> type datacounter count_interval 1m count_key referer aggregate all tag nginx.referer pattern1 facebook facebook pattern2 twitter t.co pattern3 hoku hoku.in </store> </match>
  32. 32. fluent-plugin-mackerel !も修正 # 新しく追加 <match nginx.referer.**> type mackerel api_key <投稿先オーガニゼーションのAPIキー> service <投稿先のサービス名> metrics_name access_referer.${out_keyZ ! out_keys facebook_count,twitter_count,hoku_count </match>
  33. 33. requestの数とか知りたい!! サービスメトリック accessのステータスの数とrefererでどこからのリンクから来て いるか把握
  34. 34. カスタムメトリックの設定 nginx $/etc/nginx/conf.d/default.conf server { listen 80; server_name localhost; ・・・・ location /nginx_status { stub_status on; access_log off; allow 127.0.0.1; deny all; } } $ service nginx restart
  35. 35. カスタムメトリックの設定 mackerel-agent $ apt-get install mackerel-agent-plugins ! $ vim /etc/mackerel-agent/mackerel-agent.conf [plugin.metrics.nginx] command = "/usr/local/bin/mackerel-plugin-nginx -host localhost - port 80" ! $ service mackerel-agent restart
  36. 36. カスタムメトリック Connection / Connetcion Status / Requests
  37. 37. Motivate(まとめ)
  38. 38. 監視から将来の展望を • LoadAverageがやべえええ!!!より、アクセスがや べええ!の方がテンション上がる • 仮説を立てて、fluentdとサーバーでグラフを見比べて 検証できる • fluent-pluginで自分が書いたコードがどれくらい使わ れてるか • また負荷はどうなんだろう?って気付く。
  39. 39. よりOffensiveな監視へ • どちらかと言うと問題があった時に見ること が多い印象(あとは定例作業?) • Systemの監視を行いつつ、Service状態を把握 • よりパフォーマンス管理を! • 円グラフでServiceの特徴とかピーク時間帯がmackerelで見れたらいいなぁ(ボソ イメージです>< before after
  40. 40. Reference • https://github.com/mackerelio/cookbook-mackerel-agent • https://github.com/mackerelio/cookbook-mackerel-agent/pull/19 • https://github.com/treasure-data/chef-td-agent • http://www.rubydoc.info/gems/fluent-plugin-datacounter/ • https://github.com/tksmd/fluent-plugin-mackerel • http://qiita.com/tksmd/items/1212331a5a18afe520df • http://d.hatena.ne.jp/naoya/20130209/1360381374 • http://help-ja.mackerel.io/entry/advanced/fluentd • http://qiita.com/key/items/038b7913b3bb0298c625

×