Apache Kafka : Monitoring vs Alerting
•Download as PPTX, PDF•
2 likes•1,627 views
The document discusses Kafka monitoring at LinkedIn. It provides background on Kafka including brokers, topics, partitions, producers and consumers. It then discusses the history of Kafka monitoring at LinkedIn, including the many host, JVM and broker metrics they collect. Key metrics to monitor include under replicated partitions, offline partitions, active controller count, and partitions with replication below the minimum threshold. The future of monitoring at LinkedIn will focus on service level objectives and external availability monitoring. Monitoring collects all data while alerting on very few, key metrics.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Apache Kafka : Monitoring vs Alerting
Similar to Apache Kafka : Monitoring vs Alerting (20)
Recently uploaded
Recently uploaded (20)
Apache Kafka : Monitoring vs Alerting
- 1. Apache Kafka Monitoring vs Alerting Ratish Ravindran Sr. Site Reliability Engineer - LinkedIn
- 2. Agenda Kafka 101 History of Kafka monitoring at LinkedIn Key Metrics to alert on and why Future of Kafka monitoring at LinkedIn Q&A
- 3. Kafka 101
- 4. Brokers and Clusters ● Single Kafka Server ● Multiple brokers
- 5. Topics and Partitions ● Categorized messages ● Broken down into partitions
- 6. Leader and Replica ● Broker owning partition ● Broker with partition copy
- 8. Controller ● Broker with more responsibilities ● Partition management • Leader election • Leader Switch • New topic and partition • New broker
- 9. History of Kafka Monitoring at LinkedIn
- 10. Common Metrics ● Host metrics ● JVM metrics ● Broker or Application metrics
- 11. Host Metrics ● Load average ● Memory usage (free, used, %) ● CPU ● Inbound/Outbound network stats ● Disk usage ● RAID stats ● Inode (free, used, %) ● IO stats ● Disk states ● TCP connections (est. count, error etc) ~ 320 host metrics
- 12. JVM Metrics ● Minor/Full GC time ● Minor/Full GC count ● Total thread count ● Used thread count ● Total heap size available ● Total heap size used
- 13. Broker Metrics ● Bytes and messages in/out rate ● Fetch/produce queue size ● Request handler idle ratio ● Request pool usage ● Request latencies – at different stages ● Log flush rate ● Produce and consume latency 2100 broker metrics and 165 graphs
- 15. Kafka at LinkedIn ● 2500+ Kafka brokers ● 1 PB In per day ● 3.9 PB Out per day
- 17. Kafka as a service ● Broker/Cluster Health ● Message Delivery ● Performance ● Capacity
- 18. Metrics you need to know Partitions that are not fully replicated within the cluster Partitions unavailable for produce and consume Should always be 1 URP Offline Partitions Active Controller Count of partitions with ISR < MinISR Under MinISR Count
- 19. Under Replicated Partitions ● Highly discussed ● Overall cluster health ● Replication is consumer and producer
- 23. Metrics you need to know Partitions that are not fully replicated within the cluster Partitions unavailable for produce and consume Should always be 1 URP Offline Partitions Active Controller Count of partitions with ISR < MinISR UnderMinIsrP artitionCount
- 24. Offline partition count ● Partition(s) unavailable ● All brokers hosting replica down OR ● unclean.leader.election.enabled=false ● Potential data loss
- 25. Metrics you need to know Partitions that are not fully replicated within the cluster Partitions unavailable for produce and consume Should always be 1 URP Offline Partitions Active Controller Count of partitions with ISR < MinISR UnderMinIsrP artitionCount
- 26. Active Controller Count ● What is Kafka controller ? ● Partition management ● There should only 1 controller
- 27. Active Controller Count < 1
- 28. Active Controller Count > 1
- 29. Metrics you need to know Partitions that are not fully replicated within the cluster Partitions unavailable for produce and consume Should always be 1 URP Offline Partitions Active Controller Count of partitions with ISR < MinISR UnderMinIsrP artitionCount
- 30. UnderMinIsrPartitionCount ● min.insync.replicas ● in-sync replicas count < min.insync.replicas ● Per-broker metrics
- 31. UnderMinIsrPartitionCount Screen Shot 2018-06-18 at 3.12.17 PMOffline partitions URP Under MinIsr Partition count
- 32. Operating System and Hardware Metrics ● Should I be worried ? ● What application is causing it ? ● Don’t alert unless: ● 100% clear signal ● 100% actionable
- 33. Capacity Planning ● Plan in advance ● Multi-factor ● Don’t alert for capacity
- 34. Capacity Metrics ● Request Handler Idle Ratio ● Disk Utilization ● Partition Count ● Network Utilization
- 35. Future of Kafka Monitoring at LInkedIn
- 36. Future of Monitoring Kafka at LinkedIn ● SLO based ● Monitor and alert on SLO metrics • Latency – Produce and consume • Availability – Produce and consume • Retention
- 37. Availability Monitoring ● Measured externally ● Client focused ● github.com/linkedin/kafka-monitor
- 38. Conclusion
- 39. Monitoring is not alerting ● Collect everything ● Alert on almost nothing ● SLO based monitoring
- 40. Getting (and Giving) Help • Community users@kafka.apache.org dev@kafka.apache.org • Bugs and Work https://issues.apache.org/jira/projects/KAFKA • Kafka Monitor https://github.com/linkedin/kafka-monitor • Burrow https://github.com/linkedin/Burrow • Cruise Control https://github.com/linkedin/cruise-control • kafka-tools https://github.com/linkedin/kafka-tools
- 41. Questions