The document discusses introducing log analysis to an organization. It covers log shipping architecture using file shippers, centralized buffers like Kafka and Redis, and storage and analysis using Elasticsearch, Kibana and Grafana. Specific topics covered include choosing the right shipper, buffer types, protocols, and optimizing Elasticsearch configuration, indices, and hardware for different node types like data, ingest and client nodes.
33. Buffer types
Disk || memory || combined hybrid approach
On source || centralized
App
Buffer
App
Buffer
file or local log shipper
easy scaling – fewer moving parts
often with the use of lightweight shipper
App
App
Kafka / Redis / Logstash / etc…
one place for all changes
extra features made easy (like TTL)
ES
ES
53. Elasticsearch – Indices
Index – logical place for data
Index – can be compared to database in DB
Index – built out of one or more shards
54. Elasticsearch – Indices
Index – logical place for data
Index – can be compared to database in DB
Index – built out of one or more shards
Shard – can be spread among multiple nodes
65. Daily indices are a good start
2017.11.16 2017.11.17 2017.11.20 2017.11.21. . .
Indexing is faster for smaller indices
Deletes are cheap
Search can be performed on indices that are needed
Static indices are cache friendly
indexing
most searches
We delete whole indices