Scaling splunk 101 - the easy slides

1. Scaling Splunk 101Quick Overview of Scaling Splunk with Commodity HardwareErik SwanOct, 09 ** Slides intentionally ugly, no designers were harmed during construction

2. Single Server InstallCommodity Architecture Simplest Splunk install is a single server that functions as both indexer and search head. A single box can easily index 100-200G per day, BUT for fast searching its best to use more than one box. Data from Splunk Forwarders, Syslog, Files, etc. Splunk (all in one) Users

3. Improving Search and Indexing Performance Splunk scales search and indexing performance horizontally by adding more indexers and in some cases scaling out a search tier. By spreading the incoming load across more indexers you index faster. Perhaps more importantly, by spreading the indexed data across more indexers your search performance improves linearly as well. Consider that every doubling of hardware will double your index and search performance and don’t be shy of adding 10’s of servers. RULE #1 – If your searches are slow, add another box!

4. Adding a Search Head By splitting out a Search Head, search performance is improved and load is taken off the indexer for faster indexing. Best to add sooner than later. Best for volumes between 5-100G p/day 1 Indexer 1 Search Head Data from Splunk Forwarders, Syslog, Files, etc. Spunk Indexer Splunk Search Head Users

5. Adding a second Indexer As volume goes up beyond 100G OR you want to improve search performance its best to add a second Indexer. **Remember adding indexers improves search performance linearly as well. Best for volumes 20-200G p/day 2 Indexers 1 Search Head Data from Splunk Forwarders, Syslog, Files, etc. Spunk Indexer Spunk Indexer Splunk Search Head Users

6. Adding additional Indexers For every new ~100G, or again to improve search performance add another indexer. RULE #1: If searches are slow, add an another indexer. For volumes from 200G-1T p/day TBs/day from Splunk Forwarders and Syslog Spunk Indexer Spunk Indexer Spunk Indexer (n) Indexers Splunk Search Head Users

7. Adding additional Indexers For every new ~100G, or again to improve search performance add another indexer. RULE #1: If searches are slow, add an another indexer. For volumes from 200G-1T p/day TBs/day from Splunk Forwarders and Syslog Assume 100G p/day: Use Case : Log archival and some periodic troubleshooting 1 Commodity Server Use Case #2 : Archival, troubleshooting and summary reporting 1 Index Server, 1 Search Server Use Case #3: Archival, Trouble Shooting, and Reporting 2 Index Servers, 1 Search Server Use Case #4: Many ( >2 ) users doing constant use 3+ Index Servers, 1 Search Server Spunk Indexer Spunk Indexer Spunk Indexer (n) Indexers Splunk Search Head Users

8. Adding additional Search Heads TBs/day from Splunk Forwarders and Syslog Adding more Search Heads is a convenient way to improve search performance Add an additional Search Heads when: It makes sense to partition users. Too offload summary or scheduled searches. Spunk Indexer Spunk Indexer Spunk Indexer (n) Indexers Splunk Search Head Splunk Search Head (n) Search Heads 1~ 4T each p/day Load Bal. Users

9. Adding additional Search Heads TBs/day from Splunk Forwarders and Syslog Assuming a load of 1T p/day: Use Case #1: Log archival and some periodic troubleshooting 4 Index Servers, 1 Search Server Use Case #2: Archival, trouble shooting and some summary reporting 8+ Index Server, 1 Search Server Use Case #3: Archival, Trouble Shooting, and Reporting 16+ Index Servers, 1 Search Server Use Case #4: Many ( >2 ) users doing constant use 20+ Index Servers, 1 Search Server For every new ~TB p/day, add another search head. For volumes > 2T p/day (n) Indexers each <100G p/day (m) Search Heads for every ~1T p/day Spunk Indexer Spunk Indexer Spunk Indexer (n) Indexers Splunk Search Head Splunk Search Head (n) Search Heads 1~ 4T each p/day Load Bal. Users

10. Long term storage, add a SAN TBs/day from Splunk Forwarders and Syslog Long term storage can not be kept on local commodity IO. If wanting to keep more than can be kept on local indexer disk, splunk can be configured to use SAN or other storage device. Best for keeping >30 day – multi year data. Spunk Indexer Spunk Indexer Spunk Indexer (n) Indexers Tier 1 SAN Splunk Search Head Splunk Search Head Load Bal. Users

11. Multi-datacenter or deployment If you have multiple data centers, it is often best to leave the data local and use distributed search between two deployments. If you have data that naturally partitions such that users would rarely search across the data, partitioning entire deployments can help. Obviously for DR as well.

12. Additional Scaling Topics Summary Indexing – If your searches are slow consider using summary indexing: video - http://www.splunk.com/view/SP-CAAACZW docs - http://www.splunk.com/base/Documentation/4.0.5/User/UseSummaryIndexingForIncreasedReportingEfficiency Routing High Volume data to Separate Index – If you are searching or reporting on a source that is dwarfed by the volume of another source, you can partition data such that the high volume source is in its own index: docs - http://www.splunk.com/base/Documentation/latest/Admin/Setupmultipleindexes#Why_have_multiple_indexes.3F