Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Tyler Rutschman- Kansas City
1. Growing Splunk
Tyler Rutschman - Garmin International
Tuesday, May 15, 12
2. About Me
Linux System Administrator
Husband and Father of 2 Kids
DevOps, Productivity Hacks and Tools,
The Big Lebowski
Tuesday, May 15, 12
3. OH: (during an outage)
I don’t want to live in a world
without Splunk.
Tuesday, May 15, 12
4. Backstory
Free instance installed in 2009 by the
Network Team
Single Instance on Central Log server
Upgrade to Enterprise
Tuesday, May 15, 12
I started as an IT intern my mentor had a free copy of version 2.x running on the log server.
I was tasked with finding a solution for SOX & PCI requirements. (Which was mind expanding for an intern, to say the least)
Worked with purchasing to get a small license for the enterprise features.
My project ended up piping Splunk output into a python program that no one but I understood that printed out a text report that (I felt at least) was superior to the one in place at the time. (Big
surprise, didn’t end up using it).
5. Building Blocks
Split Splunk off onto dedicated instance
License overwhelmed by single app
Limited visibility and use
Tuesday, May 15, 12
When I came back there was some cursory interest in the app, but no major users and no project champion.
Welcome, back Tyler... Splunk Expert (by Default).
I was also attached to Garmin Connect, which is our awesome fitness tracking site, after getting more comfortable in my settings, I began to integrate the site logs into Splunk
6. IF YOU HAVE MORE INPUTS THAN LICENSE
YOU’RE GONNA HAVE A BAD TIME
Tuesday, May 15, 12
Obvious, but this was my experience during the first dedicated instance. We had a small license and it was all being used by Garmin Connect. It really wasn’t taking hold
like I knew it could.
7. Plan for Expansion
Decided to make application more robust
Read the Documentation
Planned roll out
Multiple Applications
License Increase
Scalable Architecture
Tuesday, May 15, 12
After I became more comfortable in my position, I felt impelled to make the application more robust and widespread.
I went to .conf last year, attended some training sessions and read up on the Administration documentation.
8. Enterprise Architecture
Elements (so far)
Puppet Deploy
Infrastructure Layout
Gotchas
Future Plans
Tuesday, May 15, 12
Overview of the Current Architecture Elements, will then go in depth a bit more on each subject.
9. Puppet
Search, Indexer and Forwarder are “turn-key”
ex: include splunk::indexer ...done
Really Awesome for Forwarders
Tuesday, May 15, 12
Puppet makes deployment simple. Servers are built with one include statement.
Forwarders are split up based on role and inputs. Customize the inputs a bit if necessary and include the splunk forwarder class in the puppet node definition.
10. Infrastructure
Tuesday, May 15, 12
Describe layers and functions.
Search is load balanced.
Search, Index and Forwarders are horizontally scalable.
Network/Taiwan instances aren’t pictured but are separate dedicated instances. Will move the network index into the main infrastructure real soon now.
11. How We Use Splunk
Web Access Logs
Service Usage Metrics
Feature Tracking
Diagnosing Problems in Production
Internal Application Audits
Windows Security Events
Tuesday, May 15, 12
We don’t have a wide variety if inputs into Splunk at the moment.
We currently use it on all of the major IT web applications to obtain service metrics, track new features and diagnosing issues in Production.
The developers are also starting to cater their applications to output Splunk friendly logs
Windows security events are queried via WMI ad filtered to specific IDs, this helps keep the volume down while delivering value for the Windows guys.
12. Why I like Splunk
Makes Users Happy
Real Time Data
No Alternatives
Tuesday, May 15, 12
Ease of configuration, having the one stop shop for user-land configs. LDAP integration is super simple.
Able to generate detailed reports and drill into the data on the fly is a killer feature and something that you simply won't find with any other application.
User community and Documentation.
There are no real alternatives to Splunk. Some tools touch on some of the features gained with the app, but there is no offering that matches what splunk can give you. I’ve
tried SEC, logwatch, Logstash, and Spiceworks. None were as user friendly and robust as Splunk.
13. Gotchas
Don’t Index a lot of data over NFS
Shared Knowledge Bundle Time Sync
Tag and Search permissions
Tuesday, May 15, 12
Keeping up with the demand. From a license and user request perspective, I limited amount of time to handle the requests at hand. Familiar position for me at least, but a good problem to
have.
Mounted Bundles must have the same time across the board.
Watch your permissions on saved searches and tags. They are usually private when I share them with another user and they cannot access.
14. Future Plans
Fix Central Logging
Check Out Deployment Server
More Inputs
Training
Tuesday, May 15, 12
Currently only one centralized syslog server, want to scale it out and put a farm of syslog servers behind a load balancer. Splunk will be the defininitive timeline for syslog
events.
Read about Deployment Server but passed on it at the time. Would like to pick it back up and see how it could be beneficial.
Add additional inputs to the application
I’ve been tasked with training my coworkers on how to use the application. Once they pick it up and figure it out, they can do awesome things.
15. Tips and Advice
WMI Event Filter for Windows Events
Splunkbase (stack overflow engine)
Tuesday, May 15, 12
