This was a presentation I gave about Splunk to the Sydney Python group in October 2010. I talked in depth about modifying Splunk for interesting added functionality.

1. Splunk and Python
Sydney Python October 2010
Kelvin Nicholson

2. What is Splunk?
“Splunk is the world’s leading software used to
monitor, report and analyze live streaming IT data
as well as terabytes of historical data – located
on-premises or in the cloud.” -Splunk.com
“Splunk is like google for log files.” -Kelvin

3. Installing Splunk (on Ubuntu)
$ sudo dpkg -i splunk-4.1.5-85165-linux-2.6-intel.deb
$ sudo splunk enable boot-start
$ sudo /etc/init.d/splunk start

4. Splunk Welcome Screen

5. Configuring Splunk
● Configure Splunk to allow syslog traffic
● Configure devices to send syslog to Splunk
○ Linux (syslog-ng)
destination loghost {
udp("192.168.83.11" port (514));
};
log { source(s_all); destination(splunk); };
● Cisco IOS
no logging console
no logging monitor
logging 192.168.83.11
● OSSEC
<syslog_output>
<server>192.168.83.11
</server>
<port>8514</port>
</syslog_output>

6. Splunk Search Screen

7. Why I Like Splunk (Abridged)
● Dashboards of Search terms
■ Security alerts “login failed for”
■ STP network issues (“LEARNING AND FORWARDING”
■ Duplex mismatches
■ Wildcard searches, e.g. “-server2k3-”
● My “WTF” filter (easy filter building)
● Beautiful trending (“cold start” AND “switch01”)

8. Splunk Simple Filtering

9. Extending Splunk with Python
● REST API. (Search only)
● Custom search command. (iplocation)
● Configuring scripted alerts. (tweet X alert)
● Directly to backend using Splunk's built-in
modules. (Full module access)

10. Accessing Splunk Datastore
kelvinn@splunk:/opt/splunk/bin$ ./splunk cmd python
>>> import splunk.auth, splunk.search
>>> key = splunk.auth.getSessionKey('admin','changeme')
>>> my_job = splunk.search.dispatch('search sypy', namespace='search')
>>> event_list = []
>>> for event in my_job.events:
... event_list.append(event.fields)
...
>>> [{'_si':event_list
print splunk,main, 'index': main, 'sourcetype': syslog, 'source': udp:514,
'_kv': 1, 'splunk_server': splunk, '_time': 2010-10-06T19:40:37+1100, 'host':
192.168.83.5, '_sourcetype': syslog, '_raw': Oct 6 19:40:37 192.168.83.5 Oct
6 19:40:38 mini kelvinn: hello SyPy, hope you are doing well., '_serial': 0,
'_cd': 0:275}, {'_si': splunk,main, 'index': main, 'sourcetype': syslog, 'source':
udp:514, '_kv': 1, 'splunk_server': splunk, '_time': 2010-10-06T19:39:
33+1100, 'host': 192.168.83.5, '_sourcetype': syslog, '_raw': Oct 6 19:39:33
192.168.83.5 Oct 6 19:39:34 mini kelvinn: sypy, '_serial': 1, '_cd': 0:251}]
>>> event_list[0]['_raw']
Oct 6 19:40:37 192.168.83.5 Oct 6 19:40:38 mini kelvinn: hello SyPy, hope you are
doing well.

11. Splunk Architecture
CherryPy built-in, sweet. What can we do with that?

12. Built-in CherryPy Fun
kelvinn@splunk:/opt$ cat splunktest.py
import cherrypy
import splunk.auth, splunk.search
def get_splunk_data():
key = splunk.auth.getSessionKey('admin','changeme') # replace with your credentials
my_job = splunk.search.dispatch('search sypy', namespace='search', earliest_time='-24h')
event_list = []
for event in my_job.events:
event_list.append(event.raw)
return event_list
class HelloWorld:
def index(self):
splunk_list = get_splunk_data()
return str(splunk_list)
index.exposed = True
cherrypy.config.update({'server.socket_host': '0.0.0.0',
'server.socket_port': 9999,
})
kelvinn@splunk:/opt$ /opt/splunk/bin/splunk cmd python /opt/splunktest.
cherrypy.quickstart(HelloWorld())
py I'm not a CherryPy expert, but it looks pretty
P.S.

13. View CherryPy Page

14. Resources + Thanks
Splunk introduction:
http://www.splunk.com/base/Documentation/4.1.5/Installation/Splunksarchitectureandwhatgetsinstalled
Splunk REST Search (with Python httplib example):
http://www.splunk.com/base/Documentation/4.1.5/Developer/RESTCreateSearch
Custom search command (iplocation):
http://www.splunk.com/base/Documentation/latest/SearchReference/Customsearchiplocation
How to write custom alerts:
http://www.splunk.com/base/Documentation/4.1.5/Admin/Configurescriptedalerts
Using Splunk's built-in Python modules:
http://answers.splunk.com/questions/14/can-i-use-splunks-built-in-python-sdk-in-my-own-scripts
Some information about Splunk's Python SDK:
http://www.splunk.com/base/Documentation/4.1.5/Developer/PySDK
Thanks
.