Splunk is the ultimate tool for the InfoSec hunter. In this unique session, we’ll dive straight into the Splunk search interface, and interact with wire data harvested from various interesting and hostile environments, as well as some web access logs. We’ll show how you can use Splunk Enterprise with a few free Splunk applications to hunt for attack patterns. We’ll also demonstrate some ways to add context to your data in order to reduce false positives and more quickly respond to information. Bring your laptop – you’ll need a web browser to access our demo systems!
Splunk Enterprise for Information Security (Hands-On)
1. 1
https://<yourIP>/ Username: splunklive Password: security
January (A-M): 50.31.181.148 July (A-M): 50.31.181.158
January (N-Z): 50.31.181.149 July (N-Z) 50.31.181.159
February: 50.31.181.150 August: 50.31.181.160
March: 50.31.181.151 September 50.31.181.161
April (A-M): 50.31.181.152 October (A-M) 50.31.181.162
April (N-Z): 50.31.181.153 October (N-Z) 50.31.181.163
May (A-M) 50.31.181.154 November (A-M) 50.31.181.164
May (N-Z) 50.31.181.155 November (N-Z) 50.31.181.165
June (A-M): 50.31.181.156 December (A-M) 50.31.181.166
June (N-Z): 50.31.181.157 December (N-Z) 50.31.181.167
5. 5
Splunk makes machine data accessible, usable and valuable to everyone.
5
Report and
analyze
Custom
dashboards
Monitor
and alert
Ad hoc
search
Real-time
Collection and
Indexing
Developer
Platform
Local or SAN
Other
Stores
6. Mainframe
Data
VMware
Platform for Machine Data
Splunk Solutions > Easy to Adopt
Exchange PCISecurity
Relational
Databases
MobileForwarders
Syslog /
TCP / Other
Sensors &
Control Systems
Across Data Sources, Use Cases & Consumption Models
Wire
Data
IT Svc Int
Splunk Premium Solutions & Apps Rich Ecosystem of Apps
ITSI UBA
UBA
7. Machine data contains a definitive record
of all interactions
Splunk is a very effective platform to collect,
store, and analyze all of that data
Human Machine
Machine Machine
8. 8
Rapid Ascent in the Gartner SIEM Magic Quadrant*
*Gartner, Inc., SIEM Magic Quadrant 2011-2015. Gartner does not endorse any vendor, product or
service depicted in its research publication and not advise technology users to select only those
vendors with the highest ratings or other designation. Gartner research publications consist of the
opinions of Gartner’s research organization and should not be construed as statements of fact.
Gartner disclaims all warranties, express or implied, with respect to this research, including any
warranties of merchantability or fitness for a particular purpose.
2015 Leader and the only vendor to
improve its visionary position
2014 Leader
2013 Leader
2012 Challenger
2011 Niche Player
2015
9. 9
https://<yourIP>/ Username: splunklive Password: security
January (A-M): 50.31.181.148 July (A-M): 50.31.181.158
January (N-Z): 50.31.181.149 July (N-Z) 50.31.181.159
February: 50.31.181.150 August: 50.31.181.160
March: 50.31.181.151 September 50.31.181.161
April (A-M): 50.31.181.152 October (A-M) 50.31.181.162
April (N-Z): 50.31.181.153 October (N-Z) 50.31.181.163
May (A-M) 50.31.181.154 November (A-M) 50.31.181.164
May (N-Z) 50.31.181.155 November (N-Z) 50.31.181.165
June (A-M): 50.31.181.156 December (A-M) 50.31.181.166
June (N-Z): 50.31.181.157 December (N-Z) 50.31.181.167
11. 11
Open Web Application Security Project
2013 Top 10
[10] Unvalidated redirects and forwards
[9] Using components with known vulnerabilities
[8] Cross-site request forgery
[7] Missing function level access control
[6] Sensitive data exposure
[5] Security misconfiguration
[4] Insecure direct object reference
[3] Cross-site scripting (XSS)
[2] Broken authentication and session management
15. 15
The anatomy of a SQL injection attack
SELECT * FROM users WHERE email='xxx@xxx.com'
OR 1 = 1 -- ' AND password='xxx';
xxx@xxx.xxx' OR 1 = 1 -- '
xxx
admin@admin.sys
1234
An attacker might supply:
17. 17
Hands-on Environment
Our learning environment consists of:
• 20 Publically-accessible single Splunk
servers
• Each with ~5.5M events, from real
environments but massaged:
• Windows Security events
• Apache web access logs
• Bro DNS & HTTP
• Palo Alto traffic logs
• Some other various bits
19. 19
https://splunkbase.splunk.com/app/1528/
Search for possible SQL injection in your events:
looks for patterns in URI query field to see if
anyone has injected them with SQL
statements
use standard deviations that are 2.5 times
greater than the average length of your URI
query field
Macros used
• sqlinjection_pattern(sourcetype, uri query field)
• sqlinjection_stats(sourcetype, uri query field)
20. 20
SQLINJECTION_REX Macro
sqlinjection_rex is a search macro. It contains:
(?<injection>(?i)select.*?from|union.*?select|'$|delete.*?from|update.*?set|alter.*?table|([
%27|'](%20)*=(%20)*[%27|'])|w*[%27|']or)
Which means: In the string we are given, look for ANY of the following matches and put that
into the “injection” field.
• Anything containing SELECT followed by FROM
• Anything containing UNION followed by SELECT
• Anything with a ‘ at the end
• Anything containing DELETE followed by FROM
• Anything containing UPDATE followed by SET
• Anything containing ALTER followed by TABLE
• A %27 OR a ‘ and then a %20 and any amount of characters then a %20 and then a %27 OR a ‘
• Note: %27 is encoded “’” and %20 is encoded <space>
• Any amount of word characters followed by a %27 OR a ‘ and then “or”
23. 23
Summary: Web attacks/SQL injection
SQL injection provide attackers with easy access to data
Detecting advanced SQL injection is hard – use an app!
Understand where SQLi is happening on your network and put a
stop to it.
Augment your Web Application Firewall (WAF) with enterprise-
wide Splunk searches.
28. 28
This and other techniques used in destructive Sands breach…
….and many others.
29. 29
Detecting Legacy PassTheHash
Look for Windows Events:
Event ID: 4624 or 4625
Logon type: 3
Auth package: NTLM
User account is not a domain logon
…this is trivially easy in Splunk.
31. 31
Then it got harder
• Pass the Hash tools have
improved
• Adjusting of jitter, other
metrics
• So let’s detect lateral
movement differently – just
looking at Windows event
codes isn’t enough.
32. 32
Network traffic provides source of truth
I usually talk to 10 hosts
Then one day I talk to 10,000 hosts
ALARM!
39. 39
DNS exfil tends to be
overlooked within an
ocean of DNS data.
Let’s fix that!
DNS exfiltration
40. 40
https://splunkbase.splunk.com/app/2734/
DNS exfil detection – tricks of the trade
parse URLs & complicated TLDs (Top Level Domain)
calculate Shannon Entropy
List of provided lookups
• ut_parse_simple(url)
• ut_parse(url, list) or ut_parse_extended(url, list)
• ut_shannon(word)
• ut_countset(word, set)
• ut_suites(word, sets)
• ut_meaning(word)
• ut_bayesian(word)
• ut_levenshtein(word1, word2)
41. 41
Examples
• The domain aaaaa.com has a Shannon Entropy score of 1.8 (very low)
• The domain google.com has a Shannon Entropy score of 2.6 (rather low)
• A00wlkj—(-a.aslkn-C.a.2.sk.esasdfasf1111)-890209uC.4.com has a Shannon
Entropy score of 3 (rather high)
Layman’s definition: a score reflecting the randomness or measure of
uncertainty of a string
Shannon Entropy
43. 43
Detecting Data Exfiltration
RESULTS
• Exfiltrating data requires many DNS requests – look for high counts
• DNS exfiltration to mooo.com and chickenkiller.com
44. 44
Summary: DNS exfiltration
Exfiltration by DNS and ICMP is a very common technique
Many organizations do not analyze DNS activity – do not be like them!
No DNS logs? No Splunk Stream? Look at FW byte counts
46. 46
Summary
Multiple phases to modern attacks
Deploy detection across all phases
Also consider adaptive response!
Stay abreast of modern advancements
App Export:
https://splunk.box.com/SplunkLive2016ScottsdaleSec
47. 47
SEPT 26-29, 2016
WALT DISNEY WORLD, ORLANDO
SWAN AND DOLPHIN RESORTS
• 5000+ IT & Business Professionals
• 3 days of technical content
• 165+ sessions
• 80+ Customer Speakers
• 35+ Apps in Splunk Apps Showcase
• 75+ Technology Partners
• 1:1 networking: Ask The Experts and Security
Experts, Birds of a Feather and Chalk Talks
• NEW hands-on labs!
• Expanded show floor, Dashboards Control
Room & Clinic, and MORE!
The 7th Annual Splunk Worldwide Users’ Conference
PLUS Splunk University
• Three days: Sept 24-26, 2016
• Get Splunk Certified for FREE!
• Get CPE credits for CISSP, CAP, SSCP
• Save thousands on Splunk education!
- Real world problems and data
- Tons of ways to do searches
- Context-specific adjustments
- Starting points
- Because real data, real noise to filter.
- Some quirks to the data
We’ll do a intro of Splunk for security.
We’ll then move pretty quickly to our hands-on exercise in three phases where we cover web attacks, lateral movement, and DNS exfil
The splunk examples will go from very simple, to very complex – all in the course of an hour
Afterwards you will have all of the material we have gone through, and the servers will stay up through tonight
If you require some additional time, we can discuss that after the talk
Splunk is software that collects machine-generated data from virtually any source. It indexes the data, aggregating it so you can search and query the data from one place in real-time:
Once your data is in Splunk you can:
Provide in-depth search capability across machine data and mash ups of machine / structured data
Create ad hoc reports to identify trends or understand customer behavior.
Build real-time role-based dashboards to monitor for KPI across business in real-time.
Analyze user transactions, customer behavior, machine behavior, security threats and fraudulent activity.
The Splunk platform consists of multiple products and deployment models to fit your needs.
Splunk Enterprise – for on-premise deployment
Splunk Cloud – Fully managed service with 100% SLA and all the capabilities of Splunk Enterprise…in the Cloud
Splunk Light – log search and analytics for small IT environments
Hunk – for analytics on data in Hadoop
The products can pull in data from virtually any source to support multiple use cases.
Splunk Apps extend and simplify deployments by providing pre-packaged content designed for specific use cases and data types.
Splunk excels at creating a data fabric
Machine data: Anything with a timestamp, regardless of incoming format.
Throw it all in there!
Collect it. Store it in one place. Make it accessible for search/analytics/reporting/alerting.
DETECTION NOT PREVENTION! ASSUME BREACH!
So we need a place we can go to DETECT attacks. DETECT breaches. DETECT the “weird.”
So if you had a place to see “everything” that happened…
….what would that mean for your SOC and IR teams?
Our rapid ascent reflects the customer traction we have and value we deliver to customers – with thousands of security customers and 40% year-over-year growth, we are the fastest growing SIEM vendor in the market. 2011 was our first time in the MQ; In 2 short years we raced up to the top quadrant in the MQ.
- Real world problems and data
- Tons of ways to do searches
- Context-specific adjustments
- Starting points
- Because real data, real noise to filter.
- Some quirks to the data
The goal of the OWASP (Open Web Application Security Project ) Top Ten is to raise awareness about application security by identifying some of the most critical risks facing organizations today. This list is refreshed every three years – we’re currently waiting on the 2016 one. These were the top 10 through 2 for 2013. What is #1?
These are examples of injections flaws that occur when untrusted data is sent to an interpreter as part of a command or query.
The attacker essentially tricks the interpreter into executing unintended commands or bypassing authorization/authentication.
Most of us are familiar with SQL injection attacks, the risks they pose to our environments and maybe even the inner workings of the threat itself, because they have been around for about 10 years. However, the attack types can vary and mitigating them efficiently requires an understanding of the basic types of SQL injection attacks.
SQL injection has been around so long…
…so long, that this is what Google looked like when SQL injection first became a topic of discussion: 1998. With 18 years of ways to prevent SQLi, you would think that this isn’t a problem anymore, right?
Wrong. Check out this survey conducted by Imperva – ostensibly to sell their web application firewalls, but still. This survey was just conducted last year…we can see that
(MOAR HERE)
A SQL injection needs just two conditions to exist – a relational database that uses SQL, and a user controllable input which is directly used in an SQL query.
An attacker would get into your organization though unsanitized user input.
Instead of entering a “real” username or password, the hacker will enter a SQL command (or a JSON string in the case of a NoSQL database). As a result, the malicious code, the example SQL query shown here, is then sent to the database.
Cybercriminals use injection attacks against databases to export data, to delete accounts, create bogus accounts and modify data. Injection attacks can even be used to initiate a Denial of Service (DOS) attack.
There have been some notable SQL injection attacks in our recent history that have exfiltrated entire user databases, prompting system-wide password resets, such as…
Here’s two big SQLi injection breaches that happened last year TalkTalk and vTech – resulting in the theft of about 10M records between the two of them – a lot of them in the vTech breach were kids – including names, genders, birthdates, and addresses.
A little bit about our environment
Everything I’m going through up here has been pretty well documented in a word doc. You can use the link here to get that doc, or if you’re really interested in it later come see me. You won’t need it right now though.
Each of you has creds – there are 10 fairly large Amazon EC2 instances that have been provisioned for this exercise and if we’re at capacity there will be 12 of you on each. Now’s a good time to try hitting that URL and logging into Splunk.
The answer is Splunk apps! Apps come with pre-built content, like dashboards, reports, alerts and workflows that anyone can leverage as a starting point or template and adapt them accordingly to their environment. We have a community of 11,000+ Splunk customers and this is one of the 1,000+ apps on SplunkBase (stats as of Feb 2016).
New Splunk users often try to reinvent the wheel, when it turns out it is already there. In this case, a Splunk engineer wrote an app that searches for SQLI, and then published it on SplunkBase. He provides a broader regex that we are going to leverage for our next example.
In that app one of the things provided is a regular expression pre-created for you to find common patterns in your data that could indicate a SQL injection attack. So all we need to do is call the macro when we do the search.
Everything I’m going through up here has been pretty well documented in a word doc. You can use the link here to get that doc, or if you’re really interested in it later come see me. You won’t need it right now though.
Each of you has creds – there are 10 fairly large Amazon EC2 instances that have been provisioned for this exercise and if we’re at capacity there will be 12 of you on each. Now’s a good time to try hitting that URL and logging into Splunk.
If you’d like, you can take a look at the dashboards in the SQL injection app on your own time.
In summary, why does SQL injection attacks continue to be one of the most common? BECAUSE IT STILL WORK!!
Don’t waste your previous cycles to reinvent the wheel, search for an existing app, or even provide one to the community.
While there are great Web Application Firewalls and similar products that will sit inline and detect or even block these attacks, you will still benefit from looking through all of your data for any signs of SQL injection, and Splunk can help.
If you have a specific question or use case that you want to run by us, come find us at the Splunk for Security table outside of Salon 9.
What if they use that to pivot around the environment to get to other more sensitive systems?
Pass the hash – dump the hash tables from a system, then use those hashes to authenticate to other systems and move around the environment or proliferate malicious code.
Guess what – this is pretty common – the Sands breach and Sony breaches from last year used PtH to move around the environment and in both cases distribute destructive malware. I mention these two because they were prominent and well researched.
It used to be that PtH was pretty easy to find – and in some cases it still is. Basically we just look for certain Windows events, and if the user account is not domain or anonymous, or specifies an unusual domain – that’s suspicious. Today, a not-very-sophisticated actor might show up this way if we look for them…so let’s see how we can do that.
Everything I’m going through up here has been pretty well documented in a word doc. You can use the link here to get that doc, or if you’re really interested in it later come see me. You won’t need it right now though.
Each of you has creds – there are 10 fairly large Amazon EC2 instances that have been provisioned for this exercise and if we’re at capacity there will be 12 of you on each. Now’s a good time to try hitting that URL and logging into Splunk.
If you look at some of the whitepapers and other links in the app you will see that the “old” ways of detecting PtH are not enough. Instead we really need to look at user and entity behavior and find unusual network connections and login patterns in our data. Again – splunk can help here…
Everything I’m going through up here has been pretty well documented in a word doc. You can use the link here to get that doc, or if you’re really interested in it later come see me. You won’t need it right now though.
Each of you has creds – there are 10 fairly large Amazon EC2 instances that have been provisioned for this exercise and if we’re at capacity there will be 12 of you on each. Now’s a good time to try hitting that URL and logging into Splunk.
Recalling back to the magical UBA solution that David just showed with the three high-level stages of the kill chain:
We started with an intrusion through the use of SQL injection, expansion through lateral movement, and now we will discuss exfiltration via DNS.
Here is a Dave‘s credentials in clear text; <CLICK>
And what it looks like encoded <CLICK>
An attacker could send that encoded string, containing Dave‘s credentials via a DNS request bound for his own DNS server.
By the way – there are other ways to exfil data, too – you can use TXT records and chunk data into that and tunnel it. Our examples here however will exfil data right in the domain name.
Raise your hand if you log your DNS data? Do you look at it?
The point is: DNS data tends to be too chatty to log and often overlooked. The same goes for an ICMP content body.
URL Toolbox can parse URLs and complicated top level domains by dissecting a domain into its parts, such as subdomain, domain name and top level domain. It can also perform functions like shannon entropy, counting, suites, meaning ratio, bayesian analysis, etc.
But our focus today will be on the use of Shannon Entropy.
The Shannon Entropy will give us a numeric representation for how much randomness there is in the letters / numbers used in a subdomain.
It is a good indication of Domain Generation Algorithms (DGA) or of data exfiltration because compressed and/or encrypted data will have very high entropy.
Let’s all go to the DNS Exfil menu and select Detecting Data Exfiltration from the dropdown.
Again, using pre-built content, we can parse out the query and domain components and calculate the Shannon Entropy for each subdomain. Since most data exfiltration occurs via subdomain, that’s where we will be focusing our search.
If you recall back to the tornado slide, the attacker used encoded credentials as a subdomain for the DNS request.
How many DNS requests would it take to exfiltrate a MB of data? How about a GB? This is why we look for high counts.
Here, an 18K text file is being exfiltrated to chickenkiller.com and a 20MB+ zip file is being exfiltrated to mooo.com.
In summary, don’t overlook the value in your DNS and ICMP data. If you don’t log it, you could consider grabbing the data right off the wire with Splunk Stream. If you don’t have logs or Stream, at a minimum, trend your firewall byte counts.
We’re headed to the East Coast!
2 inspired Keynotes – General Session and Security Keynote + Super Sessions with Splunk Leadership in Cloud, IT Ops, Security and Business Analytics!
165+ Breakout sessions addressing all areas and levels of Operational Intelligence – IT, Business Analytics, Mobile, Cloud, IoT, Security…and MORE!
30+ hours of invaluable networking time with industry thought leaders, technologists, and other Splunk Ninjas and Champions waiting to share their business wins with you!
Join the 50%+ of Fortune 100 companies who attended .conf2015 to get hands on with Splunk. You’ll be surrounded by thousands of other like-minded individuals who are ready to share exciting and cutting edge use cases and best practices. You can also deep dive on all things Splunk products together with your favorite Splunkers.
Head back to your company with both practical and inspired new uses for Splunk, ready to unlock the unimaginable power of your data! Arrive in Orlando a Splunk user, leave Orlando a Splunk Ninja!
REGISTRATION OPENS IN MARCH 2016 – STAY TUNED FOR NEWS ON OUR BEST REGISTRATION RATES – COMING SOON!