5. Machine data contains a definitive record
of all interactions
Splunk is a very effective platform to collect,
store, and analyze all of that data
Human Machine
Machine Machine
6. Mainframe
Data
VMware
Platform for Machine Data
Splunk Solutions > Easy to Adopt
Exchange PCISecurity
Relational
Databases
MobileForwarders
Syslog /
TCP / Other
Sensors &
Control Systems
Across Data Sources, Use Cases & Consumption Models
Wire
Data
IT Svc Int
Splunk Premium Solutions & Apps Rich Ecosystem of Apps
ITSI UBA
UBA
7. 7
Rapid Ascent in the Gartner SIEM Magic Quadrant*
*Gartner, Inc., SIEM Magic Quadrant 2011-2015. Gartner does not endorse any vendor, product or
service depicted in its research publication and not advise technology users to select only those
vendors with the highest ratings or other designation. Gartner research publications consist of the
opinions of Gartner’s research organization and should not be construed as statements of fact.
Gartner disclaims all warranties, express or implied, with respect to this research, including any
warranties of merchantability or fitness for a particular purpose.
2015 Leader and the only vendor to
improve its visionary position
2014 Leader
2013 Leader
2012 Challenger
2011 Niche Player
2015
10. 10
OWASP 2013 Top 10
[10] Unvalidated redirects and forwards
[9] Using components with known vulnerabilities
[8] Cross-site request forgery
[7] Missing function level access control
[6] Sensitive data exposure
[5] Security misconfiguration
[4] Insecure direct object reference
[3] Cross-site scripting (XSS)
[2] Broken authentication and session management
15. 15
The anatomy of a SQL injection attack
SELECT * FROM users WHERE email='xxx@xxx.com'
OR 1 = 1 -- ' AND password='xxx';
xxx@xxx.xxx' OR 1 = 1 -- '
xxx
admin@admin.sys
1234
An attacker might supply:
26. 26
What have we here?
Our learning environment consists of:
• 14 Publically-accessible single Splunk
servers
• Each with ~5.5M events, from real
environments but massaged:
• Windows Security events
• Apache web access logs
• Bro DNS & HTTP
• Palo Alto traffic logs
• Some other various bits
28. 28
https://splunkbase.splunk.com/app/1528/
Search for possible SQL injection in your events:
looks for patterns in URI query field to see if
anyone has injected them with SQL
statements
use standard deviations that are 2.5 times
greater than the average length of your URI
query field
Macros used
• sqlinjection_pattern(sourcetype, uri query field)
• sqlinjection_stats(sourcetype, uri query field)
29. 29
Regular Expression FTW
sqlinjection_rex is a search macro. It contains:
(?<injection>(?i)select.*?from|union.*?select|'$|delete.*?from|update.*?set|alter.*?table|([
%27|'](%20)*=(%20)*[%27|'])|w*[%27|']or)
Which means: In the string we are given, look for ANY of the following matches and put that
into the “injection” field.
• Anything containing SELECT followed by FROM
• Anything containing UNION followed by SELECT
• Anything with a ‘ at the end
• Anything containing DELETE followed by FROM
• Anything containing UPDATE followed by SET
• Anything containing ALTER followed by TABLE
• A %27 OR a ‘ and then a %20 and any amount of characters then a %20 and then a %27 OR a ‘
• Note: %27 is encoded “’” and %20 is encoded <space>
• Any amount of word characters followed by a %27 OR a ‘ and then “or”
32. 32
Summary: Web attacks/SQL injection
SQL injection provide attackers with easy access to data
Detecting advanced SQL injection is hard – use an app!
Understand where SQLi is happening on your network and put a
stop to it.
Augment your WAF with enterprise-wide Splunk searches.
37. 37
This and other techniques used in destructive Sands breach…
….and at Sony, too.
38. 38
Detecting Legacy PtH
Look for Windows Events:
Event ID: 4624 or 4625
Logon type: 3
Auth package: NTLM
User account is not a domain logon, or Anonymous Logon
…this is trivially easy in Splunk.
40. 40
Then it got harder
• Pass the Hash tools have
improved
• Adjusting of jitter, other
metrics
• So let’s detect lateral
movement differently – just
looking at Windows event
codes isn’t enough.
41. 41
Network traffic provides source of truth
I usually talk to 10 hosts
Then one day I talk to 10,000 hosts
ALARM!
48. 48
DNS exfil tends to be
overlooked within an
ocean of DNS data.
Let’s fix that!
DNS exfiltration
49. 49
FrameworkPOS: a card-stealing program that exfiltrates data from the
target’s network by transmitting it as domain name system (DNS) traffic
But the big difference is the way how stolen data is
exfiltrated: the malware used DNS requests!
https://blog.gdatasoftware.com/2014/10/23942-new-frameworkpos-
variant-exfiltrates-data-via-dns-requests
“
”
… few organizations actually keep detailed logs or records
of the DNS traffic traversing their networks — making it an
ideal way to siphon data from a hacked network.
http://krebsonsecurity.com/2015/05/deconstructing-the-2014-sally-
beauty-breach/#more-30872
“
”
DNS exfiltration
50. 50
https://splunkbase.splunk.com/app/2734/
DNS exfil detection – tricks of the trade
parse URLs & complicated TLDs (Top Level Domain)
calculate Shannon Entropy
List of provided lookups
• ut_parse_simple(url)
• ut_parse(url, list) or ut_parse_extended(url, list)
• ut_shannon(word)
• ut_countset(word, set)
• ut_suites(word, sets)
• ut_meaning(word)
• ut_bayesian(word)
• ut_levenshtein(word1, word2)
51. 51
Examples
• The domain aaaaa.com has a Shannon Entropy score of 1.8 (very low)
• The domain google.com has a Shannon Entropy score of 2.6 (rather low)
• A00wlkj—(-a.aslkn-C.a.2.sk.esasdfasf1111)-890209uC.4.com has a Shannon
Entropy score of 3 (rather high)
Layman’s definition: a score reflecting the randomness or measure of
uncertainty of a string
Shannon Entropy
54. 54
Detecting Data Exfiltration
… | stats
count
avg(ut_shannon) as avg_sha
avg(sublen) as avg_sublen
stdev(sublen) as stdev_sublen
by ut_domain
| search avg_sha>3 avg_sublen>20
stdev_sublen<2
TIPS
Leverage our Bro DNS data
Calculate Shannon Entropy scores
Calculate subdomain length
Display count, scores, lengths,
deviations
55. 55
Detecting Data Exfiltration
RESULTS
• Exfiltrating data requires many DNS requests – look for high counts
• DNS exfiltration to mooo.com and chickenkiller.com
56. 56
Summary: DNS exfiltration
Exfiltration by DNS and ICMP is a very common technique
Many organizations do not analyze DNS activity – do not be like them!
No DNS logs? No Splunk Stream? Look at FW byte counts
58. 58
Summary
Multiple phases to modern attacks
Deploy detection across all phases
Also consider adaptive response!
Stay abreast of modern advancements
App Export:
https://splunk.box.com/v/SplunkLive2016SeattleSec
59. 59
SEPT 26-29, 2016
WALT DISNEY WORLD, ORLANDO
SWAN AND DOLPHIN RESORTS
• 5000+ IT & Business Professionals
• 3 days of technical content
• 165+ sessions
• 80+ Customer Speakers
• 35+ Apps in Splunk Apps Showcase
• 75+ Technology Partners
• 1:1 networking: Ask The Experts and Security
Experts, Birds of a Feather and Chalk Talks
• NEW hands-on labs!
• Expanded show floor, Dashboards Control
Room & Clinic, and MORE!
The 7th Annual Splunk Worldwide Users’ Conference
PLUS Splunk University
• Three days: Sept 24-26, 2016
• Get Splunk Certified for FREE!
• Get CPE credits for CISSP, CAP, SSCP
• Save thousands on Splunk education!
- Real world problems and data
- Tons of ways to do searches
- Context-specific adjustments
- Starting points
- Because real data, real noise to filter.
- Some quirks to the data
- Real world problems and data
- Tons of ways to do searches
- Context-specific adjustments
- Starting points
- Because real data, real noise to filter.
- Some quirks to the data
We’ll do a intro of Splunk for security.
We’ll then move pretty quickly to our hands-on exercise in three phases where we cover web attacks, lateral movement, and DNS exfil
The splunk examples will go from very simple, to very complex – all in the course of an hour
Afterwards you will have all of the material we have gone through, and the servers will stay up through tonight
If you require some additional time, we can discuss that after the talk
Splunk excels at creating a data fabric
Machine data: Anything with a timestamp, regardless of incoming format.
Throw it all in there!
Collect it. Store it in one place. Make it accessible for search/analytics/reporting/alerting.
DETECTION NOT PREVENTION! ASSUME BREACH!
So we need a place we can go to DETECT attacks. DETECT breaches. DETECT the “weird.”
So if you had a place to see “everything” that happened…
….what would that mean for your SOC and IR teams?
The Splunk platform consists of multiple products and deployment models to fit your needs.
Splunk Enterprise – for on-premise deployment
Splunk Cloud – Fully managed service with 100% SLA and all the capabilities of Splunk Enterprise…in the Cloud
Splunk Light – log search and analytics for small IT environments
Hunk – for analytics on data in Hadoop
The products can pull in data from virtually any source to support multiple use cases.
Splunk Apps extend and simplify deployments by providing pre-packaged content designed for specific use cases and data types.
Our rapid ascent reflects the customer traction we have and value we deliver to customers – with thousands of security customers and 40% year-over-year growth, we are the fastest growing SIEM vendor in the market. 2011 was our first time in the MQ; In 2 short years we raced up to the top quadrant in the MQ.
- Real world problems and data
- Tons of ways to do searches
- Context-specific adjustments
- Starting points
- Because real data, real noise to filter.
- Some quirks to the data
The goal of the OWASP (Open Web Application Security Project ) Top Ten is to raise awareness about application security by identifying some of the most critical risks facing organizations today. This list is refreshed every three years – we’re currently waiting on the 2016 one. These were the top 10 through 2 for 2013. What is #1?
These are examples of injections flaws that occur when untrusted data is sent to an interpreter as part of a command or query.
The attacker essentially tricks the interpreter into executing unintended commands or bypassing authorization/authentication.
Most of us are familiar with SQL injection attacks, the risks they pose to our environments and maybe even the inner workings of the threat itself, because they have been around for about 10 years. However, the attack types can vary and mitigating them efficiently requires an understanding of the basic types of SQL injection attacks.
SQL injection has been around so long…
…so long, that this is what Google looked like when SQL injection first became a topic of discussion: 1998. With 18 years of ways to prevent SQLi, you would think that this isn’t a problem anymore, right?
Wrong. Check out this survey conducted by Imperva – ostensibly to sell their web application firewalls, but still. This survey was just conducted last year…we can see that SQL injection attempts appear to happen in concentrated bursts, perhaps the result of specific campaigns. Regardless of when they happen, SQL injection attacks remain some of the most damaging to organizations in terms of records compromised and lost value.
Attribution: http://www.imperva.com/docs/HII_Web_Application_Attack_Report_Ed6.pdf
In fact, you could even say that no matter how much technologies change, injection vulnerabilities seem to remain! Last year’s SANS holiday hack challenge prominently featured injection vulnerabilities in server-side JS framework Node.JS and popular nosql database MongoDB. If you like today’s exercise, you will love the Holiday Hack Challenge.
Attribution: https://quest.holidayhackchallenge.com/ and https://holidayhackchallenge.com/
A SQL injection needs just two conditions to exist – a relational database that uses SQL, and a user controllable input which is directly used in an SQL query.
An attacker would get into your organization though unsanitized user input.
Instead of entering a “real” username or password, the hacker will enter a SQL command (or a JSON string in the case of a NoSQL database). As a result, the malicious code, the example SQL query shown here, is then sent to the database.
Cybercriminals use injection attacks against databases to export data, to delete accounts, create bogus accounts and modify data. Injection attacks can even be used to initiate a Denial of Service (DOS) attack.
There have been some notable SQL injection attacks in our recent history that have exfiltrated entire user databases, prompting system-wide password resets, such as…
Here’s two big SQLi injection breaches that happened last year TalkTalk and vTech – resulting in the theft of about 10M records between the two of them – a lot of them in the vTech breach were kids – including names, genders, birthdates, and addresses.
And so far this year – we’ve had at least 9 major breaches covered in the media that involved SQLi. You can just google “SQL injection hall of fame” to pull this up.
Attribution: The Code Curmudgeon Blog http://codecurmudgeon.com/wp/sql-injection-hall-of-shame/
We’re going to do some hands-on here, but before we do, let me set the stage a bit. How many of you have had your hands on Splunk before? OK – for the purposes of this discussion, you all are ninjas.
For the newbies – or for anyone who wants a little more help – the app we’re going to use this afternoon is very self-guided. You can literally just read along and see the search examples without having to type any Splunk searches.
This is what that looks like – the search is shown, and the results below.
You will notice that in the app there’s this “step by step” button. Don’t press it – what it does is it allows you to step through the search line by line – great for learning, but not great for this shared environment where you’re all hitting the same servers.
This is what that step by step looks like. You can use it later on your own time.
Now if you are a ninja you can simply cut and paste the searches into the Search bar within the app – that’s what I will do up here.
So what I will be doing is mousing over the search…
…and pasting it in to the search bar.
A little bit about our environment
The answer is Splunk apps! Apps come with pre-built content, like dashboards, reports, alerts and workflows that anyone can leverage as a starting point or template and adapt them accordingly to their environment. We have a community of 11,000+ Splunk customers and this is one of the 1,000+ apps on SplunkBase (stats as of Feb 2016).
New Splunk users often try to reinvent the wheel, when it turns out it is already there. In this case, a Splunk engineer wrote an app that searches for SQLI, and then published it on SplunkBase. He provides a broader regex that we are going to leverage for our next example.
In that app one of the things provided is a regular expression pre-created for you to find common patterns in your data that could indicate a SQL injection attack. So all we need to do is call the macro when we do the search.
If you’d like, you can take a look at the dashboards in the SQL injection app on your own time.
In summary, why does SQL injection attacks continue to be one of the most common? BECAUSE IT STILL WORK!!
Don’t waste your previous cycles to reinvent the wheel, search for an existing app, or even provide one to the community.
While there are great Web Application Firewalls and similar products that will sit inline and detect or even block these attacks, you will still benefit from looking through all of your data for any signs of SQL injection, and Splunk can help.
If you have a specific question or use case that you want to run by us, come find us at the Splunk for Security table outside of Salon 9.
What if they use that to pivot around the environment to get to other more sensitive systems?
Pass the hash – dump the hash tables from a system, then use those hashes to authenticate to other systems and move around the environment or proliferate malicious code.
Guess what – this is pretty common – the Sands breach and Sony breaches from last year used PtH to move around the environment and in both cases distribute destructive malware. I mention these two because they were prominent and well researched.
It used to be that PtH was pretty easy to find – and in some cases it still is. Basically we just look for certain Windows events, and if the user account is not domain or anonymous, or specifies an unusual domain – that’s suspicious. Today, a not-very-sophisticated actor might show up this way if we look for them…so let’s see how we can do that.
If you look at some of the whitepapers and other links in the app you will see that the “old” ways of detecting PtH are not enough. Instead we really need to look at user and entity behavior and find unusual network connections and login patterns in our data. Again – splunk can help here…
Recalling back to the magical UBA solution that David just showed with the three high-level stages of the kill chain:
We started with an intrusion through the use of SQL injection, expansion through lateral movement, and now we will discuss exfiltration via DNS.
Here is a Dave‘s credentials in clear text; <CLICK>
And what it looks like encoded <CLICK>
An attacker could send that encoded string, containing Dave‘s credentials via a DNS request bound for his own DNS server.
By the way – there are other ways to exfil data, too – you can use TXT records and chunk data into that and tunnel it. Our examples here however will exfil data right in the domain name.
Raise your hand if you log your DNS data? Do you look at it?
The point is: DNS data tends to be too chatty to log and often overlooked. The same goes for an ICMP content body.
There are a couple of retail breaches that we have called out here.
We will be posting all session slides to Slideshare, so you can view these articles at your leisure.
FrameworkPOS is a packaged tool that hackers can buy and put into their hacking arsenal along with dnscat.
Just like the hacker community have their tools, our Splunk community does too!
URL Toolbox can parse URLs and complicated top level domains by dissecting a domain into its parts, such as subdomain, domain name and top level domain. It can also perform functions like shannon entropy, counting, suites, meaning ratio, bayesian analysis, etc.
But our focus today will be on the use of Shannon Entropy.
The Shannon Entropy will give us a numeric representation for how much randomness there is in the letters / numbers used in a subdomain.
It is a good indication of Domain Generation Algorithms (DGA) or of data exfiltration because compressed and/or encrypted data will have very high entropy.
Let’s all go to the DNS Exfil menu and select Detecting Data Exfiltration from the dropdown.
Again, using pre-built content, we can parse out the query and domain components and calculate the Shannon Entropy for each subdomain. Since most data exfiltration occurs via subdomain, that’s where we will be focusing our search.
If you recall back to the tornado slide, the attacker used encoded credentials as a subdomain for the DNS request.
Large volumes of data exfiltration tend to have relatively similar data length because it is chunking big segments of data. A low standard deviation will help indicate that this is large volume data transfer.
How many DNS requests would it take to exfiltrate a MB of data? How about a GB? This is why we look for high counts.
Here, an 18K text file is being exfiltrated to chickenkiller.com and a 20MB+ zip file is being exfiltrated to mooo.com.
In summary, don’t overlook the value in your DNS and ICMP data. If you don’t log it, you could consider grabbing the data right off the wire with Splunk Stream. If you don’t have logs or Stream, at a minimum, trend your firewall byte counts.
We’re headed to the East Coast!
2 inspired Keynotes – General Session and Security Keynote + Super Sessions with Splunk Leadership in Cloud, IT Ops, Security and Business Analytics!
165+ Breakout sessions addressing all areas and levels of Operational Intelligence – IT, Business Analytics, Mobile, Cloud, IoT, Security…and MORE!
30+ hours of invaluable networking time with industry thought leaders, technologists, and other Splunk Ninjas and Champions waiting to share their business wins with you!
Join the 50%+ of Fortune 100 companies who attended .conf2015 to get hands on with Splunk. You’ll be surrounded by thousands of other like-minded individuals who are ready to share exciting and cutting edge use cases and best practices. You can also deep dive on all things Splunk products together with your favorite Splunkers.
Head back to your company with both practical and inspired new uses for Splunk, ready to unlock the unimaginable power of your data! Arrive in Orlando a Splunk user, leave Orlando a Splunk Ninja!
REGISTRATION OPENS IN MARCH 2016 – STAY TUNED FOR NEWS ON OUR BEST REGISTRATION RATES – COMING SOON!