This presentation was delivered at BSides Augusta in September 2016. The A/V portion is available here: https://www.youtube.com/watch?v=i6p71t9PFWM
Abstract:
"We can all agree that threat ("Evil") detection is an essential component of a functioning security monitoring program. Let's start thinking about how to take our tradecraft to the next level and hunt for insecure conditions ("Ways for Evil to do Evil things") that might allow threat actors to succeed in their mission.
This talk will run through some of the observations gathered during hunting expeditions inside the networks of multiple Fortune-ranked organizations and challenge you to expand your security operations thinking beyond signature-based detection.
- What is Hunting?
- How have we done it?
- What have we found, and what should be done about those findings?
- How might you achieve similar outcomes in your own environment?"
Speakers:
- Jacqueline Stokes (@find_evil) is an infosec enthusiast who picked up hacking as a preteen and cut her teeth over multiple years in Iraq. Her ongoing mission is to assess and advise clients on the most actionable and forward-thinking methods to improve detection, response, and containment of advanced threats. Jackie likes long walks on the beach, 90's nostalgia, and is the president and founding member of the Kevin Mandia Fan Club.
- Danny Akacki (@dakacki) was a Lead Analyst with GE Capitals' Applied Intelligence team prior to his employment with Mandiant, and now works for Bank of America's hunt team. He is a pragmatic optimist and believes we are probably screwed, but hopes we aren't. Danny enjoys finding evil on the weekends.
- Stephen Hinck (@stephenhinck) is a Senior Security Analyst at Oracle, Inc. Stephen stumbled into the information security world years ago and has since only managed to dig his way deeper to the rabbit hole. With a background in security operations, incident response and threat hunting, Stephen's experience is multi-faceted. Although he enjoys many things, he absolutely hates writing silly bios like this one.
2. Who We Are
Hunting: Defense Against The Dark Arts 2
• Jackie Stokes ....................................... @find_evil
• Danny Akacki ....................................... @dakacki
• Stephen Hinck ...................................... @stephenhinck
3. Hunting: Defense Against The Dark Arts 3
Problem Set
• Finding Evil
• Ways for Evil to do Evil Things
• Leverage data we already have / can readily obtain
• Drive maturation of monitoring & detection capabilities
4. HUNT
Drive continuous improvement
Identify opportunities for action
Use internal and external data to
of the Information Security program
Solution: Threat Hunting
Hunting: Defense Against The Dark Arts 4
5. Hunting: Defense Against The Dark Arts 5
Hunting is a collection of processes
Not
❌ Tools
❌ Alerts
❌ Automation
6. Building a Hunt Program
Hunting: Defense Against The Dark Arts 6
"Understanding is the first step to
acceptance, and only with
acceptance can there be
recovery."
— Albus Dumbledore
7. Hunting Program
Mature detection capabilities
Use Cases + Playbooks
Guiding processes for SOC / CIRT
Technology & Tools
Operationally-driven and requirements-based
SOC + CIRT
Security operations and incident response
Formalized Security Program
Chartered and backed by an executive sponsor
Hunting: Defense Against The Dark Arts 7
Hunting Capability Pyramid
Must be this
tall to ride
8. Hunting: Defense Against The Dark Arts 8
http://blog.sqrrl.com/the-cyber-hunting-maturity-model
Hunting Maturity Model
9. Building a Hunt Program
Hunting: Defense Against The Dark Arts 9
1. Establish executive sponsorship and mission charter/objectives
2. Establish and implement enterprise logging strategy
3. Aggregate, centralize, and process data
4. Make data available within searchable (fast) interface
5. Drive maturity
• Develop use cases
• Are we getting the right data?
• Review tooling and associated requirements
• Reintegrate hunt mission data to security operations
10. Hunting + IR Detection Maturation
Hunting: Defense Against The Dark Arts 10
HUNT SOC DETECT
IR USE CASE
Ongoing hunt
missions
Feed Incident
Response
activities
IR outcomes
affect
SecOps
Lessons
Learned
incorporated
to SecOps
Detection
capability
improvement
Evil
Non-Evil Risk
11. Hunt Mission Outcomes
Hunting: Defense Against The Dark Arts 11
•Benefit: Activity shown not to be present
•Next Step: Evaluate hunt mission effectiveness
No Detection
•Benefits: Activity shown to be present
Hunt mission effectiveness validated
Identify best practice / compliance issues
•Next Step: Escalate as appropriate, monitor to closure
Detection:
Non-Malicious
•Benefits: Activity shown to be present
Hunt mission effectiveness validated
Identify security incidents
•Next Step: Escalate as appropriate, monitor to closure
Detection:
Malicious
12. Sorting Out Your Data
Hunting: Defense Against The Dark Arts 12
"Not Slytherin, eh? Are you sure? You could be great, you know."
13. Data Sources
- Remote Access
- Web Proxy
- IDS / IPS
- Email
- WAF
- DNS
- DHCP
- NetFlow
- Firewall
- Router / Switch
- Wireless Infrastructure
- Agents
- Antivirus
- Operating Systems
- Active Directory
- File, Print, Database
- Other Services
External Feeds
- Paid, Free, OSINT
Internal Feeds
- Recon data
- IR Lessons Learned
- Critical Asset
Inventory
- Privilege
Management
- Approved Service
Interruptions
- Terminated Users
- Acceptable Use Policy
- Employee Work Hours
- Physical Access Data
Security
Network
Endpoint
IT
Threat
Intel
HR
Hunting: Defense Against The Dark Arts 13
14. Two Types of Events
Hunting: Defense Against The Dark Arts 14
1. Observed Originated from a device which handled the event in some way
2. Synthetic Generated through automated analysis of event data
15. What is the Right Data?
Hunting: Defense Against The Dark Arts 15
• Original source data where-ever possible
• Ensure the presence of important fields
• Generally, observed events > synthetic events
• Synthetic events can provide useful context in the form of analytics
• Logs must enable pivoting
• Minimum one extractable / consistent data point to correlate log sources
16. Ready the Spells!
Hunting: Defense Against The Dark Arts 16
• Understand the network
• Learn critical assets
• Develop enterprise logging strategy
• Ensure data sources use consistent time settings; implement NTP, use GMT
• Plug in to asset, change, and configuration management processes
• Account for other organizational use cases
• IT Operations
• Forensics / Incident Response
• Compliance / Audit
• Clean up the dataset
• Normalization
• De-duplication
• Parsing
• Enrich and contextualize the dataset...!
17. Event Enrichment
Hunting: Defense Against The Dark Arts 17
• Internally-sourced Intelligence
• Attack trees
• Red Team / Penetration test output
• TTPs from previous incidents
• Deviances from baselines / Expected behavior
• Organizational risk profile / Threat context
• Externally-sourced Intelligence
• Paid subscriptions
• OSINT
• Free feeds
• Passive DNS, WHOIS, etc.
• Geographical data
• ISAC, Infragard, etc.
• Context
• Environmental
• Refer to "Data Source" slide
• Previous hunt and IR output
• Malware analysis
• Analytics, Ex.
• Geo-infeasibility
• Beacon detection
• DNS entropy
• Data exfiltration
18. Tools of the Trade
Hunting: Defense Against The Dark Arts 18
"It is important to fight, and fight again, and keep
fighting, for only then could evil be kept at bay,
though never quite eradicated"
— Albus Dumbledore
19. Criteria for a Working Hunt Platform
Hunting: Defense Against The Dark Arts 19
• Rapid search with high quality UI and / or API
• Stacking
• Group and reduce the dataset to more easily identify outliers
• Make manual analysis of an entire environment feasible
• Pivoting
• Move laterally through the dataset
• See the whole picture
Is It Worth It? Let Me Work It
• Tagging and Enrichments
• Intelligence Integration Support
• Automation: Rules & Alerting
• Evaluation Success Criteria
• Totally sweet dance moves
20. All About The Galleons
Hunting: Defense Against The Dark Arts 20
• Budget
• Driven by Operational Requirements
• Tool/Vendor Selection Process
• Multiple Tools: Diverse Perspectives
• Free and Open Source Software!
• NXLog
• Sysmon
• Moloch
• Wireshark
• Bro Network Security Monitor
• ELK (ElasticSearch, Logstash, Kibana)
• Security Onion Linux Distribution– Da Real MVP
+ A bunch of other stuff we didn't list here...
23. Sample Hypotheses to Drive Hunt Missions
Hunting: Defense Against The Dark Arts 23
1. Sensitive corporate data stored
only in approved locations
2. Large or extended outbound data
transfers meet business needs
3. Reconnaissance activities
against DMZ hosts provide
advance warning of pending
malicious activity
4. VPN logins by users are
geographically feasible
5. Domain controller baselines are
simple and deviations rarely
occur
6. Service credentials are used only
in expected ways and for their
appropriate services
7. Web proxies are appropriately
configured to block suspicious
traffic
8. Our services communicate
using secure, encrypted
protocols
9. Tunneling HTTP traffic and other
proxy avoidance techniques are
not allowed in or out of our
network
10. The use of management tools
(such as PSExec) occurs only
within approved change
windows
11. Endpoints are not added to the
network without infosec visibility
24. More Data, More Problems
Hunting: Defense Against The Dark Arts 24
"Dobby is... free."
— Dobby the House Elf
26. 1. Remote Access
Hunting: Defense Against The Dark Arts 26
Hypothesis: Remote access to our environment is conducted using approved means
Discovery:
• Remote access is occurring over multiple protocols to / from unapproved hosts
• VNC to / from production network
• RDP to domain controllers from DMZ
• Evidence of unapproved remote access utilities such as LogMeIn, GoToMyPC, etc
Recommendation:
• Evaluate unapproved connections for mitigation or for risk acceptance
• Ensure that risk accepted software is fully patched and up to date
• Implement strong encryption, jump boxes / VPN ACLs, and two-factor
authentication where possible
27. 2. Data Storage
Hunting: Defense Against The Dark Arts 27
Hypothesis: Corporate data is only stored in approved locations
Discovery:
• Sensitive corporate data stored on unencrypted and infected external media
• Unrestricted use of common cloud data storage providers
• Unmanaged source code repositories (intellectual property)
Recommendation:
• Evaluate DLP implementation and allowed web proxy categories
• Consider establishing formalized agreement with a cloud storage provider
• Bring unmanaged data stores under management in support of development
teams
28. 3. Proxy Infrastructure
Hunting: Defense Against The Dark Arts 28
Hypothesis: Our proxy infrastructure is properly configured
Discovery:
• Not blocking known malicious categories
• Not blocking executable downloads
• Proxies not logging all necessary protocol metadata
• Ex. User Agent, Status Code, Byte Counts, X-Forward-For, etc.
Recommendation:
• Validate security operations' requirements of proxy infrastructure
• Re-evaluate proxy configurations for appropriate changes
• Ensure security operations are looped in to the change management process
29. 4. Approved Protocols
Hunting: Defense Against The Dark Arts 29
Hypothesis: Protocols transiting our network are secure and approved for use
Discovery:
• Various insecure protocols identified in use across the network
• Unencrypted: Telnet, FTP
• Deprecated: SNMP v2, cleartext SMTP
• Risky: IRC, TOR / i2p
Recommendation:
• Identify opportunities to deploy secured versions of protocols
• FTP SFTP
• Telnet SSH
• SNMP v2 SNMP v3, etc.
• Evaluate implementation of risk detection and mitigation strategies
30. 5. Approved Clients
Hunting: Defense Against The Dark Arts 30
Hypothesis: Internet access is achieved using known and approved client software
Discovery:
• Suspicious user-agents identified indicating potential latent infections
• Extremely out of date software, including client browsers, Flash, and Java
Recommendation:
• Begin incident response procedures to evaluate and triage endpoints
• Evaluate consistency of patch and vulnerability management processes
31. 6. Privilege Management
Hunting: Defense Against The Dark Arts 31
Hypothesis: Account management is rooted in best practice
Discovery:
• Service accounts used for unrelated purposes or shared by users
• Regular and privileged users with non-specific accounts
• Direct privileged logins without approved privilege escalation process (e.g. sudo)
• Suspicious usernames that do not conform to the organizational standard
• User account belonging to terminated user active on the network
Recommendation:
• Evaluate suspicious or ambiguous accounts for mitigation or for risk acceptance
• Ensure security operations are tied into the HR termination workflow
• Update organizational username standard and privilege management processes
32. 7. Security Architecture
Hunting: Defense Against The Dark Arts 32
Hypothesis: Event logs provide information needed to validate control effectiveness
Discovery:
• Non-security specific appliances with disabled security functionality
• Ex. Cisco ASA scan detection disabled
• Security specific appliances improperly placed
• Bro NSM placed post-proxy, post-NAT
Recommendation:
• Evaluate IT systems for security value (non-traditional security appliances)
• Ex. Network devices
• Modify configuration and placement of systems to meet requirements
33. 8. Process Execution
Hunting: Defense Against The Dark Arts 33
Hypothesis: Endpoints only execute processes required for business functions
Discovery:
• Obfuscated PowerShell execution
• Mimikatz and other persistence toolkit execution
• Suspicious filenames/paths/registry entries, etc.
• Users installing browser toolbars and miscellaneous adware/spyware
Recommendation:
• Call the IR Team
• Adjust detections / controls to rapidly detect and prevent future occurrences
34. 9. DNS
Hunting: Defense Against The Dark Arts 34
Hypothesis: DNS resolutions occur within the bounds of best practices
Discovery:
• "Weird" protocol deviations/padded packets suggesting exfil or C&C
• Uncontrolled resolutions that are not forced through corporate infrastructure
• Resolutions for unusual or risky domains
• Ex. Dynamic DNS domains, domains appearing to be algorithmically generated
• Initial resolutions for suspicious domains + subsequent unusual communication
Recommendation:
• Harden organizational DNS infrastructure
• Ex. Implement DNSSEC, prevent zone transfers, etc.
• Configure perimeter devices to only accept DNS requests from corporate DNS
• Implement protocol anomaly detection to identify protocol misuse
35. Thinking Ahead
Hunting: Defense Against The Dark Arts 35
"The one with the
power to vanquish
the Dark Lord
approaches..."
— Sybill Trelawney
36. Ensuring Successful Outcomes
Hunting: Defense Against The Dark Arts 36
• Goals
• Reduce attack surface
• Harden the environment
• Improve detection and monitoring
• Don't bother hunting without using the outputs!
• Lessons Learned / AAR
• Feedback loop on IR processes
• Create new or improve existing detections
• Metrics
• Cannot improve what is not measured
• The absence of something is still something
• Most metrics will trend upwards before they come down
• 'Time to Detect' and other metrics will trend downward over time
37. Hunt Methodology: From Art to Science
Hunting: Defense Against The Dark Arts 37
Begin evolution from an intuitive art form to a structured science
39. Resources
Hunting: Defense Against The Dark Arts 39
FireEye Threat Analytics Platform: Hunting at Scale
https://www.fireeye.com/products/threat-analytics-platform.html
Sqrrl: Thought leadership in the hunting space
http://blog.sqrrl.com
The Threat Hunting Project: Compendium of useful resources
http://www.threathunting.net
Loggly: Helpful logging guidelines
https://www.loggly.com/intro-to-log-management
Security Onion: Peel back the layers of your network
https://securityonion.net
Notas do Editor
Intros in order
Jackie
While gaining maximum value from:
Onetime assessments
Blue team engagements
Ongoing security operations
Stephen
How do we address the problem set? We can layer a hunt methodology as part of our detection strategy.
Using a less structured methodology than the traditional alert review process can provide opportunities to identify evil and ways for evil to do evil things outside of pre-existing definitions or signature-based rulesets
- Collective name for any manual or machine-assisted techniques used to detect security incidents
- Innovative and effective security monitoring and detection methodology
- Iterative, hypotheses-driven process
Danny
We want to be clear what hunting isn’t – It isn’t investigating alerts, or using a particular tool. You can’t throw a tool at this probem set.
Hunting is a set of methodologies for analyzing large datasets in search of incidents that can fuel future automated detections.
Find Incidents Find new ways of finding incidents.
Jackie
Stephen
Hunting Program
Development of mature detection capabilities
Use Cases + Playbooks
Guiding processes for SOC / CIRT
Technology & Tools
Operationally-driven and requirements-based
SOC + CIRT
Security operations and incident response
Formalized Security Program
Chartered and backed by an executive sponsor
Danny
- Ask the audience after describing the levels – how many of you feel your organization is at a level $level?
HM0 - Initial
At HM0, an organization relies primarily on automated alerting tools such as IDS, SIEM or antivirus to detect malicious activity across the enterprise. They may incorporate feeds of signature updates or threat intelligence indicators, and they may even create their own signatures or indicators, but these are fed directly into the monitoring systems. The human effort at HM0 is directed primarily toward alert resolution.HM0 organizations also do not collect much information from their IT systems so their ability to proactively find threats is severely limited. Organizations at HM0 are not considered to be capable of hunting.
HM1 - Minimal
An organization at HM1 still relies primarily on automated alerting to drive their incident response process, but they are actually doing at least some routine collection of IT data. These organizations often aspire to intel-driven detection (that is, they base their detection decisions in large part upon their available threat intelligence). They often track the latest threat reports from a combination of open and closed sources. HM1 organizations routinely collect at least a few types of data from around their enterprise into a central location such as a SIEM or log management product. Some may actually collect a lot of information. Thus, when new threats come to their attention, analysts are able to extract the key indicators from these reports and search historical data to find out if they have been seen in at least the recent past.
Because of this search capability, HM1 is the first level in which any type of hunting occurs, even though it is minimal.
HM2 - Procedural
If you search the Internet for hunting procedures, you will find several great ones. These procedures most often combine an expected type of input data with a specific analysis technique to discover a single type of malicious activity (e.g., detecting malware by gathering data about which programs are set to automatically start on hosts). Organizations at HM2 are able to learn and apply procedures developed by others on a somewhat regular basis, and may make minor changes, but are not yet capable of creating wholly new procedures themselves. Because most of the commonly available procedures rely in some way on least-frequency analysis (as of this writing, anyway), HM2 organizations usually collect a large (sometimes very large) amount of data from across the enterprise.
HM2 is the most common level of capability among organizations that have active hunting programs.
HM3 - Innovative
HM3 organizations have at least a few hunters who understand a variety of different types of data analysis techniques and are able to apply them to identify malicious activity. Instead of relying on procedures developed by others (as is the case with HM2), these organizations are usually the ones who are creating and publishing the procedures. Analytic skills may be as simple as basic statistics or involve more advanced topics such as linked data analysis, data visualization or machine learning. The key at this stage is for Analysts to apply these techniques to create repeatable procedures, which are documented and performed on a frequent basis.Data collection at HM3 at least as common as it is at HM2, if not more advanced.
HM3 organizations can be quite effective at finding and combating threat actor activity. However, as the number of hunting processes they develop increases over time, they may face scalability problems trying to perform them all on a reasonable schedule unless they increase the number of available analysts to match.
HM4 - Leading
An HM4 organization is essentially the same as one at HM3, with one important difference: automation. At HM4, any successful hunting process will be operationalized and turned into automated detection. This frees the analysts from the burden of running the same processes over and over, and allows them instead to concentrate on improving existing processes or creating new ones.
HM4 organizations are extremely effective at resisting adversary actions. The high level of automation allows them to focus their efforts on creating a stream of new hunting processes, which results in constant improvement to the detection program as a whole.
Jackie
Danny
Stephen
- No such thing as a “failed hunt"
Jackie
Jackie
Jackie
Jackie
Danny
Stephen
Jackie
Danny
Is It Worth It? Let Me Work It:
STACKING
Fancy word for counting. How much of this thing did this other thing do?
PIVOTING
The ability to ”move laterally” through your own data
INTEL
Homegrown or farmed out. Hunting is as much about what you know as what you don’t know.
METRICS
Keeping your bosses happy
Bro (The most useful)
Bro is a SCRIPTING LANGUAGE
Bro can separate out classes of traffic by metadata
HTTP (Proxy)
Connection (Firewall)
DNS
DHCP
Etc..
Laika Boss
Moloch
Danny
GOOD BUDGET
You’ve got the cash and the time to pick your poison
Maybe rolling your own is the answer (Enter Bro)
BROKE ASS BUDGET
ELASTIC STACK (aka ELK Stack)
Elastic Search
Logstash
Kibana
Security Onion
intrusion detection
network security monitoring
log management
contains Snort, Suricata, Bro, OSSEC, Sguil, Squert, ELSA, Xplico, NetworkMiner
Jackie
Jackie
Step 1 – We begin with a hypothesis. Starting from a hypothesis allows analysts to attempt to debunk assumptions about the environment.
Step 2 – We use tools and analysis techniques to attack the problem set and validate our hypothesis
As we identify threats and risks that are not currently able to be detected by automation, we are able to use this information to improve our monitoring program.
Jackie
Hypotheses may be be:
Intelligence-Driven: Created from threat intelligence reports, threat intelligence feeds, malware analysis, vulnerability scans
Situational-Awareness Driven: Crown Jewel analysis, enterprise risk assessments, company- or employee-level trends
Jackie
Jackie
Jackie
Stephen
Danny
Jackie
Stephen
Danny
Jackie
Stephen
Danny
Jackie
Stephen for first two bullet points
Danny for metrics
Jackie
System 1
- Intuitive
- Potentially biased
- Efficient / Fast
- Draws on available knowledge/experience/how things work in a specific env
System 2
- Conscious
- Slow
- Effort to remove bias
- Deliberate
- Includes all types of analysis including, critical thinking, structured analytics techniques, empirical/quantitative methods