Intersect

INTERSECT: Combining
Commercial/FOSS Tools with
Custom Code to Root Out
Malware

Fotios Lindiakos
Matt Pawloski
© 2011 The MITRE Corporation.
Approved for Public Release: 11-0130 ALL RIGHTS RESERVED.

INTERSECT: How to Combine All
of the Stuff You Spent Too Much
Money on With the Cool Free Stuff
Your Boss Won’t Let You Install to
Actually Do Something Useful


About Us
Fotios Matt
• Graduated 2007 from RIT • Graduated 2005 from RIT
with a BS in CS with a BS in IT
• Attending GMU for a MS in • Graduated 2010 from
ISA Capitol College with MS in
• Started as an intern at IA
MITRE and has been full • Worked at Symantec, KCG
time for 3.5 years • Been at MITRE 3 years


About MITRE
• “Not-for-profit organization chartered to work in the
public interest”
• “MITRE is a unique organization that assists the
United States government with scientific research and
analysis, development and acquisition, and/or systems
engineering and integration”
• “MITRE also has its own independent research and
development program that explores new technologies
and new uses of technologies to solve our sponsors'
problems in the near-term and in the future”
Sources: http://www.mitre.org/about/ffrdcs.html
http://www.mitre.org/about/index.html


An Axiom
• Client side attacks are the most prevalent
attack vector
– Users receiving a malicious email attachment
– Users receiving a malicious link in an email
• We need agile file examination!
• Good tools exist, but can be hard to
deploy/use
• “Real-time” is nice to have, but not practical


STATE OF THE INDUSTRY


Products
Pros Cons
• Quick indicator • Not effective against targeted attacks
Antivirus • Cheap/free
• Can block in • Same “signature problem” as
real-time antivirus
IDS/IPS • Doesn’t examine full files
• False positives can cause an outage
• Can be very • Getting files can be difficult
Home effective for
your specific
• Lots of reinvention of the wheel
• Can be unstable
Grown organization if
used properly


Most COTS Products
• Difficult to interface with
• No great “top to bottom” solution
• Expensive
• Not agile enough to meet quickly adapting
threats
• Vendors don’t meet your specific needs
• This doesn’t mean they are worthless!


If DEFCON has taught us
anything…
• “Race to Zero”
– Signature based scanning is trivial to bypass
– Examples
• Repacking
• Causing AV engines to timeout by wrapping malware
with some trivial code
– Doesn’t have to attack AV or modify malware
• Unhook AV
• Targeted defenses are needed for targeted
attacks


CURRENT THREATS


Client Side Attacks
• We have everything we need in the file
– Static analysis
• Initial file is usually just a dropper
– Behavioral analysis
• File will beacon out, download more malware, and
commence C&C
• Even a small success rate is still a success


0-day
• Everybody is sick of talking about this
• Detection sometimes possible through
– Content detonation
– Targeted profiling
• ssdeep
• HBGary FingerPrint


Targeted Attacks
• How can you expect any non-specialized tools
to find something that was made specifically
for you?
– Targeted attacks need targeted defenses
• Only targeted at a select number of users
• Phishing email so well crafted your users will
definitely click on it


What We Need
• Need to Gather Intelligence
– Accumulating Data
• You already have the files
• Different tools can tell you different things
– Correlating Data
• Need to Protect
– Detect targeted attacks
– Need to react faster than traditional solutions
– Different tools may offer overlapping protection
• Need to Measure Efficacy
– Are your tools actually doing a good job?
– Easily evaluate new technologies


WHAT WE PROPOSE


Not a Replacement!
• You still need these products
• We want to augment and integrate them
• Use conventional technology in
unconventional ways


Our Requirements
• Simple Interfaces
– Front end (user experience)
– Back end (developer experience)
• Scalable
• Resilient
• Fast
• Awesome (with a catchy name)


Our Solution…
• We call it INTERSECT


What Is It?
• Middleware that works!
• Just a framework
– Ties together all the pieces (more on them later)
– Gives the users a “single pane of glass”
– Handles all of the mundane stuff to let the
developers focus on their parts
– Helps consolidate results
• Can be used to perform correlation and alerting


Producers
• Any services/devices that see full files and can
upload
• Examples
– Web proxy
– Email server
– File server (SMB, FTP, etc)
– Full file extractor on live network stream


Producer Example
#!/usr/bin/ruby
require ‘rest_client’

RestClient.post url, {
:upload => {
:upload => File.new(filename)
},
:transfer => {
:param1 => value1,
:param2 => value2
}
}


Consumers
• Scanners that examine or possibly modify
files submitted by producers
– Start from scratch
• Code it right into your own tools
– Leverage existing tools
• Write a wrapper for a COTS product you already have
• Return results to be correlated


Examples of Consumers
COTS FOSS/Custom
• AV • AV
– Individual (ClamAV, AVG, etc)
– Individual – Aggregate (VirusTotal)
(Symantec, McAfee, etc) • Content Detonation
– Aggregate (MetaScan) – Honeynet Project
• Yara
• Content Detonation – Public signatures
– FireEye – Create your own!
• Archive Extraction
• File Profiling – Zip/Tar/ISO
– HBGary FingerPrint – DD Image. Forensics anybody?
• Covert Data Channels
– Find indicators and quickly
weaponize them


Consumer Example
def subscribe()
AMQP.start(:host => HOST ) do
amq = MQ.new
q = amq.queue(QUEUE_NAME)
ex = amq.topic(MESSAGES_EX)
q.bind(ex, :key => "image.#")

q.subscribe(:ack => true) do |hdr,body|
yield hdr,body
hdr.ack
end
end
end

Returning Results
def publish(hash)
AMQP.start(:host => HOST ) do
amq = MQ.new
ex = amq.fanout(RESULTS_EX)

ex.publish(
hash[:body],
:headers => hash[:headers]
)
end
end


INTERSECT INTERNALS


Brains
• Keep the trains running…
• Accepts file submissions
• Submit files to bus
• Collect results created by consumers
• Correlate results from consumers


Front End
• HTTP Interface
• Producers can upload files
– By POSTing files
• Analysts can view results
– Through the Ruby on Rails app
• Consumers can download files to analyze
– Via a simple static file serving


Ruby on Rails App
• Allows for rapid web development
– Abstracts away everything for you
• WEBrick
– Standard, lightweight development web server
– Holding up to pretty much whatever we throw at it

• EventMachine
– Extremely high scalability, performance and stability for
the most demanding production environments
– An API that eliminates the complexities of high-
performance threaded network programming, allowing
engineers to concentrate on their application logic


Back End
• MySQL
– Holds all metadata for files and results
• File store
– All files are stored and renamed to match their
MD5 hash to prevent duplication


Service Bus
• The unsung hero
• Disclaimer
– We know very little about ESB software. We
know just enough to say we don’t like them.
• Provides basic routing of messages between
producers, INTERSECT, and consumers
• Allows us to decouple everything
– Just connect to the bus


Tying It Together
• Consumers bind to the bus
– Simple wrapper script / class to communicate
with bus
• Use this method to quickly repurpose already existing
services and capabilities
• Use this method to integrate proprietary solutions
with limited interfaces
– Integrate directly into your consumer from the
start


Advanced Message Queuing
Protocol (AMQP)
• Awesome protocol
– Lightweight
• Developed by some financial companies to
facilitate “common business messaging”
• Protocol developed by some major technical
companies


RabbitMQ
• Easy to setup
– Servers run on Windows, *nix, OSX, OpenVMS?!
• Libraries for most languages
– Ruby, Java, Perl, Python, .NET, PHP, C, Erlang, Lisp, H
askell
• Simple to configure and manage
– Allows you to spend more time developing
– Powerful features
• Access control
• vhosts
• Load balancing / Redundancy


Exchanges
Fanout Topic
• Shotgun approach • More precise
– All services get all messages – Declare what messages you
– Queues fill up want based on routing key
– Consumers have to decide – We use filetype
• Based on libmagic for now
– Reduces load on consumers


Messages
• We keep them as small as possible
• Files are not contained in the messages
– Some consumers simply operate on metadata
• We provide a URL to get the file from
INTERSECT


Message Packet Capture


Result Packet Capture


Consumer Workflow


Load Balancing / Redundancy
• RabbitMQ
– Multiple instances can be stood up in a cluster
• Consumer
– Multiple instances can be bound to the same
queue
– Messages will be delivered to an available
instance of a service


ADVANTAGES TO OUR APPROACH


‚Single Pane of Glass‛
• Consolidate results from disparate services
– Correlate those results to find something novel
• Search through results and transfers by any
amount of metadata
• Evaluate efficacy of different services
– Did some detect maliciousness and others not?


Agile
• Easy to add new consumers
• Resubmit files to new/updated consumers
• Provide research projects with relevant test
data


Asynchronous
• Queuing allows consumers to take as much
time as they need to process files
• A failure of one consumer has no effect across
the system
– Less stable research projects can process real
data to better prove their methods


Leverage Resources
• Fully utilize your COTS tools
– It was expensive, get your money’s worth
– Many of them expect a manual workflow and go
underutilized
• Throw more hardware at it
– Run multiple copies of services


WHAT WE NEED


What We Need
• Need to Gather Intelligence
• Need to Protect
• Need to Measure Efficacy


Gather Intelligence
• You can never have too much intelligence
• Once you have all of the information in one
place
– Act on it
– Analyze it


Protect
• How we’re using it right now
• Producers
– Web proxy
– Network taps
– Mail server
• Consumers
– Lots of file scanners


Protect
• Workflow
– File comes in from the network
• Somebody is downloading or was sent a file
• Scanners do triage
• If the file is suspicious
– Alert an analyst
– They can decide what to do based on your corporate policy
• We can accumulate data on files
– Retroactively scan files when new tips/signatures come out
– Start to tie different files to the same attackers


Statistics
• Began ingesting files from multiple enterprise
producers
• Since 14-Dec
– ~175,000 unique files
– Averaging ~4500 unique files daily
– Max ~7800 files in a day
– No backlog!


Measure Efficacy
• With all of that data
– You can see how your COTS tools compare to
your research projects
– You can see how your research projects are
progressing
• Run scans with one version
• After you make some changes, run new scans and
compare


Other Possible Use Cases
• Egress Filtering
– Write scanners that look for SSN, Credit Card
Numbers, “Dirty Words”…
• File Transfer
– Make users put files into the system if they want
to bring it into the corporation
– Don’t allow them to download it directly
• They need to put it into your system and then
download it from there after if gets scanned


FUTURE IMPROVEMENTS


Correlation Engine
Integration
• We are working on utilizing Splunk
– Other SIEMs would work too
• Provide better UI, alerting, searching, etc.
• Don’t reinvent the wheel


Better Filetype Checking
• This is a difficult problem to get perfectly
correct
• Maybe we can develop services that do this
for us…
> file --mime-type INTERSECT.*
INTERSECT.doc: application/msword
INTERSECT.docm: application/zip
INTERSECT.docx: application/zip


Archive ‚Explosion‛
• Simplest form, unzip a file and resubmit the
children
– The “threat” of the archive would be an aggregate of
the threat of the children
• But what is an “archive”?
– Office 2007+ file format
• PowerPoint stores each slide as a different file, along with
each image individually
– Disk images
• Write a forensic service that can parse through and pull out
all files
• Resubmit those files to keep track of all files in a disk image


Contact Us
• Emails
– flindiakos@mitre.org
– mpawloski@mitre.org


Intersect

Recommended

Recommended

More Related Content

What's hot

What's hot (11)

Similar to Intersect

Similar to Intersect (20)

Intersect

Editor's Notes