This presentation is how to build and use the Collective Intelligence Framework to start to collect threat intelligence from open-sources and use that to protect your network. This presentation was given for Hacker Hotshots on 11/27/2013.

1. Hacker Hotshots – 11/27/2013
How to Build the
Collective Intelligence Framework
And Start to Protect Your Network
John Bambenek
Chief Forensic Examiner, Bambenek Consulting
jcb@bambenekconsulting.com

2. Problem

Lots of people product lots of data,
blacklists, indicators of badness out there.

They all have their own formats and means
of distribution.

How to take multiple datasets, normalize
them and take action?

“Does anyone know anything about X?”

3. Solution: Collective Intelligence
Framework

Developed by REN-ISAC
 http://code.google.com/p/collective-intelligence-
framework/

Does not generate data, simply takes sources
normalizes it and then outputs by given types

Not really a data sharing tool

Up to user to assess confidence in the data

Limited in the types of data it can handle

4. Data Types
URLs
 Domains
 IPs
 MD5s


Certainly more to threat intel than
this, but it’s a start

5. CIF Architecture

6. CIF Architecture

By default, everything lives in /opt/cif

Configs in /opt/cif/etc/*.cfg (CIF processes
all files ending in .cfg)




cif_smrt – queries the feeds
cif_feed – generates feeds by assessment
cif – command-line client tool
cif_crontool – used for querying all feeds
automatically

7. Requirements to Install

For a “real” instance, you would need some disk
(250 GB – 500 GB) and RAM (16 GB)
 Disk is driven by how long you want to keep old data

Memory is only needing while parsing data
 CIF can be placed in a virtual infrastructure easily

Can install it on most everything, Debian/Ubuntu
easiest mostly because the instructions are available
and clear
 Ubuntu 12 probably best, 13 has some undocumented
changes that need to be made

Some kernel tweaking is needed

8. CIF Queries

Generally an analyst investigating will use
queries to see what is in the database.

cif –q <IP ADDRESS|DOMAIN
NAME|MD5>

Will include search records in the response
(unless suppressed)

Exact matching only (can’t search for part
of a URL… yet)

9. CIF Queries

CIF also ships a browser plugin which is
a little easier for analysis

Use cif_apikeys –l to get your key, find
your amazon IP and configure it now

Can query specific items or feeds

10. CIF feeds

cif –q feed/assessment –p output type [-z 0]
 (-z 0 will prevent truncating URLs)

cif –q infrastructure/scan
 Try with a lower confidence level

cif –q url/phishing –c 45

Not all output plugins work for all feeds
 Full list at: http://code.google.com/p/collective-
intelligence-framework/wiki/API_FeedTypes_v1

11. CIF output types
bindzone
Bind zone configuration
bro
bro (network monitor)
csv
comma separated value
html
Html-ized table
iptables
iptables drop rules
json
json
pcapfilter
pcap filter (i.e. tcpdump)
snort
snort alert rules
table
ascii table (default)

12. CIF Output

There are dozens of sources (many don’t have
configs in CIF), but you can integrate them all
into CIF and/or a feed.

What to do with this now?





Snort Rules
Feed to web proxy to block/alert
Send to border device to blacklist IPs
Set up a sinkhole
You can also put your own data into CIF for
later research

13. Questions?

Thanks for tuning in!

See more courses and hacker hotshot
sessions at concise-courses.com