The seventh session from a two day course I ran for potential first responders in a large financial services client.

1. Phil Huggins
February 2004

2.  Description
 Strategy Meeting
 Documentation
 Debugging
 Log Processing
 Basic Host Analysis
 Rootkits
 Information Gathering

3.  The Scope Assessment Phase has the following
goals:
 To confirm the existence of the incident
 To identify which systems (if any) are involved in the incident
 To estimate the damage (if any) done to involved systems
 To identify if the attack is still underway
 To identify the complexity of the incident
 To gather any other data needed to make decisions on how to respond
 Sources of data:
 Logs
 Network Monitoring
 AssessmentAnalysis

4.  Large or complex incidents may require an initial
strategy meeting to coordinate efforts
 This tends to be a more technically focused meeting
than the initial team meeting discussed yesterday
 This meeting will identify who is responsible for:
 Verifying the initial report
 Verifying that similar systems were not affected
 Watching for an additional incident
 Deploying additional monitoring tools

5.  Document everything (even mistakes)
 Trust nothing on the suspect system
 Suspect systems should be modified as little
as possible
 Chain of Custody forms should be generated
for all evidence

8.  Debugging is simply “finding what’s wrong
with stuff”
 Obvious principles but MUST be applied
 Book Recommendation:
 Debugging by David J. Agans, ISBN 0-8144-7168-4
 http://www.debuggingrules.com/

9.  Understand the system
 Make it fail
 Quit thinking and look
 Divide and conquer
 Change one thing at a time
 Keep an audit trail
 Check the plug
 Get a fresh view
 If you didn’t fix it, it ain’t fixed.

10.  The goal is to find new clues and validate other findings
 Using information that is already known about the incident,
consult logs for additional clues
 Extract logs that reference suspect systems from devices
between the gateway and the suspect systems (using grep)
 If a time frame is known, extract all logs from that time on
gateways and remote access devices (stone-stepping
scenario)
 Identify additional hosts that have similar log activity or that
may have been used as a stepping stone
 Generate MD5 values of extracted logs
 Ensure that logs from the incident timeframe are not
overwritten

11.  In some cases, an analysis needs to be performed on a
compromised system before a forensic acquisition occurs
 The goal of this analysis is to identify the scope of damage
and quickly gather additional clues
 The analysis may answer:
 Have hiding mechanisms, such as a rootkit, been installed
 Who recently logged on and from where
 Were log files modified
 What files were recently created or modified
 If it is suspected that there are “time bombs” or other
“traps”, then the system should be unplugged and only
examined with a trusted kernel

12.  Document everything
 The “AccessTime” of files will be updated when you view
their contents, record which files you look at so those
times can be explained
 Send log files to an evidence server via netcat, calculate
an MD5 value, and analyze that copy
 Trust nothing on the suspect system
 Use only trusted tools from an response kit CD-ROM
 Kernel Module rootkits will hide data even with original
binaries

13.  Suspect systems should be modified as little as possible
 Use a tool such as mac-robber (http://www.sleuthkit.org/mac-
robber/index.php) or mac-daddy (www.incident-response.org) to collect the MAC
times of files before they get modified during the analysis
 Use a tool such asThe Sleuth Kit (http://www.sleuthkit.org/index.php) to analyze
the file system from the raw device (the MAC times will not be modified)
 Use tools such as Afind fromThe Forensic ToolkitVersion 2.0
(http://foundstone.com under resources and free tools) to search for recently
edited files on Windows systems..
 Stop schedulers from running commands on system
 Do not write files to the disk, it will overwrite deleted content. Instead pipe data
using netcat to the evidence server or to a floppy disk
On Evidence Server:
# nc -l -p 9000 > wtmp.log
On Suspect system:
# cat wtmp.log | nc -w 5 10.0.0.1 9000

14.  Volatile data acquisition procedures should be done
first to collect the data before it could be modified
(we will cover this later)
 netstat
 ps / pslist
 lsof / handle / fport
 etc.
 Examine the output (on the evidence server) for
suspicious processes, open ports, and logged on
users

15.  All files have at least 3 times associated with them
(Modified, Access, and Change)
 Timelines can be created with file activity at any
time
 For UNIX hosts,The Sleuth Kit can collect the data
from the raw device and not modify the file system
 An alternative is mac-robber or mac-daddy, which
will modify the access times of directories
 Both approaches will send data to an evidence
server where it is processed and analyzed

16.  Sleuth Kit:
# fls -f solaris -m / -r
/dev/rdsk/c0t0d0s0 | nc -w 5
10.0.0.1 9000
# ils -f solaris -m /dev/rdsk/c0t0d0s0 |
nc -w 5 10.0.0.1 9000
…. (repeat for each partition)
 mac-robber:
# mac-robber / | nc -w 5 10.0.0.1 9000
 mac-daddy:
# perl mac-daddy.pl / | nc -w 5 10.0.0.1
9000

17.  On the evidence server (a new file for each partition
with the Sleuth Kit):
# nc -l -p 9000 > mac_1.dat
 Sleuth Kit and mac-robber require a processing tool
from the Sleuth Kit:
# mactime -b mac_1.dat 01/01/2002 >
mac_1.tl
 Refer to the timeline.README document in the
Sleuth Kit for details (www.sleuthkit.org)

19.  DIBS MycroftV3
 http://www.dibsusa.com/products/mycroft.html
 Very fast and cheap

20.  Rootkits are installed by attackers to:
 Hide files and processes that they created
 Collect data (such as logins and passwords) from the
network or local system
 Provide a back-door method of gaining access to the
system
 Remove evidence of previous attack
 There are two major varieties of data hiding:
 Classical binary modification
 Kernel Modules

21.  The original system binaries are modified to read a
configuration file
 The configuration file contains a list of processes or
files to hide
 These can be detected by comparing the MD5 value
of current binary with one from a non-compromised
system (change management)
 In basic versions of this, running ‘strings’ on the
binary will show the location of the configuration
file (/dev/ptx0)

22.  Contents of a process config file (LRK 4)
2 slice2
2 snif
2 pscan
2 imp
3 qd
2 bs.sh
3 nn
3 egg.lin
 Contents of a file hiding config file (LRK 4)
tcp.log
slice2
scan
a
p
addy.awk
.fakeid

23.  Strings of a trojaned (LRK 4) ps binary:
<…>
90t:
u&Vh
/dev/ptyp
NR PID STACK ESP EIP
TMOUT ALARM
PID TTY MAJFLT MINFLT TRS DRS
SIZE SWAP
<…>
 /dev/ptyp file is a regular file, not a device
 /dev/ptyp0, /dev/ptyp1, etc. are valid devices

24.  Compare MD5 values of binaries with:
 Trusted system with same patch level
 Solaris Fingerprint Database (www.sun.com)
 NIST NSRL (www.nsrl.nist.gov)
 Linux RPM (with -V a flag)
 Compare output of system binaries with
trusted binaries on a CD
 chkrootkit (www.chkrootkit.org) signature
analysis

25.  Kernel Module rootkits modify the kernel
system call table instead of modifying the
binaries
 These rootkits prevent the kernel from giving
information on the processes and files that
are in a configuration file
 These are harder to detect because the MD5
of the binaries remain constant

26.  Normally, tools like ‘ps’ and ‘ls’ use theAPI to
request a list of processes or files from the
Kernel

27.  A rootkit goes between the Kernel and API
 Now, the API requests a list of processes or files
from the Rootkit, which forwards the request to the
Kernel and then filters out the “hidden” data.

28.  Trojan ‘sshd’ and ‘tcpd’ servers also exist to allow
access
 ‘ifconfig’ can be trojaned to hide the Promiscuous
flag
 Padding can be added to the end of new binaries to
match the CRC value of the original
 Use an accepted hashing algorithm such as MD5 or SHA-1

29.  New open network ports (nmap port scan)
 Promiscuous network interface (AntiSnif)
 Updated patch levels
 Modified logs
 AntiVirus software

30.  Different output from ‘nmap’ than ‘netstat’
 Different output from ‘ls’ than the Sleuth Kit
or Encase
 Carbonite
 chkrootkit
 Kstat
 Intrusion Prevention Systems

31. UNIX Windows
t0rn NetBus
Adore (LKM) Back Orifice
SLKM (LKM) Sub Seven
Linux Root
Kit(LRK)
NT Rootkit
Romanian Vanquish
Acquatica HE4Hook

32.  Check MD5 values of ‘ls’, ‘ps’, ‘netstat’, ‘sshd’
binaries
 Compare output of nmap port scan and netstat
 Look for text files in /dev/ or directories that start
with a ‘.’ in UNIX
 Compare output of ‘ls’ with that of the Sleuth Kit
 Examine a file activity timeline created by the
Sleuth Kit (not mac-robber or mac-daddy)

33.  Data can also be hidden while not using rootkits
 UNIX files and directories that start with a ‘.’ are not
shown by default:
# find / -name “.*” -print
 NTFS Alternate Data Streams are not shown by
default:
C:> echo “test” > file1.txt
C:> echo “hidden test” >
file1.txt:hidden
 Crucial ADS, sfind, and the Sleuth Kit will show their
existence

34.  Copy logs to evidence server for analysis (using
netcat as previously described)
 Look at wtmp logs on UNIX and run integrity checks
to see if it has been modified
 Look at other logs and correlate entries with
remotely stored copies or network device logs
 Copy Event Logs fromWindows and open in Event
Viewer (will be missing some application log text)
 Don’t forget to generate MD5 values

35.  To analyze a UNIX system, a CD withThe Sleuth Kit, Autopsy,
and other utilities can be created for remote analysis.
 Autopsy is HTML-based, so it is run from the CD and listens
on a given port
 The investigator connects to the port on the suspect system
and can browse the file system through the raw device
 This means that no files are modified and that rootkits will
be bypassed
 EnCase Preview offers a similar function forWindows
systems

36.  Internet-based Research
 Sanitize your location & be careful where you
visit
 Use a dial up account NOT the corporate
network
 Mailing lists may contain additional
information - google searches
 If IRC information or IP addresses are found,
it is not recommended that you join the IRC
channel or do a port scan of the host

37.  Converting an IP address to a hostname or a
hostname from an IP
 ‘dig’ collects data about domains and networks
from DNS records

38.  ‘whois’ returns contact information for an IP address

39.  traceroute may show where a host is located
(based on hostnames of back-bone devices)

40.  http://samspade.org/
 Powerful collection of ‘network detective’ tools run from
the web site.
 Windows tool for download.
 http://www.arin.net
 American IP Allocation Database
 http://www.ripe.net/db/whois/whois.html
 European IP Allocation Database
 http://www.apnic.net
 Asia Pacific IP Allocation Database

41.  This phase should answer:
 Which systems are involved and to what extent?
 How critical is each involved system?
 Which systems do we need to acquire?
 Is the attack still in progress?
 Is there an ongoing threat?
 Do we want to prosecute?
 Are more monitoring and logging needed for the
investigation?
 Are there any suspects?
 Is this from an insider?

42.  This phase collects data to identify the scope of the
incident
 The types of activities of this phase will depend on
the type of incident
 The data collected will be used in the Response
Phase, which will decide whether it is necessary to
use additional monitoring or do an acquisition
 Documentation and non-intrusive analysis are
crucial
 Chain of Custody is important if prosecution is likely