Checklist

                             Basic Forensic Checklist
                             -======================-

                              Heiko Reese (KIT-CERT)

                                    2012-08-28

                                   @GridKA-School


Find a place for your findings. Minimize/avoid changes to the existing
filesystems.

* external drive
* mount -t tmpfs none /mystuff
* mkdir /dev/shm/mystuff

Problems with these?



If possible, send data over the network:

* Listen on server:
  nc -l 45678 >> ${logfilename}.txt
* Send stuff from compromised machine:
  $CMD | nc -w 2 name_or_ip_of_server 45678
  
Untrusted network? Use openssl! Insert into cmdchain:

nc -l 45678 | openssl enc -aes128 -d -k supersecretpw >> log.txt
$CMD | openssl enc -aes128 -e -k supersecretpw | nc -w 2 name_ip_server 45678

Or use cryptcat, if available.



Want to transfer images of blockdevices?

  dd if=/dev/sdx23 | nc...

Feel like copying stuff?

  cat /usr/bin/rootkit_0.1 | nc...

The following cmdlines presume that you are putting stuff on the local disk 
(»... > file«). Substitute properly if you want to transfer files to remote 
locations.



Let's collect stuff, starting with the most volatile:

* network state
* process state
* users
* system config



# Network State #1

netstat --program --verbose -n > netstat_pTvn.txt
netstat --program --verbose > netstat_pTv.txt

arp -n > arp_n.txt
ip neigh show > ip_neigh_show.txt

for i in link addr route rule neigh ntable tunnel tuntap maddr mroute mrule; 
do
  ip $i list > ip_${i}_l.txt;
done



# Network State #2

for t in filter nat mangle raw; do
  iptables -v -n -x -L -t > iptables_vnxL_t${t};
done

for table in filter mangle raw; do ip6tables -n -t ${table} -L -v -x; done

for table in filter nat broute; do ebtables -t ${table} -L --Lmac2 --Lc; done



Save process table:

ps auxwwwe > ps_auxwwwe.txt
lsof -b -l -P -X -n -o -R -U > lsof_blPXnoRU.txt
for i in t p c t l; do ipcs -a -${i} > ipcs_a_${i}.txt;done



# Users

last > last.txt
lastlog > lastlog.txt
who > who.txt
w > w.txt



# System
dmesg > dmesg.txt
cat /proc/mounts > proc_mounts.txt
cat /proc/mdstat > proc_mdstat.txt
lspci > lspci.txt
uname -a > uname_a.txt
uptime > uptime.txt


# So you've found a suspicions process?

Stop it!

  export PID=12345 # <- insert correct PID!
  kill -STOP ${PID} # prozess stoppen

Save the executable:

  cp /proc/${PID}/exe ${PID}.exe

Dump it's memory:

  gdb -p ${PID}
  # type »gcore«
  # type quit
  # some systems provide a programm calles gcore to do that in one step

Check for shared memory segments:
  less /proc/${PID}/maps # look for /dev/shm

Save state (some of it at least):
  tar cvf proc_${PID}.tar /proc/${PID}/{auxv,cgroup,cmdline,comm,environ,
  limits,maps,sched,schedstat,sessionid,smaps,stack,stat,statm,status,syscall,
  wchan} 

Want to get rid of it (think twice before you do it)?
  kill -9 ${PID}



# rw -> ro
# Create a rudimentary timeline:

# heuristic: ymmv. Doublecheck.
sync
for mountpoint in $(sort -r /proc/mounts | \
    grep -E '(ext[234]|xfs|reiser|vfat|ntfs)' | cut -d' ' -f2)
do
  echo mount -o remount,ro "${mountpoint}"
  find "${mountpoint}" -xdev -print0 | xargs -0 stat -c "%Y %X %Z %A %U %G %n" >> 
timestamps.dat
done

ATTN: Some filesystems still change metadata when mounted read-only.



# Need the system to stay rw while you create the timeline?

Create »aliases«:

  mkdir /mnt/root_ro
  mount --bind / /mnt/root_ro
  mount -o remount,ro /mnt/root_ro
  find /mnt/root_ro -xdev -print0 | \
    xargs -0 stat -c "%Y %X %Z %A %U %G %n" >> timestamps.dat
  umount /mnt/root_ro

Repeat for all other »real« filesystems.

Or: mount -o remount,noatime


# timeline-decorator.py

#!/usr/bin/python

import sys, time

def print_line(flags, t, mode, user, group, name):
    print t, time.ctime(float(t)), flags, mode, user, group, name

for line in sys.stdin:
    line = line[:-1]
    (m, a, c, mode, user, group, name) = line.split(" ", 6)
    if m == a:
        if m == c:
            print_line("mac", m, mode, user, group, name)
        else:
            print_line("ma-", m, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)
    else:
        if m == c:
            print_line("m-c", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
        else:
            print_line("m--", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)



Get these tools:

* chkrootkit ( http://www.chkrootkit.org)
* OSSEC rootcheck (http://www.ossec.net/main/rootcheck)
* rkhunter ( http://rkhunter.sourceforge.net)
* Trojanscan ( http://www.trojanscan.org)
* CVE-Checker ( http://cvechecker.sourceforge.net/)

Happy hunting!



If applicable, compare checksums of package management with actual files:

  debsums (Debian-based distributions)
  rpm -Va (Redhat-based)

Found mismatch? Look closer!
Found nothing? Binary or database could be compromised -> keep walking, 
nothing to see here.



# Step 2: Offline Forensics

Switch off machine (_no_ shutdown). Create disk image.

* see Toby's slides
* TL;DR:
  * Binaries: strings, hexdump, objdump, elf*, gdb,
              rec (http://www.backerstreet.com/rec/rec.htm), IDAPro,…
  * Logfiles: grep, sort, log2timeline, …
  * Autosy, rkhunter, …
  * GOTO 1 if necessary

Goal: extract relevant information about attack/attacker.


# Misc

Keep calm. (Remove nervous boss/manager)

Keep track of your findings!
  * Put timestamps on everything you write down.
  * Don't postpone, you won't remember everything afterwards.

Be gentle. Take great care not to tamper with evidence.



  

FIN