Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Forensic

From EGIWiki
Jump to navigation Jump to search
EGI-CSIRT web site EGI-CSIRT Public wiki EGI-CSIRT Contacts EGI-CSIRT Activities EGI-CSIRT Private wiki


This document describes a best-effort approach for preserving and analyzing compromised Linux installations. In many situations, errors in forensics can lead to destruction of evidences, so please take precaution and contact experts if you are unsure. This document proposes commands to run to obtain or analyse some data, but the responsibility of running them is yours: if you do not know them, read the man-page and/or contact experts.

You will find useful references or explanation on this wiki: http://forensicswiki.org

Preparation before incidents

The difficulty of any forensic varies on the effort attackers took to hide themselves. However, preparing yourself and your servers can ease forensics by a large amount

Prepare yourself

Check your procedures

Some questions should be answered before an incident happens, to make sure they don't hinder you while working on forensics:

  • Who are you supposed to contact during incidents?
    • Local security contacts?
    • External CSIRT you depend on?
  • What is your policy with regard to law enforcement? Will you pursue legal action?
  • Do you legally have to report security breaches to your users?
  • Who can decide to take systems and services down?
  • Who will coordinate actions? (like: Who will answer any press contact?)

As a EGI resource centre the EGI CSIRT Security Incident Handling Procedure should be used to complement your local Incident Response Procedure.

Prepare some hardware

During forensics, you will most likely need:

  • A USB key containing:
    • A Linux distribution of your choice, with auto-mount disabled
    • A static copy of forensics tools. We recommend in particular:
  • Few large capacity USB hard drive to collect disk images and evidences
  • [Optional] A SATA->USB converter to image disks offline (potentially with a write-blocking functionality) or a disk mirroring device.

Prepare your systems

  • Collect all system and audit logs on a remote system with strict access control.
  • Try to improve your file-system evidences:
    • Disable 'prelink' (It alters binary metadata)
    • Avoid cronjobs that read all files (Removes any meaning in the 'last access' metadata on file)
    • Avoid mounting your system with 'noatime', use 'relatime' only if necessary
  • Make sure to have basic tools installed on your systems that will help you collecting forensics data (you can't install them once you think you have an incident):
    • netstat
    • lsof: list of open files
    • pstree (from psmisc): tree of processes
    • gcore (from gdb): generate core images of processes

Remarks on live analysis

Data collection order

Data should be collected in a specific order, to avoid missing or destroying evidences.

Observation changes the observed object

On a live system:

  • Every command you run will alter (at least) file system metadata (atime on that binary)
  • Every data you write on disk can override data from files which were deleted before

Forensic data is volatile

On a live system, most of the data is volatile and usually survive only for a short time:

  • Processor states, caches and RAM: Nano-seconds
  • Network states: Milli-seconds
  • Running processes: Seconds
  • Disks: minutes

(List borrowed from “Forensic Discovery”, Farmer & Venema, Addison-Wesley 2005)

Please note that except for the last one, all these data are lost when the system is rebooted!

Danger from live system

Compromised tools

When performing live analysis, one has to remember that the system might have been modified to lie to you:

  • The kernel might have been modified
  • Binaries might have modified
  • Libraries might have been modified or are injected

As a result, always doubt what you see and check if it makes sense:

  • Compare output of different tools (e.g. ps VS pstree VS ls /proc/pid)
  • Compare with external behaviour (e.g. network activity)
  • Compare with one own's knowledge (e.g. missing files/activity)

Are you alone?

If the system has been compromised, malicious actors might still be present. If you are unlucky, they might still be around, detect your actions and start deleting evidences!

As a result, as soon as the breach is confirmed and baisc network/process forensics evidences have been collected, the system should be isolated from other systems.

Live (Quick & Dirty) forensics

When you are not sure you have a case and/or for physical systems (forensics on virtual machines should concentrate on snapshots (including memory snapshots)), here is a guide to collect initial data from the live system.

All these actions MUST be performed before switching off the system, as some of the evidences would be destroyed otherwise!

Before you start

  • Log every action you take, in order, with a time-stamp.
    • Use also paper and pen.
    • Consider using script (check "man script") to log all your commands executed.
    • If the case is complex, large and/or sensitive (e.g. suspicious of highly data)
  • If you connect to the system remotely, e.g. ssh, avoid using credentials that can be reused on other systems and consider them lost: change them as soon a possible
  • Don't store data on the harddrive:
    • Set HISTFILE to /dev/null
    • Store temporary files in a filesystem backed by a tmpfs (in RAM), remotely or on a USB drive. For example:
      mount | grep tmpfs # pick one in this list, we consider /dev/shm here
      df -h /dev/shm # verify that it is big enough
      mkdir /dev/shm/work && cd /dev/shm/work

Collect live data

In your temporary location, start by collecting live data from the system:

  • Network states:
    • Network sockets:
      netstat -apn | tee netstat_apn.txt
    • Network environment:
      ip -4 neigh show | tee ip6_neigh_show.txt
      ip -4 route show | tee ip6_route_list.txt
      ip -4 link  show | tee ip6_link_show.txt
      ip -6 neigh show | tee ip6_neigh_show.txt
      ip -6 route show | tee ip6_route_list.txt
      ip -6 link  show | tee ip6_link_show.txt
  • Users' logins:
    w > w.txt
    last | tee last.txt
    lastlog | tee lastlog.txt
  • Running processes:
    ps -auxwwwe | tee ps_auxwwwe.txt
    pstree -lap | tee pstree_lap.txt
  • Open sockets/files:
    lsof -b -l -P -X -n -o -R -U | tee lsof_blPXnoRU.txt
    lsof -b -l -P -X -n -o -R | tee lsof_blPXnoR.txt
  • List of mounted devices:
    cat /proc/mounts | tee proc_mounts.txt
  • Loaded kernel modules:
    cat /proc/modules | tee proc_modules
    ls /sys/modules |tee sys_modules

Collect malicious process data

Review rapidly the data you have collected and look for oddities, for example: - Processes listening on a raw socket (not related to dhcp) - Processes listening on ports that you don't know about, can't find on the internet - Processes spawned by a process that should not do it usually - Processes with network activity that should not do it usually

For each of such process:

  • Keep the PID number in a separate variable and folder:
    export PID=12345  # <- INSERT PROCESS-ID (PID) HERE
    mkdir $PID
    cd $PID
  • Stop the process (can't be caught):
    kill -STOP ${PID}
  • Copy the executable:
    cp /proc/${PID}/exe ${PID}.exe
  • Dump a core of the process:
    gcore ${PID}
    • If you don't have gcore, you need to do it manuall:
      gdb -p ${PID}
        # Type "gcore"
        # Type "detach"
        # Type "exit"
  • Copy all interesting open files:
    • List files opened by the process:
      lsof -np ${PID}
    • Copy 'deleted' files:
      cp /proc/${PID}/fd/${FDNUM} ${FILENAME} # FDNUM and FILENAME should be taken from the output of lsof
    • Copy files stored in /dev/shm:
      cp /dev/shm/${FILENAME} ${FILENAME} # FILENAME should be taken from the output of lsof
  • Copy all files from the proc folder:
    tar cvf proc_${PID}.tar /proc/${PID}/{auxv,cgroup,cmdline,comm,environ,limits,maps,sched,schedstat,sessionid,smaps, stack,stat,statm,status,syscall,wchan}
  • Get out of your temporary folder
    cd ..

Collect filesystem metadata

For each locally mounted filesystem, you should collect their file metadata (access, modification and change times).

The list of local filesystem can usually be found with:

grep '^/dev/' /proc/mounts

For each of them:

  • Create a mount point:
    mkdir mountpoint
  • Bind mount the existing mounted point there, e.g. for '/':
    mount --bind / mountpoint
  • Remount it read-only (this cannot be done in a single step!):
    mount -o remount,ro
  • Get into the mount point:
    cd mountpoint
  • Get all metadata:
    find . -print0 | xargs -0 stat -c "%Y %X %Z %A %U %G %n" -- | tee ../root.files # replace 'root' with the actual filesystem you mounted
  • Get out of the mountpoint and unmount it:
    cd .. 
    umount mountpoint

Once you have collected all this metadata and extracted it to another location, the timeline of each filesystem can be rebuilt using this script:

#! /usr/bin/python

from __future__ import print_function

from datetime import datetime
import sys
try:
  import pytz
except:
  pass

if len(sys.argv) > 1:
  try:
    timezone = pytz.timezone(sys.argv[1])
  except:
    print("Impossible to use this timezone")
    sys.exit(-1)
else:
  timezone = None

def print_line(flags, t, mode, user, group, name):
    when = datetime.utcfromtimestamp(float(t))
    if timezone is not None:
        try:
            when = pytz.utc.localize(when).astimezone(timezone)
        except:
            print("Timezone issue!")
            sys.exit(-1)
    print(' '.join([t, when.isoformat(), flags, mode, user, group, name]))

for line in sys.stdin:
    line = line[:-1]
    (m, a, c, mode, user, group, name) = line.split(" ", 6)
    if m == a:
        if m == c:
            print_line("mac", m, mode, user, group, name)
        else:
            print_line("ma-", m, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)
    else:
        if m == c:
            print_line("m-c", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
        else:
            print_line("m--", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)

[Optional] Automated tests

You can run some automated tools to try to identify malicious activity/files, but you first need:

  • To have these tools installed (e.g. chkrootkit, rkhunter, ossec-rootcheck)
  • To first remount all your local filesystems as read only. A first approach for this remount can use the following code, but you should rather review the list and remount them manually.
    for mountpoint in $(grep '^/dev/' /proc/mounts |cut -d ' ' -f 2 |sort -r); do 
      echo mount -o remount,ro $mountpoint
    done

You can also (after remounting all filesystems read only) use your package management system to verify installed packages (save the output). On RedHat based systems, this is rpm -Va, on Debian based system, debsums.

Stopping the system

Before stopping the system, remember to take out all the data you have already collected using e.g. scp or rsync if you extracted it in a tmpfs. You might also want to run 'sync' to make sure that all data is written on disk.

There are two ways of stopping a system:

  • Run the normal shutdown procedure: this will close all running program properly, which would preserve e.g. databases, but would write a lot on the disk, potentially destroying evidences
  • Simply unplug the power cables of the system: this will kill all running programs, potentially damaging e.g. databases, but would preserve more evidences.

We usually recommend the later.

Offline analysis

Most of the analysis usually happens offline, where time pressure is more limited

Copying disks

One of the most important rules of forensics is to never worker on the original supports, thus you need to copy the data (some will even recommend to only work on copies of copies in order to avoid copying the original again if you have a corruption).

There are mainly 3 ways of copying a disk :

  • Use a disk imaging/mirroring device
  • Use an external hard drive docking bay
  • Use an internal hot-swappable connector

For the last two possibility, one has to be very cautious not to modify any data:

  • The operating system on which the disk is plugged must have any auto-mount feature disabled
  • If more than one device is connected, they should be plugged in sequentially, to make sure that their identifiers are properly mapped
  • Any command typed can irremediably destroy data and should be checked multiple time

Once drives are identified, we recommend using dd to image the disk:

dd if=/dev/sdX of=mybigfile.img bs=65536 conv=noerror,sync # inverting if and of will destroy all evidences!

Collect filesystem metadata

While filesystem metadata can be collected from live systems, collecting it from cold disk have some advantages:

  • While live system analysis can be affected by rootkit, this one cannot
  • On ext4 filesystem, normal live tools do not extract the creation time. Forensic tools do

We recommend using fls and mactime from the Sleuth Kit for collecting such metadata, see the corresponding wiki page: http://wiki.sleuthkit.org/index.php?title=Timeline

Carving unallocated blocks

Data on file-systems is not deleted when a file is deleted, but is kept on unallocated data blocks. These blocks might be overridden by the content of new files. Until then, that data can be recovered from the harddrive:

  • For a given pattern (e.g. sshd logs), it’s possible (and usually fruitful) to simply grep the raw disk
  • Dedicated tools, like PhotoRec from the testdisk utility, can try to rebuild files based on built-in patterns

Exploiting forensic data

Reading file system timelines

FIXME

  • Binary ctime or crtime modified while the package was not updated
  • Traces of compilation: atime on files in /usr/include
  • Impossible timestamp combinaisons:
    • crtime after ctime
    • crtime of a file/folder after ctime of its parent folder

Malware analysis

FIXME

Credits

This document, compiled by Vincent Brillault in 2017, is based on:

  • Trainings written by Leif Nixon
  • A forensics HowTo written by Heiko Reese