From EGIWiki
(Redirected from Forensic Howto)
Jump to: navigation, search
EGI-CSIRT web site EGI-CSIRT Public wiki EGI-CSIRT Contacts EGI-CSIRT Activities EGI-CSIRT Private wiki

This document describes a best-effort approach for preserving and analyzing compromised Linux installations. In many situations, errors in forensics can lead to destruction of evidences, so please take precaution and contact experts if you are unsure. This document proposes commands to run to obtain or analyse some data, but the responsibility of running them is yours: if you do not know them, read the man-page and/or contact experts.

You will find useful references or explanation on this wiki:


Preparation before incidents

The difficulty of any forensic varies on the effort attackers took to hide themselves. However, preparing yourself and your servers can ease forensics by a large amount

Prepare yourself

Check your procedures

Some questions should be answered before an incident happens, to make sure they don't hinder you while working on forensics:

As a EGI resource centre the EGI CSIRT Security Incident Handling Procedure should be used to complement your local Incident Response Procedure.

Prepare some hardware

During forensics, you will most likely need:

Prepare your systems

Remarks on live analysis

Data collection order

Data should be collected in a specific order, to avoid missing or destroying evidences.

Observation changes the observed object

On a live system:

Forensic data is volatile

On a live system, most of the data is volatile and usually survive only for a short time:

(List borrowed from “Forensic Discovery”, Farmer & Venema, Addison-Wesley 2005)

Please note that except for the last one, all these data are lost when the system is rebooted!

Danger from live system

Compromised tools

When performing live analysis, one has to remember that the system might have been modified to lie to you:

As a result, always doubt what you see and check if it makes sense:

Are you alone?

If the system has been compromised, malicious actors might still be present. If you are unlucky, they might still be around, detect your actions and start deleting evidences!

As a result, as soon as the breach is confirmed and baisc network/process forensics evidences have been collected, the system should be isolated from other systems.

Live (Quick & Dirty) forensics

When you are not sure you have a case and/or for physical systems (forensics on virtual machines should concentrate on snapshots (including memory snapshots)), here is a guide to collect initial data from the live system.

All these actions MUST be performed before switching off the system, as some of the evidences would be destroyed otherwise!

Before you start

Collect live data

In your temporary location, start by collecting live data from the system:

Collect malicious process data

Review rapidly the data you have collected and look for oddities, for example:

For each of such process:

Collect filesystem metadata

For each locally mounted filesystem, you should collect their file metadata (access, modification and change times).

The list of local filesystem can usually be found with:
grep '^/dev/' /proc/mounts

For each of them:

Once you have collected all this metadata and extracted it to another location, the timeline of each filesystem can be rebuilt using this script:

#! /usr/bin/python

from __future__ import print_function

from datetime import datetime
import sys
  import pytz

if len(sys.argv) > 1:
    timezone = pytz.timezone(sys.argv[1])
    print("Impossible to use this timezone")
  timezone = None

def print_line(flags, t, mode, user, group, name):
    when = datetime.utcfromtimestamp(float(t))
    if timezone is not None:
            when = pytz.utc.localize(when).astimezone(timezone)
            print("Timezone issue!")
    print(' '.join([t, when.isoformat(), flags, mode, user, group, name]))

for line in sys.stdin:
    line = line[:-1]
    (m, a, c, mode, user, group, name) = line.split(" ", 6)
    if m == a:
        if m == c:
            print_line("mac", m, mode, user, group, name)
            print_line("ma-", m, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)
        if m == c:
            print_line("m-c", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
            print_line("m--", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)

[Optional] Automated tests

You can run some automated tools to try to identify malicious activity/files, but you first need:

You can also (after remounting all filesystems read only) use your package management system to verify installed packages (save the output). On RedHat based systems, this is rpm -Va, on Debian based system, debsums.

Stopping the system

Before stopping the system, remember to take out all the data you have already collected using e.g. scp or rsync if you extracted it in a tmpfs. You might also want to run 'sync' to make sure that all data is written on disk.

There are two ways of stopping a system:

We usually recommend the later.

Offline analysis

Most of the analysis usually happens offline. Here we have less time pressure.

Copying disks

One of the most important rules of forensics is to never worker on the original supports, thus you need to copy the data (some will even recommend to only work on copies of copies in order to avoid copying the original again if you have a corruption).

There are mainly 3 ways of copying a disk :

For the last two possibility, one has to be very cautious not to modify any data:

Once drives are identified, we recommend using dd to image the disk:
dd if=/dev/sdX of=mybigfile.img bs=65536 conv=noerror,sync # inverting if and of will destroy all evidences!

Collect filesystem metadata

While filesystem metadata can be collected from live systems, collecting it from cold disk have some advantages:

We recommend using fls and mactime from the Sleuth Kit for collecting such metadata, see the corresponding wiki page:

Carving unallocated blocks

Data on file-systems is not deleted when a file is deleted, but is kept on unallocated data blocks. These blocks might be overridden by the content of new files. Until then, that data can be recovered from the harddrive:

Exploiting forensic data

There is one basic thing to remember when doing forensics: always try to assert how much you can trust what you see. Most of it is just bits on some disks, which could have been altered by an attacker, thus:

Reading file system timelines

When looking a a file system timeline, one should first remember that:

We recommend looking for the following artefacts:

Malware analysis

Malware analysis is complex and, with the ongoing competition between criminals and professional security researchers, is now mostly inaccessible without proper tool or training. If you have some doubts about a library or binary, you can take a quick look by:


This document, compiled by Vincent Brillault in 2017, is based on:

Personal tools