Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @


From EGIWiki
Revision as of 17:54, 19 May 2011 by Dfouosso (talk | contribs) (RELE)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search


release of 19 may 2011, edited by Heiko Reese <Heiko.Reese(at)>

Linux Forensics HowTo

This document is a constant work-in-progress. Comments and additions are always welcome.

This document describes a best-effort approach for preserving and analyzing compromized Linux installations. Because there are many different Linux userlands (aka distributions), some commands may require a different syntax or different commands (most notably when package management is involved) to achive the same goal. To follow the instructions in this document, at least a basic understanding of the procedures presented here is necessary.

Forensic analysis consists of (at least) these phases:

* Identify the system.
* Gather data.
* Analyzie the data.

Identify compromized systems

TODO: (logs, monitoring, netflows, suspicious or erratic behaviour, external notification, etc...)

Gather data

The data aquisition process is twofold: first, gather information from the running (live) system. After that, analyze the »cold« system.

If the system runs as a virtual machine, freeze/pause it and create dumps/images from the filesysems/blockdevices and the memory.

Try not to write to the local filesystem. Put all gathered data onto external drives, network shares or into a ramdisk.

Collect data about the system's state (consult the manpages if you are unsure about what you are doing): {{{

  1. -------------

mkdir incident_data cd incident_data ps -auxwwwe > ps_auxwwwe.txt netstat --program --notrim --verbose -n > netstat_pTvn.txt netstat --program --notrim --verbose > netstat_pTv.txt w > w.txt last > last.txt lastlog > lastlog.txt cat /proc/mounts > proc_mounts.txt arp -n > arp_n.txt ip neigh show > ip_neigh_show.txt ip route list > ip_route_list.txt ip link show > ip_link_show.txt lsof -b -l -P -X -n -o -R -U > lsof_blPXnoRU.txt for i in t p c t l; do ipcs -a -${i} > ipcs_a_${i}.txt;done

  1. -------------


If there are suspicious processes that need further analysis, preserver the original binary and dump the program's memory: {{{

  1. -------------

export PID=12345 # <- INSERT PROCESS-ID (PID) HERE kill -STOP ${PID} # stop process cp /proc/${PID}/exe ${PID}.exe

  1. some distributions have a script called "gcore" which does this in batch-mode

gdb -p ${PID}

 # type "gcore", then "detach" and "quit"
 # The program's memory is now saved as core.PID.

ls -l /dev/shm

  1. look for shared-memory-segments owned by the process
  2. by doing

grep '/dev/shm' /proc/${PID}/maps

  1. copy them if deemed neccessary

tar cvf proc_${PID}.tar /proc/${PID}/{auxv,cgroup,cmdline,comm,environ,limits,maps,sched,schedstat,sessionid,smaps,stack,stat,statm,status,syscall,wchan} kill -9 ${PID} # kill process

  1. -------------


Create a list of all files in the system: {{{

  1. -------------

mkdir /mnt/root_ro mount --bind / /mnt/root_ro mount -o remount,ro /mnt/root_ro

  1. do not combine the two previous steps, this won't work on some older kernels

find /mnt/root_ro -xdev > find_root_ro_xdev.txt umount /mnt/root_ro

  1. -------------


Install/copy chkrootkit (, rkhunter ( and ossec-rootcheck ( to the machine.

Remount all „real“ filesystems as read-only ({{{mount -o remount,ro MOUNTPOINT}}}). This is best done manually by the administratior. You may use this very simple heuristic as an alternative if needed:

grep -E ' (ext[234]

Run chkrootkit, rkhunter and ossec-rootcheck.

Some package management-systems have checksums of their installed packages. Debian-based systems offer {{{debsums}}} and Redhat has {{{rpm -Va}}}. Save the output.

Copy the collected data someplace save or remove/unount the external storage/network drive.

Do not shutdown as usual! Disconnect the power the the system.

Remove the harddisks and create images (use

Data analysis

In order to proceed from this point on, answer these questions:

* What do you already know? How can you use this knowledge to proceed?
* What is your next goal? Finding the breakin point? Understand malicious code/backdoors? Find culprits? Identify other systems involved?
* Do you have the resources/manpower to analyze now? Or should these resources be used to mitigate the threat (assuming that is still exists)?

TODO: (explain procedures for standard problems)

Here are a few pointers to helpful software and services:

* TODO: malware-foo
* TODO: network intelligence
* TODO: binary analysis