The wiki is in the process of being deprecated and migrated to other supports.

Difference between revisions of "Forensic"

From EGIWiki
Jump to navigation Jump to search
Line 32: Line 32:


Collect data about the system's state (consult the manpages if you are unsure about what you are doing):
Collect data about the system's state (consult the manpages if you are unsure about what you are doing):
 
<pre>
mkdir incident_data
{{{
cd incident_data
#-------------
ps -auxwwwe > ps_auxwwwe.txt
mkdir incident_data
netstat --program --notrim --verbose -n > netstat_pTvn.txt
cd incident_data
netstat --program --notrim --verbose > netstat_pTv.txt
ps -auxwwwe > ps_auxwwwe.txt
w > w.txt
netstat --program --notrim --verbose -n > netstat_pTvn.txt
last > last.txt
netstat --program --notrim --verbose > netstat_pTv.txt
lastlog > lastlog.txt
w > w.txt
cat /proc/mounts > proc_mounts.txt
last > last.txt
arp -n > arp_n.txt
lastlog > lastlog.txt
ip neigh show > ip_neigh_show.txt
cat /proc/mounts > proc_mounts.txt
ip route list > ip_route_list.txt
arp -n > arp_n.txt
ip link  show > ip_link_show.txt
ip neigh show > ip_neigh_show.txt
lsof -b -l -P -X -n -o -R -U > lsof_blPXnoRU.txt
ip route list > ip_route_list.txt
for i in t p c t l; do ipcs -a -${i} > ipcs_a_${i}.txt;done
ip link  show > ip_link_show.txt
 
lsof -b -l -P -X -n -o -R -U > lsof_blPXnoRU.txt
for i in t p c t l; do ipcs -a -${i} > ipcs_a_${i}.txt;done
#-------------
}}}
</pre>
If there are suspicious processes that need further analysis, preserver the original binary and dump the program's memory:
If there are suspicious processes that need further analysis, preserver the original binary and dump the program's memory:
 
<pre>
export PID=12345  # <- INSERT PROCESS-ID (PID) HERE
{{{
kill -STOP ${PID} # stop process
#-------------
cp /proc/${PID}/exe ${PID}.exe
export PID=12345  # <- INSERT PROCESS-ID (PID) HERE
# some distributions have a script called "gcore" which does this in batch-mode
kill -STOP ${PID} # stop process
gdb -p ${PID}
cp /proc/${PID}/exe ${PID}.exe
  # type "gcore", then "detach" and "quit"
# some distributions have a script called "gcore" which does this in batch-mode
  # The program's memory is now saved as core.PID.
gdb -p ${PID}
ls -l /dev/shm
  # type "gcore", then "detach" and "quit"
# look for shared-memory-segments owned by the process
  # The program's memory is now saved as core.PID.
# by doing
ls -l /dev/shm
grep '/dev/shm' /proc/${PID}/maps
# look for shared-memory-segments owned by the process
# copy them if deemed neccessary
# by doing
tar cvf proc_${PID}.tar /proc/${PID} \
grep '/dev/shm' /proc/${PID}/maps
/{auxv,cgroup,cmdline,comm,environ,limits,maps,sched,schedstat,sessionid,smaps,stack,stat,statm,status,syscall,wchan}  
# copy them if deemed neccessary
kill -9 ${PID}    # kill process
tar cvf proc_${PID}.tar /proc/${PID}/{auxv,cgroup,cmdline,comm,environ,limits,maps,sched,schedstat,sessionid,smaps,stack,stat,statm,status,syscall,wchan}  
 
kill -9 ${PID}    # kill process
 
#-------------
}}}
</pre>
Create a list of all files in the system:
Create a list of all files in the system:
 
<pre>
mkdir /mnt/root_ro
{{{
mount --bind / /mnt/root_ro
#-------------
mount -o remount,ro /mnt/root_ro
mkdir /mnt/root_ro
# do not combine the two previous steps, this won't work on some older kernels
mount --bind / /mnt/root_ro
find /mnt/root_ro -xdev > find_root_ro_xdev.txt
mount -o remount,ro /mnt/root_ro
umount /mnt/root_ro
# do not combine the two previous steps, this won't work on some older kernels
 
find /mnt/root_ro -xdev > find_root_ro_xdev.txt
umount /mnt/root_ro
#-------------
}}}
</pre>
Install/copy chkrootkit (http://www.chkrootkit.org), rkhunter (http://rkhunter.sourceforge.net) and ossec-rootcheck (http://www.ossec.net/main/rootcheck) to the machine.
Install/copy chkrootkit (http://www.chkrootkit.org), rkhunter (http://rkhunter.sourceforge.net) and ossec-rootcheck (http://www.ossec.net/main/rootcheck) to the machine.


Remount all „real“ filesystems as read-only  
Remount all „real“ filesystems as read-only ({{{mount -o remount,ro MOUNTPOINT}}}). This is best done manually by the administratior. You may use this very simple heuristic as an alternative if needed:
mount -o remount,ro MOUNTPOINT
<pre>
This is best done manually by the administratior. You may use this very simple heuristic as an alternative if needed:
{{{
 
#-------------
 
sync
sync
for mountpoint in $(sort -r /proc/mounts | grep -E ' (ext[234]|xfs|reiser|vfat|ntfs)' | cut -d' ' -f2)
for mountpoint in $(sort -r /proc/mounts | grep -E ' (ext[234]|xfs|reiser|vfat|ntfs)' | cut -d' ' -f2)
do
do
   echo mount -o remount,ro "${mountpoint}"
   echo mount -o remount,ro "${mountpoint}"
done
done
 
#-------------
 
}}}
</pre>
Run chkrootkit, rkhunter and ossec-rootcheck.
Run chkrootkit, rkhunter and ossec-rootcheck.


Some package management-systems have checksums of their installed packages. Debian-based systems offer  
Some package management-systems have checksums of their installed packages. Debian-based systems offer {{{debsums}}} and Redhat has {{{rpm -Va}}}. Save the output.
debsums  
and Redhat has  
rpm -Va.
Save the output.


Copy the collected data someplace save or remove/unount the external storage/network drive.
Copy the collected data someplace save or remove/unount the external storage/network drive.

Revision as of 08:33, 23 May 2011


| Mission | Members | Contacts
| Incident handling | Alerts | Monitoring | Security challenges | Procedures | Dissemination



FORENSIC HOWTO

You may want to see egi incident response procedure at : [ https://wiki.egi.eu/wiki/EGI_CSIRT:Policies]

Release of 19 may 2011, edited by Heiko Reese <Heiko.Reese(at)kit.edu>

Linux Forensics HowTo

This document is a constant work-in-progress. Comments and additions are always welcome.

This document describes a best-effort approach for preserving and analyzing compromized Linux installations. Because there are many different Linux userlands (aka distributions), some commands may require a different syntax or different commands (most notably when package management is involved) to achive the same goal. To follow the instructions in this document, at least a basic understanding of the procedures presented here is necessary.

Forensic analysis consists of (at least) these phases:

* Identify the system.
* Gather data.
* Analyzie the data.

Identify compromized systems

TODO: (logs, monitoring, netflows, suspicious or erratic behaviour, external notification, etc...)

Gather data

The data aquisition process is twofold: first, gather information from the running (live) system. After that, analyze the »cold« system.

If the system runs as a virtual machine, freeze/pause it and create dumps/images from the filesysems/blockdevices and the memory.

Try not to write to the local filesystem. Put all gathered data onto external drives, network shares or into a ramdisk.

Collect data about the system's state (consult the manpages if you are unsure about what you are doing):

{{{
#-------------
mkdir incident_data
cd incident_data
ps -auxwwwe > ps_auxwwwe.txt
netstat --program --notrim --verbose -n > netstat_pTvn.txt
netstat --program --notrim --verbose > netstat_pTv.txt
w > w.txt
last > last.txt
lastlog > lastlog.txt
cat /proc/mounts > proc_mounts.txt
arp -n > arp_n.txt
ip neigh show > ip_neigh_show.txt
ip route list > ip_route_list.txt
ip link  show > ip_link_show.txt
lsof -b -l -P -X -n -o -R -U > lsof_blPXnoRU.txt
for i in t p c t l; do ipcs -a -${i} > ipcs_a_${i}.txt;done
#-------------
}}}

If there are suspicious processes that need further analysis, preserver the original binary and dump the program's memory:

{{{
#-------------
export PID=12345  # <- INSERT PROCESS-ID (PID) HERE
kill -STOP ${PID} # stop process
cp /proc/${PID}/exe ${PID}.exe
# some distributions have a script called "gcore" which does this in batch-mode
gdb -p ${PID}
  # type "gcore", then "detach" and "quit"
  # The program's memory is now saved as core.PID.
ls -l /dev/shm
# look for shared-memory-segments owned by the process
# by doing
grep '/dev/shm' /proc/${PID}/maps
# copy them if deemed neccessary
tar cvf proc_${PID}.tar /proc/${PID}/{auxv,cgroup,cmdline,comm,environ,limits,maps,sched,schedstat,sessionid,smaps,stack,stat,statm,status,syscall,wchan} 
kill -9 ${PID}    # kill process
#-------------
}}}

Create a list of all files in the system:

{{{
#-------------
mkdir /mnt/root_ro
mount --bind / /mnt/root_ro
mount -o remount,ro /mnt/root_ro
# do not combine the two previous steps, this won't work on some older kernels
find /mnt/root_ro -xdev > find_root_ro_xdev.txt
umount /mnt/root_ro
#-------------
}}}

Install/copy chkrootkit (http://www.chkrootkit.org), rkhunter (http://rkhunter.sourceforge.net) and ossec-rootcheck (http://www.ossec.net/main/rootcheck) to the machine.

Remount all „real“ filesystems as read-only ({{{mount -o remount,ro MOUNTPOINT}}}). This is best done manually by the administratior. You may use this very simple heuristic as an alternative if needed:

{{{
#-------------
sync
for mountpoint in $(sort -r /proc/mounts | grep -E ' (ext[234]|xfs|reiser|vfat|ntfs)' | cut -d' ' -f2)
do
  echo mount -o remount,ro "${mountpoint}"
done
#-------------
}}}

Run chkrootkit, rkhunter and ossec-rootcheck.

Some package management-systems have checksums of their installed packages. Debian-based systems offer {{{debsums}}} and Redhat has {{{rpm -Va}}}. Save the output.

Copy the collected data someplace save or remove/unount the external storage/network drive.

Do not shutdown as usual! Disconnect the power the the system.

Remove the harddisks and create images (use http://www.gnu.org/software/ddrescue/ddrescue.html).

Data analysis

In order to proceed from this point on, answer these questions:

* What do you already know? How can you use this knowledge to proceed?
* What is your next goal? Finding the breakin point? Understand malicious code/backdoors? Find culprits? Identify other systems involved?
* Do you have the resources/manpower to analyze now? Or should these resources be used to mitigate the threat (assuming that is still exists)?

TODO: (explain procedures for standard problems)

Here are a few pointers to helpful software and services:

* TODO: malware-foo
* TODO: network intelligence
* TODO: binary analysis