Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Forensic"

From EGIWiki
Jump to navigation Jump to search
(Deprecate page, content moved to CSIRT space in confluence)
Tag: Replaced
 
(20 intermediate revisions by 4 users not shown)
Line 1: Line 1:
[[Category:EGI-CSIRT]]
[[Category:EGI-CSIRT]]
<!--{{Egi-csirt-header}}-->
{{DeprecatedAndMovedTo|new_location=https://confluence.egi.eu/display/EGIBG/Forensics+Howto}}
{{New-Egi-csirt-header}}
 
This document describes a best-effort approach for preserving and analyzing compromised Linux installations.
In many situations, errors in forensics can lead to destruction of evidences, so please take precaution and contact experts if you are unsure.
This document proposes commands to run to obtain or analyse some data, but the responsibility of running them is yours: if you do not know them, read the man-page and/or contact experts.
 
You will find useful references or explanation on this wiki: http://forensicswiki.org
 
= Preparation before incidents =
 
The difficulty of any forensic varies on the effort of attackers to hide themselves.
However, preparing yourself and your servers can ease forensics by a large amount
 
== Prepare yourself ==
 
=== Check your procedures ===
 
Some questions should be answered before an incident, to make sure they don't hinder you while working on forensics:
 
* Who are you supposed to contact during incidents?
** Local security contacts?
** External CSIRT you depend of?
* What is your policy with regard to law enforcement? Will you pursue legal action?
* Do you legally have to report security breaches to your users?
* Who can the decision to take systems and services down?
* Who will coordinate actions? Who will answer any press contact?
 
=== Prepare some hardware ===
 
During forensics, you will most likely need:
* A USB key containing:
** A Linux distribution of your choice, with auto-mount disabled
** A static copy of forensics tools. We recommend in particular:
*** [http://www.cgsecurity.org/wiki/TestDisk Testdisk]
*** [https://www.sleuthkit.org/ The Sleuth Kit]
* Few large capacity USB hard drive to collect disk images and evidences
* [Optional] A SATA->USB converter to image disks offline (potentially with a write-blocking functionality) or a disk mirroring device
 
== Prepare your systems ==
 
* Collect all system and audit logs on a remote system with highly controlled access
* Try to improve your file-system evidences:
** Disable 'prelink' (It alters binary metadata)
** Avoid cronjobs that read all files (Removes any meaning in the 'last access' metadata on file)
** Avoid mounting your system with 'noatime', use 'relatime' only if necessary
* Install basic tools that will help you for debugging (you can't install them once you think you have an incident):
** lsof: list of open files
** pstree (from psmisc): tree of processes
** gcore (from gdb): generate core images of processes
 
= Remarks on live analysis =
 
== Data collection order ==
 
Data should be collected in a specific order, to avoid missing or destroying evidences.
 
=== Observation changes the observed object ===
 
On a live system:
* Every command you run will alter (at least) file system metadata (atime on that binary)
* Every data you write on disk can override data from files which were deleted before
 
=== Forensic data is volatile ===
 
On a live system, most of the data is volatile and usually survive only for a short time:
* Processor states, caches and RAM: Nano-seconds
* Network states: Milli-seconds
* Running processes: Seconds
* Disks: minutes
(List borrowed from “Forensic Discovery”, Farmer & Venema, Addison-Wesley 2005)
 
Please note that except for the last one, all these data are lost when the system is rebooted!
 
== Danger from live system ==
 
=== Compromised tools ===
 
When performing live analysis, one has to remember that the system might have been modified to lie to you:
* The kernel might have been modified
* Binaries might have modified
* Libraries might have been modified or are injected
 
As a result, always doubt what you see and check if it makes sense:
* Compare output of different tools (e.g. ps VS pstree VS ls /proc/pid)
* Compare with external behaviour (e.g. network activity)
* Compare with one own's knowledge (e.g. missing files/activity)
 
=== Are you alone? ===
 
If the system has been compromised, malicious actors might still be present.
If you are unlucky, they might still be around, detect your actions and start deleting evidences!
 
As a result, as soon as the breach is confirmed and baisc network/process forensics evidences have been collected, the system should be isolated from other systems.
 
= Live (Quick & Dirty) forensics =
 
When you are not sure you have a case and/or for physical systems (forensics on virtual machines should concentrate on snapshots (including memory snapshots)), here is a guide to collect initial data from the live system.
 
'''All these actions MUST be performed before shutting down or rebooting the system''', as some of the evidences would be destroyed otherwise!
 
== Before you start ==
 
* Log every action you take, in order, with a time-stamp
** Consider using ''script'' to log all your action
** If the case is complex, large and/or sensitive (e.g. suspicious of highly data)
* If you connect to the system remotely, e.g. ssh, avoid using credentials that can be reused on other systems and consider them lost: change them as soon a possible
* Don't store data on the harddrive:
** Set HISTFILE to /dev/null
** Store temporary files in a filesystem backed by a tmpfs (in RAM), remotely or on a USB drive. For example:<pre>mount | grep tmpfs # pick one in this list, we consider /dev/shm here&#10;df -h /dev/shm # verify that it is big enough&#10;mkdir /dev/shm/work && cd /dev/shm/work</pre>
 
== Collect live data ==
 
In your temporary location, start by collecting live data from the system:
 
* Network states:
** Network sockets: <pre>netstat -apn | tee netstat_apn.txt</pre>
** Network environment:<pre>ip -4 neigh show | tee ip6_neigh_show.txt&#10;ip -4 route show | tee ip6_route_list.txt&#10;ip -4 link  show | tee ip6_link_show.txt&#10;ip -6 neigh show | tee ip6_neigh_show.txt&#10;ip -6 route show | tee ip6_route_list.txt&#10;ip -6 link  show | tee ip6_link_show.txt</pre>
* Users' logins:<pre>w > w.txt&#10;last | tee last.txt&#10;lastlog | tee lastlog.txt</pre>
* Running processes:<pre>ps -auxwwwe | tee ps_auxwwwe.txt&#10;pstree -lap | tee pstree_lap.txt</pre>
* Open sockets/files:<pre>lsof -b -l -P -X -n -o -R -U | tee lsof_blPXnoRU.txt&#10;lsof -b -l -P -X -n -o -R | tee lsof_blPXnoR.txt</pre>
* List of mounted devices: <pre>cat /proc/mounts | tee proc_mounts.txt</pre>
* Loaded kernel modules: <pre>cat /proc/modules | tee proc_modules&#10;ls /sys/modules |tee sys_modules</pre>
 
== Collect malicious process data ==
 
Review rapidly the data you have collected and look for oddities, for example:
- Processes listening on a raw socket (not related to dhcp)
- Processes listening on ports that you don't know about, can't find on the internet
- Processes spawned by a process that should not do it usually
- Processes with network activity that should not do it usually
 
For each of such process:
* Keep the PID number in a separate variable and folder: <pre>export PID=12345  # <- INSERT PROCESS-ID (PID) HERE&#10;mkdir $PID&#10;cd $PID</pre>
* Stop the process (can't be caught): <pre>kill -STOP ${PID}</pre>
* Copy the executable: <pre>cp /proc/${PID}/exe ${PID}.exe</pre>
* Dump a core of the process: <pre>gcore ${PID}</pre>
** If you don't have gcore, you need to do it manuall: <pre>gdb -p ${PID}&#10;  # Type "gcore"&#10;  # Type "detach"&#10;  # Type "exit"</pre>
* Copy all interesting open files:
** List files opened by the process: <pre>lsof -np ${PID}</pre>
** Copy 'deleted' files: <pre>cp /proc/${PID}/fd/${FDNUM} ${FILENAME} # FDNUM and FILENAME should be taken from the output of lsof</pre>
** Copy files stored in /dev/shm: <pre>cp /dev/shm/${FILENAME} ${FILENAME} # FILENAME should be taken from the output of lsof</pre>
* Copy all files from the proc folder: <pre>tar cvf proc_${PID}.tar /proc/${PID}/{auxv,cgroup,cmdline,comm,environ,limits,maps,sched,schedstat,sessionid,smaps,
stack,stat,statm,status,syscall,wchan}</pre>
* Get out of your temporary folder <pre>cd ..</pre>
 
== Collect filesystem metadata  ==
 
For each locally mounted filesystem, you should collect their file metadata (access, modification and change times).
The list of local filesystem can usually be found with: <pre>grep '^/dev/' /proc/mounts</pre>
 
For each of them:
* Create a mount point: <pre>mkdir mountpoint</pre>
* Bind mount the existing mounted point there, e.g. for '/': <pre>mount --bind / mountpoint</pre>
* Remount it read-only (this cannot be done in a single step!): <pre>mount -o remount,ro</pre>
* Get into the mount point: <pre>cd mountpoint</pre>
* Get all metadata: <pre>find . -print0 | xargs -0 stat -c "%Y %X %Z %A %U %G %n" -- | tee ../root.files # replace 'root' with the actual filesystem you mounted</pre>
* Get out of the mountpoint and unmount it: <pre>cd .. &#10;umount mountpoint</pre>
 
Once you have collected all this metadata and extracted it to another location, the timeline of each filesystem can be rebuilt using this script:
<pre class="toccolours mw-collapsible mw-collapsed">#! /usr/bin/python
 
from __future__ import print_function
 
from datetime import datetime
import sys
try:
  import pytz
except:
  pass
 
if len(sys.argv) > 1:
  try:
    timezone = pytz.timezone(sys.argv[1])
  except:
    print("Impossible to use this timezone")
    sys.exit(-1)
else:
  timezone = None
 
def print_line(flags, t, mode, user, group, name):
    when = datetime.utcfromtimestamp(float(t))
    if timezone is not None:
        try:
            when = pytz.utc.localize(when).astimezone(timezone)
        except:
            print("Timezone issue!")
            sys.exit(-1)
    print(' '.join([t, when.isoformat(), flags, mode, user, group, name]))
 
for line in sys.stdin:
    line = line[:-1]
    (m, a, c, mode, user, group, name) = line.split(" ", 6)
    if m == a:
        if m == c:
            print_line("mac", m, mode, user, group, name)
        else:
            print_line("ma-", m, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)
    else:
        if m == c:
            print_line("m-c", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
        else:
            print_line("m--", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)
</pre>
 
== [Optional] Automated tests ==
 
You can run some automated tools to try to identify malicious activity/files, but you first need:
* To have these tools installed (e.g. [http://www.chkrootkit.org chkrootkit], [http://rkhunter.sourceforge.net rkhunter], [http://www.ossec.net/main/rootcheck ossec-rootcheck])
* To first remount all your local filesystems as read only. A first approach for this remount can use the following code, but you should rather review the list and remount them manually. <pre>for mountpoint in $(grep '^/dev/' /proc/mounts |cut -d ' ' -f 2 |sort -r); do &#10;  echo mount -o remount,ro $mountpoint&#10;done</pre>
 
You can also (after remounting all filesystems read only) use your package management system to verify installed packages (save the output). On RedHat based systems, this is '''rpm -Va''', on Debian based system, '''debsums'''.
 
== Stopping the system ==
 
Before stopping the system, remember to take out all the data you have already collected using e.g. scp or rsync if you extracted it in a tmpfs.
You might also want to run 'sync' to make sure that all data is written on disk.
 
There are two ways of stopping a system:
* Run the normal shutdown procedure: this will close all running program properly, which would preserve e.g. databases, but would write a lot on the disk, potentially destroying evidences
* Simply cut off the power (disconnect cables): this will kill all running programs, potentially damaging e.g. databases, but would preserve more evidences.
 
We usually recommend the later.
 
= Offline analysis =
 
Most of the analysis usually happens offline, where time pressure is more limited
 
== Copying disks ==
 
One of the most important rules of forensics is to never worker on the original supports, thus you need to copy the data (some will even recommend to only work on copies of copies in order to avoid copying the original again if you have a corruption).
 
There are mainly 3 ways of copying a disk :
* Use a disk imaging/mirroring device
* Use an external hard drive docking bay
* Use an internal hot-swappable connector
 
For the last two possibility, one has to be very cautious not to modify any data:
* The operating system on which the disk is plugged must have any auto-mount feature disabled
* If more than one device is connected, they should be plugged in sequentially, to make sure that their identifiers are properly mapped
* Any command typed can irremediably destroy data and should be checked multiple time
 
Once drives are identified, we recommend using dd to image the disk: <pre>dd if=/dev/sdX of=mybigfile.img bs=65536 conv=noerror,sync # inverting if and of will destroy all evidences!</pre>
 
== Collect filesystem metadata ==
 
While filesystem metadata can be collected from live systems, collecting it from cold disk have some advantages:
* While live system analysis can be affected by rootkit, this one cannot
* On ext4 filesystem, normal live tools do not extract the '''creation''' time. Forensic tools do
 
We recommend using fls and mactime from the [https://www.sleuthkit.org/ Sleuth Kit] for collecting such metadata, see the corresponding wiki page: http://wiki.sleuthkit.org/index.php?title=Timeline
 
== Carving unallocated blocks ==
 
Data on file-systems is not deleted when a file is deleted, but is kept on unallocated data blocks. These blocks might be overridden by the content of new files.
Until then, that data can be recovered from the harddrive:
* For a given pattern (e.g. sshd logs), it’s possible (and usually fruitful) to simply grep the raw disk
* Dedicated tools, like PhotoRec from the [http://www.cgsecurity.org/wiki/TestDisk_Download testdisk] utility, can try to rebuild files based on built-in patterns
 
= Exploiting forensic data =
 
== Reading file system timelines ==
 
FIXME
 
* Binary ctime or crtime modified while the package was not updated
* Traces of compilation: atime on files in /usr/include
* Impossible timestamp combinaisons:
** crtime after ctime
** crtime of a file/folder after ctime of its parent folder
 
== Malware analysis ==
 
FIXME
 
= Credits =
 
This document, compiled by Vincent Brillault in 2017, is based on:
* Trainings written by Leif Nixon
* A forensics HowTo written by Heiko Reese

Latest revision as of 15:17, 20 October 2021

Alert.png This article is Deprecated and has been moved to https://confluence.egi.eu/display/EGIBG/Forensics+Howto.