The first thing that needs to be clarified is what kind of information we are looking for in the logs.
Logging information should include:
- USER ID (Who?)
- System activity (What happened?)
- Timestamp (When?) (TZ info)
- System identifier (Where?)
- Network address/protocol
Which events should be recorded?
- System access attempts
- Data access attempts
- System/OS status and changes
- Utilization of elevated/root privileges
- File deletions or modifications (e.g., configuration, software, keys)
- System alerts and errors
- Access control
- Authentication
- Notifications and status from security programs (e.g., EDR, firewall)
- Services and processes status
Linux systems use syslog and write the logs to /var/log by default (all systemd services).
Log directory shall be owned by root.
sudo chown root /var/log
To monitor specific files or folders, auditing should be enabled as well.
Configure Linux Auditing System
Not all the events are logged by the journal. Linux Audit Daemon allows logging different system events such as executed commands, system calls, file access information and network statistics.
Sites should enable Linux Audit Daemon (auditd):
On RHEL based distributions:
sudo dnf install auditd sudo systemctl enable auditd
On Debian based distributions:
sudo apt-get install auditd audispd-plugins
When the daemon is running, rules need to be set up (default location is /etc/audit/rules.d/audit.rules):
The rules can be set up using auditctl utility.
# auditctl -help
usage: auditctl [options]
-a <l,a> Append rule to end of <l>ist with <a>ction
-A <l,a> Add rule at beginning of <l>ist with <a>ction
-b <backlog> Set max number of outstanding audit buffers
allowed Default=64
-c Continue through errors in rules
-C f=f Compare collected fields if available:
Field name, operator(=,!=), field name
-d <l,a> Delete rule from <l>ist with <a>ction
l=task,exit,user,exclude,filesystem
a=never,always
-D Delete all rules and watches
-e [0..2] Set enabled flag
-f [0..2] Set failure flag
0=silent 1=printk 2=panic
-F f=v Build rule: field name, operator(=,!=,<,>,<=,
>=,&,&=) value
-h Help
-i Ignore errors when reading rules from file
-k <key> Set filter key on audit rule
-l List rules
-m text Send a user-space message
-p [r|w|x|a] Set permissions filter on watch
r=read, w=write, x=execute, a=attribute
-q <mount,subtree> make subtree part of mount point's dir watches
-r <rate> Set limit in messages/sec (0=none)
-R <file> read rules from file
-s Report status
-S syscall Build rule: syscall name or number
--signal <signal> Send the specified signal to the daemon
-t Trim directory watches
-v Version
-w <path> Insert watch at <path>
-W <path> Remove watch at <path>
--loginuid-immutable Make loginuids unchangeable once set
--backlog_wait_time Set the kernel backlog_wait_time
--reset-lost Reset the lost record counter
--reset_backlog_wait_time_actual Reset the actual backlog wait time counter
Example:
auditctl -w /etc/passwd -p wa -k user-modify
In /etc/audit/auditd.conf you can specify the maximum log size, log retention, for example:
log_file = /var/log/audit/audit.log log_format = RAW flush = INCREMENTAL freq = 20 num_logs = 5 max_log_file = 30MB max_log_file_action = rotate space_left_action = email action_mail_acct = root
Examples of audit rules (eg. /etc/auditd/rules.d/90-egi.rules):
- Main Recommendations:
# command_log - Record all commands executed by users
-a always,exit -F arch=b64 -S execve -k command_log
-a always,exit -F arch=b32 -S execve -k command_log
#Record commands run by privileged users
-w /etc/sudoers -p wa -k actions
#Record file deletion by users
-a always,exit -F arch=ARCH -S rmdir -S unlink -S unlinkat -S rename -S renameat -F auid>=500 -F auid!=4294967295 -k delete
# Record modifications of User/Group Information; user_modify
-w /etc/group -p wa -k user_modify
-w /etc/passwd -p wa -k user_modify
-w /etc/gshadow -p wa -k user_modify
-w /etc/shadow -p wa -k user_modify
-w /etc/security/opasswd -p wa -k user_modify
#Record attemps to change Logon and Logout Events
-w /var/log/faillog -p wa -k logins
-w /var/log/lastlog -p wa -k logins
- Other Recommendations:
# audit_time_rules - Record attempts to alter time through adjtime -a always,exit -F arch=b64 -S adjtimex -k audit_time_rules # audit_time_rules - Record attempts to alter time through settimeofday -a always,exit -F arch=b64 -S settimeofday -k audit_time_rules # audit_time_rules - Record Attempts to Alter Time Through stime -a always,exit -F arch=b64 -S adjtimex -S settimeofday -S clock_settime -k audit_time_rules # audit_time_rules - Record Attempts to Alter Time Through clock_settime -a always,exit -F arch=b64 -S clock_settime -k audit_time_rules # Record Attempts to Alter the localtime File -w /etc/localtime -p wa -k audit_time_rules # Recording changes of network configuration; audit_network_modifications -a always,exit -F arch=ARCH -S sethostname -S setdomainname -k audit_network_modifications -w /etc/hosts -p wa -k audit_network_modifications -w /etc/sysconfig/network -p wa -k audit_network_modifications -w /etc/issue -p wa -k audit_network_modifications #Record events that modify the system access controls - example for lchown (same goes for fchown, fremovexattr, fsetxattr, removexattr, setxattr etc.) -a always,exit -F arch=b32 -S lchown -F auid>=500 -F auid!=4294967295 -k perm_mod -a always,exit -F arch=b64 -S lchown -F auid>=500 -F auid!=4294967295 -k perm_mod #Record session modification attempts -w /var/run/utmp -p wa -k session -w /var/log/btmp -p wa -k session -w /var/log/wtmp -p wa -k session #Record unauthorized unsuccessful access attempts to files -a always,exit -F arch=b32 -S creat -S open -S openat -S open_by_handle_at -S truncate -S ftruncate -F exit=-EACCES -F auid>=500 -F auid!=4294967295 -k access -a always,exit -F arch=b32 -S creat -S open -S openat -S open_by_handle_at -S truncate -S ftruncate -F exit=-EPERM -F auid>=500 -F auid!=4294967295 -k access -a always,exit -F arch=b64 -S creat -S open -S openat -S open_by_handle_at -S truncate -S ftruncate -F exit=-EACCES -F auid>=500 -F auid!=4294967295 -k access -a always,exit -F arch=b64 -S creat -S open -S openat -S open_by_handle_at -S truncate -S ftruncate -F exit=-EPERM -F auid>=500 -F auid!=4294967295 -k access #Record events related to module loading/unloading -w /sbin/insmod -p x -k modules -w /sbin/rmmod -p x -k modules -w /sbin/modprobe -p x -k modules -a always,exit -F arch=b64 -S init_module -S delete_module -k modules #Make the auditd Configuration Immutable -e 2
Search the audit logs
The tool that facilitates searching the audit logs is called ausearch, the search is based on key_name. An example:
sudo ausearch -i -k user-modify
For other rules, check Linux Audit Repository
See also: RedHat Audit HowTo
Enable logging on the system
Some Linux distributions do not have "rsyslog" enabled by default, relying instead on "systemd-journald" for logging. This is common in systems like AlmaLinux 9, CentOS Stream 9, Fedora, and other RHEL-based distributions. For this reason "rsyslog" must be manually enabled. Please follow the instructions below:
dnf -y install rsyslog systemctl enable rsyslog.service systemctl start rsyslog.service
Remote logging
In the operations policy for RC, a mandatory requirement is set to use a remote logging service. In case of a compromise, logs on the compromised machine are often deleted. In this case remote logs can facilitate the analysis of the incident.
Multiple solutions are available for remote logging, such as:
Some of the solutions also provide visualisation options and alerting functionality. EGI CSIRT does not have a preference regarding the solution selected, provided that the logs are available on the remote server.
Remote logging using rsyslog
Rsyslog operates on client-server model. It enables collecting messages from different sources. Logs can be processed (filtered), based on severity, content or source. Also formatting can be modified. Logs can be transferred via UDP, TCP or TLS.
By default rsyslog uses UDP protocol for message transfers, as it is faster and does not create any overhead.
Best practices:
- Standardise log message format across the systems to ease parsing and analysis (rsyslog modification modules such as mmanon that anonymises IP, mmfields extracts specific field from logs, mmjsonparse etc).
- Use network time synchronisation on all clients and the rsyslog server.
- Use TCP for message transfers to ensure reliable log delivery.
- Enable encryption using TLS for secure transmission of log data over network (by default rsyslog sends messages in plain text).
- Apply access controls and authentication mechanisms to restrict access to rsyslog data.
Logs (outputs) can be saved either as files or databases using output modules (omfile is used by default and saves the logs as files on the local system, ommysql is used to save files to MySQL db, omelasticsearch is used to save output to Elasticsearch, ompgsql for PostGres).
Additional packages need to be installed to support this. Example for PostGresSQL:
dnf install rsyslog-pgsql
Rsyslog server configuration
The configuration file is /etc/rsyslog.conf. When modifications of the configuration files are done, the service needs to be restarted.
By default the service rsyslog listens on 514 port, for both TCP and UDP, UDP is used by default and the server is listening on localhost. If you want to specify a specific interface configure also the IP address:
input(type="imtcp" port="514" address="xx.xx.xx.xx")
Also IPv6 address can be used.
If you have SELinux enabled, run the following commands:
semanage port -a -t syslogd_port_t -p tcp 514 semanage port -a -t syslogd_port_t -p udp 514
Add the rule for Firewalld:
firewall-cmd --permanent --add-port=514/tcp firewall-cmd --permanent --add-port=514/udp firewall-cmd --reload
Optional for system admins: Logs can be seperated on a device-by-device basis
Logs can be seperated into individual directories stored in a configurable directory. In this example logs will be stored in /var/logs/remote:
- Add to `/etc/rsyslod.conf` on the machine which logs are sent to (logging server):
# Template to dynamically generate a filename based on the remote hostname template(name="RemoteLogs" type="string" string="/var/log/remote/%hostname%/%programname%.log") # Log all messages from remote machines using the template *.* ?RemoteLogs
(Can be anywhere in the file, usually put it at the end for convinience)
- Restart rsyslogd: `sudo systemctl restart rsyslogd`
- New directories will now begin to appear in `/var/log/remote/` for each device which send log files to the logging server. In each of these directories will be files containing logs for each program on the client. For example:
remote
├── server1
│ ├── CROND.log
│ ├── dnf.log
│ ├── rsyslogd.log
│ ├── run-parts.log
│ ├── sshd.log
│ ├── systemd.log
│ └── systemd-logind.log
└── server2
├── chronyd.log
├── CROND.log
├── dnf.log
├── rsyslogd.log
├── run-parts.log
├── sshd.log
├── systemd.log
└── systemd-logind.log
TLS transfers
To enable TLS for message transmission, install also rsyslog-openssl package (in some cases rsyslog-gnutls). This should be done on server and client side. TLS connection will use 1514 port. A certificate needs to be installed and configured in rsyslog.conf.
For detailed instructions see: Encrypting Syslog Traffic with TLS
Rsyslog client configuration
To send logs to the rsyslog remote server over TCP, specify this in the /etc/rsyslog.conf using double "@@":
*.* @@xx.xx.xx.xx:514
To send logs to the rsyslog remote server over UDP, a single "@" should be used:
*.* @xx.xx.xx.xx:514
Also IPv6 address can be used.
Protection
Apply access rules to limit the access to the logs.
Other options to protect the logs include:
- Cryptographic hashing (Configure rsyslog with TLS)
- append-only logs (in case there is a root compromise, logs cannot be deleted ( eg. chattr +a /var/log/auth.log),
- read-only logs.