Reading the Logs

Hardening tells you what to fix. The logs tell you what already happened, which on a competition image often points straight at the backdoor, the guessed account, or the file that was dropped.

Where the logs are

Log	Contains	Read with
/var/log/auth.log	Logins, sudo, SSH, PAM, faillock	sudo grep -E 'sshd|sudo|faillock' /var/log/auth.log
journalctl	Everything systemd sees, including the above	sudo journalctl -u ssh --since today
/var/log/syslog	General system messages, cron, kernel	sudo tail -f /var/log/syslog
/var/log/audit/audit.log	auditd events	sudo ausearch, sudo aureport
/var/log/sudo.log	Every sudo command (once the Sudo Defaults logfile is set)	sudo cat /var/log/sudo.log
/var/log/ufw.log	Blocked packets	sudo grep BLOCK /var/log/ufw.log
/var/log/apache2/, /var/log/nginx/, /var/log/mysql/, /var/log/vsftpd.log, /var/log/samba/	The server applications
wtmp, btmp, lastlog	Login history, failed logins, last login per user	last, sudo lastb, lastlog

Questions the logs answer

Who logged in, and from where?

last -a | head -n 30
sudo journalctl _COMM=sshd --since -7d | grep -E 'Accepted|Failed' | tail -n 30

Accepted password for bob from 203.0.113.9 at 3 a.m. from an address the README doesn't describe is a compromised account. Change its password and check what it did (sudo -u, cron, files).

Who's been guessing?

sudo lastb | awk '{print $3}' | sort | uniq -c | sort -rn | head
sudo grep -E 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head

Hundreds of failures from one address is a brute force. If faillock is set up, that account locked; if it isn't, this is why it matters.

Who used sudo, for what?

sudo grep -E 'sudo:.*COMMAND' /var/log/auth.log | tail -n 40
sudo ausearch -k user_emulation -i --start this-week | grep -E 'proctitle|uid=' | tail -n 40

A sudo command that adds a user, edits sudoers, or runs something in /tmp is the moment the image was compromised. The account that ran it is the one to look at.

What changed in a file you hardened?

sudo ausearch -k scope -i
sudo ausearch -k identity -i --start today
sudo ausearch -f /etc/ssh/sshd_config -i

Only works for events after auditd was installed and the rules loaded; that's the reason to install it early.

Did the service log anything odd?

sudo journalctl -p err --since yesterday
sudo tail -n 50 /var/log/apache2/access.log | grep -E '\.php\?|cmd=|\.\./'

A web log full of ?cmd= or ../../etc/passwd is a web shell in use or an attack in progress.

Keep the logs safe

Logs are owned by root:adm (or root:root), mode 640 or tighter, and /var/log is 755 (CIS 6.1.3.1). A world-writable log can be edited to hide activity.

sudo find /var/log -type f -perm /037 -exec chmod 640 {} +
sudo find /var/log -type f \( ! -user root -a ! -user syslog \) -exec chown root:adm {} +

The journal files themselves, under /var/log/journal, stay at their defaults of 640 with group systemd-journal (6.1.1.2). systemd-journald must be running (6.1.1.1), and its config in /etc/systemd/journald.conf or a drop-in should keep the journal across reboots and stop it filling the disk (6.1.1.3, 6.1.2.3, 6.1.2.4):

sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/60-journald.conf > /dev/null <<'EOF'
[Journal]
Storage=persistent
Compress=yes
SystemMaxUse=1G
SystemKeepFree=500M
RuntimeMaxUse=200M
MaxFileSec=1month
EOF
sudo systemctl restart systemd-journald

Mint and Debian both run rsyslog alongside the journal, which is why /var/log/auth.log exists. The Mint benchmark's ForwardToSyslog=no (6.1.2.2) is only for systems that use the journal alone; the Debian benchmark, which prefers rsyslog, wants the opposite, ForwardToSyslog=yes (Debian CIS 6.1.1.1.3). Leave forwarding on. The Level 2 items about systemd-journal-remote (6.1.2.1.x) are for shipping logs to another machine and don't apply to a single image.

For rsyslog itself (Debian CIS 6.1.2.1 to 6.1.2.7):

sudo apt install -y rsyslog
sudo systemctl enable --now rsyslog
grep -q '^\$FileCreateMode' /etc/rsyslog.conf || echo '$FileCreateMode 0640' | sudo tee /etc/rsyslog.d/00-filemode.conf
sudo grep -rE 'imtcp|imudp|InputTCPServerRun|UDPServerRun' /etc/rsyslog.conf /etc/rsyslog.d/     # nothing; the machine doesn't accept logs from the network
sudo systemctl restart rsyslog
cat /etc/logrotate.conf | grep -E '^(weekly|daily|rotate)'

$FileCreateMode 0640 means new log files aren't world-readable. A module(load="imtcp") or $ModLoad imudp line turns the machine into a log server listening on 514, which the README won't ask for. logrotate keeps logs from filling the disk; the defaults (weekly, keep 4) are fine. The Level 2 items about sending logs to a remote host over TLS (rsyslog-gnutls, 6.1.2.5 and 6.1.2.8 to 6.1.2.10) need a log server the image doesn't have.

Verify

systemctl is-active rsyslog systemd-journald auditd
stat -c '%a %U:%G %n' /var/log/auth.log /var/log/syslog

Example

last -a shows hacker logging in over SSH from 10.0.0.99 every night at 02:00. sudo grep 'sudo:.*hacker' /var/log/auth.log shows hacker : COMMAND=/usr/bin/crontab -e and COMMAND=/usr/sbin/useradd -o -u 0 toor. So: remove hacker and toor, clear hacker's crontab, check /etc/sudoers.d for how hacker got sudo, and set up faillock and SSH AllowUsers so the nightly login stops.

Try it

1. Log in wrong three times over SSH from another VM and find the lines in auth.log and with lastb.
2. Run sudo a few times and find each command in auth.log and sudo.log.

Build it

A whathappened.sh that prints the last 20 SSH accepts and failures, the last 20 sudo commands, last -a | head, and lastb | head. Run it during forensics questions.

Next

File Integrity with AIDE