Malware and Persistence

A program running that shouldn't be is half the problem. The other half is whatever starts it again: a cron job, a service, a login script. Kill the process, then find and remove every way it comes back.

Running processes

ps -eo pid,user,%cpu,etime,cmd --sort=-%cpu | head -n 20
sudo ss -tunap | grep ESTAB

Look for: names that mimic system processes (kworkerd, sshd with a trailing space, [kthreadd] running from a path), anything running from /tmp, /dev/shm, /var/tmp, or a home directory, a process with an outbound connection to an address you don't recognize, and high CPU from something you can't name (a miner).

sudo ls -l /proc/1234/exe /proc/1234/cwd
sudo cat /proc/1234/cmdline | tr '\0' ' '

/proc/PID/exe is the real binary, even if the process renamed itself or the file was deleted ("(deleted)" in the output means it's running from memory; copy /proc/PID/exe somewhere if you want to look at it later, then kill it).

sudo kill 1234
sudo kill -9 1234          # if it ignores the first
sudo pkill -f '/tmp/.x'    # by command line pattern

Where things restart from

Mechanism	Check	Remove
System cron	sudo ls -la /etc/cron.d /etc/cron.hourly /etc/cron.daily; sudo cat /etc/crontab	Delete the file or the line
User cron	for u in $(cut -d: -f1 /etc/passwd); do sudo crontab -l -u $u 2>/dev/null | sed "s/^/$u: /"; done	sudo crontab -r -u user, or -e to edit
systemd services	ls -l /etc/systemd/system/ /etc/systemd/system/*.wants/; systemctl list-unit-files --state=enabled	systemctl disable --now name; rm the unit; systemctl daemon-reload
systemd timers	systemctl list-timers --all	Same
User systemd units	ls /home/*/.config/systemd/user/	Same, as that user
rc.local	cat /etc/rc.local	Remove the line
Login scripts	grep -E 'curl|wget|nc |bash -i|/tmp|/dev/shm' /etc/profile /etc/bash.bashrc /etc/profile.d/* /home/*/.bashrc /home/*/.profile /root/.bashrc	Remove the line
Desktop autostart	ls /etc/xdg/autostart /home/*/.config/autostart	Delete the .desktop file
at jobs	sudo atq	sudo atrm N
Preloaded libraries	cat /etc/ld.so.preload	Should be empty or absent
Kernel modules	lsmod; ls /etc/modules-load.d	rmmod, delete the file; then Kernel Hardening
SSH keys	authorized_keys files, on the SSH page
Setuid binaries	sudo find / -perm -4000 -type f 2>/dev/null	A setuid copy of bash or nano in an odd place is a root backdoor; chmod u-s and delete
Aliases and PATH	alias; echo $PATH; cat ~/.bashrc	An alias for sudo or ls, or . at the front of PATH, runs something else when you type a command

Rootkit scan

sudo apt install -y rkhunter chkrootkit
sudo chkrootkit
sudo rkhunter --update && sudo rkhunter --check --sk

They produce false positives (hidden files they don't recognize); read the warnings rather than trusting the summary.

Verify

ps -eo pid,user,cmd | grep -E '/tmp|/dev/shm|/var/tmp' | grep -v grep
sudo crontab -l -u root; sudo ls /etc/cron.d
systemctl list-units --type=service --state=running

Example

ps shows python3 /tmp/.x/s.py using 2% CPU, connected to 198.51.100.7:4444. kill it. sudo crontab -l -u www-data has * * * * * python3 /tmp/.x/s.py; crontab -r -u www-data. /etc/systemd/system/backup.service has ExecStart=/tmp/.x/s.py; disable, delete, daemon-reload. rm -r /tmp/.x. Then check www-data's shell (should be nologin), which is how the cron job got created in the first place.

Try it

1. Add a cron job for a test user that runs touch /tmp/x every minute, find it with the listing commands, remove it.
2. Create a systemd service that runs nc -l 4444, find it from the port, from ps, and from the unit list, then remove it.

Build it

A persistence.sh that prints every user's crontab, the cron.d files, unit files in /etc/systemd/system, timers, rc.local, ld.so.preload, and setuid binaries outside the known list. Read-only.

Next

File Permissions and Ownership