Host
What's running on this box right now?
ps -eo pid,ppid,user,lstart,etime,cmd --sort=start_timeThe money command. Parent PID, start time and full command line in one list, oldest first.ps -ef --forestProcess tree. A shell hanging off nginx or a cron job spawning curl reads instantly.ps -eo pid,user,%cpu,%mem,cmd --sort=-%cpu | head -20What's burning cycles. Miners show up here.ls -l /proc/PID/exe; cat /proc/PID/cmdline | tr '\0' ' '; ls -l /proc/PID/cwdWhere the binary really lives, exactly how it was launched, and where it's working. "(deleted)" on exe means the file was removed after launch.ls -la /proc/*/exe 2>/dev/null | grep deletedEvery process whose binary has been deleted from disk. Classic fileless malware tell.systemctl list-units --type=service --state=runningRunning services. Compare against a known-good host.lsof -p PIDEvery file, socket and library a process has open.cat /proc/PID/environ | tr '\0' '\n'Environment of a process. LD_PRELOAD or odd HTTP_PROXY here is a red flag.Host
Who's talking to what?
ss -tunapAll TCP/UDP sockets with owning process. Replaces netstat; run as root to see every PID.ss -tlnpListeners only. A new port on a box that shouldn't have one.ss -tnp state establishedLive outbound connections. Sort by remote address and look for the one that doesn't belong.lsof -i -nPSame data, lsof style. -nP skips DNS and port-name lookups so it doesn't hang.cat /etc/hosts; cat /etc/resolv.confHosts file hijacks and swapped resolvers. Quick and often skipped.dig +short suspicious-domain.com; dig +short -x 203.0.113.10Forward and reverse lookups for an IOC.iptables -L -n -v; nft list rulesetFirewall rules. Attackers add accepts, or flush everything.tcpdump -i any -nn -c 200 'not port 22'Two hundred packets of whatever's happening, minus your own SSH. Add -w capture.pcap to save it.arp -a; ip neighWho this host has talked to on the local segment.Host
What's in the logs?
Debian/Ubuntu keep auth in /var/log/auth.log; RHEL/Rocky/Amazon use /var/log/secure. journalctl works on both.
journalctl --since "24 hours ago" -p warningEverything warning and above in the last day.journalctl -u sshd --since todayOne service's log. Swap sshd for cron, sudo, nginx.journalctl _COMM=sudo --since "7 days ago"Every sudo invocation this week, with who ran what.grep -E 'Accepted|Failed' /var/log/auth.log | tail -100SSH successes and failures. Accepted publickey from an IP you don't know is the scary one.grep 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | headTop attacking IPs by failed logins.last -aiF; lastb -aiF | headLogins and failed logins from wtmp/btmp with IPs and full timestamps.lastlogLast login for every account. Service accounts that suddenly have one.ausearch -k KEY --start today; aureport --auth --summaryauditd, if it's configured. aureport -x for executed programs.journalctl --list-boots; journalctl -b -1 -ePrevious boot's log. Reboots are how attackers clear their tracks.Reference
Paths worth memorizing
| Path | What's there |
|---|---|
| /var/log/auth.log, /var/log/secure | SSH, sudo, su, PAM |
| /var/log/syslog, /var/log/messages | General system log |
| /var/log/wtmp, btmp, lastlog | Login history (binary; use last / lastb / lastlog) |
| /var/log/audit/audit.log | auditd events |
| /var/log/cron* | Cron executions |
| /var/log/apache2, /var/log/nginx | Web server access and error logs |
| ~/.bash_history, ~/.zsh_history | Shell history (only if not disabled or unlinked) |
| Path | What's there |
|---|---|
| /etc/passwd, /etc/shadow, /etc/group | Accounts, hashes, groups |
| /etc/sudoers, /etc/sudoers.d/ | Who can sudo what |
| ~/.ssh/authorized_keys | SSH keys that can log in as that user |
| /etc/cron*, /var/spool/cron/ | System and user cron jobs |
| /etc/systemd/system, ~/.config/systemd/user | Custom service units and timers |
| /etc/ld.so.preload | Libraries injected into every process (should be empty or absent) |
| /tmp, /var/tmp, /dev/shm | World-writable drop zones |
Host
How did it persist?
for u in $(cut -f1 -d: /etc/passwd); do crontab -l -u $u 2>/dev/null | sed "s/^/$u: /"; doneEvery user's crontab in one pass.cat /etc/crontab; ls -la /etc/cron.*; cat /etc/cron.d/*System cron. Look for curl | sh, base64, or paths in /tmp.systemctl list-timers --allsystemd timers are the modern cron. Fewer people check them.find /etc/systemd /usr/lib/systemd /home/*/.config/systemd -name '*.service' -mtime -7Service units created or changed this week.systemctl cat NAME.serviceRead the unit. ExecStart pointing at /tmp or /dev/shm is game over.cat /etc/ld.so.preload; echo $LD_PRELOADUserland rootkit tell. This file should not exist on most systems.for h in /root /home/*; do echo "== $h"; cat $h/.ssh/authorized_keys 2>/dev/null; doneEvery SSH key that can log in. Unknown key equals unknown person with access.grep -rE 'curl|wget|nc |bash -i|/dev/tcp' /etc/profile /etc/profile.d /etc/bash.bashrc /home/*/.bashrc /home/*/.profile /root/.bashrc 2>/dev/nullShell startup files that phone home.cat /etc/rc.local; ls -la /etc/init.d/ /etc/rc*.d/Old-school init persistence. Still works on plenty of boxes.lsmod; cat /proc/modulesLoaded kernel modules. Compare against a clean host for rootkits.Host
What is this file?
sha256sum /path/to/file; md5sum /path/to/fileHash first, then check VT / your EDR / threat intel.file /path/to/file; strings -n 8 /path/to/file | lessWhat it really is regardless of extension, and the readable strings inside (URLs, IPs, commands).stat /path/to/fileAccess, modify, and change times. ctime can't be faked with touch.find / -xdev -newermt "2026-09-03" -type f -not -path '/proc/*' 2>/dev/nullEvery file modified since a date. Narrow the date to the incident window.find / -xdev -type f -mmin -60 -not -path '/proc/*' 2>/dev/nullModified in the last hour.find / -xdev -perm -4000 -type f 2>/dev/nullSUID binaries. A new one is a privilege escalation backdoor.find /tmp /var/tmp /dev/shm -type f -executable 2>/dev/nullExecutables in the drop zones.find / -xdev -nouser -o -nogroup 2>/dev/nullFiles owned by a UID that no longer exists. Leftovers from a deleted account.rpm -Va | grep '^..5'; debsums -cPackaged binaries whose hash no longer matches. Trojaned ls, ps, ss.ls -la --time-style=full-iso /path/Full timestamps and dotfiles. Filenames with a trailing space or "..." are hiding.Identity
Who's on this machine and what can they do?
id; whoami; sudo -lWho you are and what you can escalate to. First thing on any shell.w; who -aWho's logged in now, from where, running what.awk -F: '$3 == 0' /etc/passwdEvery UID 0 account. There should be exactly one.awk -F: '$3 >= 1000 && $7 !~ /nologin|false/' /etc/passwdHuman accounts with a real shell.awk -F: '$2 == ""' /etc/shadowAccounts with no password.ls -la --time-style=full-iso /etc/passwd /etc/shadow /etc/sudoersWhen the account files last changed.cat /etc/sudoers /etc/sudoers.d/* | grep -v '^#'NOPASSWD: ALL entries that weren't there before.getent group sudo wheel docker admMembers of the groups that equal root.for h in /home/* /root; do echo "== $h"; tail -50 $h/.bash_history 2>/dev/null; doneShell history. Unreliable, but attackers forget to clear it more than you'd think.Analysis
Carving logs on the command line
This is why bash matters for a SOC analyst. Any log, any format, the same six tools: grep, awk, cut, sort, uniq, and a pipe.
grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' file.log | sort | uniq -c | sort -rn | head -20Top IPs in any log, any format. Change the regex, change the answer.awk '{print $1}' access.log | sort | uniq -c | sort -rn | headTop source IPs in a web access log.awk '$9 == 404 {print $7}' access.log | sort | uniq -c | sort -rn | headMost-requested paths that 404'd. Scanner fingerprint.grep -iE 'union.*select|\.\./|/etc/passwd|cmd=|wget|curl' access.logCheap web attack grep. Add your own patterns.grep -h 'Accepted' /var/log/auth.log* | awk '{print $9, $11}' | sort | uniq -c | sort -rnUser and source IP pairs for every successful SSH login, across rotated logs.zgrep 'pattern' /var/log/auth.log.*.gzSearch rotated, compressed logs without unpacking them.awk '$0 >= "Sep 3 14:00" && $0 <= "Sep 3 15:00"' /var/log/syslogSlice a syslog to a one-hour window.cut -d' ' -f1-3 file.log | uniq -cEvents per timestamp. Bursts jump out.grep -v -E 'known-noise-1|known-noise-2' file.log | lessSubtract the noise until only the story is left.strings -n 8 memdump.raw | grep -iE 'http|\.onion|password|BEGIN RSA'Pulling the interesting bits out of a memory dump or unknown binary.tail -f /var/log/auth.log | grep --line-buffered 'Failed'Watch a log live. --line-buffered keeps grep from holding output back.base64 -d <<< 'ZWNobyBoaQo='Decode the payload from that cron job or webshell.Fleet
Remote triage
ssh admin@host 'ps -eo pid,ppid,user,cmd --sort=start_time'Run one command and get the output back, no interactive session.for h in $(cat hosts.txt); do echo "== $h"; ssh -o ConnectTimeout=5 admin@$h 'ss -tlnp'; doneSame command across a list of hosts.ansible all -i hosts.txt -m shell -a 'cat /etc/ld.so.preload 2>/dev/null' -bSame idea, faster, if Ansible is set up.scp -r admin@host:/var/log/ ./evidence/host/Pull logs before they rotate or get wiped.ssh admin@host 'sudo tar czf - /var/log /etc /home/*/.ssh /tmp' > host-evidence.tgzOne tarball of the interesting directories, streamed straight back.ssh admin@host 'sudo dd if=/dev/sda bs=4M' | pv > host-disk.imgFull disk image over SSH. Slow, but you only get one shot before something changes.Response
Contain it
kill -STOP PIDFreeze the process without killing it. Memory stays intact for a dump.cp /proc/PID/exe ./evidence/PID.bin; kill -9 PIDGrab the binary (works even if deleted from disk), then kill.systemctl stop NAME; systemctl disable NAME; systemctl mask NAMEStop it, keep it stopped, and stop anything from re-enabling it.usermod -L user; passwd -l user; pkill -KILL -u userLock the account and end its sessions.chattr +i /etc/passwd /etc/shadow /etc/sudoersMake the account files immutable while you work. chattr -i to undo.iptables -I OUTPUT -d 203.0.113.10 -j DROPBlock the C2 at the host. -I puts it first in the chain.iptables -I INPUT -s 198.51.100.0/24 -j DROPBlock an attacking range.crontab -r -u user; mv /etc/cron.d/bad /root/evidence/Remove persistence after you've copied it.Before you contain: hash it, copy it, capture memory if you can. kill -STOP first, kill -9 second. Order matters.
Practice
Habits that matter more than any single command
Start every session with script -a ~/triage-$(hostname)-$(date +%F).log. Your own commands and their output become part of the timeline.
Don't trust the box's binaries on a suspected rootkit. Bring statically linked ps, ss, ls and find on a USB or mount, or compare output against /proc directly.
Redirect stderr with 2>/dev/null on any find across the whole filesystem, and add -xdev so you don't walk into /proc, /sys, or an NFS mount.
Learn sort | uniq -c | sort -rn as one word. It turns any column of a log into a frequency table, and that's half of log analysis.
Timestamps lie in different ways. stat for ctime, date -u to know what timezone the host thinks it's in, and timedatectl to know whether it's even synced.
Collect before you contain. Copy the binary out of /proc, tar the logs, dump memory, then kill. The order is the difference between a root cause and a shrug.
Nothing matches. Try a shorter word or a path.