awk for SOC Analysts

Start here

The whole language in one idea

awk reads input one line at a time and, for each line, checks a pattern. When the pattern is true it runs the action in braces. That's it. Everything below is a variation on pattern { action }.

awk 'pattern { action }' fileThe shape of every awk program. Omit the pattern and it runs on every line. Omit the action and it prints matching lines.

awk '/Failed/' auth.logPattern only. Prints every line containing "Failed", like grep.

awk '{ print $1 }' access.logAction only. Runs on every line, prints the first field.

awk '/Failed/ { print $1 }' auth.logBoth. Only for matching lines, print the first field.

awk 'BEGIN { print "start" } { c++ } END { print c, "lines" }' fileBEGIN runs once before input, END runs once after. This is where totals and headers live.

Core

Fields: $1, $2, $NF

awk splits each line into fields on whitespace by default. This is the single most useful thing it does.

awk '{ print $1 }'First field.

awk '{ print $NF }'Last field, whatever the line length. $(NF-1) is second-to-last.

awk '{ print $1, $NF }'Comma between fields prints a space (OFS). No comma and they're jammed together.

awk '{ print $0 }'The whole line, unchanged.

awk -F: '{ print $1 }' /etc/passwd-F sets the separator. Colon here pulls the username out of passwd.

awk -F'","' '{ print $3 }' data.csvA multi-char separator for quote-comma-quote CSVs. Good enough when fields have no embedded commas.

awk 'NF' fileNF is the field count. As a bare pattern it's true when the line has fields, so this drops blank lines.

awk '{ $2=""; print }'Blank out a field. Reassigning any field rebuilds the line using OFS.

Reference

Built-in variables

Var	Is	Common use
$0	The whole line	`print $0`, or match against it
$1…$NF	Fields 1 through last	Pull columns out of a log
NF	Number of fields on this line	Find malformed rows: `NF != 7`
NR	Record (line) number, running total	`NR==1` skips a header; `NR%2`
FNR	Line number within the current file	Tell files apart when reading several
FS	Input field separator	Set with `-F` or in BEGIN
OFS	Output field separator	`BEGIN{OFS="\t"}` for TSV out
RS / ORS	Record separators (in / out)	`RS=""` reads paragraph blocks
FILENAME	Name of the current input file	Label output when globbing logs

The one to internalize: NR counts every line, NF counts fields on a line. Half of awk is those two plus $NF.

Core

Patterns: choosing the lines

awk '/regex/'Line matches a regex. Same as grep.

awk '$9 == 404'A field equals a value. Numbers compare numerically.

awk '$9 >= 500'Field comparison. Server errors in an access log.

awk '$7 ~ /admin/'Field matches a regex. !~ is "does not match".

awk '$3 == "root" && $9 == "Accepted"'Combine with && and ||. Reads like the sentence you'd say out loud.

awk 'NR > 1'Skip the header row.

awk '/START/,/STOP/'Range pattern. Every line from a START match through the next STOP match, inclusive.

awk '!seen[$0]++'Dedupe while preserving order. The classic. First time a line is seen the count is 0 (falsey after !), so it prints once.

SOC bread and butter

Counting and tallying

Associative arrays are where awk pulls ahead of grep. Any field becomes a key; you accumulate as you go and print in END.

awk '{ c[$1]++ } END { for (k in c) print c[k], k }'Count occurrences of field 1. This is sort | uniq -c without the sort passes.

awk '{ c[$1]++ } END { for (k in c) print c[k], k }' access.log | sort -rn | headTop talkers. Pipe to sort because awk arrays aren't ordered.

awk '$9==404 { c[$7]++ } END { for (k in c) print c[k], k }'Count only the 404s, keyed by URL. Filter and tally in one pass.

awk '{ bytes[$1] += $10 } END { for (k in bytes) print bytes[k], k }'Sum a field per key. Total bytes sent to each IP: data exfil signal.

awk '{ c[$1" "$7]++ } END { for (k in c) print c[k], k }'Composite key. Count IP-and-URL pairs by joining fields into the key.

awk '!seen[$5]++ { print $5 }'Unique values of a field, in first-seen order, no sort needed.

awk 'END { print NR }'Line count. Same as wc -l but composes with a pattern: add one to count only matches.

awk '/Failed/ { c[$(NF-3)]++ } END { for (k in c) if (c[k] > 10) print c[k], k }'Only report keys over a threshold. Brute-force sources with more than 10 failures.

The job

Log analysis, worked out

Real one-liners against the logs you actually touch. Field numbers assume common formats; run awk '{print NF; for(i=1;i<=NF;i++) print i, $i; exit}' on a sample line to confirm your columns first.

awk '($9 ~ /^[45]/) { c[$1]++ } END { for (k in c) print c[k], k }' access.log | sort -rn | headHosts generating the most 4xx/5xx. Scanners and broken integrations both surface here.

awk '$6 ~ /POST/ && $7 ~ /login/ { print $1 }' access.log | sort | uniq -c | sort -rnCredential stuffing: who's POSTing to the login endpoint, ranked.

awk '/Accepted/ { print $(NF-5), $(NF-3) }' /var/log/auth.log | sort | uniq -cSuccessful SSH logins as user + source-IP pairs, tallied. The unfamiliar pair is your lead.

awk '/Failed password/ { ip[$(NF-3)]++ } END { for (i in ip) print ip[i], i }' /var/log/auth.log | sort -rn | headTop SSH brute-force sources.

awk '{ split($4,d,":"); h[d[2]]++ } END { for (k in h) print k, h[k] }' access.log | sortRequests per hour. Splits the [10/Sep/2026:14:… timestamp on colons and keys on the hour. A spike at 3am is a story.

awk -F'"' '{ ua[$6]++ } END { for (k in ua) print ua[k], k }' access.log | sort -rn | headUser-Agent frequency (split on quotes; UA is the 6th quoted field in combined format). curl, python-requests, and sqlmap stand out.

awk '$10 > 1000000 { print $1, $7, $10 }' access.logResponses over ~1 MB. Large downloads from odd endpoints = possible exfil.

awk 'prev && $1==prev_ip && ($4!=prev) { print } { prev=$4; prev_ip=$1 }' access.logSkeleton for spotting same-IP requests across changing timestamps. Adapt the condition to your hunt.

awk -F, 'NR==1{next} { c[$3]++ } END { for (k in c) print c[k], k }' export.csv | sort -rnTally a column of an exported CSV (SIEM export, EDR dump), skipping the header.

awk 'NF != 12 { print FILENAME":"NR": "NF" fields" }' *.logFind malformed lines across many files. Injected or truncated log entries have the wrong field count.

Core

Reshaping and extracting text

awk '{ gsub(/[0-9]/, "#"); print }'gsub replaces every match in the line, in place. sub does just the first.

awk '{ print substr($0, 1, 15) }'substr(string, start, length). Grab the syslog timestamp prefix.

awk '{ n = split($7, parts, "/"); print parts[2] }'split() breaks a field into an array and returns the count. Pull a path segment.

awk 'match($0, /[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/) { print substr($0, RSTART, RLENGTH) }'match() sets RSTART and RLENGTH so substr can carve out exactly what matched. Extract the IP from anywhere in the line.

awk '{ print toupper($1), tolower($2) }'Case folding for normalizing before you dedupe.

awk 'BEGIN{OFS="\t"} { print $1, $4, $7 }' access.logRe-emit chosen columns as clean TSV, ready for a spreadsheet or the next tool.

awk '{ printf "%-16s %s\n", $1, $7 }'printf for aligned columns. %-16s left-justifies in 16 chars.

awk 'length($0) > 500'Absurdly long lines. Encoded payloads and log-injection attempts run long.

Core

Math, stats, and time

awk '{ sum += $10 } END { print sum }'Sum a column. Total bytes, total events.

awk '{ sum += $10 } END { print sum/NR }'Mean. Average response size.

awk 'NR==1{min=max=$1} { if($1<min)min=$1; if($1>max)max=$1 } END { print min, max }'Min and max of a column in one pass.

awk '{ a[$1]+=$10 } END { for(k in a) print a[k], k }' | sort -rn | headSum per key, ranked. Bytes per source IP, biggest first.

awk 'BEGIN { print strftime("%F %T", 1757000000) }'Turn an epoch timestamp into a readable date (gawk). Logs full of Unix time become legible.

awk '{ print systime() - $1, "seconds ago" }'systime() is now in epoch seconds (gawk). Age of an event.

awk 'BEGIN { print mktime("2026 09 03 14 00 00") }'Build an epoch from parts (gawk) to compare against log timestamps.

Keepers

Full recipes worth saving

Slightly longer programs. Drop them in a file and run with awk -f name.awk logfile when they outgrow a one-liner.

awk '{ c[$1]++ } END { for (k in c) if (c[k] > 100) print c[k]"\t"k }' access.log | sort -rnOnly sources over 100 requests, tab-separated and ranked. A rate-limit candidate list in one line.

awk 'NR==1 { for(i=1;i<=NF;i++) col[$i]=i; next } { print $col["src_ip"], $col["action"] }' export.tsvReference columns by header name instead of number. Survives a SIEM changing its column order.

awk 'FNR==NR { bad[$1]; next } ($1 in bad)' iocs.txt access.logTwo-file join. Load IOC IPs from the first file into a set, then print access-log lines whose source is in it. This is a poor-man's threat-intel match.

awk '/Failed password/ { f[$(NF-3)]++ } /Accepted/ { if (f[$(NF-5)] > 5) print $(NF-5), "succeeded after", f[$(NF-5)], "failures from", $(NF-3) }' auth.log

Brute force that eventually worked: an account with many failures that then logs in. The alert you actually want.

awk '{ t=substr($4,2,20); c[t]++ } END { for (k in c) print k, c[k] }' access.log | sort | awk '$2 > avg*3' avg=$(...)Sketch of per-minute spike detection: bucket by timestamp, then flag buckets far above baseline. Compute avg in a first pass.

awk 'BEGIN{OFS=","; print "ip,hits,last_status"} { c[$1]++; s[$1]=$9 } END { for(k in c) print k, c[k], s[k] }' access.log

Emit a CSV summary with a header row, ready to attach to a ticket.

Save yourself an hour

Gotchas

awk '$5 == "404"' vs awk '$5 == 404'String vs numeric comparison. Usually the same result, but leading zeros and whitespace behave differently. Know which you mean.

awk -v ip="$SUSPECT" '$1 == ip'Pass a shell variable in with -v. Don't splice shell vars into the program string; it breaks on quotes and is a quoting nightmare.

awk -F'\t' '...'Default FS collapses runs of whitespace and ignores leading spaces. For real TSV, set -F'\t' or a mangled column shifts everything.

gawk vs mawk vs busybox awkstrftime, systime, mktime and gensub are gawk-only. On Alpine or a minimal container you get busybox awk with far less. Check with awk --version.

awk '{ print > "out-"$1".txt" }'awk can write to many files at once, keyed by a field. Powerful, but it holds file handles open; on thousands of keys you'll hit the limit.

for (k in c) ...Array iteration order is undefined. Never assume sorted; pipe to sort or use gawk's PROCINFO["sorted_in"].

awk '/[/'An unescaped slash inside a /regex/ ends it early. Escape it as /\// or match on a string instead.

Nothing matches. Try a shorter word or a field name.