When a log is columnar, a CSV export needs one field extracted, or a report asks for a sum without opening a spreadsheet, character-by-character editors are the wrong tool. That is what awk is for. It reads a stream line by line, splits each line into fields, and lets you print, filter, and calculate from those fields.

You do not need a full awk program on day one. Start with printing a column, then layer on field separators, pattern filters, BEGIN/END blocks, and simple math as the job gets richer. The sections below walk through the commands you will reach for most often — with enough context that each one feels intentional, not magical.

Warm-up: print one column

Before diving into patterns, give yourself a small playground. The short access log below mirrors what you see in real services: client IP, method, path, status, and response size in bytes.

Create it once, then reuse it for every example in this article:

cat > access.log <<'EOF'
192.168.1.10 GET /health 200 128
192.168.1.11 GET /api/users 200 2048
192.168.1.10 POST /login 401 512
192.168.1.12 GET /api/users 500 256
192.168.1.11 GET /health 200 128
192.168.1.10 GET /api/orders 200 4096
192.168.1.12 POST /login 200 640
EOF

With the file ready, print only the request path (the third field). This is the simplest useful form of awk: a program in single quotes that refers to fields as $1, $2, $3, and so on.

awk '{print $3}' access.log

You should see one path per line:

/health
/api/users
/login
/api/users
/health
/api/orders
/login

$0 is the whole line. $NF is the last field on the line — handy when the column count varies but you always want the trailing value:

awk '{print $1, $NF}' access.log
192.168.1.10 128
192.168.1.11 2048
192.168.1.10 512
192.168.1.12 256
192.168.1.11 128
192.168.1.10 4096
192.168.1.12 640

Note: By default awk splits on any run of whitespace. That covers most logs and command output. When fields are comma-separated or use another delimiter, set the separator explicitly (next section) instead of fighting with sed.

Change the field separator

Not every file is space-delimited. Exports, /etc/passwd, and CSV-ish dumps need a custom separator. -F sets the field separator for the whole run.

Try a colon-separated inventory that looks like a mini passwd-style table:

cat > users.txt <<'EOF'
alice:dev:1001:/home/alice
bob:ops:1002:/home/bob
carol:dev:1003:/home/carol
EOF

Print username and home directory ($1 and $4) with -F::

awk -F: '{print $1, $4}' users.txt
alice /home/alice
bob /home/bob
carol /home/carol

Inside a program you can also set FS in a BEGIN block — useful when the separator is easier to express there, or when you want a different output separator with OFS:

awk -F: 'BEGIN {OFS=","} {print $1, $3, $2}' users.txt
alice,1001,dev
bob,1002,ops
carol,1003,dev

Note: Reach for -F',' (or -F,) on simple CSV. Nested quotes and embedded commas need a real CSV parser — awk is fine for tidy, machine-generated columns, not for messy spreadsheet exports.

Filter rows with patterns

Like sed and grep, awk can keep only the lines that match a condition. Patterns sit before the action block. Matching lines run the action; non-matching lines are skipped.

Show only failed HTTP responses (status 400 and above) by comparing the fourth field:

awk '$4 >= 400 {print $0}' access.log
192.168.1.10 POST /login 401 512
192.168.1.12 GET /api/users 500 256

Regex patterns work too. Print lines whose path contains /api/:

awk '$3 ~ /\/api\// {print $1, $3, $4}' access.log
192.168.1.11 /api/users 200
192.168.1.12 /api/users 500
192.168.1.10 /api/orders 200

Combine conditions with && and ||. Client 192.168.1.10 with successful responses only:

awk '$1 == "192.168.1.10" && $4 == 200 {print $2, $3}' access.log
GET /health
GET /api/orders

When the action is just {print $0}, you can omit it — a bare pattern prints matching lines:

awk '$4 == 500' access.log

Note: Prefer awk over grep when the filter depends on a field (column 4 is 500) rather than a substring anywhere on the line. Prefer grep when you only care that a string appears somewhere and never plan to touch columns.

Line numbers and field counts

NR is the current record (line) number across the input. NF is how many fields the current line has. Together they help you skip headers, spot broken rows, and annotate output.

Prefix every line with its line number:

awk '{print NR, $3, $4}' access.log
1 /health 200
2 /api/users 200
3 /login 401
4 /api/users 500
5 /health 200
6 /api/orders 200
7 /login 200

Skip the first line (treat it as a header) with NR > 1:

awk 'NR > 1 {print $1, $3}' access.log

Flag rows that do not have exactly five fields — useful when a bad exporter injects blank or truncated lines:

awk 'NF != 5 {print "bad row", NR ": " $0}' access.log

On the clean sample file this prints nothing. Add a broken line and rerun to see the guard fire:

echo 'broken line' >> access.log
awk 'NF != 5 {print "bad row", NR ": " $0}' access.log
bad row 8: broken line

Restore a clean playground before continuing:

head -n 7 access.log > access.log.tmp && mv access.log.tmp access.log

Note: FNR is the line number within the current file when you pass multiple files. Use NR for a single stream; reach for FNR when stitching several inputs and you care about per-file position.

BEGIN and END: setup and summaries

BEGIN runs once before any input. END runs once after the last line. That is how you print headers, initialize counters, and emit totals without wrapping awk in a shell script.

Print a small header, then the IP and status for each row:

awk 'BEGIN {print "ip status"} {print $1, $4}' access.log
ip status
192.168.1.10 200
192.168.1.11 200
192.168.1.10 401
192.168.1.12 500
192.168.1.11 200
192.168.1.10 200
192.168.1.12 200

Sum response bytes ($5) and print the total after reading the file:

awk '{bytes += $5} END {print "total_bytes", bytes}' access.log
total_bytes 7808

Count errors and successes in one pass:

awk '
  $4 >= 400 {err++}
  $4 < 400 {ok++}
  END {print "ok", ok+0; print "err", err+0}
' access.log
ok 5
err 2

The +0 forces a missing counter to print as 0 instead of a blank — a small habit that keeps summary lines consistent.

Note: Put one-off setup in BEGIN and roll-ups in END. Keep the middle block for per-line work. If you find yourself writing multi-page awk, that is a signal to move the logic into a real script file (awk -f report.awk).

Reshape and report in pipelines

awk belongs in the middle of a pipe as often as it does on a file. Take grepped lines, keep the interesting columns, or feed unique values into sort.

List unique client IPs that hit /login:

awk '$3 == "/login" {print $1}' access.log | sort -u
192.168.1.10
192.168.1.12

Build key=value lines for a downstream script — environment-style exports from columnar input:

awk '$4 == 200 {print "path="$3}' access.log
path=/health
path=/api/users
path=/health
path=/api/orders
path=/login

Average response size for successful requests only:

awk '$4 == 200 {n++; sum += $5} END {if (n) print sum/n; else print 0}' access.log
1408

When a colleague already filtered with grep, awk can still finish the job by slicing fields:

grep ' /api/' access.log | awk '{print $4, $5}'

Note: If you only need a fixed column and never filter or compute, cut is often enough. Reach for awk when the job mixes field selection with conditions, arithmetic, or a final summary.

Quick reference card

Keep this nearby until the patterns become muscle memory:

GoalCommand
Print a fieldawk '{print $3}' file
First and lastawk '{print $1, $NF}' file
Custom separatorawk -F: '{print $1}' file
Output separatorawk 'BEGIN {OFS=","} {print $1,$2}' file
Field comparisonawk '$4 >= 400' file
Regex on a fieldawk '$3 ~ /api/' file
Combined conditionsawk '$1=="x" && $4==200' file
Line numberawk '{print NR, $0}' file
Skip headerawk 'NR > 1' file
Sum a columnawk '{s+=$5} END {print s}' file
Header + rowsawk 'BEGIN {print "h"} {print $1}' file
From a pipecmd | awk '{print $1}'

Practice drills

Use the sample access.log (recreate it from the warm-up if you changed it) and try these without peeking. The point is to choose the field and pattern with intent, not to memorize syntax under pressure.

  1. Print method and path for every request that returned status 200.
  2. Show only rows from IP 192.168.1.12, with a client= prefix on the IP.
  3. Using users.txt, print name:shellish-home as alice -> /home/alice (username and home).
  4. Compute the total bytes transferred by POST requests only.
  5. Print a one-line summary: number of lines read and the largest response size ($5).

When you are ready to compare, here are solid answers — not the only ones, but clear and portable:

awk '$4 == 200 {print $2, $3}' access.log
awk '$1 == "192.168.1.12" {print "client="$1, $2, $3, $4}' access.log
awk -F: '{print $1, "->", $4}' users.txt
awk '$2 == "POST" {s += $5} END {print s}' access.log
awk 'BEGIN {max=0} {if ($5 > max) max=$5} END {print NR, max}' access.log

If you can work through those five comfortably, you already cover most real awk work: column extraction, field filters, custom separators, pipeline reshaping, and END-block summaries. Start with {print $n}, then tighten with -F, patterns, and totals only when the first pass gets you most of the way there.