Linux Troubleshooting Toolkit:Diagnose Crashes and High Load

Two pages show up again and again: the box is crawling under load, or it died — reboot, OOM, unexplained hang. You do not need every flag from every tool. You need a repeatable order: observe with htop / top, name the hot PID or unit, read journalctl, check the kernel with dmesg, then see what the process still holds with lsof.

This article is that order of operations. The linked guides own deep syntax. Here you learn when to open each tool and which one-liners to run first.

Warm-up: the triage order

Memorize five steps. Skip ahead only when the symptom already points there (for example, jump to dmesg after a hard reboot).

1. Observe live pressure — htop, or a batch top snapshot if you need something pasteable.
2. Name the hot PID (and the systemd unit when there is one).
3. Pull logs for that unit or time window with journalctl.
4. Check the kernel for OOM, I/O, and hardware with dmesg / journalctl -k.
5. Inspect held resources — files, ports, deleted-open leaks — with lsof.

Quick batch snapshot you can drop into a ticket before opening a full-screen UI:

nproc
uptime
top -b -n 1 -o %CPU | head -n 15

Note: The monitor answers who is expensive. Logs and the kernel buffer answer why. Do not kill or restart until you can name the PID or unit with confidence — see ps/pgrep when the UI alone is ambiguous.

High load: start in htop

Open a live view when SSH still works and the machine is slow rather than dead:

htop

If htop is missing, classic top is enough. Sort by CPU first, then by memory if RES / %MEM looks tight or swap is climbing. High %wa with modest user CPU often means disk or NFS pressure, not only a hot loop.

Batch form when you cannot keep an interactive session open:

top -b -n 1 -o %CPU | head -n 15
top -b -n 1 -o %MEM | head -n 15

Confirm the offender’s full command line once you have a candidate PID (replace 12345):

ps -p 12345 -o pid,user,%cpu,%mem,args

Header reading, sort keys, and htop/btop UI details live in the top, htop, and btop guide. When this step ends, you should have a PID and ideally a unit name.

Note: Load average alone is not a diagnosis. Compare it to nproc, then identify whether the top consumers are CPU-bound, memory-bound, or waiting on I/O before you touch logs.

Service and user-space clues: journalctl

With a unit or a time window, pull errors instead of drowning in the full journal.

systemctl status nginx
journalctl -u nginx -n 50 --no-pager
journalctl -u nginx --since "10 min ago" -p err..alert --no-pager

After a reboot, the previous boot is often where the crash story lives:

journalctl -b -1 -p err..alert --no-pager
journalctl -b -1 -u nginx --no-pager

Follow live only while you reproduce:

journalctl -u nginx -f

Filter stacking, boots, and severity ranges are covered in the journalctl guide. Prefer systemctl restart / stop for managed services once you understand the failure — raw kills race the supervisor; see kill, pkill, and killall when you must signal an orphan you own.

Note: Empty unit logs do not mean “healthy.” Confirm the unit name with systemctl status or the process’s cgroup before widening to _SYSTEMD_UNIT searches or the whole journal.

Kernel and crash clues: dmesg

Jump here when user-space logs are thin, the host rebooted, or you suspect OOM, disk, USB, or driver faults.

Persistent / boot-spanning kernel lines (preferred after a reboot):

journalctl -k -b -1 --no-pager
journalctl -k -b -1 -p err..alert --no-pager

Live ring buffer with human timestamps:

dmesg -T | tail -n 50
dmesg -T | grep -i -E 'out of memory|oom-kill|killed process'
dmesg -T | grep -i -E 'I/O error|reset|usb|nvme|ext4|xfs'

Use journalctl -k when you need the last boot or a time-filtered kernel history. Use dmesg -T when you are on a live box chasing fresh hardware noise. Hunt patterns and permissions live in the dmesg guide.

Note: An OOM kill in the kernel log often explains a “random” process death that never appeared clearly in the app’s own stdout. Capture the victim command name from that line before you blame the redeploy.

What the process is holding: lsof

Once you have a PID, ask what it still owns — busy files, bound ports, or deleted logs that keep disk usage weird while load is high.

lsof -nP -p 12345
lsof -nP -iTCP -sTCP:LISTEN | head
lsof -nP -i :8080
sudo lsof -nP +L1 | head

-p scopes to one process. -i :port answers “who took my bind?” Deleted-but-open (+L1) catches the classic “df is full but du looks fine” leak during or after a load incident.

Column primers and selectors are in the lsof guide. Use this step to decide whether a restart will free a socket or a deleted file — not as your first command on a saturated CPU.

Note: Prefer lsof after you know the PID. A system-wide lsof without filters is slow and noisy on a box that is already struggling.

Playbook A — high load

Run this short chain when the host is up but painful. Replace unit and PID with what you find.

# 1) Observe
nproc; uptime
htop
# or: top -b -n 1 -o %CPU | head -n 15

# 2) Name the offender
ps -p 12345 -o pid,user,%cpu,%mem,args
systemctl status nginx   # when it is a managed service

# 3) Logs for the incident window
journalctl -u nginx --since "15 min ago" -p err..alert --no-pager

# 4) Kernel only if OOM / I/O still suspected
dmesg -T | grep -i -E 'oom-kill|I/O error' | tail

# 5) What the PID holds
lsof -nP -p 12345 | head

# 6) Act with the manager when possible
# systemctl restart nginx
# otherwise escalate signals carefully (see kill guide)

Playbook B — crash or sudden reboot

When the box came back from the dead, start in the previous boot, then the kernel, then only use live tools if a culprit process is still around.

# 1) Previous boot, errors first
journalctl -b -1 -p err..alert --no-pager | tail -n 80

# 2) Kernel story from that boot
journalctl -k -b -1 -p err..alert --no-pager
journalctl -k -b -1 --no-pager | grep -i -E 'out of memory|oom-kill|I/O error|panic|BUG'

# 3) Live ring buffer if the host is unstable again now
dmesg -T | tail -n 50

# 4) If a related PID survived the event
ps -p 12345 -o pid,user,args
lsof -nP -p 12345 | head

Note: After a reboot, the hot PID from before is gone. Reconstruct from journal and kernel lines first; htop only helps if the problem is still happening on the current boot.

Quick reference card

Symptom	First commands	Deep-dive
Box feels slow	htop or top -b -n 1 -o %CPU | head	top/htop/btop
Need PID / args	ps -p PID -o pid,user,args	ps/pgrep
Service misbehaving	journalctl -u UNIT --since … -p err..alert	journalctl
After reboot	journalctl -b -1 -p err..alert	journalctl
OOM / disk / USB	journalctl -k -b -1 or dmesg -T | grep …	dmesg
Port or file held	lsof -nP -i :PORT / lsof -nP -p PID	lsof
Safe stop / signal	Prefer systemctl; else TERM→KILL	kill/pkill/killall

Practice drills

Use read-only commands on shared hosts. Do not practice stops on PID 1, sshd, or production units without intent.

1. Capture a paste-friendly high-load snapshot: CPU count, uptime, and top CPU offenders.
2. Pick any local unit you are allowed to inspect and show its last 30 journal lines plus errors from the last hour.
3. Search the live kernel buffer (or current boot kernel journal) for OOM or I/O-style tokens.
4. Choose a PID you own (your shell $$ is fine) and list its open files with numeric hosts/ports.
5. Write the first three commands you would run after an unexplained reboot (previous-boot errors, then kernel).

When you are ready to compare, here are solid answers — not the only ones, but clear and portable:

nproc; uptime; top -b -n 1 -o %CPU | head -n 15

journalctl -u ssh -n 30 --no-pager
journalctl -u ssh --since "1 hour ago" -p err..alert --no-pager

dmesg -T | grep -i -E 'oom-kill|I/O error' | tail
# or: journalctl -k -b 0 --no-pager | grep -i -E 'oom-kill|I/O error' | tail

lsof -nP -p $$

journalctl -b -1 -p err..alert --no-pager | tail -n 80
journalctl -k -b -1 -p err..alert --no-pager
dmesg -T | tail -n 50

If you can work through those five comfortably, you already have the toolkit’s muscle memory: observe in htop/top, explain with journalctl, confirm kernel truth with dmesg, and inspect leftovers with lsof. Memorize the order; open a deep-dive only when a step needs more flags.