Linux strace:See What a Stuck Process Is Waiting On

The process is alive. CPU is idle. journalctl is quiet, dmesg has nothing new, and lsof only names the files still held open. You already walked the Linux Troubleshooting Toolkit; the box still will not move. The remaining question is which syscall the process is blocked in. That is what strace answers.

This article is the sequel for an alive but stuck PID. Find that PID with ps and pgrep first. Restart with systemctl only after you understand the wait.

Warm-up: trace a command you launch

Start by tracing a process you own from birth. strace prints every syscall (or a filtered set) as the kernel sees it. Launch ls under a file-related filter so the firehose stays readable.

strace -e trace=file ls /tmp

You should see ls open the directory (and a few libraries) before it exits. Paths and library names will differ; the shape is the point:

execve("/usr/bin/ls", ["ls", "/tmp"], 0x7ffd...) = 0
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/tmp", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
+++ exited with 0 +++

A return value of 3 is a file descriptor. = 0 on execve means the new program started. A hang, by contrast, is a line that never closes — the syscall name, arguments, and no = yet.

Note: Trace commands you started, not production sshd and not PID 1. A short ls or a sleep you own is the right lab.

Attach to a PID you own

When the stuck process is already running, attach instead of relaunching it. Start a decoy that will sit in nanosleep, then attach (replace 12345 with the PID the shell printed).

sleep 60 &
echo $!

strace -p 12345

Typical first line after a successful attach:

strace: Process 12345 attached
restart_syscall(<... resuming interrupted nanosleep ...>

restart_syscall means you attached during an existing wait. The kernel is showing the syscall that was already in flight — here nanosleep. Leave strace running a few seconds; when sleep finishes you will see the call return and the process exit.

Write the trace to a file and stamp each line with wall-clock time when you need a paste for a ticket:

strace -p 12345 -tt -o /tmp/sleep.strace

-tt prints timestamps with microseconds. -o keeps the terminal free and gives you a log you can grep later. Watch it live with tail -f /tmp/sleep.strace if you still want the hang on screen.

Detach with Ctrl-C in the strace terminal. That usually detaches and leaves the target running. Do not SIGKILL the target to “get out of strace,” and do not kill -9 the strace process itself — an unclean tracer death can leave the target in tracing stop (t in ps).

Note: If attach fails with Operation not permitted, check Yama:

cat /proc/sys/kernel/yama/ptrace_scope

1 (common default) lets you trace children you started, not arbitrary same-uid processes. Retry with sudo strace -p PID when you are allowed to, or start the command under strace as in the warm-up. Never attach to PID 1 or a production SSH daemon without a deliberate incident plan.

Follow children with -f

Workers often block in a child, not the PID you first named. -f follows fork / clone so you see the whole tree.

Trace a tiny shell that starts a background sleep and waits for it:

strace -f -e trace=process bash -c 'sleep 2 & wait'

You should see a clone, then a wait, with a new PID in brackets:

clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, ...) = 23456
[pid 23456] execve("/usr/bin/sleep", ["sleep", "2"], ...) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 23456

Without -f, you only see the parent’s wait4 and miss the child’s nanosleep. That is the classic “I straced the supervisor and learned nothing” miss.

When many children would mix on one stream, split one file per PID:

strace -ff -o /tmp/follow bash -c 'sleep 1 & sleep 1 & wait'
ls /tmp/follow.*

Note: -f multiplies volume and overhead. Pair it with -e trace= (next section) so you are not printing every write from every worker.

Filter with -e trace=

Unfiltered strace on a chatty server is unreadable. Restrict the set with -e trace= — either a class or a comma-separated list of syscall names.

File-related (open, stat, access, …):

strace -e trace=file ls /etc/hostname

Network-related (socket, connect, sendto, recvfrom, …):

strace -e trace=network curl -sI https://example.com | head

Named syscalls when you already suspect the wait:

strace -e trace=open,openat,connect,futex,read -p 12345

Useful classes on modern strace: file, network, process, memory, signal, ipc. Mix a class with extra names if you need both.

Note: Filtering does not make attach free. strace still stops the process at syscall boundaries it cares about. Use it to read a hang, not to profile a hot path all afternoon.

Read a hang: futex, read, connect

A stuck line is the diagnosis. The syscall name tells you what the process is waiting for; lsof on the same PID tells you which file or socket that FD is.

Blocking read (a FIFO you own)

Create a named pipe and cat it. With no writer, cat blocks in open:

mkfifo /tmp/strace-fifo
cat /tmp/strace-fifo &
CATPID=$!
strace -p "$CATPID"

You should see an unfinished openat (or open) on the FIFO path:

strace: Process 34567 attached
openat(AT_FDCWD, "/tmp/strace-fifo", O_RDONLY

Ctrl-C to detach. Clean up:

kill "$CATPID"
rm -f /tmp/strace-fifo

A process that already opened the FD and is waiting for data looks like read(3, or recvfrom(6, with no return. The number is the descriptor — map it with lsof -nP -p PID.

Connect that never finishes

A TCP connect that does not return is usually “SYN sent, no reply”: routing, firewall, or a remote that is down. Illustrative line:

connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("192.0.2.1")}, 16

Reproduce safely against the documentation-range address 192.0.2.1 (expect a timeout, not a login):

timeout 4 strace -e trace=connect -tt curl -s --connect-timeout 3 http://192.0.2.1/ || true

futex: waiting on another thread

A line like this means the thread is asleep on a lock or condition variable — often a mutex another thread never released, or a pool waiting for work:

futex(0x7f8a1c000ba0, FUTEX_WAIT_PRIVATE, 2, NULL

strace -f matters here: the blocked thread may not be the PID you attached to first. -tt (and -T, time spent in the current syscall) tells you this wait has been sitting for minutes, not milliseconds.

Note: Do not strace PID 1 or production sshd as a casual lab. strace is heavy: every traced syscall is a stop. For always-on production tracing you want lower-overhead tools (eBPF / perf); those are a different tutorial. Use strace for a short attach, read the hang, detach with Ctrl-C, then decide whether a systemctl restart is justified.

Quick reference card

Keep this nearby until the flags become muscle memory:

Goal	Command
Trace a command you start	strace cmd …
File syscalls only	strace -e trace=file cmd
Network syscalls only	strace -e trace=network cmd
Named syscalls	strace -e trace=open,connect,futex -p PID
Attach	strace -p PID
Follow forks	strace -f -p PID
One file per child	strace -ff -o /tmp/t -p PID
Timestamps	strace -tt -p PID
Log to a file	strace -o /tmp/trace.out -p PID
Time in current syscall	strace -T -p PID
Detach	Ctrl-C on the strace terminal

Practice drills

Use a sleep or FIFO you started. Do not attach to PID 1, sshd, or anyone else’s process.

1. Trace ls so you only see file-related syscalls, and confirm an openat on the directory you listed.
2. Start sleep 30 in the background, attach with timestamps, and name the syscall it is blocked in. Detach with Ctrl-C (leave sleep running).
3. Run a bash -c that backgrounds a sleep and waits, with -f and trace=process, and point to the child’s PID in the output.
4. Recreate the FIFO hang: mkfifo, background cat, attach, and identify the unfinished syscall. Clean up afterward.
5. Write the strace command you would use to attach to a stuck worker, follow children, and log only futex, read, and connect to /tmp/wait.strace.

When you are ready to compare, here are solid answers — not the only ones, but clear and portable:

strace -e trace=file ls /tmp

sleep 30 &
strace -p $! -tt
# Ctrl-C detaches; leave sleep running (kill it later if you started it)

strace -f -e trace=process bash -c 'sleep 2 & wait'

mkfifo /tmp/strace-drill
cat /tmp/strace-drill &
strace -p $!
# Ctrl-C, then:
kill $!; rm -f /tmp/strace-drill

strace -f -e trace=futex,read,connect -o /tmp/wait.strace -p 12345

If you can work through those five comfortably, you already have the strace move for a quiet, living hang: attach to a PID you identified with ps/pgrep, filter the syscall set, read restart_syscall / futex / read / connect, detach without killing the target, and only then reach for systemctl.