CI says green but Slack alerts says “prod looks wrong.”
Before you dig into logs or roll anything back, you need a sharper question answered: which pods are running, on which nodes, with which image and resources — and what just happened in this namespace?
That is everyday kubectl inspection. You do not need every API object on day one. Start by confirming context and namespace, find your workload with labels, then read placement, resources, images, and events as the story narrows. The sections below walk through the commands you will reach for most often after a deploy — with enough context that each one feels intentional, not magical.
Warm-up: context, namespace, and columns
Your playground is a live cluster (or a local kind/minikube). Blind kubectl get pods in the wrong context or default namespace is how “nothing is deployed” becomes a false alarm.
Confirm where you are pointing, then list namespaces:
kubectl config current-context
kubectl get ns
Typical context line (names differ):
prod-eu-west-1
Prefer an explicit namespace on every triage command so output stays scoped:
kubectl get pods -n payments -o wide
Wide output adds Pod IP and Node. Columns look like this:
NAME READY STATUS RESTARTS AGE IP NODE
payments-api-7d9f8c6b5-xk2m4 1/1 Running 0 12m 10.244.2.18 worker-3
payments-api-7d9f8c6b5-p9q1r 1/2 Running 1 12m 10.244.1.44 worker-1
Fields you will use constantly:
- READY — ready containers over total (
1/2means one container is not ready yet) - STATUS — phase summary (
Running,Pending, CrashLoopBackOff, …) - RESTARTS — how often containers have restarted
- IP / NODE — Pod address and the machine it landed on
Note: STATUS can say Running while READY is still 1/2. Treat Ready counts as the deploy-health signal; Status alone is not enough.
Find the right workload (not a random pod)
Pods are ephemeral. Listing everything in a busy namespace buries the deploy you care about. Prefer a label selector that matches your app (often app=… or app.kubernetes.io/name=…):
kubectl get pods -n payments -l app=payments-api -o wide
One step up the ownership chain shows the intended replica set alongside the pods:
kubectl get deploy,rs,pods -n payments -l app=payments-api
Typical ownership view:
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/payments-api 2/2 2 2 40d
NAME DESIRED CURRENT READY AGE
replicaset.apps/payments-api-7d9f8c6b5 2 2 2 12m
replicaset.apps/payments-api-5c4b8d9f7 0 0 0 40d
NAME READY STATUS RESTARTS AGE
pod/payments-api-7d9f8c6b5-xk2m4 1/1 Running 0 12m
pod/payments-api-7d9f8c6b5-p9q1r 1/1 Running 0 12m
The Deployment carries the desired replica count and container image. ReplicaSets show which revision is current (DESIRED > 0) versus scaled to zero after a rollout. Pods are the running instances of the active set.
Note: If label queries return nothing, check how the chart or manifest labels objects (kubectl get deploy -n payments --show-labels) before assuming the workload is missing.
Placement and identity: get -o wide
When networking, node pressure, or “is this the box I think it is?” matters, -o wide is the fast map: Pod IP plus Node for every matching pod.
kubectl get pods -n payments -l app=payments-api -o wide
For a tighter triage table, ask only for the columns you need:
kubectl get pods -n payments -l app=payments-api \
-o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[*].ready,NODE:.spec.nodeName,IMAGE:.spec.containers[*].image
Example custom output:
NAME READY NODE IMAGE
payments-api-7d9f8c6b5-xk2m4 true worker-3 ghcr.io/acme/payments-api:a1b2c3d
payments-api-7d9f8c6b5-p9q1r true worker-1 ghcr.io/acme/payments-api:a1b2c3d
Note: Custom columns are for human triage snapshots. Keep the field paths short; reach for jsonpath or -o yaml when you need exact scripting.
Deep config: describe pod
describe is the human-readable dump for one object: image and ports, CPU/memory requests and limits, conditions, and the last handful of events for that pod.
kubectl describe pod -n payments payments-api-7d9f8c6b5-xk2m4
Skim these sections first:
- Containers — image, ports, readiness/liveness probes
- Limits / Requests — allocated CPU and memory (or “none,” which is its own finding)
- Conditions —
Ready,PodScheduled, and friends - Events — recent pulls, kills, mounts, and scheduling notes for this pod
Excerpt shape (trimmed):
Containers:
payments-api:
Image: ghcr.io/acme/payments-api:a1b2c3d
Port: 8080/TCP
Limits:
cpu: 500m
memory: 512Mi
Requests:
cpu: 100m
memory: 256Mi
Conditions:
Type Status
Ready True
PodScheduled True
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12m default-scheduler Successfully assigned payments/payments-api-7d9f8c6b5-xk2m4 to worker-3
Normal Pulled 12m kubelet Successfully pulled image "ghcr.io/acme/payments-api:a1b2c3d"
Normal Created 12m kubelet Created container payments-api
Normal Started 12m kubelet Started container payments-api
Note: describe is for humans. When you automate checks or paste into scripts, prefer -o jsonpath or -o yaml so the output stays stable.
Prove the image: tag and digest
CI green does not prove the cluster is on the build you think. Print the image reference from the Pod spec:
kubectl get pod -n payments payments-api-7d9f8c6b5-xk2m4 \
-o jsonpath='{.spec.containers[*].image}{"\n"}'
Typical result:
ghcr.io/acme/payments-api:a1b2c3d
Tags can be overwritten (:latest, or a reused :v1.2.3). For stronger proof that this pull resolved to a specific content digest, read the runtime image ID from status:
kubectl get pod -n payments payments-api-7d9f8c6b5-xk2m4 \
-o jsonpath='{.status.containerStatuses[*].imageID}{"\n"}'
Digest-style output looks like:
ghcr.io/acme/payments-api@sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
Compare the Deployment’s intended image to a running Pod when you suspect a stuck rollout:
kubectl get deploy -n payments payments-api \
-o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'
kubectl get pod -n payments -l app=payments-api \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'
If the Deployment already shows the new tag but pods still show the old one, the rollout has not finished (or cannot).
Note: Treat a matching tag as a hint, not a guarantee. Prefer digest / imageID when you must prove “exactly this build” landed in the cluster.
What just happened: events
Pod describe shows events for one object. For a chronological namespace timeline — what the scheduler, kubelet, and controllers just did — list events sorted by time:
kubectl get events -n payments --sort-by=.lastTimestamp
Recent lines might look like:
LAST SEEN TYPE REASON OBJECT MESSAGE
2m Warning Unhealthy pod/payments-api-7d9f8c6b5-p9q1r Readiness probe failed: HTTP probe failed with statuscode: 503
5m Normal Killing pod/payments-api-5c4b8d9f7-aabb1 Stopping container payments-api
12m Normal Scheduled pod/payments-api-7d9f8c6b5-xk2m4 Successfully assigned ... to worker-3
12m Normal Pulled pod/payments-api-7d9f8c6b5-xk2m4 Successfully pulled image ...
Watch for scheduling failures, image pull errors, OOMKilled, FailedMount, and probe failures right after a deploy. Those often explain “CI green, traffic sad” before you open logs.
Cluster-wide noise is a deliberate choice:
kubectl get events -A --sort-by=.lastTimestamp | tail -n 40
Note: Without -A / --all-namespaces, kubectl get events is namespace-scoped (current context namespace, or -n). Empty output usually means the wrong namespace — not a quiet cluster.
Quick reference card
Keep this nearby until the workflow becomes muscle memory:
| Goal | Command |
|---|---|
| Current context | kubectl config current-context |
| Pods in a namespace (wide) | kubectl get pods -n NS -o wide |
| Pods by label | kubectl get pods -n NS -l app=NAME -o wide |
| Deploy + RS + pods | kubectl get deploy,rs,pods -n NS -l app=NAME |
| Human deep dive | kubectl describe pod -n NS POD |
| Print container images | kubectl get pod -n NS POD -o jsonpath='{.spec.containers[*].image}{"\n"}' |
| Print image digests | kubectl get pod -n NS POD -o jsonpath='{.status.containerStatuses[*].imageID}{"\n"}' |
| Deployment image | kubectl get deploy -n NS NAME -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}' |
| Namespace events (sorted) | kubectl get events -n NS --sort-by=.lastTimestamp |
| All-namespace events | kubectl get events -A --sort-by=.lastTimestamp |
Practice drills
Use a cluster you can read (prod with care, or a local kind/minikube playground). Pick one labeled app and one namespace. The point is to choose the next command with intent, not to memorize flags under pressure. Answers sit under each step — solid shapes, not the only ones (replace payments, payments-api, and the pod name with yours).
-
List wide pods for one label in a namespace and name the node for each pod.
kubectl get pods -n payments -l app=payments-api -o wide -
From a pod name, extract all container images with
jsonpath.kubectl get pod -n payments payments-api-7d9f8c6b5-xk2m4 \ -o jsonpath='{.spec.containers[*].image}{"\n"}' -
Find CPU request and limit for the first container via
describe(orjsonpathon.spec.containers[0].resources).kubectl describe pod -n payments \ payments-api-7d9f8c6b5-xk2m4 \ | sed -n '/Limits:/,/Requests:/p;/Requests:/,/Environment:/p' -
Sort events in that namespace and identify the newest non-
Normalevent (or confirm there is none).kubectl get events -n payments \ --sort-by=.lastTimestamp \ | awk '$2=="Warning"' -
Compare the Deployment’s image to each running Pod’s image for the same app label.
kubectl get deploy -n payments payments-api \ -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}' kubectl get pods -n payments -l app=payments-api \ -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'
If you can work through those five comfortably, you already cover most real deploy-verification work with kubectl: find the right objects, read placement and resources, prove the image (tag and digest), and skim what just happened. Start with context and labels, use describe when a human needs the full picture, and save logs, exec, and rollout undo for the next pass when inspection alone is not enough.