Container debugging — logs, exec, inspect, stats

Reading a container from the outside in: logs, exec, inspect, stats

Section 1 of 5 · ~3 min

You can build and run images now. The day something breaks — a container that won’t stay up, an app returning 500s, a box running hot — you need to read the container instead of guessing. Docker gives you four lenses, and knowing which one answers which question is the whole skill.

docker logs shows what the container’s main process wrote to stdout and stderr. It’s your first move on almost any problem, because a well-behaved container narrates its own startup and errors there. -f follows the stream live (like tail -f), and --tail 50 shows just the last 50 lines instead of the entire history — essential on a container that’s been up for a week.

docker exec -it <name> sh opens a shell inside a still-running container, so you can read files, check config, test a connection, or run the app’s own CLI from where it actually lives. -i keeps stdin open and -t allocates a terminal. The catch that trips everyone: exec needs a running process to join — you cannot exec into a container that has already crashed.

docker inspect <name> dumps the full JSON Docker holds about a container: its config, environment, mounts, networks, restart policy, and current state including the exit code. It’s a firehose, so you pipe it through jq to pull one branch — docker inspect web | jq '.[0].NetworkSettings.IPAddress'. Note the .[0]: inspect returns an array, one element per target you named.

docker stats is the live vital-signs monitor — CPU %, memory against its limit, network and block I/O, and PID count, refreshed each second. docker top <name> complements it by listing the processes running inside. Together they answer “is this container starved, leaking, or spinning?”

Real world: Think of a doctor with a patient. docker logs is the patient describing their symptoms; docker exec is going in with a stethoscope to examine directly; docker inspect is the medical chart with history and medications; docker stats is the bedside monitor beeping out heart rate and blood pressure. You reach for a different instrument depending on the question — and a good diagnosis usually reads more than one.

Diagnosing a crash-looping container

The hardest case is a container that keeps restarting. docker ps shows its status as Restarting (1) — the number is the exit code the process died with, and any non-zero code means it failed. Because it isn’t currently running, exec won’t work; instead you read docker logs (they persist across restarts) to see what it printed before dying, and docker inspect | jq '.[0].State' to read the exit code, error string, and whether a --restart policy is bouncing it. The loop is almost always the same failure repeating: a missing env var, a config file that isn’t there, a port it can’t bind. Logs tell you which.

A named example makes it concrete. Datadog’s container agent doesn’t invent its metrics — it reads the very same per-container CPU and memory counters that docker stats surfaces, because Docker exposes them straight from the Linux cgroup the container runs in. So the numbers you eyeball with docker stats on your laptop are the same signals a production platform graphs and alerts on; learning to read them by hand is learning what the dashboards mean.

Four lenses, one habit: logs first, then exec, inspect or stats depending on what the logs point at.

Hands-On Lab

Section 2 of 5 · ~3 min

Budget about 20 minutes. Open your WSL2 Ubuntu 24.04 terminal with Docker 27+ (install jq first if needed: sudo apt-get install -y jq). You’ll debug a healthy container with all four lenses, then build a container that crash-loops on purpose and diagnose it. Container IDs, IPs, PIDs and resource figures are unique to each run — yours will differ from the samples below.

# 1. Start a healthy nginx container in the background to debug against.
docker run -d --name web nginx:1.27-alpine
# Output (a 64-char container ID — yours will differ):
# 3f9a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8

# 2. Read the last few log lines. --tail limits history; -f would follow live (blocks until Ctrl-C).
docker logs --tail 5 web
# Output (representative nginx startup — timestamps will differ):
# /docker-entrypoint.sh: Configuration complete; ready for start up
# 2026/07/11 09:14:22 [notice] 1#1: using the "epoll" event method
# 2026/07/11 09:14:22 [notice] 1#1: nginx/1.27.4
# 2026/07/11 09:14:22 [notice] 1#1: start worker processes

# 3. Get a shell INSIDE the running container and read a config file from where the app lives.
docker exec -it web sh
# Output (your prompt becomes the container's shell):
# / # ls /etc/nginx/conf.d
# default.conf
# / # exit        # leaves the shell; the container keeps running

# 4. Pull ONE branch out of docker inspect with jq — note the .[0], inspect returns an array.
docker inspect web | jq '.[0].Config.Env'
# Output (the image's baked-in environment):
# [
#   "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
#   "NGINX_VERSION=1.27.4"
# ]
# (image also sets NJS_VERSION, NJS_RELEASE, PKG_RELEASE, DYNPKG_RELEASE)

# 5. Read the network branch — the container's IP and which network it joined.
docker inspect web | jq '.[0].NetworkSettings.Networks.bridge.IPAddress'
# Output (address from the default bridge pool — yours will differ):
# "172.17.0.2"

# 6. Snapshot live resource usage. --no-stream prints one sample instead of refreshing forever.
docker stats --no-stream web
# Output (CPU/MEM figures vary by machine and moment — yours will differ):
# CONTAINER ID   NAME   CPU %   MEM USAGE / LIMIT   MEM %   NET I/O      BLOCK I/O   PIDS
# 3f9a1b2c3d4e   web    0.00%   3.4MiB / 7.6GiB     0.04%   1.1kB / 0B   0B / 0B     2

# 7. List the processes running INSIDE the container, seen from the host.
docker top web
# Output (host-side PIDs will differ):
# UID     PID    PPID   C   STIME   TTY   TIME       CMD
# root    2451   2430   0   09:14   ?     00:00:00   nginx: master process nginx -g daemon off;
# nginx   2502   2451   0   09:14   ?     00:00:00   nginx: worker process

# 8. Now make a container that FAILS on start and keeps retrying — a real crash loop to debug.
docker run -d --name crasher --restart on-failure alpine sh -c "echo starting; exit 1"
# Output (container ID — yours will differ):
# 7b2e4d9c1a83f60c5e8d2a4b6f1093e7c2d5a8b4e0f3c9d17a2b6e5c8d4f0a19

# 9. Read its status: Restarting, and the (1) is the exit code it keeps dying with.
docker ps -a --filter name=crasher
# Output (STATUS cycles between Restarting and Exited — yours will differ):
# CONTAINER ID   IMAGE    COMMAND                 CREATED         STATUS                         NAMES
# 7b2e4d9c1a83   alpine   "sh -c 'echo startin…"  8 seconds ago   Restarting (1) 2 seconds ago   crasher

# 10. exec would fail (it isn't running) — read logs instead; they persist across every restart.
docker logs crasher
# Output (the same line printed once per failed attempt):
# starting
# starting
# starting

# 11. Confirm the exit code and error state in the container's live State branch.
docker inspect crasher | jq '.[0].State | {Status, ExitCode, Error, Restarting}'
# Output (ExitCode 1 = the process failed; Restarting true = the policy is bouncing it):
# {
#   "Status": "restarting",
#   "ExitCode": 1,
#   "Error": "",
#   "Restarting": true
# }

# 12. Stop the restart loop and clean up both containers.
docker rm -f web crasher
# Output:
# web
# crasher

Read the last blocks back: docker ps flagged the crash loop, docker logs showed the exact line printed before each death, and docker inspect confirmed exit code 1 with the restart policy bouncing it — three lenses converging on one cause. That’s the debugging reflex you’ll carry into every later phase: read first, restart never blindly.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These three trip up almost everyone the first time they debug a container in anger. Read the error text slowly — learning to parse it is the actual skill.

Common error: Trying to docker exec into a container that has crashed or exited — for example docker exec -it crasher sh on the crash-looping container above:

Error response from daemon: container 7b2e4d9c1a83 is not running

Why: docker exec starts a new process inside a container’s namespaces, which needs a live main process to join. A crashed or exited container has no running process, so there’s nothing to exec into. This is the number-one confusion when debugging a crash loop — the container you most want to look inside is exactly the one you can’t shell into.

Fix: For a stopped container, read its history with docker logs <name> (logs survive the exit) and read the exit code from docker inspect <name> | jq '.[0].State.ExitCode'. If you need its filesystem, docker cp files out, or start a fresh container from the same image with an overridden command like sh to look around.

How you’d spot it in prod: An on-call engineer trying to shell into a restarting container (or a Kubernetes CrashLoopBackOff pod) hits exactly this — the move is to pivot to logs and the exit code, not to keep retrying the shell.

Common error: Filtering docker inspect with jq but forgetting that inspect returns an array — writing docker inspect web | jq '.Config.Env':

jq: error (at <stdin>:NN): Cannot index array with "Config"

Why: docker inspect always returns a JSON array — one element per object you named, even when you named just one. jq '.Config.Env' tries to index that array with a string key and fails. The data is fine; the filter is aimed at the wrong level.

Fix: Select the first element first: docker inspect web | jq '.[0].Config.Env'. To inspect several at once, iterate with jq '.[].Config.Env'. The same rule applies to Go templates — docker inspect --format '{{ (index .Config.Env) }}' already operates per-object because --format runs once per target.

How you’d spot it in prod: A monitoring or automation script that parses docker inspect works against one container and breaks the moment it’s pointed at several (or vice-versa) — an array-versus-object mismatch in the jq filter, not a Docker fault.

Common error: docker logs on a container that’s clearly working shows nothing at all:

(docker logs web prints no output, even though the app is serving traffic)

Why: The default json-file logging driver captures only what the main process writes to stdout and stderr. An app configured to log to a file inside the container — say /var/log/app/app.log — produces nothing for docker logs to show. The logs aren’t lost; they’re just not on the stream Docker is watching.

Fix: Configure the app to log to stdout/stderr — the 12-factor convention containers expect, usually a one-line config switch. As a stopgap, docker exec -it web sh and tail the file directly, or mount its directory as a volume so you can read it from the host.

How you’d spot it in prod: A healthy service whose docker logs (or kubectl logs) is empty is almost always logging to a file, not the stream — which also means your central log aggregator is collecting nothing. Redirect it to stdout so the logs actually flow.

Container Debugging Interview Questions

Section 4 of 5 · ~1 min

Debugging questions separate people who’ve only run containers from people who’ve operated them — a calm answer that names the right lens for each symptom lands well. Cover each answer, say your own version out loud first, then compare; recalling before revealing is what makes it stick for interview day. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 5 min — Run docker stats with no --no-stream and watch the live refresh, then in another terminal docker events while you start and stop a container — the daemon narrates every lifecycle change in real time.
• 10 min — Learn docker inspect --format with Go templates: docker inspect --format '{{ .State.Status }} {{ .State.ExitCode }}' web. It avoids the jq .[0] trap because --format runs once per container, and it’s what you’ll paste into scripts.
• 15 min — Read the debugging and logging sections of the Docker for DevOps guide for how logs, exec and inspect fit the wider container workflow you build across this phase.