Docker is the tool that finally made “it works on my machine” a non-excuse. By packaging an application together with its exact dependencies into a portable image, Docker lets the same artifact run identically on a laptop, a CI runner, and a production server. This guide is a deep, practical tour for DevOps engineers: how images and layers actually work, how to write efficient and secure Dockerfiles, how containers talk to each other over networks, how to persist data with volumes, how to compose multi-service stacks, and how to take all of it to production. Work through it top to bottom, or jump to the part you need. For a structured learning path beyond this guide, see the Docker roadmap.
The Problem Docker Solves
Section 1 of 67 · ~4 min
The “Works on My Machine” Story
It’s 2019. A developer at a startup has been grinding for 3 weeks on a Node.js app — a real-time dashboard for tracking delivery agents. The UI is slick, the APIs are fast, the PM is happy. He types npm start and everything just works. Perfectly.
He pushes the code to staging. The CI pipeline goes green. He Slacks his team: “Deployed! Please test.”
3 minutes later:
text
Error: Cannot find module 'optional-chaining-polyfill'Error: node: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.28' not foundnpm ERR! peer dep missing: react@^17.0.0, required by react-query@3.39.0
His laptop: Node 14.18, npm 6.14, Ubuntu 22.04. The staging server: Node 12.22, npm 5.6, CentOS 7. Three days of debugging. Three days of “but it works on my machine!”
This problem was not unique — every developer at every company was facing it. Docker was the solution.
Dependency Hell — Real-World Examples
The “Works on My Machine” problem is a symptom of a deeper disease: dependency hell. Here are scenarios every developer has lived through:
Scenario
The Conflict
Pain Level
Two Python projects on same machine
Project A needs Python 2.7, Project B needs Python 3.9. pip install breaks both.
High
npm version conflict
App needs npm 6 for legacy scripts, another needs npm 8. Switching breaks lockfiles.
Medium
OpenSSL version mismatch
Dev has OpenSSL 3.0, prod server has 1.1. Node.js crypto breaks silently.
High
System library versions (glibc)
Compiled on Ubuntu 22, deployed on CentOS 7. Binary won’t even load.
Critical
Ruby gem native extensions
Gem compiled against libxml2 on Mac, server has different libxml2. Segfault.
Critical
Note: Think of Docker like sharing a React component with node_modules — not just the code, but the exact node version, npm version, and OS libraries. Docker packages your application together with its entire environment.
Deployment Nightmares Before Docker
Before Docker, deploying an app to a new server looked like this:
bash
# The legendary "deployment runbook" (50 pages of this)ssh root@production-server-ip# Step 1: Install the right Node version (pray it's compatible)curl -fsSL https://deb.nodesource.com/setup_14.x | bash -apt-get install -y nodejs# Step 2: Hope npm is the right versionnpm install -g npm@6.14.15# Step 3: Clone appgit clone https://github.com/company/app.git /var/www/appcd /var/www/appnpm install # Often fails here due to native modules# Step 4: Set up environment variables (done manually, often wrong)export DB_HOST=192.168.1.100export NODE_ENV=production# (Someone always forgets one. App crashes at 2 AM)# Step 5: Set up PM2 or systemd (manual, error-prone)npm install -g pm2pm2 start app.js --name "myapp"pm2 startup # Different command on every distro# Step 6: Set up nginx (manual config, copy-paste errors)# ... 20 more steps ...# Total time: 4-8 hours per server# Success rate: ~60% first try# Consistency across servers: ZERO
And if you need to deploy to 10 servers? Do all of this 10 times — and something different will happen each time. This is called configuration drift — over time servers diverge from each other, and no one knows why production behaves differently.
The VM Tax — Resource Waste at Scale
Companies tried solving this with Virtual Machines — one VM per application. But VMs have a massive overhead:
Resource
Each VM Consumes
10 Apps = 10 VMs
RAM
2–4 GB minimum (OS alone)
20–40 GB wasted on OS
Disk
20–40 GB per VM image
200–400 GB
Boot time
1–5 minutes
Can’t scale fast
CPU
Always-on OS processes
Significant idle overhead
A startup with 20 microservices using VMs wastes a minimum of 40 GB RAM just for operating systems. The same 20 services in Docker containers consume 2–5 GB of overhead — a dramatic difference that translates directly to cloud cost savings.
Docker’s Shipping Container Analogy
Before the 1950s, international shipping was chaos. Every ship had a different design, every port used different equipment, and cargo had to be repacked repeatedly. A shipment from Europe to America could take months, and costs were astronomical.
In 1956, Malcolm McLean invented the standardized shipping container. One fixed size (20ft/40ft), standard corners, works on any ship, any port, any truck, any train. Load once in Shanghai — unload in Rotterdam without opening the box.
Result: Global trade exploded. Shipping costs dropped 90%.
Docker is the shipping container of software:
Shipping World
Software World
Different ships
Different servers/OS
Different ports
Different cloud providers
Cargo re-packing
Re-configuring for each environment
Standard container
Docker container
Load once, ship anywhere
Build once, run anywhere
McLean standardized trade
Docker standardized deployment
Tip: When asked “Why do you use Docker?” in an interview — use the shipping container analogy. Then give concrete numbers: “Our deployment time went from 4 hours to 15 minutes. Configuration drift eliminated. Dev-prod parity achieved.”
Why Every Company Adopted Docker
By 2026, over 80% of cloud workloads run in containers. Here’s why adoption was inevitable:
Dev-prod parity — Same container in dev, test, staging, production. No more surprises.
Faster onboarding — New developer joins? One command: docker compose up. Ready in 5 minutes, not 2 days.
Microservices enablement — Each service in its own container. Scale independently.
CI/CD becomes simple — Build a container, test it, push to registry, deploy. Repeatable every time.
Cost reduction — Pack more apps per server. AWS/GCP bills drop 40–60%.
Kubernetes compatibility — Kubernetes orchestrates containers. You cannot use Kubernetes without containers.
Think of a VM like separate apartments in a large building. Each apartment has its own kitchen, its own bathroom, its own living room. What happens in one apartment is completely invisible to the other.
A Virtual Machine runs a complete operating system on top of a hypervisor — software that emulates physical hardware. Each VM thinks it owns the whole computer.
Hypervisor (VMware, VirtualBox, KVM, Hyper-V) sits between hardware and VMs
Each VM gets a full Guest OS (Windows, Ubuntu, etc.) — 1–4 GB just for the OS
Hardware is emulated — virtual CPU, virtual RAM, virtual disk
Strong isolation — a crash in VM1 doesn’t affect VM2
Boot time: 1–5 minutes (full OS boot)
Containers — The Shared House Analogy
Think of containers like rooms in a shared house. You share the same kitchen and water supply, but each person has their own private room and lives independently.
A Container shares the host OS kernel but runs in an isolated process namespace. No hardware emulation, no duplicate OS.
Shared kernel — uses the host’s Linux kernel directly
Each container gets an isolated view of processes, network, filesystem
No hypervisor overhead — near-native performance
Lightweight — MBs instead of GBs
Boot time: Milliseconds to seconds
Containers skip the Guest OS layer that VMs require, cutting overhead from gigabytes per app to near zero.
Detailed Comparison Table
Feature
Virtual Machine
Container
Startup Time
1–5 minutes (full OS boot)
Seconds (process start)
Size
2–40 GB per VM image
10 MB–1 GB (app + libs only)
OS
Full Guest OS per VM
Shares Host OS Kernel
Isolation Level
Hardware-level (very strong)
Process-level (strong, not absolute)
Performance Overhead
5–15% CPU/RAM overhead
<1–2% overhead (near-native)
Portability
Limited (hypervisor-specific)
Excellent (any Linux/Mac/Win with Docker)
Resource Usage
Heavy (duplicate OS per VM)
Light (shared kernel)
Density (per server)
5–20 VMs typical
50–500 containers typical
Security Isolation
Very strong (hardware boundary)
Good (namespaces + cgroups)
Best Use Case
Legacy apps, strong isolation, Windows workloads
Microservices, CI/CD, cloud-native apps
Tip: Real-world production numbers: Server utilization jumped from 30–40% (with VMs) to 80%+ (with containers). Deployment time dropped from 45 minutes to 8 minutes. Auto-scaling: containers spin up in 3 seconds vs 4 minutes for VMs — critical during peak load events.
When to Use VM vs Container
Use VM When…
Use Container When…
Running Windows workloads on Linux host
Running microservices/APIs
Need kernel-level isolation (security-critical)
CI/CD pipelines (build, test, deploy)
Legacy apps that need specific OS versions
Stateless applications
Running untrusted code (multi-tenant SaaS)
Development environments
Database servers needing dedicated resources
Auto-scaling workloads
Note: In enterprise environments, both are used together. VMs run Docker. In the cloud, an EC2 VM runs Docker containers on top. This is not a binary choice — they are complementary technologies.
The Origin Story — From DotCloud to Changing the World
2008: Solomon Hykes co-founds DotCloud — a Platform-as-a-Service (PaaS) company in San Francisco. They’re building tools to deploy customer apps on their platform, and internally they develop a container management tool.
March 21, 2013: PyCon, Santa Clara. Hykes gives a 5-minute lightning talk called “The Future of Linux Containers.” He demos Docker — their internal tool they’re open-sourcing. The audience is small. The impact: massive.
A 5-minute talk at a Python conference launched a technology that today runs in every data center in the world. That’s what happens when the right solution meets the right problem at the right time.
2013: Docker open-sourced, goes viral on GitHub
2014: Google, Microsoft, Amazon begin supporting Docker
2017: Kubernetes wins the orchestration wars, but Docker remains the build standard
2019: Docker Inc. sells its enterprise business to Mirantis, refocuses on developers
2023+: Docker Desktop 4.x, Docker Scout (security), AI-powered features
2026: Docker is as fundamental as Git — you cannot be a backend/DevOps engineer without it
Docker vs Other Container Runtimes
Docker is not just one tool — it is an ecosystem. But the container runtime market has other players:
Runtime
Used By
Key Feature
Should You Learn?
Docker
Developers, small teams, CI/CD
Best Developer Experience (DX), most popular, huge ecosystem
Yes, first priority
containerd
Kubernetes production clusters
Lightweight production runtime, CNCF graduated project
Learn after Docker
Podman
Red Hat / Fedora users, security-conscious teams
Rootless (no daemon), daemonless, Docker-compatible CLI
Be aware of it
CRI-O
OpenShift, Kubernetes
Minimal footprint, built for Kubernetes CRI interface
Be aware of it
runc
Low-level, all runtimes use it
OCI reference implementation, the actual container runner
Deep-dive later
Tip: When asked “What is the difference between Docker and containerd?” — say this: Docker is the full toolchain (build + run + push + compose). containerd is just the runtime that Kubernetes uses to actually run containers. Docker internally uses containerd. Think of it as: Docker = Full Car, containerd = The Engine.
Docker’s Place in the DevOps Stack
Docker is the central connective tissue of the modern DevOps pipeline:
Docker knowledge significantly expands your career options:
Skill Level
Typical Role
US (USD)
India (INR)
Without Docker
Junior/Mid Developer (Frontend/Backend)
$80k–120k
₹15–20 LPA
Docker + Kubernetes
DevOps / SRE (2–5 yrs exp)
$120k–170k
₹30–50 LPA
Docker + AWS + K8s + CI/CD
Senior DevOps / Staff SRE (5+ yrs exp)
$170k+
₹50+ LPA
Ranges are rough 2026 signals and vary widely by region, company size, and cost of living.
A React/frontend background is a unique advantage in DevOps — engineers who understand both frontend containerization and backend infrastructure are in high demand.
Docker’s magic comes from three specific Linux kernel features. Understanding them is what separates a senior engineer from a beginner — and this knowledge is invaluable in interviews when asked “How does Docker work internally?”
Docker containers are not magic — they’re three Linux kernel features working together:
Namespaces — What a container can see (isolation of view)
Cgroups — What a container can use (resource limits)
Union Filesystem (OverlayFS) — How a container’s filesystem is built
Linux Namespaces — The Isolation Magic
A namespace makes a container feel like it is the only process on the machine — like an actor on a film set who is made to feel they are actually in a royal palace, when they are actually standing on a soundstage.
Linux has 7 types of namespaces. Docker uses 6 of them:
PID Namespace — Process ID Isolation
The container believes it has its own process tree. Container’s PID 1 = Host’s PID 5678.
bash
# Inside container: sees its own process tree$ ps auxPID USER CMD1 root node server.js ← thinks it's PID 1!12 root /bin/sh# On host: sees the real PID$ ps aux | grep node5678 root node server.js ← actually PID 5678
The container cannot see parent processes. Completely isolated process tree.
NET Namespace — Network Isolation
Each container gets its own network interfaces, IP address, routing table, and firewall rules.
bash
# Container 1: has its own IP$ docker exec container1 ip addr172.17.0.2/16 ← container's IP# Container 2: different IP$ docker exec container2 ip addr172.17.0.3/16 ← different IP, same host# Host: sees docker0 bridge$ ip addr show docker0172.17.0.1/16 ← bridge connects them
MNT Namespace — Filesystem Isolation
The container has its own filesystem view. It appears to have a complete Linux filesystem — /etc, /usr, /var all separate.
bash
# Container sees its own filesystem$ docker exec mycontainer ls /bin dev etc home lib procroot sys tmp usr var app# Host files are NOT visible in container# (unless explicitly mounted with -v)
UTS Namespace — Hostname Isolation
Each container can have its own hostname, independent of the host machine.
bash
# Set custom hostname for container$ docker run --hostname=myapp-prod ubuntu hostnamemyapp-prod# Host machine sees its own hostname$ hostnameprod-server-01
USER Namespace — User ID Mapping
Maps the container’s root user to an unprivileged host user. Container “root” is NOT host root — important for security!
bash
# Container root → host user 1000# Even if container is "root",# on host it's unprivileged UID 1000# Security: container escape doesn't# give attacker host root access
IPC Namespace — Inter-Process Communication Isolation
Containers cannot access each other’s shared memory, semaphores, or message queues.
bash
# Share IPC namespace between containers# (for high-performance apps like Redis)docker run --ipc=container:redis-container my-app# Default: isolated IPC per container# No shared memory leaks between apps
Cgroups — Resource Control
Control Groups (cgroups) control how much resource a container can use. If namespaces define what a container can see, cgroups define how much it can consume.
Think of cgroups as a resource budget per container:
bash
# Memory limit: container cannot use more than 512MBdocker run --memory=512m nginx# CPU limit: container gets max 50% of one CPU coredocker run --cpus=0.5 nginx# Both together (production-ready)docker run \ --memory=512m \ --memory-swap=512m \ --cpus=0.5 \ --cpu-shares=512 \ nginx# I/O limits (read/write speed)docker run \ --device-read-bps /dev/sda:100mb \ --device-write-bps /dev/sda:50mb \ nginx# See current container resource usagedocker stats --no-stream# Output:# CONTAINER CPU % MEM USAGE / LIMIT NET I/O# nginx 0.1% 12MiB / 512MiB 1.2kB / 0B
Cgroup Subsystem
What It Controls
Docker Flag
memory
Max RAM usage, OOM killer behavior
--memory=512m
cpu
CPU time allocation
--cpus=0.5
cpuset
Which CPU cores to use
--cpuset-cpus=0,1
blkio
Block I/O limits (disk read/write)
--device-read-bps
net_cls
Network bandwidth tagging
(via tc/iptables)
pids
Max number of processes in container
--pids-limit=100
Caution: Always set memory and CPU limits on every production container. Without limits, a runaway container can crash the entire host server — the “noisy neighbour” problem. During peak traffic, if one container hogs CPU, all other containers will slow down. Limits protect all containers on the host.
Union Filesystem — OverlayFS
A Docker image is made of read-only layers. When you run a container, Docker adds a thin writable layer on top. This is OverlayFS (Overlay Filesystem).
bash
# Example Dockerfile - creates layersFROM ubuntu:22.04 # Layer 1: Base Ubuntu (~70MB)RUN apt-get update # Layer 2: Package index (~20MB)RUN apt-get install -y curl # Layer 3: curl binary (~5MB)COPY app.js /app/ # Layer 4: Your app (~1MB)CMD ["node", "app.js"] # Layer 5: Metadata only# See the layers of an image$ docker history my-node-appIMAGE CREATED SIZE COMMENTa1b2c3d4 2 hours ago 1.2MB COPY app.jsf5e6d7c8 2 hours ago 5.1MB apt-get install curlg9h0i1j2 2 hours ago 22MB apt-get updateubuntu:22.04 2 weeks ago 72MB base layer
Copy-on-Write (CoW) — The Secret Sauce
When a container modifies a file from a read-only layer:
Docker copies that file to the writable layer
Makes the modification in the writable layer
The original read-only layer is untouched
100 containers sharing the same image = base layers shared, only diffs stored
If 100 containers are using the same nginx image, there is still only one copy on disk. 100x efficiency!
bash
# See OverlayFS in action on Linux host$ docker inspect mycontainer | grep -A 10 GraphDriver"GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/abc123/diff: /var/lib/docker/overlay2/def456/diff", "MergedDir": "/var/lib/docker/overlay2/xyz789/merged", "UpperDir": "/var/lib/docker/overlay2/xyz789/diff", "WorkDir": "/var/lib/docker/overlay2/xyz789/work" }}# LowerDir = read-only image layers (stacked)# UpperDir = writable container layer (your changes)# MergedDir = what container sees (unified view)
Note (Senior Engineer Interview): “Both Docker containers and VMs provide isolation — so why are containers considered less secure?” Answer: VMs provide hardware-level isolation (hypervisor = hard boundary). Containers only use Linux kernel features — if there is a kernel vulnerability (like Dirty COW, runc CVE), container escape is possible. This is why multi-tenant environments running untrusted code prefer VMs or gVisor/Kata Containers.
Tip (Quick Summary): Docker container = Linux process + Namespaces (isolation of view: can’t see others’ processes, network, files) + Cgroups (resource limits: max CPU/RAM) + OverlayFS (layered filesystem: shared base + writable top). Everything else — Dockerfile, docker-compose, Kubernetes — is built on top of these three.
Docker follows a client-server architecture. When you type docker run nginx, a lot is happening under the hood — think of it as an assembly line where each component performs a specific job.
The Docker CLI is a thin REST client; all heavy lifting happens inside the Docker daemon, which coordinates containerd and runc to run containers and communicates with registries for image storage.
1. Docker Client (CLI)
The Docker Client is the tool you use — the docker command. When you type docker run nginx, the client only translates CLI arguments into a REST API call and sends it to the Docker Daemon. The client itself does nothing else — it is just a translator.
bash
# Docker CLI is just making REST API calls to the daemon# You can curl the Docker socket directly# List running containers (same as docker ps)curl --unix-socket /var/run/docker.sock http://localhost/v1.43/containers/json# Start a container via APIcurl --unix-socket /var/run/docker.sock \ -X POST \ -H "Content-Type: application/json" \ -d '{"Image": "nginx"}' \ http://localhost/v1.43/containers/create# Check Docker version via APIcurl --unix-socket /var/run/docker.sock http://localhost/v1.43/version | jq# See all API endpointscurl --unix-socket /var/run/docker.sock http://localhost/v1.43/_ping
Tip: You can control a remote machine’s Docker: export DOCKER_HOST=tcp://192.168.1.100:2376, then run normal docker commands. In production, always use TLS (port 2376), never plain port 2375!
2. Docker Daemon (dockerd)
The Docker Daemon (dockerd) is the background process that actually does the work. It accepts REST API requests and manages containers, images, networks, and volumes.
Feature
Details
Default Socket
unix:///var/run/docker.sock
TCP (TLS)
Port 2376 (production)
TCP (No TLS)
Port 2375 (NEVER in production!)
Config File
/etc/docker/daemon.json
Logs
journalctl -u docker.service
Restart
sudo systemctl restart docker
3. Docker Registry
Registry is where images are stored. Docker Hub is the default public registry, but enterprises use private registries:
Docker Hub — Default public registry. Free tier available. docker.io/library/nginx
AWS ECR — Amazon Elastic Container Registry. IAM integration.
GCR / GAR — Google Container/Artifact Registry. Seamless with GKE.
Artifactory — JFrog’s universal artifact manager. Docker + Maven + npm in one place.
4. containerd — High-Level Runtime
containerd is a CNCF project that sits between the Docker Daemon and actual containers. It manages image pulling, storage, and container lifecycle. Kubernetes also uses containerd directly (without Docker).
bash
# containerd has its own CLI: ctr# Note: operates in a different namespace from Docker# List images in containerdsudo ctr images ls# List containerssudo ctr containers ls# Pull image directly via containerdsudo ctr images pull docker.io/library/nginx:latest# Run container via containerd (low-level)sudo ctr run docker.io/library/nginx:latest mynginx# containerd socketls -la /run/containerd/containerd.sock# Check containerd statussudo systemctl status containerd
Note: Kubernetes v1.24+ removed Docker support. Kubernetes now talks directly to containerd via CRI (Container Runtime Interface). Docker’s “removal” was only the removal of the Docker daemon — containers still run the same way because containerd and runc are the same underneath.
5. runc — Low-Level Runtime
runc is the reference implementation of the OCI Runtime Specification. It directly makes Linux kernel syscalls — creates namespaces, sets up cgroups, and starts the process. It is the actual tool that makes a “container” exist.
bash
# runc version checkrunc --version# runc is not normally used directly# But for understanding: containerd calls runc like this:# runc create --bundle /path/to/bundle container_id# runc start container_id# runc delete container_id
Docker Installation on Ubuntu 24.04 (Complete Guide)
Section 6 of 67 · ~5 min
Prerequisites Check
Caution: Check if Docker is already installed before starting. If an old version exists, remove it first: sudo apt remove docker docker-engine docker.io containerd runc
bash
# System info checkuname -alsb_release -awhoami# Check if Docker already installeddocker --version 2>/dev/null || echo "Docker not installed"# Remove old versions if they existsudo apt remove -y docker docker-engine docker.io containerd runc 2>/dev/nullsudo apt autoremove -y
Step 1: System Update
bash
sudo apt update && sudo apt upgrade -y# If the kernel was updated, reboot (optional but recommended)# sudo reboot
Note: The GPG key ensures packages come from Docker’s official server and have not been intercepted. This protects against man-in-the-middle attacks. Always verify packages before installing.
Step 4: Add Docker Repository
bash
echo \ "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] \ https://download.docker.com/linux/ubuntu \ "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null# Verify repository was addedcat /etc/apt/sources.list.d/docker.list# Expected output:# deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg]# https://download.docker.com/linux/ubuntu noble stable# Update apt cache with new repositorysudo apt update
Step 5: Install Docker Engine
bash
sudo apt install -y \ docker-ce \ docker-ce-cli \ containerd.io \ docker-buildx-plugin \ docker-compose-plugin# Package breakdown:# docker-ce → Docker Community Edition (daemon)# docker-ce-cli → Docker CLI (the 'docker' command)# containerd.io → containerd runtime# docker-buildx-plugin → BuildKit for advanced image building# docker-compose-plugin → Docker Compose v2 (docker compose, not docker-compose)# To install a specific version:# List available versionsapt-cache madison docker-ce# Install specific version# sudo apt install docker-ce=5:26.0.0-1~ubuntu.24.04~noble \# docker-ce-cli=5:26.0.0-1~ubuntu.24.04~noble \# containerd.io docker-buildx-plugin docker-compose-plugin
Step 6: Verify Installation
bash
sudo docker run hello-world# Expected output:# Hello from Docker!# This message shows that your installation appears to be working correctly.# Service status checksudo systemctl status dockersudo systemctl status containerd
Step 7: Post-Installation — Run Without sudo
bash
# Create 'docker' group (usually already exists)sudo groupadd docker 2>/dev/null || echo "Group already exists"# Add current user to docker groupsudo usermod -aG docker $USER# Apply group changes (no logout/login needed for this session)newgrp docker# Verify — run docker without sudodocker run hello-world# Permanent verificationgroups $USER# Output should include 'docker'
Caution: Adding a user to the docker group is practically equivalent to giving root access. Through the Docker socket (/var/run/docker.sock), anyone can mount a volume and access the host filesystem. Add users carefully on production servers. Rootless Docker is a more secure option.
# Version infodocker --version# Docker version 26.1.4, build 5650f9bdocker compose version# Docker Compose version v2.27.1# System info (detailed)docker info# Test containerdocker run hello-world# Interactive ubuntu containerdocker run -it ubuntu bash# Nginx web server testdocker run -d -p 8080:80 nginxcurl http://localhost:8080# Cleanupdocker stop $(docker ps -q)docker system prune -f
Common Installation Errors & Fixes
Error
Cause
Fix
permission denied while trying to connect
User not in docker group
sudo usermod -aG docker $USER && newgrp docker
Cannot connect to the Docker daemon
dockerd not running
sudo systemctl start docker
dial unix /var/run/docker.sock: connect: no such file
Docker not installed or not started
Install Docker OR sudo systemctl start docker
Failed to connect to bus: No such file
systemd not available (WSL/container)
Logout and login again, or sudo service docker start
E: Package 'docker-ce' has no installation candidate
Repository not added
Redo steps 3-4
toomanyrequests from Docker Hub
Rate limit hit (unauthenticated)
Run docker login or use a mirror
Docker Rootless Mode
bash
# Install prerequisitessudo apt install -y uidmap# Install rootless Docker (as regular user, NOT sudo)dockerd-rootless-setuptool.sh install# Set environment variablesexport DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sockexport PATH=$HOME/bin:$PATH# Add to ~/.bashrc for persistenceecho 'export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock' >> ~/.bashrcecho 'export PATH=$HOME/bin:$PATH' >> ~/.bashrc# Enable service (user-level systemd)systemctl --user enable dockersystemctl --user start docker# Verifydocker info | grep -i rootless# Should show: rootless: true
Note: In security-critical environments, use rootless Docker or Podman. In rootless mode, the impact of container breakout attacks is drastically reduced because the daemon is not root. In Kubernetes environments, configure securityContext with non-root users.
Note (Production Must-Haves):log-driver + log-opts: By default Docker logs can consume unlimited disk. Set max-size: 100m and max-file: 3 — total 300MB max per container. live-restore: true: Containers will not stop when the Docker daemon restarts. Critical for zero-downtime daemon updates! storage-driver: overlay2: Best performance on modern kernels. Also the default on Ubuntu.
Note (containerd-shim): containerd-shim is an intermediate process spawned for each container. Its job: (1) relay stdin/stdout/stderr, (2) report exit status, (3) keep the container alive even if the daemon crashes. This is why live-restore: true works!
OCI — Open Container Initiative
Specification
Defines
Implemented By
Image Spec
Image format: layers, manifest, config JSON
Docker, containerd, Buildah
Runtime Spec
How to run a container: config.json + rootfs
runc, crun, kata-containers
Distribution Spec
How images are pushed/pulled from registries
Docker Hub, ECR, GCR, Harbor
Tip (Interview Question):Q: “What is the difference between Docker and containerd?”A: Docker is a complete platform (CLI + daemon + build tools + compose). containerd is just a container runtime — lightweight, CRI-compatible, perfect for Kubernetes. Docker internally uses containerd. Kubernetes bypasses the Docker daemon and talks directly to containerd (since v1.24).
An image is an immutable blueprint. Think of it this way: a class Component extends React.Component — you don’t directly “run” a class, you create an instance. Similarly, Image = Class, Container = Instance. An image is built once, then thousands of containers can be created from it. The image itself never changes.
Technically, a Docker image is a read-only filesystem snapshot + metadata (env vars, exposed ports, entry command), organized in layers — each layer representing an incremental change.
Image Layers — The Onion Model
text
┌─────────────────────────────────────────┐│ Container Layer — WRITABLE │ ← New files, modifications go here.│ New files, modifications go here. │ Deleted when container stops.│ Deleted when container stops. │├─────────────────────────────────────────┤│ ↑ WRITABLE / READ-ONLY ↓ │├─────────────────────────────────────────┤│ Layer 4: COPY . /app │ ← Your application source code (~5 MB)├─────────────────────────────────────────┤│ Layer 3: RUN npm install │ ← node_modules directory (~80 MB)├─────────────────────────────────────────┤│ Layer 2: WORKDIR /app │ ← Creates /app directory (metadata only)├─────────────────────────────────────────┤│ Layer 1: FROM node:18-alpine │ ← Base OS + Node.js runtime (~120 MB,│ │ shared with other node images)└─────────────────────────────────────────┘ OverlayFS merges all layers → appears as single filesystem to container
Note: OverlayFS is a Linux kernel filesystem driver. It merges multiple read-only layers and one read-write layer into a single unified filesystem. The container sees a normal filesystem, but it is actually a stack of layers. Copy-on-Write is used — when a container modifies a file, it is first copied from the read-only layer to the read-write layer (upperdir), then modified.
Image Manifest and Digest
bash
# View image manifest (JSON format)docker manifest inspect nginx:latest# Image digest — immutable SHA256 referencedocker images --digests nginx# REPOSITORY TAG DIGEST IMAGE ID# nginx latest sha256:a484819eb60... e784f4560448# Pull by digest (PRODUCTION BEST PRACTICE)# Tags can change (latest is mutable), digests never change!docker pull nginx@sha256:a484819eb60211f5299034ac80f6a681b06f89e65866ce91f356ed7c72af059c# Tags are mutable — this is dangerous:docker pull nginx:latest # Today pulls one version, tomorrow may pull another# Safe production pull (pinned digest):docker pull nginx@sha256:exact_digest_from_registry# Get digest of a local imagedocker inspect nginx --format='{{index .RepoDigests 0}}'
Caution: The :latest tag is a moving target. If you pull today you get 1.25; if CI/CD pulls tomorrow it might get 1.26 — with breaking changes. In production, always pin semantic versions: nginx:1.25.3-alpine or use digests. :latest is only acceptable for local development.
Image Storage on Disk
bash
# Where Docker images are stored on diskls /var/lib/docker/# Image layers (overlay2 storage driver)sudo ls /var/lib/docker/overlay2/# Each directory is a layer# Image metadatasudo ls /var/lib/docker/image/overlay2/imagedb/content/sha256/# Each file is an image manifest/config# Disk usage breakdown — important for DevOps!docker system df# TYPE TOTAL ACTIVE SIZE RECLAIMABLE# Images 15 3 2.1GB 1.8GB (85%)# Containers 5 2 45MB 23MB (51%)# Local Volumes 8 3 560MB 300MB (53%)# Build Cache 0 0 0B 0B# Detailed breakdowndocker system df -v# View layer sharing (images share base layers)docker history nginx:latest --no-truncdocker history nginx:alpine --no-trunc# Common layers are shared — only one copy on disk!
Image Tags vs Digests vs IDs
Concept
Example
Mutable?
Use Case
Image ID
e784f4560448
No (content-addressed)
Local reference, scripting
Tag
nginx:1.25.3
Yes (can be moved)
Human-readable, dev/staging
Digest
nginx@sha256:abc...
No (cryptographic)
Production deployments, security
latest tag
nginx:latest
Yes (frequently moves)
Local dev ONLY, never production
Tip (Interview Question):Q: “What is the difference between a Docker image and a container?”A: An image is a read-only blueprint/template — a static filesystem snapshot. A container is a running instance of that image, with a writable layer added on top. You can create thousands of containers from one image. When a container is stopped, the writable layer is deleted (unless committed or a volume is used).
Purpose: Download images from a registry to your local machine. Pull without docker run when you want the image without starting a container.
bash
# Basic pull — pulls latest tag (avoid in production!)docker pull nginx# Specific tag pull (recommended)docker pull nginx:1.25.3-alpine# Specific digest pull (MOST RECOMMENDED for production)docker pull nginx@sha256:a484819eb60211f5299034ac80f6a681b06f89e65866ce91f356ed7c72af059c# Pull from private registrydocker pull myregistry.company.com/myapp:v1.2.3# Pull from AWS ECR (login first)aws ecr get-login-password --region ap-south-1 | \ docker login --username AWS --password-stdin \ 123456789012.dkr.ecr.ap-south-1.amazonaws.comdocker pull 123456789012.dkr.ecr.ap-south-1.amazonaws.com/myapp:latest# Pull from GCRdocker pull gcr.io/my-project/myapp:v2.0.0# Pull from GitHub Container Registrydocker pull ghcr.io/myorg/myapp:main# Specify platform (ARM/AMD64)docker pull --platform linux/amd64 nginx:latestdocker pull --platform linux/arm64 nginx:latest # For Apple Silicon
Caution: Always specify a tag when pulling. docker pull nginx = docker pull nginx:latest. If the latest tag is updated tomorrow, your CI/CD will pull a different image. Fix: docker pull nginx:1.25.3 or use a digest.
docker images — Listing Images
bash
# Basic listdocker imagesdocker image ls # Same command, new syntax# Table format with sizedocker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}\t{{.CreatedAt}}"# Filter dangling images (untagged, build artifacts)docker images -f dangling=true# Filter by labeldocker images -f label=env=production# Image IDs only (useful for scripting)docker images -q# Show images for a specific repositorydocker images nginx# Custom formatdocker images --format "{{.ID}}: {{.Repository}}:{{.Tag}} — {{.Size}}"# All images (including intermediates)docker images -a# Images with digestsdocker images --digests# Sort by sizedocker images --format "{{.Size}}\t{{.Repository}}:{{.Tag}}" | sort -h
docker rmi — Deleting Images
bash
# Remove by namedocker rmi nginxdocker image rm nginx# Specify tag (when multiple tags exist)docker rmi nginx:1.25.3-alpine# Remove by image IDdocker rmi e784f4560448# Remove multiple images at oncedocker rmi nginx:latest ubuntu:20.04 redis:alpine# Force remove — even if a container is using it (dangerous!)docker rmi -f nginx:latest# Remove dangling images (safe cleanup)docker image prune# Remove all unused images (aggressive cleanup)docker image prune -a# Remove without confirmationdocker image prune -fdocker image prune -a -f# Nuclear option — delete all images (careful!)docker rmi $(docker images -q)# Delete by age filterdocker image prune -a --filter "until=24h" # Older than 24 hoursdocker image prune -a --filter "until=168h" # Older than 7 days
Caution: An image used by a running container cannot be deleted — docker rmi will fail. Stop and remove the container first: docker stop <container> && docker rm <container>, then docker rmi <image>.
docker tag — Tagging Images
bash
# Tag for Docker Hubdocker tag myapp:latest myusername/myapp:v1.0.0docker tag myapp:latest myusername/myapp:latest# Tag for AWS ECRdocker tag myapp:latest \ 123456789012.dkr.ecr.ap-south-1.amazonaws.com/myapp:v1.0.0# Tag for GCRdocker tag myapp:latest gcr.io/my-gcp-project/myapp:v1.0.0# Tag for GitHub Container Registrydocker tag myapp:latest ghcr.io/myorg/myapp:v1.0.0docker tag myapp:latest ghcr.io/myorg/myapp:main-abc1234 # Git commit hash# CI/CD common pattern (git hash + version)GIT_HASH=$(git rev-parse --short HEAD)VERSION="1.2.3"docker tag myapp:latest myregistry.io/myapp:${VERSION}-${GIT_HASH}docker tag myapp:latest myregistry.io/myapp:${VERSION}docker tag myapp:latest myregistry.io/myapp:latest
Tip (Tagging Strategy for CI/CD): Good tagging strategy: (1) Git commit hash tag — always, immutable and traceable. (2) Semantic version tag. (3) Branch name tag (e.g., main, develop). (4) latest only on main/master branch. This strategy makes both rollback and debugging easy.
docker save & docker load — Transferring Images
bash
# Save a single imagedocker save -o myapp.tar myapp:latest# Save multiple images into one tardocker save -o all-images.tar nginx:latest redis:alpine postgres:15# Compressed save (saves disk space)docker save myapp:latest | gzip > myapp.tar.gz# Load from tar filedocker load -i myapp.tar# Load compressedgunzip -c myapp.tar.gz | docker load# Transfer via SSH (no file on disk!)docker save myapp:latest | ssh user@remote-server docker load# Verify loaded imagedocker images | grep myapp
docker history — Viewing Image Layers
bash
# View image historydocker history nginx# Full commands (not truncated)docker history --no-trunc nginx# Custom formatdocker history --format "table {{.ID}}\t{{.Size}}\t{{.CreatedBy}}" nginx# Identify large layersdocker history nginx --format "{{.Size}}\t{{.CreatedBy}}" | \ grep -v "^0B" | \ sort -rh | head -5
Place stable layers (base image, dependencies) below volatile ones (source code) so Docker's layer cache is reused on every incremental build.
OverlayFS — Technical Deep Dive
OverlayFS Component
Docker Equivalent
Description
lowerdir
Image layers (read-only)
Base image + all intermediate layers — never modified
upperdir
Container’s writable layer
Container-specific changes go here
workdir
Internal OverlayFS use
Staging area for atomic operations
merged
Container’s filesystem view
Unified mount — what the container sees
bash
# View OverlayFS mount for a running containerdocker run -d --name mynginx nginx# Get mount infodocker inspect mynginx | jq '.[0].GraphDriver'# {# "Data": {# "LowerDir": "/var/lib/docker/overlay2/abc.../diff:/var/lib/docker/overlay2/def.../diff",# "MergedDir": "/var/lib/docker/overlay2/xyz.../merged",# "UpperDir": "/var/lib/docker/overlay2/xyz.../diff",# "WorkDir": "/var/lib/docker/overlay2/xyz.../work"# },# "Name": "overlay2"# }# View actual filesystem layering on hostsudo ls /var/lib/docker/overlay2/# View mount point on hostcat /proc/mounts | grep overlay# Demonstrate Copy-on-Writedocker exec mynginx sh -c "echo 'hello' > /usr/share/nginx/html/test.html"# The file went into the container's upperdir, NOT into the image.# The image remains unchanged.
Layer Caching — The Secret of Build Speed
bash
# First build — everything downloaded and executed (slow)docker build -t myapp:v1 .# Step 1/5 : FROM node:18-alpine → DOWNLOADING (slow)# Step 2/5 : WORKDIR /app → Running# Step 3/5 : COPY package*.json ./ → Running# Step 4/5 : RUN npm install → Installing (slow!)# Step 5/5 : COPY . . → Running# Second build (only source code changed)docker build -t myapp:v2 .# Step 1/5 : FROM node:18-alpine → Using cache ✓ (instant!)# Step 2/5 : WORKDIR /app → Using cache ✓ (instant!)# Step 3/5 : COPY package*.json ./ → Using cache ✓ (package.json unchanged)# Step 4/5 : RUN npm install → Using cache ✓ (instant!)# Step 5/5 : COPY . . → Running ← Only this re-runs!# Total time: 2 seconds vs 3 minutes!# Force bypass cachedocker build --no-cache -t myapp:fresh .# Use cache from external source (CI/CD optimization)docker build \ --cache-from myregistry.io/myapp:latest \ -t myapp:new .
Cache Invalidation — When Cache Breaks
Instruction
Cache Invalidates When
Note
FROM
Base image digest changes
Rarely — unless you explicitly pull new base
RUN
Command text changes
Even whitespace change = cache miss!
COPY / ADD
File content changes (checksum)
Even 1 byte change = cache miss for this layer + all subsequent
ENV
Value changes
All subsequent layers also invalidated
ARG
Build arg value changes
Be careful with VERSION args — they invalidate builds
WORKDIR
Path changes
Usually stable
Caution: A cache miss in one layer causes ALL subsequent layers to re-run. This is why layer ordering is so important — put frequently changing instructions last, stable instructions first.
Layer Ordering — Cache Optimization
dockerfile
# ❌ BAD — npm install runs EVERY TIME any file changesFROM node:18-alpineWORKDIR /appCOPY . . # Source code AND package.json copied togetherRUN npm install # Re-runs even if only App.jsx changed!
dockerfile
# ✅ GOOD — npm install only when package.json changesFROM node:18-alpineWORKDIR /app# Step 1: Copy ONLY package files firstCOPY package.json package-lock.json ./# Step 2: Install dependencies (CACHED as long as package.json is unchanged)RUN npm install# Step 3: Now copy source code (this layer changes with source changes)COPY . .# BuildRUN npm run build# ServeCMD ["node", "server.js"]
dockerfile
# ✅ BEST — Python/pip example follows the same principleFROM python:3.11-slimWORKDIR /app# Dependencies first (stable layer)COPY requirements.txt .RUN pip install --no-cache-dir -r requirements.txt# Source code later (frequently changing)COPY . .CMD ["python", "app.py"]
dockerfile
# ✅ BEST — Java/Maven exampleFROM maven:3.9-eclipse-temurin-17 AS buildWORKDIR /app# Copy pom.xml first (dependencies are stable)COPY pom.xml .RUN mvn dependency:go-offline -q # Download all dependencies (cached!)# Source code laterCOPY src ./srcRUN mvn package -q -DskipTestsFROM eclipse-temurin:17-jre-alpineCOPY --from=build /app/target/*.jar app.jarCMD ["java", "-jar", "app.jar"]
Note (Golden Rule of Layer Ordering): Think of it this way: “What changes least goes first.” Order of stability: Base image > System packages > Dependency files > Dependencies install > Config files > Source code > Build artifacts. Following this order will dramatically speed up CI/CD builds.
Layer Sharing — Memory & Disk Efficiency
When 50 Node.js microservices all use the same node:18-alpine base image:
This means more containers fit per Kubernetes node, node startup is faster (layer already cached), and network bandwidth is saved during pulls.
Layer Analysis Tools
bash
# Method 1: docker history (built-in)docker history --no-trunc myapp:latest# Method 2: docker inspect layersdocker inspect myapp:latest | jq '.[0].RootFS.Layers'# Method 3: dive tool (BEST for optimization)# Installationwget https://github.com/wagoodman/dive/releases/download/v0.12.0/dive_0.12.0_linux_amd64.debsudo apt install ./dive_0.12.0_linux_amd64.deb# Interactive layer analysisdive myapp:latest# Left panel: layer list + size# Right panel: filesystem changes per layer# Shows wasted space — e.g., files deleted in later layers# Gives an efficiency score# CI mode (automated check)dive myapp:latest --ci # Fails if efficiency is below threshold# Method 4: BuildKit inline --progress=plainDOCKER_BUILDKIT=1 docker build --progress=plain -t myapp:latest .# Shows each step's timing and cache status
Advanced Layer Optimization Techniques
dockerfile
# Technique 1: Combine RUN commands (fewer layers)# ❌ BAD — 3 layers for apt operationsRUN apt updateRUN apt install -y curl wgetRUN rm -rf /var/lib/apt/lists/*# ✅ GOOD — Single layer, cleanup in SAME layerRUN apt update && \ apt install -y curl wget && \ rm -rf /var/lib/apt/lists/*# IMPORTANT: Cleanup must be in the SAME RUN instruction!# Cleaning in a separate RUN does NOT reduce layer size# (the layer with apt cache already exists)# Technique 2: Multi-stage builds (drastically reduce SIZE)# Build stageFROM node:18 AS builderWORKDIR /appCOPY package*.json ./RUN npm ci --production=false # All deps including devDependenciesCOPY . .RUN npm run build # Create production build# Production stage (runtime only!)FROM node:18-alpine # Slim base (50MB vs 1GB!)WORKDIR /appCOPY --from=builder /app/dist ./distCOPY --from=builder /app/node_modules ./node_modules# Build tools, devDeps, source code do not go into the production image!EXPOSE 3000CMD ["node", "dist/server.js"]# Result: 1.2GB → 180MB image!# Technique 3: .dockerignore (like .gitignore for Docker)# node_modules/ ← exclude from COPY# .git/ ← unnecessary# dist/ ← build output (will be rebuilt)# *.log ← log files# .env ← Secrets! NEVER in image# coverage/ ← test coverage reports# Technique 4: Specific COPY (avoid COPY . .)# ❌ BADCOPY . . # Copies everything including .git, tests, docs# ✅ GOODCOPY src/ ./src/COPY public/ ./public/COPY package*.json ./
Tip (Senior Interview Question):Q: “A Docker image has grown to 2GB — how would you optimize it?”A: (1) Use docker history --no-trunc myimage to identify large layers. (2) Use multi-stage builds — don’t include build tools in production. (3) Use Alpine base image (50MB vs 900MB). (4) Run apt-get clean && rm -rf /var/lib/apt/lists/* in the same RUN instruction. (5) Use .dockerignore to exclude unnecessary files. (6) Use the dive tool to find wasted space. (7) Combine RUN instructions. Typically, 2GB → 150-200MB is achievable!
Every Docker container passes through multiple states during its life. Think of them like React component lifecycle (mount, update, unmount) — but more complex:
CREATED — Container created via docker create but not started
RUNNING — Container is active
PAUSED — Frozen, consuming no CPU
EXITED — Stopped or done (via docker stop, docker kill, exit, or crash)
RESTARTING — Auto-restart loop in progress
DEAD — Restart attempts failed, max retries exceeded
REMOVED — Container permanently deleted via docker rm
Image vs Container
Aspect
Image
Container
Nature
Blueprint / Template
Running instance
State
Immutable, static (never changes)
Mutable, stateful (can change)
Storage
Read-only layers
Image layers + writable layer on top
Multiple instances
Shared by all containers using it
Each container fully independent
Lifecycle
Build once, use forever
Create → Run → Stop → Delete
Disk space
Stored once even if 100 containers use it
Only writable layer extra per container
Analogy
Recipe
The actual dish
React analogy
React Component class/function
Component instance rendered in DOM
Container Lifecycle Commands — Quick Reference
bash
# Full lifecycle — a container's complete journeydocker create --name myapp nginx # Create but don't startdocker start myapp # Start itdocker pause myapp # Freeze it (stops CPU usage)docker unpause myapp # Unfreeze itdocker stop myapp # Graceful stop (SIGTERM then SIGKILL)docker start myapp # Start againdocker restart myapp # Stop + Start in one commanddocker kill myapp # Immediate kill (SIGKILL)docker rm myapp # Remove container (permanently delete)# One-liner (create + start + remove when done — for temporary tasks)docker run --rm --name temp-app nginx# Check current state of all containersdocker ps -a
Tip:docker run = docker create + docker start + docker attach (in foreground mode). CI/CD pipelines almost always use docker run. docker create is useful when you want to set up a container first and start it later.
Caution: If a container enters the DEAD state, it cannot be restarted. You can only remove it with docker rm. This happens when restart policy max retries are exhausted. Monitor DEAD containers in production!
docker run creates a container, starts it, and optionally attaches to it. Understanding its options is essential.
bash
# Full syntaxdocker run [OPTIONS] IMAGE [COMMAND] [ARG...]# -d: Detached mode (run in background)docker run -d nginxdocker run -d --name my-nginx nginx# -p: Port mapping (host_port:container_port)docker run -d -p 8080:80 nginx # Host 8080 → Container 80docker run -d -p 127.0.0.1:8080:80 nginx # Bind to specific IPdocker run -d -p 8080:80 -p 443:443 nginx # Multiple portsdocker run -d -P nginx # Auto-map ALL EXPOSED ports# -v: Volume mount (old syntax)docker run -d -v /host/path:/container/path nginxdocker run -d -v myvolume:/app/data nginxdocker run -d -v $(pwd):/app node:18 # Mount current directory# --mount: Volume mount (new, preferred syntax)docker run -d --mount type=bind,source=/host/path,target=/container/path nginxdocker run -d --mount type=volume,source=myvolume,target=/data nginxdocker run -d --mount type=tmpfs,target=/tmp nginx# -e: Environment variablesdocker run -d -e NODE_ENV=production node:18docker run -d -e DB_HOST=localhost -e DB_PORT=5432 myappdocker run -d --env-file .env myapp # Load from .env file# --name: Give the container a namedocker run -d --name my-api myapp:v1.0# --rm: Automatically remove when it exitsdocker run --rm ubuntu echo "hello" # Perfect for one-off commandsdocker run --rm -v $(pwd):/app node:18 npm test# --network: Connect to a networkdocker run -d --network mynetwork nginxdocker run -d --network host nginx # Use host machine's network# --restart: Restart policydocker run -d --restart no nginx # Never restart (default)docker run -d --restart always nginx # Always restartdocker run -d --restart on-failure nginx # Restart only on failuredocker run -d --restart on-failure:3 nginx # Max 3 retriesdocker run -d --restart unless-stopped nginx # Restart unless manually stopped# -it: Interactive Terminal (enter the container)docker run -it ubuntu bash # Start bash sessiondocker run -it --rm ubuntu bash # Session ends → container deleteddocker run -it node:18 node # Node.js REPL# Resource Limits (CRITICAL IN PRODUCTION!)docker run -d --memory=512m nginx # Max 512MB RAMdocker run -d --memory=512m --memory-swap=1g nginx # RAM + swap limitdocker run -d --cpus=0.5 nginx # 50% of one CPUdocker run -d --cpus=2 nginx # 2 full CPUs# --user: Run as a specific user (security)docker run -d --user 1000:1000 nginxdocker run -d --user nobody nginx# Real-world Production Example — all togetherdocker run -d \ --name my-react-app \ -p 3000:3000 \ -e NODE_ENV=production \ -e API_URL=https://api.myapp.com \ -v /var/log/app:/app/logs \ --memory=256m \ --cpus=0.5 \ --restart unless-stopped \ --network app-network \ myreactapp:v1.2.3
Note (Production): Without resource limits, a single buggy container can consume all server memory. A memory limit means the OOM killer will kill the container — not the entire server. CPU limits ensure fair sharing between containers.
docker ps — List Containers
bash
# Running containers onlydocker ps# ALL containers (including stopped)docker ps -a# IDs only (useful in scripts)docker ps -q# Last 5 created containersdocker ps -n 5# Custom formatdocker ps --format "table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Ports}}"docker ps --format "{{.Names}}: {{.Status}}"# Filter by statusdocker ps -f status=runningdocker ps -f status=exiteddocker ps -f name=myappdocker ps -f ancestor=nginx # Containers using nginx image# Show container sizesdocker ps -s
docker stop / start / restart
bash
# Graceful stop — sends SIGTERM, waits 10 sec, then SIGKILLdocker stop myappdocker stop --time=30 myapp # Wait 30 sec before SIGKILL# Stop multiple containers at oncedocker stop container1 container2 container3# Stop all running containersdocker stop $(docker ps -q)# Start a stopped containerdocker start myappdocker start -a myapp # Start + attach to output# Restart = stop + startdocker restart myappdocker restart --time=5 myapp # Wait 5 sec before SIGKILL
docker pause / unpause — Freeze a Container
bash
# Freeze container (uses cgroups freezer)# Container state is preserved, no CPU consumeddocker pause myapp# Unfreeze — container continues from where it left offdocker unpause myapp# When to use:# - Load testing: pause one container, test another# - Debug: capture exact state without losing data# - Resource saving: temporarily free CPU
Note:docker pause = process freeze (still in memory, state preserved, no CPU). docker stop = process terminate (SIGTERM → SIGKILL). Unpause is very fast because the process doesn’t restart — it just unfreezes.
docker kill — Immediate Termination
bash
# Immediate kill (SIGKILL — no grace period)docker kill myapp# Send a specific signaldocker kill --signal=SIGTERM myapp # Graceful shutdown requestdocker kill --signal=SIGHUP myapp # Nginx config reload (without restart!)docker kill --signal=9 myapp # SIGKILL (same as default kill)docker kill --signal=SIGUSR1 myapp # App-specific custom signal
docker rm — Remove Containers
bash
# Remove a stopped containerdocker rm myapp# Force remove a running container (think before you do this!)docker rm -f myapp# Remove container and its anonymous volumedocker rm -v myapp# Remove all stopped containers at once (recommended!)docker container prunedocker rm $(docker ps -aq -f status=exited)# Remove ALL containers (running + stopped) — nuclear option!docker rm -f $(docker ps -aq)
docker logs — View Logs (First Debug Tool)
When something is not working, look at logs first!
bash
# Basic logsdocker logs myapp# Follow (like tail -f, real-time updates)docker logs -f myapp# Last 100 lines onlydocker logs --tail 100 myapp# Show with timestampsdocker logs -t myapp# Logs since a specific timedocker logs --since 2024-01-01 myappdocker logs --since 1h myapp # Last 1 hour of logs# Most useful combination — last 50 lines follow with timestampsdocker logs -f -t --tail 50 myapp
Tip:docker logs works when the application writes to stdout/stderr. If the app writes to a file (e.g., /var/log/app.log), use docker exec -it myapp tail -f /var/log/app.log, or mount a volume and read from the host.
docker exec — Enter a Running Container
This command is used daily for debugging. Enter a running container and investigate.
bash
# Run a command inside the containerdocker exec myapp ls /appdocker exec myapp cat /etc/nginx/nginx.conf# Interactive bash session (most common use case)docker exec -it myapp bash# Alpine-based containers: use sh (bash not available)docker exec -it myapp sh# Login as root (when full access is needed)docker exec -it --user root myapp bash# With environment variabledocker exec -e DEBUG=true myapp node debug.js# Real-world examplesdocker exec -it postgres psql -U admin -d mydb # Enter PostgreSQLdocker exec -it redis redis-cli # Open Redis CLIdocker exec -it nginx nginx -t # Test nginx configdocker exec -it nginx nginx -s reload # Gracefully reload nginxdocker exec -it myapp cat /app/config.json # Read config filedocker exec -it myapp env # View all env variablesdocker exec -it myapp ps aux # View running processes
Caution:docker exec starts a new process inside the container. docker attach connects to the container’s main process — pressing Ctrl+C will STOP the container! For debugging, always use exec.
docker inspect — Extract Full Details
bash
# Full JSON infodocker inspect myapp# Extract specific info with Go templatesdocker inspect --format '{{.NetworkSettings.IPAddress}}' myappdocker inspect --format '{{.Config.Env}}' myappdocker inspect --format '{{.Config.Image}}' myappdocker inspect --format '{{.State.StartedAt}}' myapp# Port mappingsdocker inspect --format '{{json .NetworkSettings.Ports}}' myapp | jq# Volume/mount infodocker inspect --format '{{json .Mounts}}' myapp | jq# Container exit code (for debugging)docker inspect --format '{{.State.ExitCode}}' myapp
# View processes inside container (from host, without entering container)docker top myappdocker top myapp auxdocker top myapp -ef# Real-time resource usage (live dashboard)docker statsdocker stats myappdocker stats --no-stream # One snapshotdocker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"# Copy files between container and hostdocker cp myapp:/app/config.json ./config.json # Container → hostdocker cp ./config.json myapp:/app/config.json # Host → containerdocker cp myapp:/var/log/app.log ./app.log# Works with stopped containers too!# Create a new image from a modified containerdocker commit myapp myapp-modified:v2docker commit -m "Added custom nginx config" -a "Author" myapp myapp:custom# NOTE: Use Dockerfiles in production — docker commit is not reproducible!
Foreground vs Background — Attached vs Detached Mode
bash
# Foreground (attached) — output visible, Ctrl+C stops the containerdocker run nginx# Terminal is blocked, nginx logs appear here# Background (detached) — runs in background, terminal is freedocker run -d nginx# Returns container ID, you can do other work# Attach to a running container's outputdocker attach myapp# WARNING: Ctrl+C stops the container!# To detach without stopping: Ctrl+P then Ctrl+Q
Detaching Without Stopping — Critical Shortcut
bash
# Start an interactive sessiondocker run -it ubuntu bash# To leave WITHOUT stopping the container:# Press Ctrl+P then Ctrl+Q (both quickly)# Container keeps running, you get your terminal back# Return laterdocker attach ubuntu_container_name# exec vs attach — which to use?# exec — starts a new process (SAFE, always use this)docker exec -it myapp bash# attach — connects to main process (RISKY for debugging)docker attach myapp # Ctrl+C = container STOP!
Caution: New users often use docker attach and then press Ctrl+C thinking they are only detaching from the terminal — but the container STOPS! In production, always use docker exec -it myapp bash. If you must use attach, detach with Ctrl+P Ctrl+Q.
Container Exit Codes
Exit codes tell you why a container stopped. Visible in docker ps -a in the STATUS column as “Exited (137)”.
Permission error — command found but not executable
127
Command not found
Wrong command or PATH issue, typo in CMD/ENTRYPOINT
130
Ctrl+C (SIGINT)
Manual interruption
137
SIGKILL (128+9)
OOM killer or docker kill / docker stop timeout
139
Segmentation fault
Application memory error
143
SIGTERM (128+15)
docker stop — graceful shutdown signal
Tip: Signal-based exit codes = 128 + signal_number. SIGKILL = signal 9, exit code = 137. SIGTERM = signal 15, exit code = 143. Exit 137 almost always means OOM or force kill.
Restart Policies — Detailed
bash
# no (default) — Never restartdocker run --restart no nginx# Use case: one-off scripts, batch jobs, dev environment# always — Always restart (ignores exit code)docker run --restart always nginx# Note: Also auto-starts on Docker daemon restart!# Use case: critical services that must always be running# on-failure — Restart only on failure (non-zero exit)docker run --restart on-failure nginx # Unlimited retriesdocker run --restart on-failure:5 nginx # Max 5 retries, then DEAD# Use case: services that may occasionally crash# unless-stopped — Like always, but respects manual stopdocker run --restart unless-stopped nginx# docker stop → will NOT restart on daemon restart# Use case: Production services (RECOMMENDED for most cases)# Change policy on a running container (no restart required)docker update --restart unless-stopped myappdocker update --restart no myapp # Disable restart
Policy
Restart on crash?
Restart on daemon restart?
Restart after docker stop?
no
Never
No
No
always
Yes
Yes
Yes (even after docker stop!)
on-failure
Yes (non-zero exit only)
Yes (if was running)
No
unless-stopped
Yes
Yes
No (respects manual stop)
Resource Monitoring
bash
# Real-time stats dashboard (updates every 1 second)watch -n 1 docker stats --no-stream# Docker system overall usagedocker system df # Disk usage — images, containers, volumes, build cachedocker system events # Live event streamdocker system info # Complete Docker system info# Monitor events (powerful debugging tool)docker events # All events real-timedocker events --filter 'type=container' # Container events onlydocker events --filter 'event=die' # Container deathsdocker events --filter 'event=start' # Container startsdocker events --since 1h # Events from last 1 hour# Container health check statusdocker inspect --format '{{.State.Health.Status}}' myappdocker inspect --format '{{json .State.Health}}' myapp | jq
Cleanup Commands
bash
# Check disk usage firstdocker system df# TYPE TOTAL ACTIVE SIZE RECLAIMABLE# Images 25 3 12.5GB 10.2GB (82%)# Containers 8 2 150MB 130MB (86%)# Local Volumes 15 3 4.2GB 3.8GB (90%)# Build Cache - - 2.1GB 2.1GB# Individual cleanup (specific resources)docker container prune # Remove stopped containersdocker image prune # Remove dangling imagesdocker image prune -a # Remove all unused imagesdocker volume prune # Remove unused volumesdocker network prune # Remove unused networks# Nuclear options — use carefully!docker system prune # Remove everything unused (no volumes)docker system prune -a --volumes # Remove everything including volumes# Non-interactive (for CI/CD scripts)docker system prune -fdocker system prune -af --volumes # Full nuclear, no confirmation# Smart cleanup — only remove old itemsdocker container prune --filter "until=24h" # Containers older than 24 hoursdocker image prune --filter "until=168h" # Dangling images older than 1 weekdocker system prune --filter "until=72h" # Everything older than 3 days
Caution (Production Cleanup Strategy): On CI/CD servers, set up a daily cron job: 0 2 * * * docker system prune -f. On production servers, only docker container prune and docker image prune are safe — never run docker volume prune on production! An accidental docker volume prune has permanently deleted production data for multiple teams.
Tip (Interview — SIGTERM vs SIGKILL):docker stop first sends SIGTERM — a graceful shutdown request. The application has 10 seconds to clean up (close connections, complete in-flight requests, flush data). If it doesn’t exit within 10 seconds, SIGKILL is sent — forced termination, no cleanup. docker kill sends SIGKILL directly — no grace period. Use docker stop in production so applications can shut down properly. In Node.js: process.on('SIGTERM', gracefulShutdown). Exit codes: SIGTERM = 143, SIGKILL = 137.
A Dockerfile is a text file containing instructions for how to build an image. Think of it as a recipe card — ingredients and steps. Each instruction creates a layer, and layers together form the final Docker image.
Why build a custom image?
Your app requires a specific Node/Python version
Custom environment variables or config are needed
Pre-installed tools (curl, git, etc.) are required
Reproducible builds — every developer gets the same environment
Where to put the Dockerfile
Usually in the project root — the same directory as package.json or requirements.txt.
text
my-project/├── Dockerfile ← here├── Dockerfile.dev ← for dev environment├── Dockerfile.prod ← for prod environment├── .dockerignore ← CRITICAL! Never forget this├── package.json├── src/└── dist/
docker build — Building an Image
bash
# Build with current dir as contextdocker build -t myapp:latest .# Build with specific Dockerfiledocker build -f Dockerfile.prod -t myapp:prod .# Build from a remote Git repodocker build -t myapp https://github.com/user/repo.git# Build with no cache (fresh build)docker build --no-cache -t myapp .# Build with build argsdocker build --build-arg NODE_ENV=production -t myapp .# Build with multiple tagsdocker build -t myapp:latest -t myapp:v1.0.0 .# Build with verbose output (useful for CI)docker build --progress=plain -t myapp .
Tip: When you write docker build -t myapp ., the . (dot) is the build context — all files in this directory are sent to the Docker daemon. If your node_modules is 500MB and you have no .dockerignore, that entire directory gets sent. This is why .dockerignore is critical.
.dockerignore — Never Skip This
This file works exactly like .gitignore — it tells the Docker daemon which files to exclude from the build context.
Caution: Never let .env files into a Docker image. If AWS_SECRET_KEY or a database password ends up in an image, anyone can run docker history myapp and see it — even if you deleted it in a later layer!
dockerfile
# .dockerignore - put this in EVERY project!# Dependencies (will be installed inside container)node_modules/vendor/__pycache__/*.pyc.pytest_cache/venv/.venv/# Build outputsdist/build/.next/out/target/# Environment files (SECURITY RISK!).env.env.local.env.*.local.env.development.env.production*.pem*.keysecrets/# Version control.git/.gitignore# Logs*.loglogs/# OS files.DS_StoreThumbs.db# IDE/Editor files.idea/.vscode/# CI/CD & Docker meta.github/.dockerignoreDockerfile*# DocumentationREADME.md*.mddocs/# Test & Coveragecoverage/.nyc_output__tests__/
Build Cache
Docker caches every layer. If a layer has not changed, Docker uses the cache — builds become very fast.
Scenario
Result
Speed
Layer unchanged
Cache HIT
Instant
Layer changed
Cache MISS
Slow (rebuild)
After a cache miss, all subsequent layers
Force rebuild
Slow
--no-cache flag
Skip all cache
Slowest
Tip: What changes least goes on top. What changes most (source code) goes at the bottom. If package.json hasn’t changed, npm install won’t re-run — only source code copy will.
Image Naming Conventions
bash
# Tag formatsmyapp:latest # ❌ Avoid in production!myapp:v1.0.0 # ✅ Semantic version (recommended)myapp:1.0.0-alpine # ✅ Version + variantmyapp:2024-01-15 # ✅ Date-basedmyapp:abc1234 # ✅ Git commit hash (traceable)myapp:prod-v1.2.3 # ✅ Environment-specific# With registrydocker.io/myuser/myapp:v1.0.0 # Docker Hubghcr.io/myorg/myapp:v1.0.0 # GitHub Container Registry123456.dkr.ecr.us-east-1.amazonaws.com/myapp:v1.0.0 # AWS ECR# Build with git commit tag (common in CI/CD)docker build -t myapp:$(git rev-parse --short HEAD) .
Caution: Using :latest in production means if you push a new image with :latest, any server that restarts will pull the new image — without any warning! Always use specific version tags for reproducible deployments.
# Basic — always specify a versionFROM ubuntu:22.04# Specific digest (production best practice — immutable!)FROM ubuntu@sha256:abc123def456...# Multi-stage naming (used later with COPY --from)FROM node:18-alpine AS builderFROM nginx:alpine AS production# ARG before FROM (dynamic base image version)ARG NODE_VERSION=18FROM node:${NODE_VERSION}-alpine# ✅ BEST: Pinned version + variantFROM node:18.19.0-alpine3.19# ❌ BAD: Unpredictable, breaks buildsFROM node:latest
Tip (Alpine vs Slim vs Full):alpine: uses musl libc, 5MB, fastest — but some native modules may not compile. slim: Debian-based, ~75MB, more compatible. full: Most compatible, but large image. For most Node.js apps: start with node:18-alpine.
LABEL — Image Metadata
dockerfile
# ✅ BEST: All together in one layerLABEL \ maintainer="you@company.com" \ version="1.0.0" \ description="React DevOps Guide App" \ org.opencontainers.image.source="https://github.com/user/repo" \ org.opencontainers.image.created="2024-01-15" \ org.opencontainers.image.revision="abc1234"# Inspect labelsdocker inspect myapp --format='{{json .Config.Labels}}'
RUN — Execute Commands
dockerfile
# Shell form (runs in /bin/sh -c)RUN apt-get update# Exec form (no shell — preferred for clarity)RUN ["apt-get", "update"]# ❌ BAD: Multiple RUN = multiple layers = big imageRUN apt-get updateRUN apt-get install -y curlRUN apt-get install -y git# ✅ BEST: All in one RUN + clean cache in SAME layerRUN apt-get update && \ apt-get install -y --no-install-recommends \ curl \ git \ build-essential && \ rm -rf /var/lib/apt/lists/* # CRITICAL: clean apt cache!# Node.jsRUN npm ci --only=production && \ npm cache clean --force# PythonRUN pip install --no-cache-dir -r requirements.txt# Alpine (apk package manager)RUN apk add --no-cache \ curl \ git \ bash
Caution: If you don’t include rm -rf /var/lib/apt/lists/*, the apt cache (which can be several MB) stays in the image — size grows unnecessarily. Always clean in the SAME RUN instruction, not in a separate one (cleaning in a separate layer does not reduce the layer size).
CMD — Default Command
dockerfile
# ✅ Exec form (PREFERRED — no shell processing, better signal handling)CMD ["nginx", "-g", "daemon off;"]CMD ["node", "server.js"]CMD ["python", "app.py"]CMD ["uvicorn", "main:app", "--host", "0.0.0.0"]# ❌ Shell form (avoid — shell processing, harder to handle signals)CMD nginx -g "daemon off;"CMD node server.js# ⚠️ Multiple CMD — only the LAST one counts!CMD ["node", "app.js"]CMD ["node", "server.js"] # ← This one runs; the above is ignored# Override at runtimedocker run myapp node debug.js # CMD overridden
ENTRYPOINT — Fixed Executable
dockerfile
# Exec form (preferred)ENTRYPOINT ["node", "server.js"]# ENTRYPOINT + CMD combination (powerful pattern!)ENTRYPOINT ["node"]CMD ["server.js"] # docker run myapp → node server.js # docker run myapp debug.js → node debug.js (CMD overridden) # docker run myapp --version → node --version# Common pattern: entrypoint script for setup logicCOPY entrypoint.sh /entrypoint.shRUN chmod +x /entrypoint.shENTRYPOINT ["/entrypoint.sh"]CMD ["server"]# entrypoint.sh example:#!/bin/shset -eecho "Starting container..."# Run migrations, wait for DB, etc.exec "$@" # "$@" = pass CMD arguments through
CMD vs ENTRYPOINT — Most Important Comparison
Feature
CMD
ENTRYPOINT
Override easily?
Yes — docker run myapp npm test
No — requires --entrypoint flag
Purpose
Provide default arguments
Set a fixed executable
When used together
CMD = default args for ENTRYPOINT
ENTRYPOINT = command, CMD = args
Override example
docker run myapp bash
docker run --entrypoint bash myapp
Use case
Flexible containers (same image for dev/prod)
Single-purpose containers
Tip (Interview Question): “What is the difference between CMD and ENTRYPOINT?” — Short answer: CMD is easily overridden, ENTRYPOINT is not. Use them together — ENTRYPOINT as the executable, CMD as the default arguments.
COPY and ADD
dockerfile
# COPY — preferred for most casesCOPY package.json /app/COPY . /app/COPY package.json package-lock.json /app/# With ownership (chown)COPY --chown=node:node . /app/# From a multi-stage build (POWERFUL!)COPY --from=builder /app/dist /usr/share/nginx/html/COPY --from=builder /app/.next/standalone ./# ADD — supports URLs and auto-extracts tar filesADD https://example.com/file.tar.gz /tmp/ # ❌ Prefer curl in RUNADD myarchive.tar.gz /app/ # ✅ Auto-extracts tar
Feature
COPY
ADD
Local files
Yes
Yes
Download from URLs
No
Yes
Auto-extract tar
No
Yes
Caching
Predictable
Less predictable
Recommended
For most cases
Only for tar extraction
WORKDIR, ENV, ARG
dockerfile
# WORKDIR — sets working directoryWORKDIR /appWORKDIR /usr/src/app# ❌ BAD: Use cd commandRUN cd /app && npm install # Only valid for this RUN# ✅ GOOD: Use WORKDIRWORKDIR /appRUN npm install # Runs in /app# ENV — environment variables (build + runtime)ENV NODE_ENV=production \ PORT=3000 \ LOG_LEVEL=info \ TZ=Asia/Kolkata# Override at runtimedocker run -e NODE_ENV=development myapp# ⚠️ Never put secrets in ENV — visible in docker inspect!# ARG — build-time arguments (NOT available at runtime)ARG NODE_VERSION=18FROM node:${NODE_VERSION}-alpineARG BUILD_DATEARG GIT_COMMITARG APP_VERSION=1.0.0# Pass at build timedocker build \ --build-arg NODE_VERSION=20 \ --build-arg BUILD_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ) \ --build-arg GIT_COMMIT=$(git rev-parse --short HEAD) \ -t myapp .# Convert ARG to ENV (to make it available at runtime)ARG REACT_APP_API_URLENV REACT_APP_API_URL=$REACT_APP_API_URL
Feature
ARG
ENV
Available at
Build time only
Build + Runtime
Persists in container
No
Yes
Can use before FROM
Yes
No
Visibility
In layer history
In docker inspect
EXPOSE, VOLUME, USER
dockerfile
# EXPOSE — documentation only (does NOT actually open the port!)EXPOSE 3000EXPOSE 80 443EXPOSE 8080/tcpEXPOSE 53/udp# Actual port publishing requires -p flag at runtime:docker run -p 8080:3000 myapp # host:container# VOLUME — declare mount pointsVOLUME /app/dataVOLUME /var/lib/postgresql/dataVOLUME ["/app/logs", "/app/uploads"]# In production, use named volumes:docker run -v postgres_data:/var/lib/postgresql/data postgres# USER — run as non-root (security!)# ❌ BAD: Running as root (default if USER not specified)# ✅ Method 1: Named userRUN groupadd -r appuser && useradd -r -g appuser appuserUSER appuser# ✅ Method 2: Numeric UID (more secure)RUN addgroup --system --gid 1001 nodejs && \ adduser --system --uid 1001 nextjsUSER 1001# ✅ Node.js: Use the built-in 'node' userFROM node:18-alpineWORKDIR /appCOPY package*.json ./RUN npm ci --only=productionCOPY --chown=node:node . .USER nodeEXPOSE 3000CMD ["node", "server.js"]
Note: Kubernetes and AWS ECS policies typically prohibit containers running as root. Always specify a non-root USER. In Kubernetes, set runAsNonRoot: true in PodSecurityPolicy.
HEALTHCHECK
dockerfile
# HTTP health check (common for web apps)HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ CMD curl -f http://localhost:3000/health || exit 1# Node.js health check (when curl is not available)HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ CMD node -e "require('http').get('http://localhost:3000/health', r => r.statusCode === 200 ? process.exit(0) : process.exit(1))"# Disable inherited health checkHEALTHCHECK NONE# Options:# --interval=30s → check every 30 seconds (default: 30s)# --timeout=3s → fail if no response within 3s (default: 30s)# --start-period=5s → 5s grace period at start (container boot time)# --retries=3 → 3 consecutive failures = unhealthy# Check statusdocker inspect --format='{{.State.Health.Status}}' container_namedocker ps # STATUS column shows (healthy) or (unhealthy)
Shell Form vs Exec Form — Signal Handling
dockerfile
# Shell form: runs via /bin/sh -cCMD nginx -g "daemon off;"ENTRYPOINT node server.js# Exec form: runs directly (no shell)CMD ["nginx", "-g", "daemon off;"]ENTRYPOINT ["node", "server.js"]# KEY DIFFERENCE: Signal handling!# Shell form:# /bin/sh (PID 1) → nginx (child process)# SIGTERM goes to /bin/sh, not nginx → dirty shutdown!# Exec form:# nginx (PID 1) directly# SIGTERM goes directly to nginx → graceful shutdown!# RULE: ALWAYS use exec form in CMD and ENTRYPOINT
Tip (Interview): “What is the difference between shell form and exec form?” — Shell form runs via /bin/sh -c (does not become PID 1), exec form executes directly (becomes PID 1). In production, exec form is preferred because signals are properly received.
Problems: Ubuntu base (~80MB), Node+npm via apt (~200MB), multiple apt layers, all node_modules included, no cache optimization, running as root, shell form CMD.
After (180 MB image):
dockerfile
FROM node:18-alpineWORKDIR /appCOPY package*.json ./RUN npm ci --only=productionCOPY . .USER nodeEXPOSE 3000CMD ["node", "server.js"]
Benefits: Alpine base (~5MB), official Node image, layered cache optimization, production deps only, non-root user, exec CMD form.
Layer Ordering — Cache Optimization
dockerfile
# ❌ BAD ORDER: package.json copied with source codeFROM node:18-alpineWORKDIR /appCOPY . . # Source code AND package.json togetherRUN npm ci # Runs on every code change!RUN npm run buildEXPOSE 3000CMD ["node", "server.js"]# ✅ OPTIMAL ORDER: What changes least goes firstFROM node:18-alpine # Layer 1: base image (almost always cached)WORKDIR /app # Layer 2: almost always cachedCOPY package*.json ./ # Layer 3: rebuilt only when package.json changesRUN npm ci # Layer 4: rebuilt only when package.json changesCOPY . . # Layer 5: rebuilt on any code changeRUN npm run build # Layer 6: rebuilt on any code changeEXPOSE 3000CMD ["node", "server.js"]# RESULT:# If only src/App.js changed:# → Layers 1-4: CACHE HIT (npm ci skipped!)# → Layers 5-6: rebuild# Build time: 10s instead of 60s!
Tip: Thumb rule: “What changes least goes on top.” Order: Base image → System deps → Package manifest (package.json/requirements.txt) → Dependencies install → Source code copy → Build → Configuration.
Base Image Choices
Base Image
Size
Use Case
Pros/Cons
ubuntu:22.04
~80MB
Legacy apps needing apt
Familiar, large
debian:slim
~75MB
Debian apps, smaller
Compatible, slim
alpine:3.19
~5MB
Smallest possible
musl libc issues sometimes
node:18-alpine
~50MB
Node.js apps
Best for Node
python:3.11-slim
~130MB
Python apps
Best for Python
gcr.io/distroless/nodejs
~30MB
Security-focused
No shell — very secure
scratch
0MB
Static Go/Rust binaries
Smallest possible
Security Best Practices
dockerfile
# 1. Pin specific versionsFROM node:18.19.0-alpine3.19 # ✅ Exact versionFROM node:latest # ❌ Unpredictable# 2. Non-root user (always!)USER node# 3. NEVER put secrets in ENV/ARG permanently# ❌ WRONG — visible in docker history!RUN export API_KEY=secret123 && ./setup.sh# ✅ RIGHT — use runtime env varsdocker run -e API_KEY=secret123 myapp# 4. Don't install unnecessary packagesRUN apt-get install -y --no-install-recommends \ curl && \ rm -rf /var/lib/apt/lists/*# 5. Read-only filesystem (at runtime)docker run --read-only \ --tmpfs /tmp \ --tmpfs /app/logs \ myapp# 6. Drop all capabilities (at runtime)docker run --cap-drop ALL \ --cap-add NET_BIND_SERVICE \ myapp# 7. Scan images for vulnerabilitiesdocker scout cves myapp:latesttrivy image myapp:latest
Note (Production Security Checklist): Non-root USER | .dockerignore with .env | No secrets in Dockerfile | Specific base image version | HEALTHCHECK defined | Image vulnerability scan in CI/CD | Read-only filesystem where possible.
Multi-stage builds keep the build toolchain in a throw-away stage and ship only the compiled artifact in a minimal final image, cutting image size by 80–90%.
Why Multi-Stage Builds?
The problem: build tools are needed at build time (compiler, test runner, devDependencies) but should not be in the production image. Without multi-stage builds, everything ends up in the final image — size explodes.
Without Multi-Stage
With Multi-Stage
Build tools included (1.2 GB)
Only production artifacts (120 MB)
devDependencies included
Build tools discarded
Source code included
Source code discarded
Security risk (unnecessary tools)
Minimal attack surface
Slow to push/pull
Fast to push/pull
dockerfile
# BEFORE multi-stage: build tools in final imageFROM node:18WORKDIR /appCOPY . .RUN npm install # devDependencies too — 500MB!RUN npm run buildCMD ["node", "server.js"]# Final image: ~1.2 GB# AFTER multi-stage: only production artifacts# ---- Stage 1: Builder ----FROM node:18 AS builderWORKDIR /appCOPY package*.json ./RUN npm ci # ALL deps (devDeps for build)COPY . .RUN npm run build # Create production bundle# ---- Stage 2: Runner (final image) ----FROM node:18-alpine AS runnerWORKDIR /appCOPY package*.json ./RUN npm ci --only=production # Production deps onlyCOPY --from=builder /app/dist ./dist # Only built files!USER nodeEXPOSE 3000CMD ["node", "dist/server.js"]# Final image: ~120 MB (90% size reduction!)
Tip:COPY --from=builder only copies the files you specify — not the entire builder stage. Everything else (node_modules, devDeps, source code, compiler) is automatically discarded. The final image contains only the content from the LAST FROM onwards.
# ---- Stage 1: Go Binary Build ----FROM golang:1.21-alpine AS builderWORKDIR /appCOPY go.mod go.sum ./RUN go mod downloadCOPY . .# Static binary build (no external dependencies)RUN CGO_ENABLED=0 \ GOOS=linux \ GOARCH=amd64 \ go build \ -ldflags="-w -s" \ -a -installsuffix cgo \ -o app .# ---- Stage 2: Scratch (empty base!) ----FROM scratch# SSL certificates (for HTTPS calls)COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/# Only the binaryCOPY --from=builder /app/app /appEXPOSE 8080ENTRYPOINT ["/app"]# Final image: ~8-10MB (just the binary!)
Java Spring Boot Multi-Stage
dockerfile
# ---- Stage 1: Maven Build ----FROM maven:3.9-openjdk-17-slim AS builderWORKDIR /app# Copy pom.xml first (dependency caching!)COPY pom.xml .RUN mvn dependency:go-offline -B # Download all dependenciesCOPY src ./srcRUN mvn clean package -DskipTests -B# ---- Stage 2: JRE Only (no JDK!) ----FROM eclipse-temurin:17-jre-alpine AS productionWORKDIR /appRUN addgroup --system --gid 1001 spring && \ adduser --system --uid 1001 springUSER spring# Only the JAR (not Maven, not JDK)COPY --from=builder /app/target/*.jar app.jarEXPOSE 8080HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \ CMD curl -f http://localhost:8080/actuator/health || exit 1ENTRYPOINT ["java", \ "-XX:+UseContainerSupport", \ "-XX:MaxRAMPercentage=75.0", \ "-jar", "app.jar"]
Building Specific Stages
bash
# Build up to a specific stage (useful for debugging!)docker build --target builder -t myapp:builder .docker build --target production -t myapp:prod .# Open a shell in the builder stagedocker run --rm -it myapp:builder sh# Separate development vs production buildsdocker build --target development -t myapp:dev .docker build --target production -t myapp:prod .
These templates are copy-paste ready for production use. All are fully commented and follow security best practices. Customize the EXPOSE port, user names, and CMD for your project.
# ============================================# Next.js App — Production Dockerfile# Features: Standalone output, optimized,# non-root, telemetry disabled# Final size: ~100-150MB# ============================================FROM node:18-alpine AS baseRUN apk add --no-cache libc6-compatWORKDIR /app# ---- Stage 2: Dependencies ----FROM base AS depsCOPY package*.json ./RUN npm ci# ---- Stage 3: Build ----FROM base AS builderCOPY --from=deps /app/node_modules ./node_modulesCOPY . .ENV NEXT_TELEMETRY_DISABLED=1# REQUIRED: next.config.js must have output: 'standalone'RUN npm run build# ---- Stage 4: Runner ----FROM base AS runnerENV NODE_ENV=productionENV NEXT_TELEMETRY_DISABLED=1ENV PORT=3000LABEL maintainer="you@company.com" \ description="Next.js Production Image"RUN addgroup --system --gid 1001 nodejs && \ adduser --system --uid 1001 nextjsCOPY --from=builder /app/public ./publicCOPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/staticUSER nextjsEXPOSE 3000HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \ CMD node -e "require('http').get('http://localhost:3000/api/health', r => \ r.statusCode === 200 ? process.exit(0) : process.exit(1))"CMD ["node", "server.js"]# ============================================# REQUIRED: next.config.js must include:# const nextConfig = {# output: 'standalone',# }# module.exports = nextConfig# ============================================
Caution: For Next.js Docker images, output: 'standalone' in next.config.js is required. Without it, the .next/standalone folder will not be created and COPY will fail.
Quick Reference: Which Template to Use
Project Type
Template
Final Size
Server
React (CRA, Vite)
Template 1
~25-30MB
Nginx
Node.js / Express / Fastify
Template 2
~80-120MB
Node + dumb-init
Python Flask (sync)
Template 3
~130-160MB
Gunicorn
Python FastAPI (async)
Template 4
~130-150MB
Uvicorn
Next.js (full-stack)
Template 5
~100-150MB
Node standalone
Note (Common Patterns in Every Template):
Non-root USER — mandatory for security
HEALTHCHECK — so Docker/Kubernetes knows container health
Multi-stage builds — build tools not in final image
Cache optimization — package.json first, source code second
By default, every container runs in an isolated environment — its own filesystem, its own process space, and its own network namespace. However, real-world applications require containers to communicate with each other, connect to external databases, and serve clients. Understanding Docker’s networking model is therefore essential.
Scenarios that require networking:
Container-to-Container: A frontend container needs to communicate with a backend API
Container to External Services: An application container needs to connect to AWS RDS or a third-party API
External Clients to Containers: A browser sends a request and a container serves it
Multi-host Communication: Containers on different servers still communicate with each other (Overlay networks)
Container Networking Model (CNM)
Docker defines a standard networking architecture called the Container Networking Model (CNM). It has three core concepts:
Concept
Description
Real-World Analogy
Network
Isolated communication group where containers can talk to each other
An office floor — only people on that floor can communicate directly
Endpoint
A virtual NIC (network interface card) for a container on a network
An employee’s desk phone extension
Sandbox
Container’s isolated network namespace — its own IP, routing table, and DNS
An employee’s complete communication profile
Note:libnetwork — Docker’s Networking Brain. CNM is implemented by libnetwork — Docker’s networking subsystem. It is written in Go and supports a pluggable driver architecture. This means you can use bridge, overlay, macvlan — any driver — on top of libnetwork.
Network Driver Architecture: libnetwork uses a plugin system. Built-in drivers include: bridge, host, none, overlay, and macvlan. Third-party drivers can also be installed (e.g., Weave, Calico).
Docker Networking Model — Architecture Overview
Internet traffic arrives at the host → is forwarded to the docker0 bridge → passes through to Container A on port 80 via host port 8080. Container A and Container B can communicate directly with each other using their 172.17.x.x IP addresses through the bridge.
Tip:veth Pair — The Magic Behind Container Networking. When Docker starts a container, it creates a virtual ethernet pair (veth) — like two ends of a virtual cable. One end lives inside the container (eth0) and the other end attaches to the docker0 bridge. Data flows seamlessly from one end to the other.
The docker0 bridge connects containers on the same host using private IPs; a published port (-p 8080:80) uses iptables NAT to forward external traffic from the host into a specific container.
Bridge Network — The Most Common
When you start a container without the --network flag, it automatically joins the default bridge network (docker0). However, the default bridge has a significant limitation — DNS resolution does not work!
Caution:Common Beginner Mistake — No DNS on Default Bridge! On the default bridge, containers cannot find each other by name — only by IP address. The IP address can change on every restart. Always use a user-defined bridge!
bash
# Default bridge — no DNS!docker run -d --name db mysql:8 # joins default bridgedocker run -d --name myapp myimage # joins default bridge# myapp CANNOT connect to "db" by hostname!# Must use IP: mysql -h 172.17.0.2 (fragile!)# User-defined bridge — DNS built-in!docker network create mynetworkdocker run -d --name db --network mynetwork mysql:8docker run -d --name myapp --network mynetwork myimage# Now myapp can connect directly using the "db" hostname!# MONGODB_URI=mongodb://db:27017/mydb ← works!# Create a user-defined bridge with optionsdocker network create \ --driver bridge \ --subnet 172.20.0.0/16 \ --gateway 172.20.0.1 \ my-custom-network
Feature
Default Bridge
User-Defined Bridge
DNS Resolution (by name)
No
Yes (container name!)
Automatic Discovery
No
Yes
Recommended?
Legacy only
Always use this
Network Isolation
All containers share it
Per-network isolation
Connect/disconnect from running container
No
Yes
Host Network — Maximum Performance
bash
# Container shares the host's network namespacedocker run --network host nginx# nginx now listens directly on host port 80# -p flag is not needed (and does not work)# Benefits:# - Best performance: no NAT overhead# - Low latency: direct kernel network stack access# Drawbacks:# - No isolation: container can see all host interfaces# - Port conflicts: container and host services cannot share the same port# Use cases:# - High-performance networking applications# - Monitoring agents (Prometheus node_exporter)# - Network packet inspection tools
None Network — Complete Isolation
bash
# No networking at all — completely isolateddocker run --network none myapp# Inside the container:# - No internet access# - No communication with other containers# - Only loopback (127.0.0.1) is available# Use cases:# - Batch data processing jobs (file input/output only)# - Security-sensitive computations# - Testing network-agnostic code# - Malware analysis environments
Overlay Network — Multi-Host Communication (Swarm)
bash
# Multi-host networking — containers on different servers communicate with each other# Requires: Docker Swarm mode (or Kubernetes)# Initialize Swarm modedocker swarm init# Create an overlay networkdocker network create --driver overlay myoverlay# Attachable overlay (standalone containers can join too)docker network create --driver overlay --attachable myoverlay# Uses VXLAN tunneling automatically# Containers are unaware they are on different hosts!# 10.0.0.2 (Server 1) <-> 10.0.0.3 (Server 2) — transparent!
Note:Overlay = VXLAN Tunneling. Inside an overlay network, Docker uses VXLAN (Virtual Extensible LAN) — one packet is wrapped inside another. A packet from a container on Server A is wrapped in UDP and sent to Server B, where it is unwrapped. Containers have no idea they are running on different machines.
Macvlan — Container as a Physical Device
bash
# Container gets its own MAC address# Appears on the network like a real physical devicedocker network create \ --driver macvlan \ --subnet=192.168.1.0/24 \ --gateway=192.168.1.1 \ -o parent=eth0 \ mymacvlandocker run -d \ --name legacy-app \ --network mymacvlan \ --ip 192.168.1.50 \ my-legacy-app# The router treats 192.168.1.50 as a physical machine!# Use cases:# - Legacy applications that expect direct network access# - Migrating VMs to containers# - Network monitoring tools
Note:Bridge vs Host Network.Bridge: Container has its own network namespace, communicates with the host via NAT, and requires port mapping. Host: Container shares the host’s network namespace — no NAT, no port mapping, maximum performance. Use Bridge when isolation is required; use Host when performance is critical and isolation can be sacrificed.
Tip:Always Use User-Defined Bridges. Never use the default bridge network in production. Create a dedicated user-defined network for each application stack. This provides DNS resolution, isolation, and security all in one. Docker Compose automatically creates a user-defined network for each project.
# List all networksdocker network ls# NETWORK ID NAME DRIVER SCOPE# a1b2c3d4e5f6 bridge bridge local# f6e5d4c3b2a1 host host local# 0000000000000 none null local# Show full IDsdocker network ls --no-trunc# Filter by driverdocker network ls --filter driver=bridge# Filter by namedocker network ls --filter name=mynetwork# Custom formatdocker network ls --format "table {{.ID}}\t{{.Name}}\t{{.Driver}}\t{{.Scope}}"
# Connect a running container to a new networkdocker network connect mynetwork mycontainer# Assign a specific IPdocker network connect --ip 172.20.0.100 mynetwork mycontainer# Add an alias (other containers can reach this container via the alias)docker network connect --alias db-primary mynetwork mycontainer# Disconnect (container keeps running)docker network disconnect mynetwork mycontainer# Force disconnectdocker network disconnect --force mynetwork mycontainer# Real use case: zero-downtime maintenance# 1. Start new container on the new network# 2. Gradually migrate old containers# 3. Disconnect from the old network
Tip:Multiple Networks — One Container on Many Networks! A container can be connected to more than one network simultaneously. For example, a “gateway” container can be connected to the frontend network on one side and the backend network on the other. This is useful in DMZ architectures where you want to control traffic flow.
docker network rm / prune — Cleanup
bash
# Remove a network (disconnect all containers first!)docker network rm mynetwork# Error: network mynetwork has active endpoints — if containers are attached# Remove containers first, then the networkdocker rm -f $(docker ps -q --filter network=mynetwork)docker network rm mynetwork# Remove multiple networks at oncedocker network rm net1 net2 net3# Remove all unused networks (none with connected containers)docker network prune# Skip confirmation promptdocker network prune -f# Filter by label before pruningdocker network prune --filter label=env=staging
Caution:docker network prune — Be Careful! This command deletes all “unused” networks — meaning any network not connected to at least one container. If you created a network for future use, it will be deleted too. Always run docker network ls before running prune in production.
Consider deploying a MERN (MongoDB, Express, React, Node) application. All three services need to communicate with each other. Here is how to set this up in Docker:
bash
# Step 1: Create a dedicated networkdocker network create mern-network# Step 2: Start MongoDB first (others depend on it)docker run -d \ --name mongodb \ --network mern-network \ -v mongo_data:/data/db \ mongo:7# Step 3: Start the Node.js API# Use "mongodb" as the hostname — Docker DNS resolves it!docker run -d \ --name api \ --network mern-network \ -e MONGODB_URI=mongodb://mongodb:27017/mydb \ -e PORT=5000 \ -p 5000:5000 \ myapi:latest# Step 4: Start the React frontend# Use "api" as the hostnamedocker run -d \ --name frontend \ --network mern-network \ -e REACT_APP_API_URL=http://api:5000 \ -p 3000:3000 \ myfrontend:latest# Communication flow:# Browser → Host:3000 → frontend container# frontend → api:5000 (DNS resolution via Docker) → api container# api → mongodb:27017 (DNS resolution via Docker) → mongodb container
Testing DNS Resolution
bash
# Test connectivitydocker exec frontend ping api -c 4# PING api (172.20.0.3): 56 data bytes# 64 bytes from 172.20.0.3: icmp_seq=0 ttl=64 time=0.089 ms# DNS lookupdocker exec api nslookup mongodb# Server: 127.0.0.11 <- Docker's built-in DNS server!# Address 1: 127.0.0.11# Name: mongodb# Address 1: 172.20.0.2 mongodb.mern-network# Test HTTP connectiondocker exec api wget -qO- http://mongodb:27017# It looks like you are trying to access MongoDB over HTTP...# (Connection succeeded — MongoDB responded)# Test the API from the frontend using curldocker exec frontend curl http://api:5000/health# {"status":"ok","db":"connected"}
Note:127.0.0.11 — Docker’s Secret DNS Server. Whenever you use a user-defined network, Docker provides an embedded DNS server at 127.0.0.11 inside every container. It resolves container names to IP addresses. This happens automatically — no configuration required.
Service Discovery Patterns — Best Practices
Pattern
Example
When to Use
Container Name as Hostname
mongodb://mongodb:27017
Simple setups, same host
Docker Compose Service Name
http://api:5000
Compose-based deployments
Network Alias
--network-alias db
When the container name may change
Environment Variables
DB_HOST=mongodb
Externalizing configuration
Caution:Never Hardcode IPs! A very common mistake: MONGODB_URI=mongodb://172.17.0.2:27017. The IP can change when the container restarts. Always use the container name as the hostname: mongodb://mongodb:27017. The container name is stable; the IP is not.
Note:Production Pattern: Network Segmentation. In real production environments, create separate networks: frontend-net (frontend + API) and backend-net (API + database). The database will not be directly accessible from the frontend — only through the API. This is the Zero Trust networking principle. In Docker Compose, this is easily configured with the networks: key.
This is a classic point of confusion for beginners. Both EXPOSE and -p deal with ports, but they serve different purposes.
bash
# EXPOSE in Dockerfile — documentation only!EXPOSE 3000# This only says "this container listens on port 3000"# It does NOT publish anything to the HOST# Other containers can read this information via inspect# -p flag: ACTUALLY publishes the port on the hostdocker run -p 8080:3000 myapp# host:8080 → container:3000# Requests arriving at host port 8080 are forwarded to container port 3000# -P flag (capital P): publish ALL EXPOSEd ports to random host portsdocker run -P myapp# If Dockerfile has EXPOSE 3000 3306:# Docker allocates: 32768:3000, 32769:3306 (random!)# Check which port was assigneddocker port myapp# 3000/tcp -> 0.0.0.0:32768docker port myapp 3000# 0.0.0.0:32768
Tip:Mnemonic: EXPOSE = Advertisement, -p = Actually Open the Door. EXPOSE is a “menu card” — it only describes what is available. The -p flag actually opens the door to the outside world. Without -p, services inside the container are not accessible from the host (though other containers on a custom network can still access them without port mapping).
Port Mapping Patterns
bash
# 1. Specific mapping (most common)docker run -p 8080:80 nginx # Host 8080 → Container 80# 2. Multiple portsdocker run -p 80:80 -p 443:443 nginx# 3. Default binding: 0.0.0.0 (all interfaces)docker run -p 8080:80 nginx# Any IP can reach port 8080 and hit the container# curl localhost:8080 works# curl 192.168.1.100:8080 works (accessible on LAN too)# 4. Bind to localhost only (security)docker run -p 127.0.0.1:8080:80 nginx# Only accessible from the same machine# curl localhost:8080 works# curl 192.168.1.100:8080 does not work# 5. Random port on all interfacesdocker run -p 0.0.0.0::80 nginx# Docker chooses a random high portdocker port <container> 80# 0.0.0.0:49153# 6. UDP portdocker run -p 53:53/udp my-dns-server# 7. Range of portsdocker run -p 8000-8010:8000-8010 myapp
Handling Port Conflicts
bash
# Error: Bind: address already in use# docker: Error response from daemon: driver failed programming# external connectivity on endpoint ...: Bind for 0.0.0.0:8080# failed: port is already allocated# Step 1: Find out who is using port 8080sudo lsof -i :8080sudo netstat -tulpn | grep 8080# Step 2: Options:# Option A: Use a different host portdocker run -p 9090:80 nginx# Option B: Stop the conflicting processsudo kill -9 <PID># Option C: Check if another Docker container is using the portdocker ps --format "table {{.Names}}\t{{.Ports}}" | grep 8080docker stop old-nginxdocker run -p 8080:80 nginx# Useful: view all mapped portsdocker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"
Caution:Security: Bind to Localhost in Development. When you use -p 8080:80, Docker automatically adds an iptables rule — even if UFW or firewalld is disabled. This means the service may be accessible from the internet. In development, always use -p 127.0.0.1:8080:80 for sensitive services.
Note:EXPOSE vs -p — Interview Answer.EXPOSE is only metadata/documentation — it does not actually publish any port. The -p flag creates an actual port mapping at runtime between the host and container using iptables rules. EXPOSE without -p is useless for host access (though it does provide information to other containers within the network).
The most important concept in storage: containers are ephemeral (temporary) by default. When a container is deleted, all its data is deleted too.
Caution:“The container was deleted and the data went with it!” — Classic Production Disaster. Imagine you started a MySQL container, added some data, and then accidentally ran docker rm -f mysql. The entire database is gone. This happens frequently in production. Solution: always use volumes for databases.
Container storage layers use a union filesystem (overlay2):
text
┌─────────────────────────────────────────┐│ [Writable Layer] ← Temporary! ││ The container's write area ││ Container deleted → THIS DATA IS GONE! │├─────────────────────────────────────────┤│ [App Layer] ← Read-only ││ Your application code │├─────────────────────────────────────────┤│ [Base Image Layer] ← Read-only ││ ubuntu:22.04 or node:18 etc. │└─────────────────────────────────────────┘# The bottom layers are part of the image — shared across containers!# Only the writable layer is container-specific — and it is temporary
Storage Drivers — The Underlying Filesystem
Driver
Description
When to Use
overlay2
Default; uses Linux overlayfs
Almost always — best choice
devicemapper
Block device based
Legacy RHEL/CentOS (outdated)
btrfs
Btrfs filesystem
Only on Btrfs hosts
zfs
ZFS filesystem
Only on ZFS hosts
vfs
No layering, simple copy
Testing only — very slow
bash
# Check the current storage driverdocker info | grep "Storage Driver"# Storage Driver: overlay2# Where does Docker store data?/var/lib/docker/ ├── containers/ # Per-container metadata, logs ├── image/ # Image metadata and manifests ├── overlay2/ # Image layers (most data is here!) ├── volumes/ # Named volumes └── network/ # Network configs# Check sizessudo du -sh /var/lib/docker/*# 1.2G /var/lib/docker/overlay2 # Most data here!# 200M /var/lib/docker/volumes# 50M /var/lib/docker/containers# Docker summarydocker system df# TYPE TOTAL ACTIVE SIZE RECLAIMABLE# Images 15 5 3.2GB 2.1GB (65%)# Containers 8 3 120MB 90MB (75%)# Local Volumes 5 2 1.1GB 500MB (45%)# Detailed breakdowndocker system df -v
Tip:Copy-on-Write (CoW) — The Secret of Efficiency. overlay2 uses a Copy-on-Write strategy. If 10 containers use the same base image, the image layers are stored only once on disk. Only when a container writes something does it create its own copy of that file. This saves both disk space and startup time.
Note:Production Rule: Databases Always Need Volumes! MySQL, PostgreSQL, MongoDB, Redis — whenever you run any database in a container, volumes are MANDATORY. Without a volume, all data is permanently lost when the container crashes or restarts. This is the number one cause of production disasters.
Named volumes are Docker’s managed volumes — Docker decides where to store the data (/var/lib/docker/volumes/). They survive container restarts and deletions.
bash
# Create a volumedocker volume create mydata# Use with -v syntaxdocker run -v mydata:/app/data myapp# Use with --mount syntax (recommended)docker run --mount type=volume,source=mydata,target=/app/data myapp# Use with a databasedocker run -d \ --name postgres \ -v postgres_data:/var/lib/postgresql/data \ postgres:16# Delete the container — THE VOLUME STAYS SAFE!docker rm -f postgresdocker volume ls# postgres_data ← Still there!# Start a new container with the same volume — data is back!docker run -d --name postgres-new -v postgres_data:/var/lib/postgresql/data postgres:16
Bind Mounts — Development’s Superpower
With bind mounts, you specify the host path explicitly. This is extremely useful in development because your local changes are instantly reflected inside the container — without rebuilding!
bash
# Specify the host path explicitlydocker run -v /home/pushkar/myproject:/app node:18# Mount the current directory (most common in development!)docker run -v $(pwd):/app node:18# Edit code locally — changes appear inside the container immediately!# Hot reload works!# Node.js development exampledocker run -d \ --name dev-server \ -v $(pwd):/app \ -w /app \ -p 3000:3000 \ node:18 \ npm run dev# Read-only bind mount (for security — container cannot modify)docker run -v /etc/nginx/conf:/etc/nginx/conf:ro nginx# --mount syntaxdocker run --mount type=bind,source=$(pwd),target=/app node:18docker run --mount type=bind,source=/config,target=/config,readonly nginx
Tip:For React Developers: Bind Mount = Live Reload!docker run -v $(pwd):/app -p 3000:3000 node:18 npm start — edit code in VS Code and React hot reload fires automatically inside the container. No rebuild needed for each change. This is a game-changer for development workflow.
tmpfs Mounts — Sensitive Data In-Memory
bash
# Stored in memory, never written to disk!docker run --tmpfs /tmp nginx# With a size limitdocker run --mount type=tmpfs,target=/tmp,tmpfs-size=100m myapp# Multiple tmpfs locationsdocker run \ --tmpfs /tmp \ --tmpfs /run \ myapp# Use cases:# - Session data (passwords, tokens stored temporarily)# - Sensitive computation results# - Cache data that should never touch disk# - High-speed temporary storage# Container stops → data gone immediately (from RAM)# Container crashes → data gone immediately# Never persists to disk = more secure
Comparison — Which Type to Use When?
Feature
Named Volume
Bind Mount
tmpfs
Storage Location
/var/lib/docker/volumes
Any host path
Memory
Docker Managed
Yes
No
N/A
Data Persists
Yes
Yes
No (memory only)
Dev Workflow
OK
Best (hot reload!)
N/A
Production DB
Best
OK (path management)
Never
Easy Backup
Yes
Yes
No
Performance
Good
Best (NFS will be slow)
Excellent
Security
Container isolated
Host exposed
Most secure
Note:Named Volume vs Bind Mount — Interview Answer.Named Volume: Docker manages the location; it is portable, easy to back up, and best for production. Bind Mount: You control the specific host path; perfect for development (host files accessible directly in container), but path is host-specific — less portable. Prefer named volumes in production; use bind mounts in development for code changes.
# Simple named volumedocker volume create mydb-data# Explicitly use the local driver (this is already the default)docker volume create --driver local mydb-data# NFS-backed volume (remote storage!)docker volume create \ --driver local \ --opt type=nfs \ --opt o=addr=192.168.1.100,rw,vers=4 \ --opt device=:/exports/mydata \ nfs-volume# With labels (for management)docker volume create \ --label env=production \ --label app=myapp \ --label team=backend \ mydb-data# Volumes are automatically created if referenced in a run commanddocker run -v newvolume:/data myapp# "newvolume" is created automatically if it does not exist
docker volume ls, inspect, rm, prune
bash
# List all volumesdocker volume ls# DRIVER VOLUME NAME# local mydb-data# local postgres_data# local mongo_data# Find dangling volumes (not connected to any container)docker volume ls -f dangling=true# Filter by labeldocker volume ls --filter label=env=production# Inspect — full detailsdocker volume inspect mydb-data# [# {# "Name": "mydb-data",# "Driver": "local",# "Mountpoint": "/var/lib/docker/volumes/mydb-data/_data",# "Labels": {"env": "production"},# ...# }# ]# Get only the mount pathdocker volume inspect --format '{{.Mountpoint}}' mydb-data# /var/lib/docker/volumes/mydb-data/_data# Remove — stop the container first!docker volume rm mydb-data# Error: volume is in use — if a container is running# Correct approach: remove the container firstdocker rm -f mycontainerdocker volume rm mydb-data# Remove multiple at once# (Remove containers first, then volumes)docker volume prune# Skip confirmationdocker volume prune -f# Filter by label before pruningdocker volume prune --filter label=env=staging
—mount vs -v Syntax — The Modern Way
bash
# ─── OLD -v SYNTAX ──────────────────────────────────────# Older style, still works, but less readable# Named volumedocker run -v myvolume:/app/data myapp# Bind mountdocker run -v /host/path:/container/path myapp# Read-only bind mountdocker run -v /host/path:/container/path:ro myapp# ─── NEW --mount SYNTAX ──────────────────────────────────# Newer, explicit, recommended# Named volumedocker run --mount type=volume,source=myvolume,target=/app/data myapp# Bind mountdocker run --mount type=bind,source=/host/path,target=/app/path myapp# Read-only bind mountdocker run --mount type=bind,source=/host/path,target=/app/path,readonly myapp# tmpfsdocker run --mount type=tmpfs,target=/tmp,tmpfs-size=100m myapp# Why --mount is better:# - More explicit — type is clearly stated# - Better error messages# - Supports all options (tmpfs-size, etc.)# - Recommended format in Docker Docs# - More readable in scripts# - Named parameters catch typos
Note:-v vs —mount: One Important Difference. A subtle distinction: -v /nonexistent:/data will cause Docker to automatically create the host directory (sometimes with the wrong type). --mount type=bind,source=/nonexistent,target=/data returns an error if the source does not exist. —mount is stricter and more predictable — which makes it better for production.
Caution:docker volume prune — Be Very Careful in Production! This command deletes “dangling” (unused) volumes. However, if you have temporarily stopped a container (intending to restore its data), its volume may be deleted too. In production, first run docker volume ls -f dangling=true to see which volumes are actually unused, then selectively use docker volume rm.
# Step 1: Create the volumedocker volume create postgres_data# Step 2: Run PostgreSQL with a proper production setupdocker run -d \ --name postgres \ -e POSTGRES_USER=admin \ -e POSTGRES_PASSWORD=securepassword123 \ -e POSTGRES_DB=myapp \ -v postgres_data:/var/lib/postgresql/data \ -p 127.0.0.1:5432:5432 \ --restart unless-stopped \ --health-cmd="pg_isready -U admin" \ --health-interval=10s \ postgres:16-alpine# Verifydocker ps # Check status and healthdocker exec postgres psql -U admin -c "\l" # List databases
MongoDB with Multiple Volumes
bash
# MongoDB has two important paths:# /data/db — actual database files# /data/configdb — configuration datadocker run -d \ --name mongodb \ -v mongo_data:/data/db \ -v mongo_config:/data/configdb \ -e MONGO_INITDB_ROOT_USERNAME=admin \ -e MONGO_INITDB_ROOT_PASSWORD=securepass \ -p 127.0.0.1:27017:27017 \ --restart unless-stopped \ mongo:7# Verifydocker exec mongodb mongosh -u admin -p securepass --eval "db.adminCommand('listDatabases')"
Backup Strategies
bash
# ─── VOLUME BACKUP (any type of data) ─────────────────────# Use a temporary container — read the volume, create a tar archivedocker run --rm \ -v mydb_data:/data \ -v $(pwd)/backups:/backup \ alpine \ tar czf /backup/mydb-$(date +%Y%m%d-%H%M%S).tar.gz /data# Creates a file in /backup: mydb-20240115-142030.tar.gz# ─── VOLUME RESTORE ────────────────────────────────────────docker run --rm \ -v mydb_data:/data \ -v $(pwd)/backups:/backup \ alpine \ tar xzf /backup/mydb-20240115-142030.tar.gz -C /# ─── POSTGRES DUMP (recommended for PostgreSQL) ────────────# Logical backup — portable and version-awaredocker exec postgres pg_dump -U admin myapp > backup-$(date +%Y%m%d).sql# Restoredocker exec -i postgres psql -U admin myapp < backup-20240115.sql# Compressed backupdocker exec postgres pg_dump -U admin myapp | gzip > backup.sql.gz# Restore compressedgunzip -c backup.sql.gz | docker exec -i postgres psql -U admin myapp# ─── MONGODB DUMP ──────────────────────────────────────────docker exec mongodb mongodump -u admin -p securepass --out /tmp/backupdocker cp mongodb:/tmp/backup ./mongodb-backup# Restoredocker cp ./mongodb-backup mongodb:/tmp/backupdocker exec mongodb mongorestore -u admin -p securepass /tmp/backup
Shared Volumes Between Containers
bash
# Pattern: one container writes, another reads# No network communication required between them!docker volume create shared-logs# App container — writes logsdocker run -d \ --name app1 \ -v shared-logs:/var/log/app \ myapp:latest# Log viewer — read-only access to the same volumedocker run -d \ --name log-viewer \ -v shared-logs:/logs:ro \ log-viewer-app:latest# log-viewer can now read all of app1's logs# No network communication needed!# Real use case: Nginx serving React static filesdocker volume create static-files# Build container: build React, copy to volumedocker run --rm \ -v static-files:/app/build \ react-builder:latest# Nginx container: serve files from the same volumedocker run -d \ --name nginx \ -v static-files:/usr/share/nginx/html:ro \ -p 80:80 \ nginx:alpine
Cloud Volume Drivers — AWS EBS, Azure Disk
bash
# AWS EBS-backed volumes (rexray plugin)# If the host is replaced, data remains safe on AWS EBS!# Install the plugindocker plugin install rexray/ebs \ EBS_ACCESSKEY=AKIAIOSFODNN7EXAMPLE \ EBS_SECRETKEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY# Create an EBS-backed volumedocker volume create \ --driver rexray/ebs \ --opt size=20 \ --opt type=gp3 \ --opt iops=3000 \ myapp-data# Use it — exactly like a regular volumedocker run -d \ --name postgres \ -v myapp-data:/var/lib/postgresql/data \ postgres:16# If the host crashes, detach and reattach the EBS volume to a new host# Data is fully safe on AWS!# For Azure: the cloudstor plugin is available# For GCP: use the gcepd driver
Note:Production Reality: Cloud Volumes = True Persistence. Local named volumes are tied to the Docker host — if the host machine is lost, the volumes are gone (EBS is not). In real production, use AWS EFS/EBS, Azure Managed Disks, or GCP Persistent Disk via volume drivers. Or better yet, use managed databases (RDS, Atlas) — that is the preferred approach.
Tip:Volume Best Practices Summary.
Production databases: Named volumes or cloud-backed volumes (prefer RDS)
Development: Bind mounts for code, named volumes for databases
Secrets/session data: tmpfs
Backups: Automated cron + pg_dump/mongodump stored on separate storage
Never: Store critical data only in the container’s writable layer
Also make sure your .env files are correctly structured — you can validate them with the .env Example Checker before deploying. Once your stack is wired up, put these skills to work with real scenarios in Hands-on DevOps Projects.
One Compose file declares the full application stack; docker compose up creates all services on a shared network with automatic DNS so containers resolve each other by service name.
Why Docker Compose Exists
Real-world applications do not fit in a single container. A MERN stack application requires:
MongoDB — database
Express/Node — backend API
React — frontend (served via Nginx)
Redis — caching / sessions
Without Compose, everything must be managed manually:
bash
# Without Compose — run all of this by hand!docker network create myapp-netdocker run -d \ --name mongodb \ --network myapp-net \ -e MONGO_INITDB_ROOT_USERNAME=admin \ -e MONGO_INITDB_ROOT_PASSWORD=secret \ -v mongo_data:/data/db \ mongo:7.0docker run -d \ --name api \ --network myapp-net \ -e MONGODB_URI=mongodb://admin:secret@mongodb:27017/myapp \ -p 5000:5000 \ myapp-api:latestdocker run -d \ --name frontend \ --network myapp-net \ -p 3000:80 \ myapp-frontend:latest# Repeat all of this on every restart...
Tip: Just as package.json lists all npm dependencies and npm install sets everything up — docker-compose.yml defines all your containers and docker compose up starts everything in one command.
With Compose, the same setup becomes:
bash
# With Compose — just this!docker compose up -d# To shut everything down:docker compose down
Compose v1 vs Compose v2 — What Is the Difference?
Feature
Compose v1 (Old)
Compose v2 (Current)
Command
docker-compose (with dash)
docker compose (no dash!)
Installation
Separate Python binary
Built-in Docker plugin
Status
Deprecated (2023)
Active, standard
Performance
Slower
Faster (Go-based)
Features
Limited
Full feature set
2026 standard
Do not use
Use this!
Caution: Use docker compose (no dash) — this is Compose v2 and the current standard. If an older tutorial uses docker-compose (with a dash), it is using v1 syntax. V2 is mostly backward compatible, but make it a habit to use v2.
bash
# Check version — should show v2.x.xdocker compose version# Output: Docker Compose version v2.24.5# Old v1 (deprecated — do not use!)docker-compose version# If the Compose plugin is missing:# Ubuntu/Debiansudo apt update && sudo apt install docker-compose-plugin# RHEL/CentOS/Fedorasudo dnf install docker-compose-plugin# macOS (auto-bundled with Docker Desktop)# Windows (auto-bundled with Docker Desktop)
Compose vs Kubernetes — When to Use Which?
Use Case
Compose
Kubernetes
Local development
Perfect choice
Overkill
Small apps (< 5 containers)
Yes
Overkill
Single server deployment
Yes
Not needed
Auto-scaling
No
Yes
Multi-host clusters
No
Yes
Zero-downtime deploys
Limited
Yes
Production at scale
Limited
Yes
Learning curve
Easy (YAML)
Steep
Startup / small teams
Best fit
Overkill initially
Note: Start with Compose. Run a MERN stack locally using Compose; Compose is sufficient for a single VPS in production too. When traffic grows and auto-scaling becomes necessary, then invest in learning Kubernetes. The progression is: Docker → Compose → Kubernetes.
Understanding the full structure allows you to read any Compose file with ease:
yaml
# docker-compose.yml# In Compose v2, the version field is optional/deprecated# version: '3.8' ← no longer required!services: # All containers are defined here web: # Service name (also becomes the DNS hostname!) image: nginx # Which image to use # --- OR --- build: . # Build from a local Dockerfile api: # Another service build: context: ./api # Build context folder dockerfile: Dockerfile.prod # Specific Dockerfile name db: # Database service image: postgres:16networks: # Define custom Docker networks frontend: # Frontend network (public-facing) backend: # Backend network (private, for DB communication)volumes: # Named volumes (data persistence) db_data: # Database files stored here redis_data: # Redis data persistenceconfigs: # Inject config files nginx_config: file: ./nginx.confsecrets: # Sensitive data (mainly for Swarm mode) db_password: file: ./secrets/db_password.txt
Tip:Service Name = DNS Hostname! In Compose, the service name automatically becomes the DNS hostname. If a service is named db, other containers can connect to it via db:5432 — not localhost or an IP. Compose handles this automatically via Docker networking.
YAML Best Practices in Compose
yaml
# Use quotes for strings containing special charactersenvironment: - DATABASE_URL="postgresql://user:pass@db:5432/mydb"# Use literal block (|) for multiline commandscommand: | sh -c " until pg_isready; do sleep 1; done python manage.py migrate python manage.py runserver 0.0.0.0:8000 "# YAML Anchors — avoid repetition (DRY principle)x-common-env: &common-env # &common-env = define the anchor NODE_ENV: production LOG_LEVEL: info TZ: Asia/Kolkataservices: api: environment: <<: *common-env # *common-env = use the anchor (merge) PORT: 3000 SERVICE_NAME: api worker: environment: <<: *common-env # Reuse the same env vars WORKER_TYPE: email CONCURRENCY: "4"
Note:YAML Anchors — DRY Principle. YAML anchors (&name) and aliases (*name) let you write common configuration once and reuse it across multiple services. <<: *common-env means “merge all key-value pairs here.” This is the same as the spread operator (...commonProps) in JavaScript.
Override Files Pattern — Multiple Environments
One codebase, different configurations per environment — this pattern is used in production-grade applications:
bash
# File structure:docker-compose.yml # Base config (common settings)docker-compose.override.yml # Dev overrides — AUTOMATICALLY loaded!docker-compose.dev.yml # Development specificsdocker-compose.prod.yml # Production specificsdocker-compose.test.yml # Testing / CI specifics# Development (auto-loads base + override):docker compose up -d# Production (specify files explicitly):docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d# Testing:docker compose -f docker-compose.yml -f docker-compose.test.yml up -d# Validate the final merged config:docker compose -f docker-compose.yml -f docker-compose.prod.yml config
yaml
# docker-compose.yml (base)services: api: image: myapp-api:latest environment: NODE_ENV: production---# docker-compose.override.yml (dev — auto-loaded!)services: api: build: . # Build locally in dev volumes: - ./src:/app/src # Bind mount for hot reload environment: NODE_ENV: development DEBUG: "true" ports: - "5000:5000" # Expose port directly in dev---# docker-compose.prod.ymlservices: api: restart: unless-stopped deploy: resources: limits: memory: 512M logging: driver: json-file options: max-size: "10m"
Tip:Best Practice: Override Files. Keep common configuration in the base docker-compose.yml. Put environment-specific settings in override files. docker-compose.override.yml loads automatically for development — place bind mounts, debug ports, and dev-only tools there. The production file should contain resource limits, proper restart policies, and logging configuration.
All the options available in a service, annotated:
yaml
services: myapp: # ────────────────────────────────────────────── # IMAGE vs BUILD — choose one # ────────────────────────────────────────────── # Option 1: Use a pre-built image image: nginx:1.25-alpine # Option 2: Build from a local Dockerfile build: context: . # Build context (folder with Dockerfile) dockerfile: Dockerfile # Which Dockerfile? (default: Dockerfile) args: # Build-time arguments (ARG in Dockerfile) NODE_ENV: production APP_VERSION: "1.0.0" target: production # Specific stage in a multi-stage build # ────────────────────────────────────────────── # CONTAINER NAME # ────────────────────────────────────────────── container_name: my-nginx # Warning: setting container_name prevents scaling! # docker compose up --scale myapp=3 will fail if container_name is set # ────────────────────────────────────────────── # PORTS — host:container # ────────────────────────────────────────────── ports: - "8080:80" # host 8080 → container 80 - "443:443" # HTTPS - "127.0.0.1:9090:9090" # Localhost only (secure!) # ────────────────────────────────────────────── # ENVIRONMENT VARIABLES # ────────────────────────────────────────────── environment: NODE_ENV: production DB_HOST: db # Use the service name — not localhost! DB_PORT: "5432" # Use quotes for numeric values DB_NAME: mydb # Load from env_file env_file: - .env # Main .env file - .env.production # Production-specific variables # ────────────────────────────────────────────── # VOLUMES — data persistence and bind mounts # ────────────────────────────────────────────── volumes: - ./src:/app/src:ro # Bind mount, read-only (dev hot reload) - db_data:/var/lib/postgresql/data # Named volume (production data) - /etc/localtime:/etc/localtime:ro # Host timezone inside container # ────────────────────────────────────────────── # NETWORKS # ────────────────────────────────────────────── networks: - frontend # Public-facing network - backend # Private network (for DB communication) # ────────────────────────────────────────────── # DEPENDS_ON — startup order # ────────────────────────────────────────────── depends_on: db: condition: service_healthy # Wait until db is healthy redis: condition: service_started # Just started, not necessarily healthy # ────────────────────────────────────────────── # RESTART POLICY # ────────────────────────────────────────────── restart: unless-stopped # Options: # no — do not restart (default) # always — always restart # on-failure — restart only on failure # unless-stopped — always restart unless manually stopped # ────────────────────────────────────────────── # COMMAND and ENTRYPOINT # ────────────────────────────────────────────── command: npm start # String form command: ["node", "server.js"] # Array form (preferred — shell injection safe) entrypoint: ["./docker-entrypoint.sh"] # ────────────────────────────────────────────── # HEALTHCHECK # ────────────────────────────────────────────── healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3000/health"] interval: 30s # Check every 30 seconds timeout: 10s # Response must arrive within 10 seconds retries: 3 # Mark unhealthy after 3 failures start_period: 40s # Wait this long after startup before checking # ────────────────────────────────────────────── # DEPLOY — resource limits # ────────────────────────────────────────────── deploy: resources: limits: cpus: "0.50" # Max 50% of one CPU core memory: 512M # Max 512 MB RAM reservations: cpus: "0.25" # Minimum guaranteed CPU memory: 256M # Minimum guaranteed RAM # ────────────────────────────────────────────── # PROFILES — optional services # ────────────────────────────────────────────── profiles: - debug # Only starts with: docker compose --profile debug up # ────────────────────────────────────────────── # LOGGING # ────────────────────────────────────────────── logging: driver: json-file options: max-size: "10m" # Log file max 10 MB max-file: "3" # Max 3 log files (then rotate) # ────────────────────────────────────────────── # EXTRA HOSTS — /etc/hosts entries # ────────────────────────────────────────────── extra_hosts: - "host.docker.internal:host-gateway" # Connect to host machine # ────────────────────────────────────────────── # DNS — custom DNS servers # ────────────────────────────────────────────── dns: - 8.8.8.8 - 8.8.4.4
depends_on and Healthchecks — A Common Gotcha
Caution:The Default depends_on Behavior Is Misleading! A simple depends_on: [db] only guarantees that the db container has started — not that it is ready. PostgreSQL takes 5–10 seconds to initialize. If the api service tries to connect immediately, the connection will fail. Solution: use condition: service_healthy and define a healthcheck on the db service.
yaml
services: db: image: postgres:16-alpine healthcheck: test: ["CMD-SHELL", "pg_isready -U myuser -d mydb"] interval: 10s timeout: 5s retries: 5 start_period: 10s # Give PostgreSQL time to initialize api: build: . depends_on: db: condition: service_healthy # Wait until HEALTHY, not just STARTED!
Tip:Profiles — Make Dev Tools Optional. Using profiles, you can make certain services optional. For example, pgadmin (a DB GUI) is only needed in development — give it profiles: [debug]. It will not start during a normal docker compose up. It will only start with docker compose --profile debug up. This prevents debug tools from accidentally running in production.
# Basic — start in foreground (Ctrl+C to stop)docker compose up# Detached — start in background (for production)docker compose up -d# Rebuild images and start (after code changes)docker compose up --build# Force recreate containers (even if nothing changed)docker compose up --force-recreate# Start only specific servicesdocker compose up -d api db# Run multiple instances (for load balancing)docker compose up -d --scale api=3# Note: container_name must not be set when scaling!# Start with a specific profiledocker compose --profile debug up -d# Use a custom .env filedocker compose --env-file .env.prod up -d# Remove orphan containers (not defined in the compose file)docker compose up -d --remove-orphans
docker compose down — Stop and Clean Up
bash
# Standard — remove containers and networksdocker compose down# Also remove volumes (DATA LOSS — be careful!)docker compose down -v# Also remove imagesdocker compose down --rmi all # All imagesdocker compose down --rmi local # Only locally built images# Stop but keep containers (do not remove)docker compose stop# Complete teardown — clean up everythingdocker compose down -v --rmi all --remove-orphans
Caution:docker compose down -v — Data Will Be Deleted! The -v flag removes all named volumes. This means all database data is gone. This is useful for a clean restart in development, but never use it carelessly in production. Take a backup first.
docker compose logs — Debugging
bash
# Logs from all servicesdocker compose logs# Follow (live stream)docker compose logs -f# Logs from a specific servicedocker compose logs -f api# Last 100 linesdocker compose logs --tail 100# With timestampsdocker compose logs -t# Multiple services at oncedocker compose logs -f api db# Last 1 hourdocker compose logs --since 1h# From a specific timedocker compose logs --since "2024-01-15T10:00:00"
docker compose exec vs run — What Is the Difference?
Feature
exec
run
Container
Runs inside an existing running container
Starts a new container
Use case
Debug a running service
One-off commands (migrations, etc.)
Service running?
Must be running
Service does not need to be running
Cleanup
Container remains as-is
Auto-cleanup with --rm
bash
# ── docker compose exec ──────────────────────────────# Run a command in a running containerdocker compose exec api bashdocker compose exec db psql -U admin -d mydbdocker compose exec redis redis-cli# Without TTY (for automation/scripts)docker compose exec -T api npm test# As a specific userdocker compose exec --user root api bash# ── docker compose run ──────────────────────────────# One-off command in a new containerdocker compose run api npm run migratedocker compose run api python manage.py createsuperuser# Auto-cleanup after rundocker compose run --rm api npm test# Override the entrypointdocker compose run --entrypoint bash api# With custom env varsdocker compose run -e DEBUG=true api npm start
Tip:exec vs run — Simple Rule. Need to do something in a running service (debug, inspect)? → exec. Need to run a one-off task alongside the service (DB migration, seed data, test run)? → run --rm. docker compose run --rm web python manage.py migrate is the standard pattern for Django/Rails migrations.
Other Useful Commands
bash
# List servicesdocker compose psdocker compose ps -a # Include stopped servicesdocker compose ps --format json # JSON output# Build only (do not start)docker compose builddocker compose build --no-cache api # Ignore cachedocker compose build --parallel # Build all in parallel# Pull latest imagesdocker compose pull# Push images to registrydocker compose push# Validate the compose file (shows final merged config)docker compose config# Show running processesdocker compose top# Restart servicesdocker compose restartdocker compose restart api# Pause/Unpause (stop CPU usage, keep memory)docker compose pausedocker compose unpause# Create without startingdocker compose create# Stream eventsdocker compose events
Tip:Validate Your Compose File.docker compose config is a very useful command that merges all override files and shows the final YAML. Catch typos, missing variables, and syntax errors before deploying. Use it in CI/CD too: docker compose -f docker-compose.yml -f docker-compose.prod.yml config --quiet.
Most relevant for React developers — a complete MERN stack with Compose:
yaml
# docker-compose.yml — MERN Stack# Usage: docker compose up -d# Variables are loaded from a .env fileservices: # ──────────────────────────────── # MongoDB Database # ──────────────────────────────── mongodb: image: mongo:7.0-jammy container_name: mern-mongodb environment: MONGO_INITDB_ROOT_USERNAME: ${MONGO_USER:-admin} MONGO_INITDB_ROOT_PASSWORD: ${MONGO_PASSWORD:-password} MONGO_INITDB_DATABASE: ${MONGO_DB:-myapp} volumes: - mongo_data:/data/db - mongo_config:/data/configdb - ./mongo-init.js:/docker-entrypoint-initdb.d/init.js:ro networks: - backend # Backend network only — no public access! restart: unless-stopped healthcheck: test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"] interval: 30s timeout: 10s retries: 5 start_period: 30s logging: driver: json-file options: max-size: "10m" max-file: "3" # ──────────────────────────────── # Node.js API Server # ──────────────────────────────── api: build: context: ./api dockerfile: Dockerfile target: production # Use the production stage of a multi-stage build container_name: mern-api environment: NODE_ENV: production MONGODB_URI: mongodb://${MONGO_USER:-admin}:${MONGO_PASSWORD:-password}@mongodb:27017/${MONGO_DB:-myapp}?authSource=admin JWT_SECRET: ${JWT_SECRET:-changeme-in-production} PORT: 5000 volumes: - api_logs:/app/logs networks: - backend # For MongoDB communication - frontend # For receiving requests from React depends_on: mongodb: condition: service_healthy restart: unless-stopped healthcheck: test: ["CMD", "node", "-e", "require('http').get('http://localhost:5000/health', r => process.exit(r.statusCode === 200 ? 0 : 1))"] interval: 30s timeout: 5s retries: 3 start_period: 20s deploy: resources: limits: cpus: "0.50" memory: 512M logging: driver: json-file options: max-size: "10m" max-file: "3" # ──────────────────────────────── # React Frontend (served via Nginx) # ──────────────────────────────── frontend: build: context: ./frontend dockerfile: Dockerfile args: REACT_APP_API_URL: ${API_URL:-http://localhost:5000} container_name: mern-frontend ports: - "${FRONTEND_PORT:-3000}:80" networks: - frontend depends_on: api: condition: service_healthy restart: unless-stopped healthcheck: test: ["CMD", "wget", "-qO-", "http://localhost/health"] interval: 30s timeout: 5s retries: 3# ────────────────────────────────# Volumes# ────────────────────────────────volumes: mongo_data: driver: local labels: - "com.myapp.description=MongoDB data volume" mongo_config: api_logs:# ────────────────────────────────# Networks# ────────────────────────────────networks: frontend: driver: bridge # Public traffic comes here backend: driver: bridge internal: true # No external access! Container-to-container only
Note:Production Pattern: Internal Networks. Apply internal: true to the backend network. This means MongoDB is never directly accessible from the internet — only the api service can reach it. This is a security best practice. The flow is: Frontend → API (via frontend network), API → MongoDB (via backend network). MongoDB has no public port exposed.
Example 2: Python Flask + PostgreSQL + Redis + Celery
Full-stack Python application with background task processing:
Tip:Same Image, Different Command! Flask web, Celery worker, and Celery beat all use the same build: . (same Docker image) but with different command values. This is efficient: build one image, run different processes. When code changes, only one image needs to be rebuilt.
Note:What the Monitoring Stack Tracks.cAdvisor — CPU/RAM/network/disk usage for every Docker container. Node Exporter — hardware metrics for the host machine. Prometheus — collects and stores all data. Grafana — visualizes everything in dashboards. This monitoring setup is essential in production: you cannot improve what you cannot measure.
Note:Production Checklist for Compose Apps.
Add restart: unless-stopped to all services
Configure healthcheck on database services
Use internal: true on networks for sensitive services
Store passwords in a .env file — never hardcode them in the compose file
Add the .env file to .gitignore
Set resource limits (deploy.resources.limits)
Configure log rotation (max-size, max-file)
Keep images updated with regular docker compose pull
Just as npm registry stores packages, a Docker registry stores images. You build an image, push it to a registry, and anyone in the world can pull it.
Note: A Docker registry is a storage and distribution system for Docker images. Docker Hub is the default public registry, but you can use private registries for proprietary images. Think of it as GitHub for code — but for images.
Image Naming Convention
Every Docker image has a fully qualified name. The format is:
bash
[registry/][username/]image[:tag][@digest]# Examples — from simple to fully qualified:ubuntu # Docker Hub official (implicit library/)nginx:1.25-alpine # Docker Hub official with taglibrary/ubuntu:22.04 # Explicit official image pathusername/myapp:v1.0 # Docker Hub user image# Cloud registriesghcr.io/username/myapp:latest # GitHub Container Registry123456789.dkr.ecr.us-east-1.amazonaws.com/myapp:v1.0 # AWS ECR (region-scoped)asia.gcr.io/myproject/myapp:latest # Google Container Registryregistry.gitlab.com/group/project/myapp:1.0 # GitLab Registry# Digest (immutable — safer than a tag!)ubuntu@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2
Image Tagging Strategy
Tag Type
Example
Use Case
Stable?
latest
myapp:latest
Development
No — changes!
Semantic Version
myapp:v1.2.3
Production releases
Yes
Git SHA
myapp:a3f1b2c
CI/CD traceability
Yes
Branch name
myapp:main
Branch-based testing
No
Date-based
myapp:20240115
Nightly builds
Yes
Digest
myapp@sha256:abc...
Pinned dependencies
Immutable
Caution:Avoid the latest Tag! Using the latest tag in production is dangerous. If a new image is pushed, your deployment may update unexpectedly. Always use specific version tags in production — myapp:v1.2.3 or a Git SHA.
Tip:Best Practice: Multiple Tags. Tag a single image with multiple tags: myapp:v1.2.3, myapp:v1.2, myapp:v1, and myapp:latest. This gives users flexibility while maintaining immutable specific versions.
Docker Hub is the default public registry — like npm for JavaScript packages. Start with a free account and explore private registries once you hit the limits.
Note: Create an account at hub.docker.com. Free accounts include 1 private repository and unlimited public repositories. Use access tokens rather than passwords — they are safer.
Complete Docker Hub Workflow
bash
# Step 1: Login to Docker Hubdocker login# Username: yourname# Password: (enter password or access token)# Better: login with an access token (Settings → Security → New Access Token)docker login -u username --password-stdin <<< "your_access_token_here"# Step 2: Tag your local image for Docker Hubdocker tag myapp username/myapp:v1.0.0docker tag myapp username/myapp:latest# Step 3: Push to Docker Hubdocker push username/myapp:v1.0.0docker push username/myapp:latest# Step 4: Pull from anywhere in the worlddocker pull username/myapp:v1.0.0# Step 5: Logout (important on shared machines!)docker logout
Rate Limits — Important for CI/CD
Account Type
Pull Limit
Time Window
Anonymous (no login)
100 pulls
Per 6 hours / IP
Free account (authenticated)
200 pulls
Per 6 hours
Pro / Team / Business
Unlimited
—
bash
# Authenticate pulls in CI/CD to avoid rate limitsecho "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin# Check your current rate limit statusTOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)curl -s --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest | grep -i ratelimit
Rate Limit Workaround — Mirror Setup
bash
# /etc/docker/daemon.json — configure a Docker Hub mirrorsudo nano /etc/docker/daemon.json
# Restart Docker daemon after the config changesudo systemctl restart docker# Verify the mirror is activedocker info | grep -A2 "Registry Mirrors"
Caution:CI/CD Rate Limit Problem. If your CI/CD pipeline pulls images anonymously (no docker login), rate limits are hit quickly — especially in large teams. Solution: always authenticate in CI, or mirror images to your own private registry.
Tip:Use Access Tokens, Not Passwords. Docker Hub Settings → Security → New Access Token. Tokens can be given specific permissions (read-only, read-write). If a token is compromised, only that token needs to be revoked — not the full account. Always use access tokens in CI/CD.
Docker Hub is not the only option. If you deploy to AWS, ECR is the best choice. If you use GitHub Actions, ghcr.io is a perfect fit. Here is how to use each.
AWS Elastic Container Registry (ECR)
Note:When to use ECR: When your application is deployed to AWS (ECS, EKS, Lambda). ECR images integrate automatically with AWS services — no extra authentication is needed when running on AWS with proper IAM roles.
bash
# Prerequisites: AWS CLI installed and configured (aws configure)# Step 1: Authenticate with ECR (generates a 12-hour token)aws ecr get-login-password --region ap-south-1 | \ docker login --username AWS --password-stdin \ 123456789.dkr.ecr.ap-south-1.amazonaws.com# Step 2: Create a repository in ECRaws ecr create-repository \ --repository-name myapp \ --region ap-south-1 \ --image-scanning-configuration scanOnPush=true \ --encryption-configuration encryptionType=AES256# Step 3: Tag the image for ECRECR_URI="123456789.dkr.ecr.ap-south-1.amazonaws.com"docker tag myapp:latest $ECR_URI/myapp:latestdocker tag myapp:latest $ECR_URI/myapp:v1.0.0# Step 4: Pushdocker push $ECR_URI/myapp:latestdocker push $ECR_URI/myapp:v1.0.0# Step 5: Pull (from any authenticated source)docker pull $ECR_URI/myapp:latest
# Authenticate gcloud with Dockergcloud auth configure-docker asia.gcr.io# New Artifact Registry (recommended over GCR)gcloud auth configure-docker asia-south1-docker.pkg.dev# Tag and pushdocker tag myapp asia.gcr.io/my-project/myapp:v1.0docker push asia.gcr.io/my-project/myapp:v1.0# Artifact Registry (newer, more features)docker tag myapp asia-south1-docker.pkg.dev/my-project/my-repo/myapp:v1.0docker push asia-south1-docker.pkg.dev/my-project/my-repo/myapp:v1.0
Registry Comparison
Registry
Type
Cost
Best For
Docker Hub
Cloud
Free (limited) / Paid
Open source, small teams
AWS ECR
Cloud
Per storage + transfer
AWS deployments
Google GCR/AR
Cloud
Per storage + transfer
GCP deployments
GitHub Packages
Cloud
Free for public
Open source + CI/CD
Harbor
Self-hosted
Free (infra cost)
Enterprise, full control
Nexus Repository
Self-hosted
Free/Commercial
Enterprise artifact management
Tip: Deploying to AWS → use ECR (pick the Region closest to your users). Using GitHub Actions → use ghcr.io (free for public repos). Startup or open source → start with the Docker Hub free tier.
Integrating the registry workflow with CI/CD is the core of production-ready DevOps. Here is a complete, real-world workflow for a React application.
Note:Workflow Overview: Build → Tag (with version + Git SHA) → Authenticate → Push to Registry → Verify. Every step matters. Tagging with the Git SHA lets you know exactly which commit is running in production.
Complete Registry Push Workflow
bash
#!/bin/bash# Full CI/CD workflow for a React appset -e # Exit on any errorIMAGE_NAME="myreactapp"REGISTRY="123456789.dkr.ecr.ap-south-1.amazonaws.com"# Generate meaningful version tagsVERSION=$(git describe --tags --always --dirty 2>/dev/null || echo "dev")GIT_SHA=$(git rev-parse --short HEAD)DATE=$(date +%Y%m%d)BRANCH=$(git rev-parse --abbrev-ref HEAD | tr / -)echo "Building version: $VERSION (SHA: $GIT_SHA)"# Step 1: Build with build args for traceabilitydocker build \ --build-arg BUILD_DATE=$DATE \ --build-arg GIT_COMMIT=$GIT_SHA \ --build-arg VERSION=$VERSION \ -t $IMAGE_NAME:$VERSION \ -t $IMAGE_NAME:$GIT_SHA \ -t $IMAGE_NAME:latest \ .echo "Build complete"# Step 2: Authenticate with ECRaws ecr get-login-password --region ap-south-1 | \ docker login --username AWS --password-stdin $REGISTRYecho "Authenticated with ECR"# Step 3: Tag all versions for registrydocker tag $IMAGE_NAME:$VERSION $REGISTRY/$IMAGE_NAME:$VERSIONdocker tag $IMAGE_NAME:$GIT_SHA $REGISTRY/$IMAGE_NAME:$GIT_SHAdocker tag $IMAGE_NAME:latest $REGISTRY/$IMAGE_NAME:latest# Step 4: Push all tagsdocker push $REGISTRY/$IMAGE_NAME:$VERSIONdocker push $REGISTRY/$IMAGE_NAME:$GIT_SHAdocker push $REGISTRY/$IMAGE_NAME:latestecho "All tags pushed"# Step 5: Verify push was successfulecho "Recent images in ECR:"aws ecr describe-images \ --repository-name $IMAGE_NAME \ --region ap-south-1 \ --query 'sort_by(imageDetails, &imagePushedAt)[-5:].{Tags:imageTags,Size:imageSizeInBytes,Pushed:imagePushedAt}' \ --output table
GitHub Actions — Full CI/CD Pipeline
yaml
name: Build, Scan, and Pushon: push: branches: [main] pull_request: branches: [main]env: REGISTRY: 123456789.dkr.ecr.ap-south-1.amazonaws.com IMAGE_NAME: myreactappjobs: build-and-push: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 with: fetch-depth: 0 # Full history for git describe - name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v4 with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: ap-south-1 - name: Login to Amazon ECR uses: aws-actions/amazon-ecr-login@v2 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 - name: Generate image metadata id: meta uses: docker/metadata-action@v5 with: images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} tags: | type=sha,prefix=sha- type=ref,event=branch type=semver,pattern={{version}} type=raw,value=latest,enable={{is_default_branch}} - name: Build and push uses: docker/build-push-action@v5 with: context: . push: ${{ github.event_name == 'push' }} # Only push on merge to main tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} cache-from: type=gha # GitHub Actions cache cache-to: type=gha,mode=max
Caution:Do Not Push on Pull Requests! Pull request builds should only build and test — do not push images. Push only when merging to main/master. That is why push: ${{ github.event_name == 'push' }} is used.
Tip:Speed Up Builds with Cache.cache-from: type=gha and cache-to: type=gha,mode=max use GitHub Actions cache. The first build will be slow; subsequent builds are significantly faster — especially when dependencies have not changed.
Docker containers share the host kernel. This means — if a container escapes, an attacker can reach the host machine. Do not treat security as an afterthought; you must think about it while writing your Dockerfile.
Caution: Container escape is real. In 2019, a runc vulnerability (CVE-2019-5736) allowed attackers to access the host machine from inside a container. In 2020, a Docker Desktop vulnerability followed. This is not theoretical — production systems have been compromised. Always follow security fundamentals.
Common Container Vulnerabilities
#
Vulnerability
Risk Level
Impact
1
Running as root
Critical
Container escape, host access
2
Outdated base images (CVEs)
Critical
Known exploits in production
3
Hardcoded secrets in image
Critical
Credential theft via docker history
4
Privileged mode
Critical
Full host access
5
Excessive capabilities
High
Privilege escalation
6
No resource limits
High
Denial of Service (fork bomb)
7
Unscanned images
High
Unknown vulnerabilities
8
Docker socket exposed
Critical
Complete host compromise
Real Attack Scenarios
bash
# Scenario 1: Secret leakage via docker historydocker history myapp:latest --no-trunc# IMAGE CREATED CREATED BY# abc123 ... /bin/sh -c export API_KEY=sk-prod-secret123 && node setup.js# ^^^^ VISIBLE TO ANYONE who pulls the image!# Scenario 2: Crypto mining via compromised public image# Attacker pushes malicious "ubuntu:latest" to a similarly-named registry# Your CI/CD pulls it, runs a crypto miner in your cloud (you pay the bill!)# Scenario 3: Docker socket exposuredocker run -v /var/run/docker.sock:/var/run/docker.sock myapp# Attacker inside container can now:docker run -v /:/host alpine chroot /host # Read/write host filesystem!# Scenario 4: Privileged container escapedocker run --privileged alpine# Inside container:mount /dev/sda1 /mnt # Mount host diskchroot /mnt # Root access to host!
Caution: Never mount /var/run/docker.sock into a container in production unless absolutely necessary (such as for CI agents). Any container that can access the Docker socket effectively has root access to the entire machine.
Note: Container security is layered: secure Dockerfile → runtime security flags → network policies → image scanning → secrets management → monitoring. If one layer fails, the next catches it. This is the “defense in depth” model.
Security starts at build time. An insecure Dockerfile cannot be fixed at runtime. Follow these best practices for production-ready Dockerfiles.
Complete Secure Dockerfile Template
dockerfile
# ============================================# SECURE DOCKERFILE TEMPLATE — React/Node App# ============================================# Best Practice #1: Use specific versions — never "latest" or just "node"# The "latest" tag can change, breaking your buildFROM node:18.19.0-alpine3.19# Best Practice #2: Use minimal base images# Security risk order (most to least): ubuntu > debian > debian-slim > alpine > distroless# Alpine has 98% fewer packages = smaller attack surface# Best Practice #3: Update packages in base image (apply security patches)RUN apk update && \ apk upgrade && \ rm -rf /var/cache/apk/*# Note: Alpine uses apk; Debian/Ubuntu uses apt-get# Best Practice #4: Create a non-root user EARLY# Never run the app as root!RUN addgroup --system --gid 1001 appgroup && \ adduser --system --uid 1001 --ingroup appgroup appuserWORKDIR /app# Best Practice #5: Copy dependency files FIRST (caching + security)# Set proper ownership during COPY with --chownCOPY --chown=appuser:appgroup package*.json ./# Best Practice #6: Install ONLY production dependenciesRUN npm ci --only=production && \ npm cache clean --force# npm ci = exact versions from package-lock.json (more reproducible)# Best Practice #7: Copy app source with proper ownershipCOPY --chown=appuser:appgroup . .# Best Practice #8: Build (if needed) while still root for permissions# Then switch to non-root for runtimeRUN npm run build# Best Practice #9: Remove dev files and build artifactsRUN rm -rf src/ tests/ *.test.js .env.* coverage/# Best Practice #10: Switch to non-root user — everything below runs as appuserUSER appuser# Best Practice #11: Expose ONLY required ports (documentation, not enforcement)EXPOSE 3000# Best Practice #12: Add HEALTHCHECK — K8s/Docker knows if app is actually workingHEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \ CMD node -e "require('http').get('http://localhost:3000/health', r => process.exit(r.statusCode===200?0:1))" || exit 1# Best Practice #13: Use exec form CMD (not shell form)# Shell form: CMD "node server.js" → /bin/sh -c "node server.js" (signals don't propagate!)# Exec form: CMD ["node", "server.js"] → direct process (signals work correctly)CMD ["node", "server.js"]
Security Anti-Patterns — Never Do This
dockerfile
# ❌ DANGEROUS: Hardcoded secrets visible in docker history!ENV DATABASE_PASSWORD=mysupersecretpasswordENV AWS_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY# ❌ DANGEROUS: Even in RUN, it stays in layer historyRUN export API_KEY=abc123secret && npm run setup# ❌ BAD: Copying entire directory without .dockerignoreCOPY . .# This might include: .env, .git/, node_modules/, passwords.txt, etc.# ❌ BAD: Using latest tagFROM node:latest # Which version? Will break unexpectedly!# ❌ BAD: Running as root (default if no USER instruction)# root inside container → if compromised, easier to escalate# ❌ BAD: Adding unnecessary packagesRUN apt-get install -y curl wget vim git build-essential# More packages = larger attack surface# ❌ BAD: Ignoring package-lock.jsonRUN npm install # Might install different versions each time!
.dockerignore — First Line of Defense
text
# Always maintain a comprehensive .dockerignore!# Secrets and config.env.env.*.env.local.env.productionsecrets/*.pem*.key*.cert# Development artifactsnode_modules/npm-debug.log*yarn-error.log# Git history (can contain old secrets!).git/.gitignore# Tests and coverage (not needed in production image)tests/__tests__/*.test.js*.spec.jscoverage/.nyc_output/# DocumentationREADME.mddocs/*.md# IDE files.vscode/.idea/*.swp# CI/CD configs (not needed in image).github/.gitlab-ci.ymlJenkinsfile# Build artifacts (will be regenerated)dist/build/
Caution:.env MUST be in .dockerignore. If you use COPY . . and .env is not excluded, all your secrets end up in the image. Anyone can see them via docker history myapp or image inspection. Real data breaches have happened this way.
Tip: Multi-stage builds give you both security and smaller image sizes. Build tools (gcc, make, npm, pip) do not belong in a production image. Copy only the runtime artifacts into the final stage — no build tools means a smaller attack surface and 60–80% smaller images.
Note: Google’s distroless images contain no shell, no package manager, and no utilities — only the app runtime. An attacker cannot even open a shell. Use gcr.io/distroless/nodejs18-debian11. The trade-off is harder debugging — use the debug variant for that purpose.
After building a secure Dockerfile, apply security flags at runtime. These flags keep the container in a tight sandbox — even if the container is compromised, the damage is contained.
Runtime Security Flags
bash
# 1. Drop ALL capabilities, add back only what's needed# Linux capabilities are specific privileges that root hasdocker run --cap-drop ALL --cap-add NET_BIND_SERVICE nginx# Drops 38 capabilities and adds back only port binding < 1024# 2. Read-only filesystem — prevent tamperingdocker run --read-only --tmpfs /tmp --tmpfs /var/run myapp# --read-only: container filesystem is read-only# --tmpfs /tmp: in-memory temp storage (data lost on restart)# 3. No new privileges — prevent sudo/setuid escalationdocker run --security-opt no-new-privileges myapp# Even if a setuid binary exists, it cannot escalate privileges# 4. Custom seccomp profile — limit system callsdocker run --security-opt seccomp=/path/to/seccomp-profile.json myapp# Default Docker seccomp blocks 44 of 300+ syscalls# A custom profile can be more restrictive# 5. AppArmor profile — Mandatory Access Controldocker run --security-opt apparmor=docker-default myapp# 6. Resource limits — prevent Denial of Servicedocker run \ --memory=512m \ # Max RAM --memory-swap=512m \ # No swap (set equal to memory) --cpus=0.5 \ # Max 0.5 CPU cores --pids-limit=100 \ # Max 100 processes (prevents fork bombs!) --ulimit nofile=1024:1024 \ # Max open file descriptors myapp# 7. NEVER use --privileged in production!# docker run --privileged myapp ← Gives container host-level access!# Complete secure run commanddocker run \ --cap-drop ALL \ --cap-add NET_BIND_SERVICE \ --read-only \ --tmpfs /tmp \ --security-opt no-new-privileges \ --memory=512m \ --cpus=0.5 \ --pids-limit=100 \ --user 1001:1001 \ myapp:v1.0.0
Linux Capabilities — What to Grant and What to Deny
Capability
Purpose
Use in Production?
NET_BIND_SERVICE
Bind ports < 1024 (80, 443)
Only if port < 1024
NET_ADMIN
Configure networking
Only for network tools
CHOWN
Change file ownership
Very rarely
DAC_OVERRIDE
Bypass file permissions
Never
FOWNER
Bypass owner restrictions
Never
SYS_ADMIN
Broad system admin access
NEVER (equivalent to root!)
SYS_PTRACE
Process tracing (debuggers)
Never in production
KILL
Send signals to processes
Rarely needed
User Namespace Remapping (Advanced)
json
// /etc/docker/daemon.json — User namespace remapping// Container root (UID 0) becomes an unprivileged user on the host!{ "userns-remap": "default"}// After this: sudo systemctl restart docker// Container root → host UID 100000 (non-privileged)
Caution: Never use --privileged in production. The --privileged flag effectively makes the container root on the host, bypassing the security context entirely. If any tutorial asks you to use --privileged without explaining why, that tutorial is wrong. Use it only for specific, well-understood cases such as certain CI/CD systems.
Tip: Docker’s default seccomp profile already blocks 44+ dangerous syscalls. To tighten further, create a custom profile. Docker’s documentation provides the default profile as a starting point.
Your image is built and pushed — but does it contain known vulnerabilities? Image scanning tools automatically check for CVEs (Common Vulnerabilities and Exposures) in your base image and dependencies.
Note: A CVE (Common Vulnerability and Exposure) is the unique identifier for a publicly known security vulnerability. For example, CVE-2021-44228 was the Log4Shell vulnerability. A scanner compares your image’s packages against a CVE database.
Trivy — Most Popular Free Scanner
bash
# Install Trivy (Linux/macOS)curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | \ sh -s -- -b /usr/local/bin# macOS via Homebrewbrew install trivy# Basic image scantrivy image nginx:latest# Scan with severity filter — only show HIGH and CRITICALtrivy image --severity HIGH,CRITICAL nginx:latest# Fail CI/CD if CRITICAL vulnerabilities found (exit code 1)trivy image --exit-code 1 --severity CRITICAL myapp:latest# Scan your local Dockerfile for misconfigurationstrivy config ./Dockerfile# Scan docker-compose.yml for security issuestrivy config ./docker-compose.yml# Output as JSON for automated processingtrivy image --format json --output results.json myapp:latest# Scan a GitHub repository (without cloning)trivy repo https://github.com/user/myrepo# Generate SBOM (Software Bill of Materials)trivy image --format cyclonedx --output sbom.json myapp:latest
# Docker Scout (built-in since Docker Desktop 4.17)# Quick vulnerability overviewdocker scout quickview myapp:latest# Detailed CVE listdocker scout cves myapp:latest# Filter by severitydocker scout cves --only-severity critical myapp:latest# Compare two versions (what changed?)docker scout compare --to myapp:v1.0 myapp:v2.0# Get recommendations (what base image to use?)docker scout recommendations myapp:latest# Local filesystem scandocker scout cves fs://./ # Scan current directory
Snyk — Enterprise Scanner
bash
# Install Snyk CLInpm install -g snyk# Authenticate (free account at snyk.io)snyk auth# Scan Docker imagesnyk container test nginx:latest# Scan with fail thresholdsnyk container test --severity-threshold=high nginx:latest# Monitor over time (track vulnerabilities as they are discovered)snyk container monitor nginx:latest# Snyk in CI/CD (GitHub Actions)- name: Run Snyk to check Docker image for vulnerabilities uses: snyk/actions/docker@master env: SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }} with: image: myapp:latest args: --severity-threshold=high
Tip: For development: use Docker Scout (built-in, no setup). For CI/CD: use Trivy (free, fast, no registration). For enterprise: use Snyk or Aqua Security (paid, more features). ECR users: enable “Scan on Push” in repository settings — it is automatic and free.
Note: Achieving zero vulnerabilities is an unrealistic goal. Having LOW/MEDIUM vulnerabilities in production images is normal — fixing everything is not feasible. Focus on: always fix CRITICAL, fix or document exceptions for HIGH, address MEDIUM only if exploitable. False positives also occur — triage carefully.
Secrets Management — Never Bake Secrets Into Images
Section 41 of 67 · ~3 min
The most common mistake in container security is hardcoding secrets (passwords, API keys, tokens) into Docker images. Once an image is pushed to a registry, anyone who pulls it can inspect the secrets.
Caution:docker history is a secret leakage tool. Every Dockerfile instruction creates an image layer. ENV API_KEY=secret is permanently baked into that layer — even if you unset it later. Anyone can see it via docker history --no-trunc myapp. Real incident: a company put API keys in their Dockerfile, pushed the image as public to Docker Hub, and received a $50,000 AWS bill from crypto mining.
Never Do This
dockerfile
# ❌ EXTREMELY DANGEROUS — visible in docker history forever!ENV DATABASE_PASSWORD=mysupersecretpassword123ENV AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYENV JWT_SECRET=my-jwt-secret-key# ❌ ALSO DANGEROUS — even in RUN, it stays in the layer!RUN export API_KEY=sk-prod-abc123 && ./configure.sh# ❌ BAD — copying .env file into imageCOPY .env /app/.env # .env is now inside the image layers!# Check what's visible (run this on your image)docker history --no-trunc myapp | grep -i "password\|secret\|key\|token"
Approach 1: Runtime Environment Variables
bash
# Pass at runtime — not baked into the imagedocker run -e DATABASE_PASSWORD=secret myapp# From a file — more manageable (file security is your responsibility)docker run --env-file secrets.env myapp# secrets.env format (add this file to .gitignore!)# DATABASE_URL=postgresql://user:pass@host:5432/db# API_KEY=sk-prod-abc123# JWT_SECRET=super-secret-jwt-key# CAVEAT: Still visible in 'docker inspect'!docker inspect myapp | jq '.[0].Config.Env'# [# "DATABASE_PASSWORD=secret", ← Visible to anyone with docker inspect!# "PATH=/usr/local/sbin:..."# ]
Approach 2: Docker Secrets (Swarm Mode)
bash
# Create a Docker secret (stored encrypted in Swarm Raft)echo "mysupersecretpassword" | docker secret create db_password -printf "sk-prod-abc123secret" | docker secret create api_key -# List secrets (values NOT shown)docker secret ls# Use in service (mounted as file at /run/secrets/secretname)docker service create \ --secret db_password \ --secret api_key \ --env DB_PASSWORD_FILE=/run/secrets/db_password \ --env API_KEY_FILE=/run/secrets/api_key \ myapp# Inside container, secrets are files:cat /run/secrets/db_password # mysupersecretpasswordcat /run/secrets/api_key # sk-prod-abc123secret
yaml
# docker-compose.yml with secrets (Swarm mode)version: "3.8"secrets: db_password: external: true # Secret already created via docker secret create api_key: external: trueservices: api: image: myapp:latest secrets: - db_password - api_key environment: # Point to secret files, not actual values - DB_PASSWORD_FILE=/run/secrets/db_password - API_KEY_FILE=/run/secrets/api_key
Approach 3: AWS Secrets Manager (Production Best Practice)
bash
#!/bin/sh# entrypoint.sh — Fetch secrets at startup from AWS Secrets Manager# Container uses an IAM role (no hardcoded AWS credentials needed!)set -eecho "Fetching secrets from AWS Secrets Manager..."# Fetch database passwordexport DB_PASSWORD=$(aws secretsmanager get-secret-value \ --secret-id prod/myapp/db_password \ --query SecretString \ --output text \ --region ap-south-1)# Fetch API keyexport API_KEY=$(aws secretsmanager get-secret-value \ --secret-id prod/myapp/api_key \ --query SecretString \ --output text \ --region ap-south-1)echo "Secrets loaded successfully"# Execute the main command (e.g., node server.js)exec "$@"
# syntax=docker/dockerfile:1FROM node:18-alpineWORKDIR /app# Secret mounted ONLY during this RUN — not stored in the image layer!RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \ npm install # Can access private npm registry using .npmrc secret# Private repo clone during buildRUN --mount=type=ssh \ git clone git@github.com:myorg/private-repo.git /app/lib
bash
# Build with secret (secret never enters the image!)docker build \ --secret id=npmrc,src=$HOME/.npmrc \ -t myapp .# SSH forwarding for private reposeval $(ssh-agent)ssh-add ~/.ssh/id_rsadocker build --ssh default -t myapp .
Tip: Always add .env to both .gitignore and .dockerignore. Commit a .env.example template without values so team members know which variables are required. In production, never use .env files — use a proper secrets management solution.
Note: Secrets manager options include: AWS Secrets Manager (paid, excellent rotation support), HashiCorp Vault (free, powerful, complex), Kubernetes Secrets (base64-encoded — not truly secret!), Doppler (developer-friendly SaaS), Azure Key Vault, and GCP Secret Manager. Choose based on your cloud provider.
BuildKit is the default build engine in Docker 23+. It is much faster, safer, and more feature-rich than the older builder. Parallel builds, better caching, secret mounts — all of these come from BuildKit.
Note: Old builder: sequential layers, no cache import/export, no secret mounts. BuildKit: parallel stage builds, registry cache, secret mounts, SSH forwarding, multi-arch support. Master BuildKit once and CI/CD build times improve dramatically.
Enabling BuildKit
bash
# Check if BuildKit is availabledocker buildx version# github.com/docker/buildx v0.12.0 docker-desktop# Docker 23+ enables it automatically# For older versions:# Option 1: Per-build environment variableDOCKER_BUILDKIT=1 docker build -t myapp .# Option 2: Enable permanently in daemon.jsonsudo nano /etc/docker/daemon.json# Add: { "features": { "buildkit": true } }sudo systemctl restart docker# Option 3: Set DOCKER_BUILDKIT in shell profileecho 'export DOCKER_BUILDKIT=1' >> ~/.bashrc
BuildKit Special Features — syntax=docker/dockerfile:1
dockerfile
# This comment enables the latest BuildKit Dockerfile syntax# syntax=docker/dockerfile:1FROM node:18-alpineWORKDIR /app# Feature 1: Secret mounts — never stored in image layers!RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \ npm install --legacy-peer-deps# Feature 2: Cache mounts — persist between builds!RUN --mount=type=cache,target=/root/.npm \ npm ci # npm cache persists across builds = MUCH faster!# Feature 3: Bind mounts — mount host directory without COPYRUN --mount=type=bind,source=package.json,target=package.json \ --mount=type=bind,source=package-lock.json,target=package-lock.json \ npm ci# Feature 4: SSH forwardingRUN --mount=type=ssh \ git clone git@github.com:private/repo.git
docker buildx Commands
bash
# List all buildersdocker buildx ls# Create a new builder with docker-container driver (more features)docker buildx create \ --name mybuilder \ --driver docker-container \ --bootstrapdocker buildx use mybuilder# Inspect builder (shows supported platforms)docker buildx inspect mybuilder# Build for current platform (like regular docker build)docker buildx build -t myapp .# Build for a specific platformdocker buildx build --platform linux/amd64 -t myapp:amd64 --load .docker buildx build --platform linux/arm64 -t myapp:arm64 --load .# Build and push multi-arch image in one command!docker buildx build \ --platform linux/amd64,linux/arm64 \ -t username/myapp:latest \ --push \ .# Remove builderdocker buildx rm mybuilder
Registry Cache — CI/CD Game Changer
bash
# Export cache to registry (persists between CI runs)docker buildx build \ --cache-to type=registry,ref=myapp:buildcache,mode=max \ --cache-from type=registry,ref=myapp:buildcache \ -t myapp:latest \ --push \ .# mode=max: Cache all intermediate layers (maximum cache hits)# mode=min: Cache only final layer (smaller cache size)
yaml
# GitHub Actions — Optimal BuildKit caching- name: Set up Docker Buildx uses: docker/setup-buildx-action@v3- name: Build and push with cache uses: docker/build-push-action@v5 with: context: . push: true tags: myapp:latest cache-from: type=gha # GitHub Actions cache (free!) cache-to: type=gha,mode=max # Save cache for next run # Result: 5-minute builds → 45-second builds on cache hit!
Tip: Without caching: a Node app build takes 4–5 minutes. With caching (dependencies unchanged): 30–60 seconds. For large enterprise apps: 20 minutes → 2 minutes. This difference directly affects production deployment frequency.
Note: With multi-arch builds, --load (local image) and --push (registry) cannot be used simultaneously in most cases. For single-arch builds use --load; for multi-arch, push directly to the registry with --push.
Multi-Architecture Images — Supporting ARM and x86
Section 43 of 67 · ~2 min
You develop on a MacBook M1/M2/M3 (ARM64) and deploy to AWS EC2 (amd64). Multi-arch images work on both platforms without issues — no need for separate Dockerfiles.
Note: Apple Silicon Macs (ARM64) have become standard dev machines. AWS Graviton instances (ARM64) offer the same performance at 40% lower cost than x86. If your image only supports amd64, ARM machines will produce a “wrong platform” error. Build multi-arch images to support everyone.
Setup and Build
bash
# Step 1: Install QEMU (enables emulation of other architectures)docker run --privileged --rm tonistiigi/binfmt --install all# Installs QEMU user-space emulators for ARM, etc.# Verify QEMU installationls /proc/sys/fs/binfmt_misc/# Should show: qemu-aarch64, qemu-arm, etc.# Step 2: Create a multi-arch capable builderdocker buildx create \ --name multiarch-builder \ --driver docker-container \ --bootstrapdocker buildx use multiarch-builder# Step 3: Inspect — verify which platforms are supporteddocker buildx inspect multiarch-builder --bootstrap# Platforms: linux/amd64, linux/arm64, linux/arm/v7, linux/386...# Step 4: Build and push multi-arch image in one commanddocker buildx build \ --platform linux/amd64,linux/arm64 \ --tag username/myapp:latest \ --tag username/myapp:v1.0.0 \ --push \ .# Step 5: Verify multi-arch manifestdocker manifest inspect username/myapp:latest# Shows separate manifests for each architecture
Platform-Specific Handling in Dockerfile
dockerfile
# syntax=docker/dockerfile:1# BuildKit injects these automaticallyARG TARGETPLATFORMARG TARGETARCHARG TARGETOSFROM node:18-alpine# Platform-specific commands (if needed)RUN case "$TARGETARCH" in \ amd64) echo "Building for x86_64" ;; \ arm64) echo "Building for ARM64" ;; \ arm) echo "Building for ARM v7" ;; \ esac# Most Node.js apps need no platform-specific codeWORKDIR /appCOPY package*.json ./RUN npm ci --only=productionCOPY . .USER nodeEXPOSE 3000CMD ["node", "server.js"]
Note: QEMU emulation makes ARM64 builds 5–10x slower on an amd64 machine. Solutions: use native ARM builders (GitHub has paid ARM64 runners), or run matrix builds (amd64 and arm64 in parallel). QEMU is fine for small images.
Tip: AWS Graviton3 (arm64) instances are 20–40% cheaper than equivalent x86 instances for the same workload. Build one multi-arch image, deploy on Graviton — significant cost savings at scale. t4g instances are cheaper than t3, c7g instances are cheaper than c6i.
Docker contexts let you manage multiple Docker hosts from one machine — local, staging server, production server — all with the same docker command. Switch context and all commands go there.
Note: Contexts are useful when you want to run Docker commands on remote servers without SSH-ing in manually, or when managing multiple environments (dev, staging, production) from a CI/CD workflow.
Context Management Commands
bash
# List all contexts (* = currently active)docker context ls# NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT# default * ... unix:///var/run/docker.sock# remote-server ... ssh://deploy@203.0.113.10# Create context for a remote server (uses SSH)docker context create remote-server \ --description "Production server" \ --docker "host=ssh://deploy@203.0.113.10"# Create context with a specific SSH keydocker context create staging \ --docker "host=ssh://ubuntu@staging.myapp.com"# Switch to remote contextdocker context use remote-server# All docker commands now go to the remote host!docker ps # Shows containers on remote server!docker images # Shows images on remote server!docker run -d nginx # Runs on remote server!# Switch back to localdocker context use default# One-off command in a specific context (without switching)docker --context remote-server psdocker --context remote-server logs myapp -f# Inspect a contextdocker context inspect remote-server
Export/Import Contexts
bash
# Export context (share with team)docker context export remote-server > production.dockercontext# Import on another machinedocker context import production-server production.dockercontext# Remove a contextdocker context rm old-server
Context with AWS ECS
bash
# Create ECS context (deploy directly to ECS!)docker context create ecs myecscontext# Prompts for AWS credentials/profile# Switch to ECS contextdocker context use myecscontext# Now docker compose commands deploy to ECS!docker compose up # Deploys to ECS Fargate!
Tip: Define hosts in ~/.ssh/config, then reference them in a context: docker context create prod --docker "host=ssh://prod-server". SSH config can include jump hosts and identity files — complex infrastructure can be managed cleanly this way.
Caution: A common mistake is switching to a remote context, doing some work, and forgetting about it. The next local command runs on the remote host instead. Always check the current context with docker context ls. Consider showing the current context in your shell prompt using Starship or oh-my-zsh plugins.
Docker plugins extend Docker daemon capabilities with custom volume drivers, network drivers, and authorization plugins. They are useful in enterprise environments and for integrating cloud storage.
Note: Plugin types — Volume plugins: connect cloud storage (AWS EBS, NFS, GlusterFS) to Docker volumes. Network plugins: custom networking (Weave, Calico, Flannel). Authorization plugins: centralized access control defining what each container is allowed to do.
# SSHFS Plugin — Mount a remote directory as a Docker volume# Useful for: shared config files, remote storage# First install the plugindocker plugin install vieux/sshfs# Create volume using plugindocker volume create \ --driver vieux/sshfs \ --opt sshcmd=user@remote-host:/path/to/dir \ --opt password=mysshpassword \ remote-data# Use volume in containerdocker run -v remote-data:/app/data myapp# AWS EBS volume plugin (for persistent EBS volumes)docker plugin install rexray/ebs \ EBS_REGION=ap-south-1docker volume create \ --driver rexray/ebs \ --opt size=20 \ my-ebs-volume
Authorization Plugins — Enterprise Security
bash
# OPA (Open Policy Agent) authorization plugin# Define policies: what each container is allowed to do# Installdocker plugin install openpolicyagent/opa-docker-authz-local:v0.2 \ policy-file=/etc/docker/policies/authz.rego# authz.rego policy example (Rego language)# package docker.authz# allow {# not deny# }# deny {# input.Body.HostConfig.Privileged == true # Block privileged containers# }# Enable in daemon.json:# {# "authorization-plugins": ["openpolicyagent/opa-docker-authz-local:v0.2"]# }
Caution: Docker plugins run with daemon-level access. Only use trusted, well-maintained plugins. Review permissions carefully when installing — a malicious plugin can compromise the entire Docker daemon.
Tip: Many plugin use cases are now better handled by Kubernetes and CSI (Container Storage Interface) drivers. If you are using Kubernetes, explore CSI drivers instead of Docker plugins — the ecosystem is better and offers more options.
Without resource limits, a misbehaving container can consume all the memory and CPU on the host, crashing everything else. Apply limits to every container in production.
Caution::(){ :|:& };: is a bash fork bomb. Without --pids-limit, this can crash an entire machine from a single container. In the real world, buggy code leaks memory or spawns infinite processes. Always set limits.
Memory Limits
bash
# Hard memory limit — OOM killer triggers if exceededdocker run --memory=512m myapp# Memory + swap limitdocker run --memory=512m --memory-swap=1g myapp# RAM: 512MB, Swap: 512MB (total 1GB, swap = total - ram)# Disable swap for containerdocker run --memory=512m --memory-swap=512m myapp# Soft limit — suggestion, not enforced (for low-memory situations)docker run --memory-reservation=256m myapp# OOM kill disable (risky — lets container grow without being killed)docker run --oom-kill-disable myapp # Use with extreme caution!
CPU Limits
bash
# Hard CPU limit — cannot exceed X coresdocker run --cpus=1.5 myapp # Max 1.5 CPU coresdocker run --cpus=0.25 myapp # Max 25% of one CPU core# CPU shares — relative weight (soft limit, when contention exists)docker run --cpu-shares=512 myapp # Default is 1024; gets half the CPU when contesteddocker run --cpu-shares=2048 myapp # Gets double the default CPU priority# Pin to specific CPU cores (NUMA optimization, isolation)docker run --cpuset-cpus=0,1 myapp # Only use CPUs 0 and 1docker run --cpuset-cpus=2-5 myapp # Use CPUs 2, 3, 4, 5
I/O and Other Limits
bash
# Disk I/O limitsdocker run --device-read-bps /dev/sda:100mb myapp # Max 100MB/s readdocker run --device-write-bps /dev/sda:50mb myapp # Max 50MB/s writedocker run --device-read-iops /dev/sda:1000 myapp # Max 1000 IOPS read# PID limit (prevent fork bombs!)docker run --pids-limit=100 myapp # Max 100 processes/threads# Open file descriptor limitdocker run --ulimit nofile=1024:1024 myapp# Format: soft_limit:hard_limit# Update limits on a RUNNING container (no restart needed!)docker update --memory=1g --cpus=2 myapp# Check resource usage (real-time)docker stats # All containersdocker stats myapp # Specific containerdocker stats --no-stream myapp # Snapshot, not live
Tip: Strategy for setting limits: first run without limits and observe actual usage with docker stats. Then set limit = actual_peak × 1.5 (with buffer). Test on staging to verify the app works correctly within limits. Deploy to production and track OOM kills and CPU throttling via monitoring.
“It works on my machine” — everyone has heard this. Containers have their own issues. Here is a systematic debugging methodology for quickly resolving production problems.
Note: Debugging order: Status → Logs → Resources → Exec into container → Processes → Network → Events. Follow this order — 80% of issues are resolved via logs or exec. Advanced tools are only needed when the basics fail.
Systematic Debugging Playbook
bash
# ===== STEP 1: Container Status =====docker ps -a # -a shows stopped containers too# STATUS: Up 2 hours (good) | Exited (1) 5 min ago (bad) | Restarting (crash loop!)# Detailed statusdocker inspect myapp | jq '.[0].State'# {# "Status": "exited",# "Running": false,# "ExitCode": 137, ← 137 = OOM kill! 1 = app error, 0 = clean exit# "OOMKilled": true,# "Error": ""# }# ===== STEP 2: Logs =====docker logs myapp # All logsdocker logs myapp --tail 100 # Last 100 linesdocker logs myapp -f # Follow (like tail -f)docker logs myapp --since 30m # Last 30 minutesdocker logs myapp 2>&1 | grep -i error # Filter errors# ===== STEP 3: Resource Usage =====docker stats myapp --no-stream# CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O# myapp 95.2% 490MiB / 512MiB 95.7% 1.2GB/500MB 0B/0B# ← 95% memory → OOM kill is imminent!# ===== STEP 4: Shell into Running Container =====docker exec -it myapp bash # bash shell (Debian/Ubuntu)docker exec -it myapp sh # sh shell (Alpine)docker exec -it myapp /bin/sh # explicit path# Run a specific command without interactive shelldocker exec myapp env # Check environment variablesdocker exec myapp cat /app/config.json# ===== STEP 5: Check Processes =====docker top myapp # Processes running in containerdocker exec myapp ps aux # More detailed process list# ===== STEP 6: Network Debugging =====docker inspect myapp | jq '.[0].NetworkSettings.Networks'docker exec myapp netstat -tulpn # Listening portsdocker exec myapp curl -v http://othercontainer:8080/health# Test DNS resolution in containerdocker exec myapp nslookup db # Can container resolve service name?docker exec myapp ping -c3 db # Can container reach db?# ===== STEP 7: Docker Events =====docker events --filter 'container=myapp'docker events --filter 'container=myapp' --since 1h# Shows: die, kill, oom, start, stop, restart events
Debugging Containers That Won’t Start
bash
# Container crashes immediately — override the entrypointdocker run -it --entrypoint /bin/sh myapp:latest# Or bash:docker run -it --entrypoint /bin/bash myapp:latest# Then manually run the CMD to see the exact error:$ node server.js# Error: Cannot find module 'express' ← npm install was never run!# Debug with a minimal commanddocker run -it myapp:latest echo "container works"docker run -it myapp:latest ls /app # Check if files are presentdocker run -it myapp:latest env # Check env vars# Check image layers/history for cluesdocker history myapp:latestdocker inspect myapp:latest | jq '.[0].Config'
Advanced Network Debugging — netshoot
bash
# nicolaka/netshoot — network debugging toolkit# Contains: curl, wget, dig, nslookup, netstat, tcpdump, etc.# Share network namespace with target containerdocker run -it \ --net container:myapp \ nicolaka/netshoot# Now you have full network tools in myapp's network namespace$ netstat -tulpn # See what myapp is listening on$ curl localhost:3000 # Test myapp's API$ tcpdump -i eth0 # Capture traffic# Share both network AND pid namespacedocker run -it \ --net container:myapp \ --pid container:myapp \ nicolaka/netshoot# Check for OOM kills in system logsdmesg | grep oomjournalctl -k | grep oom | tail -20
Tip: Decoding exit codes — Exited (0): clean exit (intentional). Exited (1): application error. Exited (137): killed by OOM killer (memory limit exceeded). Exited (139): segfault. Exited (143): SIGTERM received (graceful stop). Exit codes reveal a lot before you even read the logs.
Caution: Alpine base images do not include bash, curl, or netstat by default. Install them temporarily for debugging: docker exec myapp apk add --no-cache curl bash net-tools. For distroless images, use ephemeral debug containers (a Kubernetes feature).
A running container does not mean the application is healthy. The process may be alive but unable to connect to the database, or not listening on the expected port. Health checks tell Docker whether the app is actually working.
Note: Health check states — starting: within the start_period, checks are ignored. healthy: the most recent check passed. unhealthy: retries exhausted, all checks failed. Docker Swarm and Kubernetes automatically restart unhealthy containers.
Dockerfile Health Check Options
dockerfile
# HTTP endpoint check (for REST APIs)HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \ CMD curl -f http://localhost:3000/health || exit 1# --interval: How often to check (default: 30s)# --timeout: How long to wait for one check (default: 30s)# --start-period: Startup grace period — failed checks don't count (default: 0s)# --retries: How many times to retry before marking unhealthy (default: 3)# curl -f: Fails silently on HTTP errors (4xx, 5xx)# TCP port check (for databases or non-HTTP services)HEALTHCHECK --interval=30s --timeout=3s \ CMD nc -z localhost 5432 || exit 1# nc -z: Check if port is open (no data sent)# Node.js application check (when curl is not available)HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \ CMD node -e "require('http').get('http://localhost:3000/health', (r) => { process.exit(r.statusCode === 200 ? 0 : 1) })" || exit 1# Custom script (for complex checks)COPY healthcheck.sh /usr/local/bin/healthcheckRUN chmod +x /usr/local/bin/healthcheckHEALTHCHECK --interval=30s --timeout=10s --retries=3 \ CMD /usr/local/bin/healthcheck# Disable inherited healthcheck (from base image)HEALTHCHECK NONE
Complex Health Check Script
bash
#!/bin/sh# healthcheck.sh — Multiple checks in one scriptset -e# Check 1: HTTP endpoint responsehttp_status=$(curl -sL -w "%{http_code}" http://localhost:3000/health -o /dev/null --max-time 3)if [ "$http_status" != "200" ]; then echo "UNHEALTHY: HTTP check failed with status $http_status" exit 1fi# Check 2: Database connectivityif ! nc -z db 5432 2>/dev/null; then echo "UNHEALTHY: Cannot reach database on port 5432" exit 1fi# Check 3: Disk space (prevent crash due to full disk)disk_usage=$(df /app | awk 'NR==2 {print $5}' | sed 's/%//')if [ "$disk_usage" -gt 90 ]; then echo "UNHEALTHY: Disk usage too high: ${disk_usage}%" exit 1fiecho "HEALTHY: All checks passed"exit 0
Health Check in Docker Compose
yaml
version: "3.8"services: db: image: postgres:15-alpine environment: POSTGRES_PASSWORD: secret healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 10s timeout: 5s retries: 5 start_period: 30s # PostgreSQL takes time to start! api: image: myapi:latest healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3000/health"] interval: 30s timeout: 5s retries: 3 start_period: 40s depends_on: db: condition: service_healthy # Wait for db to be HEALTHY, not just running! worker: image: myworker:latest depends_on: api: condition: service_healthy # Start worker only when api is healthy db: condition: service_healthy# depends_on with service_healthy = proper startup ordering!# Without this: api starts before db is ready → connection error!
Checking Health Status
bash
# docker ps shows health statusdocker ps# CONTAINER ID IMAGE STATUS# abc123 myapp Up 5 minutes (healthy)# def456 mydb Up 2 minutes (starting)# ghi789 worker Up 10 minutes (unhealthy) ← Problem!# Get health statusdocker inspect --format '{{.State.Health.Status}}' myapp# healthy# Detailed health info with historydocker inspect --format '{{json .State.Health}}' myapp | jq# {# "Status": "healthy",# "FailingStreak": 0,# "Log": [# {# "Start": "2024-01-15T...",# "End": "2024-01-15T...",# "ExitCode": 0,# "Output": "HEALTHY: All checks passed"# }# ]# }# Wait for container to be healthy (useful in scripts)until [ "$(docker inspect --format '{{.State.Health.Status}}' myapp)" == "healthy" ]; do echo "Waiting for myapp to be healthy..." sleep 5doneecho "myapp is healthy!"
Tip: Design a dedicated /health endpoint that returns HTTP 200 when healthy and 503 when unhealthy. It should check database connectivity, required external services, and application state. Response time should be under 100ms — the health check itself should not be slow.
Caution: If start_period is too short and your app starts slowly, the container will immediately be marked “unhealthy” and enter a restart loop. Java apps may take 30–60 seconds. Node.js typically takes 5–10 seconds. Set the value based on your app’s actual startup time.
Note: Docker’s HEALTHCHECK instruction does not work in Kubernetes — it is a Docker Compose feature. Kubernetes has its own livenessProbe, readinessProbe, and startupProbe. The concept is the same, but the syntax differs. When migrating to Kubernetes, convert Dockerfile HEALTHCHECKs to Kubernetes probes.
Docker for Local Development — The Real Game Changer
Docker development environments ensure every developer — whether on Windows, Mac, or Linux — runs the exact same stack. New developer onboarding becomes: git clone && docker compose up — done in under 5 minutes.
Tip: Docker dev environments eliminate “works on my machine” issues permanently. Every developer gets an identical stack, regardless of operating system.
yaml
# docker-compose.dev.yml - Development setupservices: api: build: context: . dockerfile: Dockerfile.dev # Dev dockerfile with hot reload volumes: - .:/app # Mount source code! - /app/node_modules # Keep container's node_modules environment: NODE_ENV: development DEBUG: "true" ports: - "9229:9229" # Node.js debug port command: npm run dev # Hot reload command # Development tools mailhog: image: mailhog/mailhog # Email testing ports: - "8025:8025" # Web UI - "1025:1025" # SMTP profiles: - dev-tools redis-commander: image: rediscommander/redis-commander environment: REDIS_HOSTS: "local:redis:6379" ports: - "8081:8081" profiles: - dev-tools
Use profiles to start dev tools only when needed: docker compose --profile dev-tools up
Dockerfile.dev — Node.js with Hot Reload
dockerfile
# Dockerfile.dev - Development onlyFROM node:18-alpine# Install nodemon for hot reloadRUN npm install -g nodemonWORKDIR /app# Copy package filesCOPY package*.json ./RUN npm install # All deps including devDependencies# Don't COPY source - it's mounted via volume!EXPOSE 3000 9229# Hot reload with nodemonCMD ["nodemon", "--inspect=0.0.0.0:9229", "src/server.js"]
Development vs Production — Key Differences
Aspect
Development
Production
Base Image
node:18 (full)
node:18-alpine
devDependencies
Installed
Excluded
Source Code
Via bind mount
COPYed into image
Hot Reload
nodemon
Not needed
Debug Ports
9229 exposed
No debug
Image Size
~1GB acceptable
< 200MB target
Multi-stage Build
Not needed
Yes
Benefits of Docker for Development
Same Environment: Every developer runs an identical stack. “Works on my machine” becomes irrelevant.
Instant Onboarding: New developer joins: git clone && docker compose up = fully running stack in 5 minutes.
No Version Conflicts: Switch between Node 16, 18, 20 projects without juggling version managers. Each project is isolated.
Exact Production Parity: Test with the exact same database version (Postgres 16, MongoDB 7) as production.
You can debug a Node.js app running inside a container using Chrome DevTools or VS Code.
bash
# Start Node.js container in debug modedocker run -d \ --name api-debug \ -p 9229:9229 \ -v $(pwd):/app \ node:18 \ node --inspect=0.0.0.0:9229 src/server.js# Then open Chrome: chrome://inspect# Your server will appear under "Remote Target"
# Python debugging with debugpydocker run -d \ -p 5678:5678 \ -v $(pwd):/app \ python:3.11 \ python -m debugpy --listen 0.0.0.0:5678 --wait-for-client app.py# In VS Code, use the Python: Remote Attach configuration# Port: 5678, Host: localhost
Network Debugging with netshoot
Tip:nicolaka/netshoot contains curl, tcpdump, netstat, nslookup, traceroute, dig — everything you need. Attach to any container’s network namespace and debug immediately.
bash
# Network debugging with netshootdocker run -it --rm \ --network container:myapp \ nicolaka/netshoot# Now inside myapp's network namespace with full tooling:# curl, tcpdump, netstat, nslookup, traceroute, dig, etc.# Check DNS resolution inside container networkdocker run --rm --network mynetwork nicolaka/netshoot nslookup api# Check if port is reachable between servicesdocker run --rm --network mynetwork nicolaka/netshoot nc -zv api 5000# Capture TCP trafficdocker run --rm --network container:myapp nicolaka/netshoot tcpdump -i eth0 port 5000
General Container Debugging Commands
bash
# Open a shell inside the containerdocker exec -it myapp sh# View logs with timestampsdocker logs -f --timestamps myapp# Process list inside containerdocker exec myapp ps aux# Check environment variablesdocker exec myapp env | grep -i api# Full container configdocker inspect myapp | jq '.[0].NetworkSettings'# Browse the filesystemdocker exec -it myapp find /app -name "*.log"# Copy files out of container for inspectiondocker cp myapp:/app/logs/error.log ./error.log
Caution: Before going to production, verify all items below. Common mistakes in production Dockerfiles and compose files create security vulnerabilities, crashes, or performance issues. Bookmark this checklist.
Production Dockerfile Checklist
Check
Why It Matters
Status
Specific base image version (not latest)
Reproducible builds, no surprise updates
☐
Multi-stage build used
Smaller final image, no build tools in prod
☐
Non-root user configured
Security — limits blast radius of a breach
☐
.dockerignore present
Smaller build context, no secrets leaked
☐
HEALTHCHECK configured
Orchestrators can detect unhealthy containers
☐
WORKDIR explicitly set
Predictable file paths
☐
package*.json COPY before source COPY
Better layer caching
☐
No secrets in image/Dockerfile
Security — images can be inspected
☐
Image size < 500MB (ideally < 200MB)
Faster pulls, lower storage cost
☐
LABEL with version/maintainer info
Traceability in production
☐
Graceful shutdown handled (SIGTERM)
Zero-downtime deployments
☐
Graceful Shutdown in Node.js
When Docker stops a container, it sends SIGTERM first, then SIGKILL after 10 seconds. If you do not handle SIGTERM, in-flight requests are dropped.
javascript
// server.js - Handle SIGTERM for graceful shutdownprocess.on('SIGTERM', () => { console.log('SIGTERM received, shutting down gracefully'); server.close(() => { console.log('HTTP server closed'); // Close DB connections mongoose.connection.close(false, () => { console.log('MongoDB disconnected'); process.exit(0); }); }); // Force exit after 30 seconds setTimeout(() => { console.error('Could not close connections in time, forcing exit'); process.exit(1); }, 30000);});
Image Versioning Strategy
bash
# Tagging strategy for productionIMAGE="myapp"REGISTRY="123456789.dkr.ecr.ap-south-1.amazonaws.com"GIT_TAG=$(git describe --tags --always)GIT_SHA=$(git rev-parse --short HEAD)DATE=$(date +%Y%m%d)# Always push with multiple tagsdocker build -t $IMAGE .docker tag $IMAGE $REGISTRY/$IMAGE:$GIT_TAG # v1.2.3docker tag $IMAGE $REGISTRY/$IMAGE:$GIT_SHA # abc1234docker tag $IMAGE $REGISTRY/$IMAGE:$DATE # 20240115docker tag $IMAGE $REGISTRY/$IMAGE:latest # latest (don't use in deploys!)# Deploy using a specific version, not latest!docker pull $REGISTRY/$IMAGE:v1.2.3
Caution: Never deploy using the :latest tag in production. If someone pushes and the service restarts, an unexpected version may be deployed. Always use a git SHA or semantic version tag for deployments.
Test: Docker image builds with test target → tests run inside the container.
Build and Push: Multi-arch production image builds and pushes to registry with proper tags.
Deploy: ECS task definition updates → service performs a rolling deploy with zero downtime.
Note: “Our CI/CD pipeline runs on GitHub Actions. Every push to main automatically builds the Docker image, runs tests inside the container, then pushes to ECR and does an ECS rolling deploy — all automated. Manual production deployments never happen.”
Note: Swarm is now mostly legacy. In 2026, Kubernetes is the industry standard. That said, mastering Docker Compose and core Docker concepts makes learning Kubernetes significantly easier — the concepts are the same, just with greater complexity.
When Do You Need Orchestration?
Scale: 10+ Containers: Docker Compose runs on a single machine. Deploying across multiple machines requires orchestration.
High Availability: If a node dies and the app automatically shifts to another node — that is orchestration.
Auto-scaling: Automatically scale containers up when traffic increases and scale down when it drops.
Zero-downtime Deploys: Rolling deployments where the old version keeps running until the new version is healthy.
Transition Path
Tip: Learning path: Docker basics → Docker Compose (multi-container) → Kubernetes (multi-host, production-grade). Each step builds on the previous one. Master Docker concepts and Kubernetes will feel natural. When you are ready to take the next step, the Kubernetes for DevOps guide picks up exactly where containers leave off.
bash
# Docker Swarm quick demo (educational only)docker swarm init # Initialize swarm on current nodedocker service create --name web nginx # Create a servicedocker service scale web=5 # Scale to 5 replicas (auto-distributed!)docker service ls # List servicesdocker service ps web # See where each replica runsdocker swarm leave --force # Leave swarm# Equivalent nginx deployment in Kuberneteskubectl create deployment web --image=nginxkubectl scale deployment web --replicas=5kubectl get deploymentskubectl get pods # See 5 pods running
Task Definition — Blueprint for container configuration, similar to a docker-compose.yml. Defines image, ports, environment variables, CPU and memory limits.
Task — A running instance of a task definition. Equivalent to a running container or group of containers.
Service — Maintains the desired task count, automatically replaces failed tasks, and integrates with a load balancer.
Cluster — A group of EC2 instances or Fargate compute resources on which tasks are scheduled.
Note: Never store secrets directly in environment variable definitions. Use AWS Secrets Manager or Parameter Store. ECS automatically injects secrets as environment variables at runtime — they are not stored in the image.
Complete Deployment Script — React App to AWS Fargate
Tip: Copy this script into your project, set the variables to match your setup, and deploy your entire application with a single command: bash deploy.sh
bash
#!/bin/bash# deploy.sh - Complete deployment scriptset -e # Stop the script if any step fails# Variables — set your own values hereAPP_NAME="myreactapp"AWS_ACCOUNT_ID="123456789012"AWS_REGION="ap-south-1"ECR_REGISTRY="$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com"IMAGE_TAG=$(git rev-parse --short HEAD) # Use Git SHA as tagecho "Starting deployment of $APP_NAME:$IMAGE_TAG"# Step 1: Build imageecho "Building Docker image..."docker build \ --build-arg REACT_APP_API_URL=https://api.myapp.com \ -t $APP_NAME:$IMAGE_TAG \ .# Step 2: Authenticate with ECRecho "Authenticating with ECR..."aws ecr get-login-password --region $AWS_REGION | \ docker login --username AWS --password-stdin $ECR_REGISTRY# Step 3: Tag for ECRdocker tag $APP_NAME:$IMAGE_TAG $ECR_REGISTRY/$APP_NAME:$IMAGE_TAGdocker tag $APP_NAME:$IMAGE_TAG $ECR_REGISTRY/$APP_NAME:latest# Step 4: Push to ECRecho "Pushing to ECR..."docker push $ECR_REGISTRY/$APP_NAME:$IMAGE_TAGdocker push $ECR_REGISTRY/$APP_NAME:latest# Step 5: Update ECS service (force new deployment)echo "Updating ECS service..."aws ecs update-service \ --cluster production \ --service $APP_NAME-service \ --force-new-deployment \ --region $AWS_REGION# Step 6: Wait for deployment to stabilizeecho "Waiting for deployment to stabilize..."aws ecs wait services-stable \ --cluster production \ --services $APP_NAME-service \ --region $AWS_REGIONecho "Deployment successful! Image: $ECR_REGISTRY/$APP_NAME:$IMAGE_TAG"
AWS Fargate with Application Load Balancer
bash
# Create ECS cluster (Fargate mode)aws ecs create-cluster --cluster-name production# Create ALB target groupaws elbv2 create-target-group \ --name myapp-tg \ --protocol HTTP \ --port 3000 \ --vpc-id vpc-12345 \ --target-type ip \ # Use 'ip' for Fargate, not 'instance' --health-check-path /health# Create ECS service with Fargate — 2 replicas for high availabilityaws ecs create-service \ --cluster production \ --service-name myapp-service \ --task-definition myapp:1 \ --desired-count 2 \ --launch-type FARGATE \ --network-configuration "awsvpcConfiguration={subnets=[subnet-1,subnet-2],securityGroups=[sg-123],assignPublicIp=ENABLED}" \ --load-balancers "targetGroupArn=arn:aws:elasticloadbalancing:...,containerName=myapp,containerPort=3000"
Tip: Build all of these projects and push them to GitHub. Each project can become a resume bullet point. Recruiters and interviewers want to see working code — that is far more impactful than simply claiming Docker knowledge.
# Start the complete MERN stackdocker compose up -d --build# Check all servicesdocker compose ps# Follow API logsdocker compose logs -f backend# Access points:# Frontend: http://localhost:3000# Backend API: http://localhost:5000# MongoDB GUI: http://localhost:8081# Stop the stack (data is preserved)docker compose down
Tip: Resume bullet — “Containerized full MERN stack using Docker Compose with service discovery, named volumes for data persistence, and Mongo Express GUI for development; reduced environment setup time from 2 hours to 5 minutes.”
Project 5: WordPress + MySQL — Classic Stack
Difficulty: Intermediate | Stack: WordPress, MySQL 8 | Concepts: Health checks with conditions, restart policies, named volumes
docker compose up -d# MySQL health check will be satisfied before WordPress startsdocker compose ps# Visit: http://localhost:8080# Complete the WordPress setup wizard
Project 6: Microservices with Nginx API Gateway
Difficulty: Intermediate | Stack: Node.js microservices, Nginx gateway, PostgreSQL | Concepts: API gateway pattern, service isolation, multiple databases
yaml
services: # Single entry point — all traffic routes through here api-gateway: image: nginx:alpine ports: - "80:80" volumes: - ./nginx/gateway.conf:/etc/nginx/conf.d/default.conf depends_on: - user-service - product-service user-service: build: ./user-service environment: DB_URL: postgresql://postgres:password@user-db:5432/users # No ports exposed externally — only accessible through the gateway depends_on: - user-db product-service: build: ./product-service environment: DB_URL: postgresql://postgres:password@product-db:5432/products depends_on: - product-db # Each service has its own database user-db: image: postgres:16-alpine environment: POSTGRES_DB: users POSTGRES_PASSWORD: password volumes: - user_db_data:/var/lib/postgresql/data product-db: image: postgres:16-alpine environment: POSTGRES_DB: products POSTGRES_PASSWORD: password volumes: - product_db_data:/var/lib/postgresql/datavolumes: user_db_data: product_db_data:
Tip: Resume bullet — “Designed and containerized microservices architecture with Nginx API gateway, service isolation, and per-service databases using Docker Compose; demonstrating event-driven service communication patterns.”
Project 7: Image Size Optimization Challenge
Difficulty: Intermediate | Goal: 1.2 GB → under 100 MB | Concepts: Multi-stage builds, Alpine, production deps, .dockerignore
dockerfile
# BEFORE: Naive approach — DO NOT USE in productionFROM ubuntu:22.04RUN apt-get update && apt-get install -y nodejs npmCOPY . /appWORKDIR /appRUN npm install # All deps including devDependencies# Final image size: ~1.2 GB
dockerfile
# AFTER: Optimized multi-stage buildFROM node:18-alpine AS builderWORKDIR /appCOPY package*.json ./RUN npm ci # All deps for buildingCOPY . .RUN npm run build # Compile/transpileFROM node:18-alpine # Fresh slim baseWORKDIR /appCOPY package*.json ./RUN npm ci --only=production # Production deps onlyCOPY --from=builder /app/dist ./distUSER node # Non-rootCMD ["node", "dist/server.js"]# Final image size: ~85 MB (93% reduction)
bash
# Compare before and after sizesdocker build -f Dockerfile.before -t app:before .docker build -f Dockerfile.after -t app:after .docker images | grep app# Detailed layer analysisdocker history app:after# Layer-by-layer analysis with divedocker run --rm -it \ -v /var/run/docker.sock:/var/run/docker.sock \ wagoodman/dive app:after
Tip: Resume bullet — “Implemented end-to-end CI/CD pipeline using GitHub Actions: automated Docker image build and test on every push, Docker Hub publish with semantic versioning, and zero-touch SSH deployment to production server.”
Project 10: Full Observability Stack (ELK + Jaeger + Prometheus)
Difficulty: Advanced | Stack: OpenTelemetry, Jaeger, Prometheus, ELK Stack | Concepts: Metrics + Tracing + Logging — the three pillars of observability
The following errors are encountered by almost every Docker developer at some point. Each includes the root cause, solution, and prevention guidance.
Error: Cannot connect to the Docker daemon
text
Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Root Cause: The Docker daemon (dockerd) has not started or has crashed.
bash
# Check daemon statussudo systemctl status docker# Start the daemonsudo systemctl start docker# Enable on bootsudo systemctl enable docker# Verifydocker info
Prevention: Always enable the Docker service on boot with systemctl enable docker. On Mac, ensure Docker Desktop is running before using the CLI.
Error: Permission denied on Docker socket
text
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock
Root Cause: The current user is not a member of the docker group.
bash
# Add user to docker groupsudo usermod -aG docker $USER# Apply group change immediately (or log out and back in)newgrp docker# Verifydocker run hello-world
Prevention: Add the user to the docker group during initial Docker installation. In CI/CD pipelines, add the runner user to the docker group.
Error: Port is already allocated
text
docker: Error response from daemon: driver failed programming external connectivity on endpoint myapp: Bind for 0.0.0.0:8080 failed: port is already allocated.
Root Cause: Host port 8080 is already in use by another process or container.
bash
# Find what is using the portsudo lsof -i :8080sudo ss -tulpn | grep :8080# Kill the process (replace PID)kill -9 <PID># OR use a different host portdocker run -p 8081:80 nginx# Find running containers using the portdocker ps | grep 8080
Prevention: Use environment variables for port configuration. Run docker ps before starting new containers.
Error: No space left on device
text
ERROR: failed to solve: ... write /var/lib/docker/...: no space left on device
Root Cause: The Docker daemon’s disk space is exhausted — old images, containers, and volumes have consumed all available space.
bash
# Check disk usagedf -hdocker system df# Full cleanup (removes everything unused)docker system prune -a --volumes# Selective cleanupdocker image prune -a # Remove unused imagesdocker container prune # Remove stopped containersdocker volume prune # Remove unused volumes
Prevention: Set up a cron job for docker system prune. Monitor /var/lib/docker size. Use multi-stage builds to keep images small.
Error: Container exits immediately
Root Cause: The container’s main process (CMD/ENTRYPOINT) completed, or it crashed with an error.
bash
# Check logs firstdocker logs container_iddocker logs container_id --tail 50# Check exit codedocker inspect --format '{{.State.ExitCode}}' container_id# Run interactively to debugdocker run -it myapp bash# Bad: CMD nginx (exits immediately — starts as daemon)# Good: CMD ["nginx", "-g", "daemon off;"]# Override entrypoint for debuggingdocker run -it --entrypoint bash myapp
Prevention: Use foreground commands in CMD. Test CMD locally first. Add proper health checks.
Error: Pull access denied
text
Error response from daemon: pull access denied for myapp, repository does not exist or may require 'docker login'
Root Cause: The image name is incorrect, authentication is missing for a private registry, or the image does not exist.
bash
# Login to Docker Hubdocker login# Login to a private registrydocker login registry.mycompany.com -u username -p password# Check exact image name and tagdocker pull myapp:latest# Full path required for private registriesdocker pull registry.mycompany.com/team/myapp:v1.0
Prevention: Use full image paths. Store credentials in Docker credential stores, not plain text. Set up registry mirrors for CI/CD.
Error: Build context is extremely large
text
Sending build context to Docker daemon 2.5GB
Root Cause: A .dockerignore file is missing, so Docker sends the entire project directory (including node_modules, .git, etc.) to the daemon.
Prevention: Always create a .dockerignore when starting a new Docker project. Commit it to version control alongside your code.
Error: Docker Hub rate limit exceeded
text
Error response from daemon: toomanyrequests: You have reached your pull rate limit.
Root Cause: Docker Hub anonymous pull limit has been hit (100 pulls per 6 hours unauthenticated, 200 for free accounts).
bash
# Authenticate first to increase the limitdocker logindocker pull nginx# OR configure a registry mirror in /etc/docker/daemon.json:{ "registry-mirrors": ["https://mirror.gcr.io"]}# Restart Docker after configuration changesudo systemctl restart docker
Prevention: Always authenticate in CI/CD. Use ECR or GCR as a pull-through cache.
Error: Container OOMKilled — exit code 137
Root Cause: The container exceeded its memory limit. The Linux OOM Killer forcefully terminated the process. Exit code 137 = 128 + 9 (SIGKILL).
Prevention: Profile application memory in development. Set realistic limits based on actual usage plus a 20% buffer. Add memory monitoring alerts in production.
Error: ECONNREFUSED connecting to database
text
Error: connect ECONNREFUSED 127.0.0.1:5432
Root Cause: The application inside the container is using 127.0.0.1 or localhost to connect to the database. Inside a container, localhost refers to the container itself — not another container.
bash
# WRONG - localhost inside container = the container itselfDB_HOST=localhost # ❌DB_HOST=127.0.0.1 # ❌# CORRECT - use the container name or service nameDB_HOST=postgres # ✅ (with docker compose)DB_HOST=db # ✅ (compose service name)# Verify connectivitydocker exec myapp ping postgresdocker exec myapp nslookup postgres
Prevention: Never hardcode localhost for inter-container communication. Use environment variables for hostnames. Use user-defined networks for DNS resolution.
Error: Volume mount permission denied
text
Error: Permission denied: '/app/data'
Root Cause: The container process user does not own the host volume directory.
bash
# Option 1: Match user/groupdocker run -u $(id -u):$(id -g) -v $(pwd)/data:/app/data myapp# Option 2: Fix host directory permissionssudo chown -R 1000:1000 ./data# Option 3: Fix permissions in DockerfileRUN mkdir -p /app/data && chown -R node:node /app/dataUSER node# Option 4: Use named volume (Docker manages permissions)docker run -v myapp_data:/app/data myapp
Error: DNS resolution failure between containers
text
Error: getaddrinfo ENOTFOUND api-service
Root Cause: The default bridge network does not provide automatic DNS resolution between containers. Only user-defined networks include the embedded DNS server.
bash
# Create a user-defined networkdocker network create app-network# Run both containers on the same networkdocker run --name api-service --network app-network myapidocker run --name frontend --network app-network myfrontend# Now 'api-service' is resolvable by namedocker exec frontend curl http://api-service:3000
Prevention: Always use user-defined networks for multi-container applications. Docker Compose sets this up automatically.
Error: COPY file not found in build context
text
COPY failed: file not found in build context or excluded by .dockerignore: stat package.json: file not found
Root Cause: The file does not exist in the build context, .dockerignore has excluded it, or the Dockerfile is being run from the wrong directory.
bash
# Check if file existsls -la package.json# Check .dockerignore isn't excluding itcat .dockerignore | grep package# Build from the correct directorydocker build -t myapp . # note the dot — build context = current directory# To build from a parent directorydocker build -t myapp -f ./subdir/Dockerfile .
Error: exec format error
text
exec /docker-entrypoint.sh: exec format error
Root Cause: The image was built for a different CPU architecture (e.g., built on an M1 Mac for arm64, then run on a linux/amd64 server).
bash
# Check image platformdocker inspect myapp | jq '.[0].Architecture'# Build for a specific platformdocker build --platform linux/amd64 -t myapp .# Build a multi-platform image using buildxdocker buildx create --usedocker buildx build --platform linux/amd64,linux/arm64 \ -t myapp:latest --push .
Prevention: Always use --platform linux/amd64 when building for production servers from an M1/M2 Mac.
Error: Cache keeps invalidating — npm install runs every time
Root Cause:COPY . . is placed before RUN npm install, so any file change invalidates the cache and forces npm install to re-run.
dockerfile
# WRONG - cache breaks on every file changeFROM node:18WORKDIR /appCOPY . . # ❌ Any change = cache missRUN npm install # Re-runs every timeCMD ["node", "server.js"]# CORRECT - npm install only re-runs when package.json changesFROM node:18WORKDIR /appCOPY package*.json ./ # ✅ Only copy dependency manifestRUN npm ci # Only runs when package.json changesCOPY . . # Source code (frequently changing)CMD ["node", "server.js"]
Prevention: Always separate dependency installation from source code copying. This is one of the most important Dockerfile optimizations.
Error: No configuration file provided (Docker Compose)
text
no configuration file provided: not found
Root Cause: Docker Compose cannot find a docker-compose.yml or compose.yml in the current directory.
bash
# Check the current directoryls -la | grep -i compose# Navigate to the correct directory and run againcd /path/to/your/projectdocker compose up# OR specify the file explicitlydocker compose -f /path/to/docker-compose.yml up# Compose searches for these files in order:# 1. compose.yaml# 2. compose.yml# 3. docker-compose.yaml# 4. docker-compose.yml
Error: Container status is unhealthy
Root Cause: The HEALTHCHECK command is failing. The application has not properly started, or the health endpoint is not responding.
bash
# Inspect health check outputdocker inspect --format '{{json .State.Health}}' myapp | jq .# See the last 5 health check resultsdocker inspect myapp | jq '.[0].State.Health.Log'# Manually run the health check commanddocker exec myapp curl -f http://localhost:3000/health# Temporarily disable to debugdocker run --no-healthcheck myapp
Prevention: Test health check commands manually before adding them to the Dockerfile. Use --start-period for applications that take time to start. Implement a /health endpoint in your application.
Error: OOMKilled on Python — ModuleNotFoundError
text
ModuleNotFoundError: No module named 'flask'
Root Cause:requirements.txt was not copied correctly, or pip install ran in the wrong directory.
# Debug: check if packages are installeddocker exec -it myapp pip list | grep flask# Rebuild without cachedocker build --no-cache -t myapp .
Error: nginx bind permission denied on port 80
text
nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)
Root Cause: A non-root user cannot bind to privileged ports below 1024.
bash
# Option 1: Use port 8080 (non-privileged)# In nginx.conf: listen 8080;docker run -p 80:8080 nginx-custom# Option 2: Add NET_BIND_SERVICE capabilitydocker run --cap-add NET_BIND_SERVICE -p 80:80 nginx-custom# Option 3: Use the nginx-unprivileged image (uses port 8080 by default)FROM nginxinc/nginx-unprivileged
Tip: When using a non-root user with nginx, always configure nginx.conf to listen on port 8080 or higher. Use nginxinc/nginx-unprivileged as the base image for a secure, non-root nginx setup.
Tip:docker logs and docker inspect are your best debugging tools. 90% of Docker errors can be diagnosed with just these two commands.
When something stops working, follow this systematic checklist. Do not make random changes without first diagnosing the problem.
Note: Approach Docker debugging like a doctor: check symptoms first, then diagnose, then treat. Never make changes without a clear diagnosis.
Step 1: Is the Container Running?
bash
# Show running containersdocker ps# Show ALL containers (including stopped)docker ps -a# Pretty format with useful columnsdocker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"# Filter by container namedocker ps -a --filter name=myapp# Check the STATUS column: "Exited (1) 2 minutes ago" = crashed# "Exited (0) 2 minutes ago" = completed normally
Step 2: What Do the Logs Say?
bash
# Last 100 lines of logsdocker logs --tail 100 myapp# Follow logs in real timedocker logs -f myapp# Logs from the last 1 hourdocker logs --since 1h myapp# Logs with timestampsdocker logs -t myapp# Filter for errorsdocker logs myapp 2>&1 | grep -i "error\|fatal\|critical"# Logs persist even for stopped containers until the container is removeddocker logs myapp
Step 3: What Is the Exit Code?
bash
# Get exit codedocker inspect --format '{{.State.ExitCode}}' myapp# Get full state informationdocker inspect --format '{{json .State}}' myapp | jq .# Exit code reference:# 0 = Normal exit (success, or CMD completed)# 1 = General error (application error)# 2 = Misuse of shell command# 125 = Docker daemon error# 126 = Container command cannot be invoked# 127 = Container command not found# 130 = Container terminated by Ctrl+C (SIGINT)# 137 = SIGKILL (OOM kill or docker kill)# 139 = Segmentation fault# 143 = SIGTERM (graceful shutdown)
Step 4: Can You Enter the Container?
bash
# Enter a running container (bash for ubuntu/debian)docker exec -it myapp bash# For Alpine Linux (no bash by default)docker exec -it myapp sh# Run specific commands insidedocker exec myapp ls -la /appdocker exec myapp cat /app/config.jsondocker exec myapp env | grep DB_# If the container keeps crashing, override entrypoint to debugdocker run -it --entrypoint bash myappdocker run -it --entrypoint sh myapp # Alpine
Step 5: Check Resource Usage
bash
# Live stats for a specific containerdocker stats myapp# Live stats for all containersdocker stats# One-time snapshot (no streaming)docker stats --no-stream# Check process list inside containerdocker top myapp# Check resource limits set on containerdocker inspect --format '{{json .HostConfig.Memory}}' myapp
Step 6: Network Inspection
bash
# Inspect a networkdocker network inspect app-network# Check container's network settingsdocker inspect --format '{{json .NetworkSettings}}' myapp | jq .# Test DNS resolution from inside the containerdocker exec myapp nslookup postgresdocker exec myapp nslookup google.com# Test connectivitydocker exec myapp curl -v http://api-service:3000/healthdocker exec myapp ping -c 3 postgres# Check which networks the container is ondocker inspect --format '{{json .NetworkSettings.Networks}}' myapp | jq 'keys'# Show port mappingsdocker port myapp
Step 7: Check Events
bash
# All events in the last hourdocker events --since 1h# Filter by containerdocker events --since 1h --filter container=myapp# Filter by event typedocker events --filter type=container --filter event=oom# Watch events livedocker events# Events by imagedocker events --filter image=myapp:latest
Note: Do not include debugging tools in production images — they increase image size and expand the attack surface. Instead, maintain a separate debug image built from your production image plus debugging tools.
The following 15 questions cover the most important Docker topics you will encounter in engineering interviews. Each includes a concise answer, a detailed explanation, and common pitfalls to avoid.
Q1. What is the difference between a Docker image and a container?
A Docker image is an immutable, read-only template made of layered filesystems containing everything needed to run an application. A container is a running instance of that image — it adds a thin writable layer on top. Multiple containers can run from the same image simultaneously.
Red flag: Saying “image and container are the same thing” immediately signals a foundational misunderstanding.
Q2. How does Docker image layer caching work?
Docker caches each instruction’s result as a layer. On rebuild, if an instruction and its inputs are unchanged, Docker reuses the cached layer and skips re-execution. Once any layer is invalidated, all subsequent layers are rebuilt. This is why Dockerfile instruction order matters — place rarely-changing instructions (install dependencies) before frequently-changing ones (copy source code).
dockerfile
# Good - npm install only re-runs when package.json changesCOPY package*.json ./RUN npm ciCOPY . . # source changes only invalidate this layer and below
Red flag: Not structuring Dockerfiles to exploit layer caching. A 3-minute npm install running on every CI build is an immediate code review concern.
Q3. What are multi-stage builds and why use them?
Multi-stage builds use multiple FROM statements in one Dockerfile. You compile/build in a full environment, then COPY --from=stage only the compiled artifacts into a minimal runtime image. This keeps build tools out of the final image, dramatically reducing size and attack surface.
dockerfile
FROM golang:1.21 AS builderWORKDIR /appCOPY go.mod go.sum ./RUN go mod downloadCOPY . .RUN CGO_ENABLED=0 go build -o server .FROM scratch AS runtimeCOPY --from=builder /app/server /serverCMD ["/server"]# Result: ~10 MB image vs ~800 MB with full Go toolchain
Red flag: Not knowing that you can have more than two stages, or that you can target specific stages with docker build --target=stagename.
Q4. What is the difference between CMD and ENTRYPOINT?
ENTRYPOINT sets the fixed executable that always runs. CMD provides default arguments that can be overridden at runtime. Together: ENTRYPOINT is “what to run” and CMD is “with what default arguments.” CMD alone is fully overridable by passing arguments to docker run.
Red flag: Using shell form for CMD/ENTRYPOINT (CMD node server.js). Shell form runs your command via /bin/sh -c, meaning the shell is PID 1 — not your application. SIGTERM is not forwarded, breaking graceful shutdown.
Q5. What is the difference between volumes and bind mounts?
Bind mounts link a specific host path into the container — you control the location. Named volumes are managed by Docker, stored at /var/lib/docker/volumes/. Named volumes are preferred in production for persistent data (databases). Bind mounts are preferred in development for live code reloading.
Type
Storage
Best For
Named Volume
Docker-managed
DB data, persistent app data
Bind Mount
Specific host path
Development hot reload, config files
tmpfs
Host RAM (never on disk)
Secrets, temporary cache
Red flag: Using bind mounts in production — they create host-container coupling and can expose host files if misconfigured.
Q6. Explain Docker networking — bridge, host, and overlay.
bridge (default): Isolated virtual network on the host. Containers communicate via container names on user-defined bridge networks (with automatic DNS). Requires -p to expose ports to the host.
host: Container shares the host’s network stack directly. No NAT overhead, but no network isolation. Only works on Linux.
overlay: Spans multiple Docker hosts using VXLAN tunneling. Required for Docker Swarm multi-host communication.
The default bridge network does not provide automatic DNS — containers must use IPs. User-defined bridge networks resolve container names via Docker’s embedded DNS at 127.0.0.11.
Red flag: Using the default bridge network in production. Default bridge has no DNS and all containers on it can communicate with each other — no isolation.
Q7. What are Dockerfile best practices?
Key best practices:
Use minimal base images (alpine, distroless)
Use multi-stage builds for compiled languages and build-step frameworks
Exploit layer caching — copy dependency manifests before source code
Use .dockerignore to exclude node_modules, .git, .env
Run as a non-root user (USER node)
Use exec form for CMD and ENTRYPOINT (["node", "server.js"])
Combine RUN commands to minimize layers, clean up in the same instruction
Add HEALTHCHECK for production images
Pin base image versions — avoid :latest
Never pass secrets via ARG or ENV
Q8. What does depends_on do in Docker Compose — does it guarantee the service is ready?
depends_on controls startup order — it ensures the dependent container starts before the dependent service. By default it only waits for the container to start (service_started), not for the application inside to be ready.
To actually wait for a service to be ready, combine depends_on with condition: service_healthy and a proper healthcheck:
Red flag: Using depends_on without condition: service_healthy and then being surprised when the application crashes on startup because the database was not ready.
Q9. How do you secure a Docker container in production?
A hardened container uses multiple layers of defense:
bash
docker run \ --user 1000:1000 \ # non-root user --cap-drop=ALL \ # drop all capabilities --cap-add=NET_BIND_SERVICE \ # add only what is needed --read-only \ # read-only root filesystem --tmpfs /tmp:size=50m \ # writable RAM-backed temp --no-new-privileges \ # prevent privilege escalation --security-opt seccomp=/etc/docker/seccomp.json \ myapp
Checklist: non-root USER, no --privileged, --read-only filesystem, drop all capabilities, minimal base image, no docker.sock mount, scan with Trivy, use BuildKit secret mounts (never ARG/ENV for secrets).
Q10. What is the difference between docker stop and docker kill?
docker stop sends SIGTERM to allow graceful shutdown, waits 10 seconds (configurable), then sends SIGKILL. docker kill immediately sends SIGKILL without waiting.
Graceful shutdown matters: SIGTERM lets applications finish active requests, close database connections, and flush logs. Applications should handle SIGTERM explicitly. Always use docker stop in production scripts — docker kill can cause data corruption if the app is mid-write.
Q11. What is BuildKit and what advantages does it provide?
BuildKit is Docker’s next-generation build engine (default since Docker 23.0). Key advantages over the classic builder:
Secret mounts (RUN --mount=type=secret) — secrets are never stored in any layer
SSH agent forwarding for private repositories
Multi-platform builds with buildx
Significantly faster build times
bash
# BuildKit secret mount (secret is never baked into image)RUN --mount=type=secret,id=npmrc \ cp /run/secrets/npmrc ~/.npmrc && \ npm install && \ rm ~/.npmrcdocker build --secret id=npmrc,src=$HOME/.npmrc .
Red flag: Not knowing BuildKit secret mounts exist, and instead passing secrets via ARG or ENV, which appear in docker history.
Q12. How do you implement zero-downtime deployments with Docker?
Use blue-green or rolling update strategies. The key principle: start the new container and verify it is healthy before stopping the old one — never the reverse.
bash
# Blue-green pattern# 1. Start new (green) containerdocker run -d --name myapp-green --network app myapp:new# 2. Health check greendocker exec myapp-green curl -sf http://localhost:3000/health# 3. Switch traffic (update nginx upstream, then reload)# nginx -s reload# 4. Stop old (blue)docker stop myapp-blue && docker rm myapp-blue
In Kubernetes, use rolling updates with maxUnavailable: 0 and maxSurge: 1 combined with readiness probes.
Q13. How do you handle secrets in Docker?
Never store secrets in ARG, ENV, or files copied into an image. The correct approaches are:
Build-time secrets: Use BuildKit --mount=type=secret — the secret is available during the build RUN instruction but is never stored in any image layer.
Runtime secrets: Inject via environment variables at docker run time (values are not in the image), Docker Compose secrets (mounted as files at /run/secrets/), or external secret managers (AWS Secrets Manager, HashiCorp Vault).
Verify: docker history myimage and docker inspect myimage should never reveal secret values.
Q14. What is the difference between docker exec and docker attach?
docker exec starts a new process inside a running container — it is safe for debugging. docker attach connects your terminal to the container’s existing PID 1 process. If you press Ctrl+C while attached, you send SIGINT to PID 1 and stop the container.
Always use docker exec -it mycontainer /bin/sh for debugging. Use docker attach only when you specifically need to interact with PID 1, and detach with Ctrl+P, Ctrl+Q.
Q15. How do you optimize Docker builds in CI/CD for speed?
Five key techniques:
Registry-based layer caching — reuse layers from previous builds with --cache-from
Dockerfile layer order — dependency installation before source code copy
BuildKit inline cache — export cache to registry with --cache-to type=registry,mode=max
Multi-stage parallelism — BuildKit builds independent stages in parallel
GitHub Actions/GitLab cache — use type=gha cache for persistent caching between runs
Caution: Never run docker rmi on the last known-good image until the new deployment has been stable for at least 30 minutes. Tag the last working image as :stable or :prev to enable instant rollback.
Monthly Maintenance
bash
# Clean up unused resources older than 30 daysdocker system prune -a --filter "until=720h"docker volume prune -f# Update base images and rebuild with latest security patchesdocker pull node:18-alpinedocker build --no-cache .
Zero-Downtime Deployment Pattern
bash
#!/bin/bash# zero-downtime-deploy.shset -eNEW_IMAGE="registry/myapp:$1"CONTAINER_NAME="myapp"BLUE="${CONTAINER_NAME}-blue"GREEN="${CONTAINER_NAME}-green"echo "Pulling new image..."docker pull $NEW_IMAGE# Determine which container is activeif docker ps | grep -q $BLUE; then ACTIVE=$BLUE; INACTIVE=$GREENelse ACTIVE=$GREEN; INACTIVE=$BLUEfiecho "Starting $INACTIVE..."docker run -d --name $INACTIVE --network app $NEW_IMAGE# Health check loopfor i in {1..30}; do if docker exec $INACTIVE curl -sf http://localhost:3000/health; then echo "Health check passed!" break fi sleep 2done# Switch traffic (update nginx/load balancer upstream here)echo "Switching traffic to $INACTIVE..."# Stop old containerdocker stop $ACTIVE && docker rm $ACTIVEecho "Deployment complete! Active: $INACTIVE"
# docker-compose.override.yml (development overrides — add to .gitignore)# Auto-loaded when you run: docker compose upservices: api: build: . # Build locally instead of pulling volumes: - .:/app # Live code reload - /app/node_modules # Preserve installed node_modules environment: DEBUG: "true" NODE_ENV: development DATABASE_URL: postgres://user:pass@db:5432/myapp command: npm run dev # Use dev server with hot reload # Development-only extras mailhog: image: mailhog/mailhog ports: ["8025:8025"] # Fake email server for testing redis-ui: image: rediscommander/redis-commander ports: ["8081:8081"] profiles: [debug] # Only starts with --profile debug
# Start core services onlydocker compose up -d# Start with monitoring stackdocker compose --profile monitoring up -d# Start with testing dependenciesdocker compose --profile testing up -d# Start everythingdocker compose --profile monitoring --profile testing up -d
Pattern 3: Dependency Management with Health Checks
yaml
services: db: image: postgres:15-alpine healthcheck: test: ["CMD-SHELL", "pg_isready -U user -d myapp"] interval: 10s timeout: 5s retries: 5 start_period: 30s redis: image: redis:7-alpine healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 10s retries: 3 api: build: . depends_on: db: condition: service_healthy # Wait for db health check to pass redis: condition: service_healthy # Wait for redis health check to pass environment: DATABASE_URL: postgres://user:pass@db:5432/myapp REDIS_URL: redis://redis:6379
Note: Plain depends_on only waits for the container to start — not for the service inside to be ready. Always combine with condition: service_healthy and a proper healthcheck block to avoid startup race conditions.
Lint your Dockerfiles for best practice violations, security issues, and layer ordering problems.
bash
# Run via Docker (no installation required)docker run --rm -i hadolint/hadolint < Dockerfile# Install locally on macOSbrew install hadolinthadolint Dockerfile
Trivy — Security Scanner
Comprehensive vulnerability scanner for container images. Detects CVEs in OS packages, language dependencies, and misconfigurations.
bash
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh# Scan image for HIGH and CRITICAL CVEs onlytrivy image --severity HIGH,CRITICAL myapp:latest# Fail CI on CRITICAL findingstrivy image --exit-code 1 --severity CRITICAL myapp:latest# Scan a Dockerfile for misconfigurationstrivy config Dockerfile
Watchtower — Automatic Container Updates
Automatically updates running containers when new images are pushed to the registry. Recommended for development and staging environments only.
Real-time CPU, memory, network, and I/O metrics in a clean terminal interface — like htop for Docker.
bash
# macOSbrew install ctop# Any OS via Dockerdocker run --rm -ti \ --name ctop \ -v /var/run/docker.sock:/var/run/docker.sock:ro \ quay.io/vektorlab/ctop
Tools Comparison
Tool
Type
Best For
Free?
Portainer
Web GUI
Team management, beginners
Yes (CE)
Lazydocker
Terminal UI
Developers, quick inspection
Yes
dive
CLI
Image size optimization
Yes
hadolint
Linter
Dockerfile quality in CI
Yes
trivy
Security
CVE scanning, compliance
Yes
watchtower
Daemon
Auto-updates (dev/staging)
Yes
ctop
TUI
Real-time monitoring
Yes
Docker Desktop
GUI
Mac/Windows developers
Freemium
VS Code Extension
IDE
Development workflow
Yes
Tip: Recommended starter stack — install Lazydocker for daily use, dive to check image size before every push, and integrate hadolint and trivy into your CI/CD pipeline. These four tools will catch most common issues automatically.
Free browser-based Docker playground — no installation needed
TechWorld with Nana
YouTube
youtube.com/@TechWorldwithNana
Best free Docker course on YouTube, extremely clear explanations
Bret Fisher
YouTube / Course
youtube.com/@BretFisher
Deep DevOps focus, Docker Swarm, real-world production tips
KodeKloud
Course Platform
kodekloud.com
Hands-on browser labs, great for beginners, Kubernetes learning path
Docker Deep Dive (Nigel Poulton)
Book
Amazon / Leanpub
Best Docker book — concise, practical, updated regularly
awesome-docker
GitHub List
github.com/veggiemonk/awesome-docker
Curated list of Docker tools, tutorials, and resources
Docker Official Blog
Blog
docker.com/blog
Latest Docker updates, best practices, case studies
Ivan Velichko’s Blog
Blog
iximiuz.com
Deep internals: containers, namespaces, cgroups explained with clarity
4-Week Practice Plan
text
Week 1: Install Docker, run official images (nginx, postgres, redis, ubuntu) → docker run, docker ps, docker logs, docker exec → Understand port mapping, volume mounting, environment variablesWeek 2: Write Dockerfiles for your existing React/Node projects → .dockerignore, layer caching, multi-stage builds → Optimize: get below 200 MB for your application imageWeek 3: Docker Compose — build a full MERN/PERN stack locally → depends_on, healthchecks, named volumes → Development vs production compose files, profilesWeek 4: Security, multi-stage builds, push to ECR, deploy to ECS/Fargate → Non-root USER, trivy scan, hadolint check → GitHub Actions pipeline: build → push → deploy
Certification Paths
Docker Certified Associate (DCA)
Official Docker certification covering installation, configuration, networking, security, and orchestration. A solid credential for validating foundational knowledge.
Format: 55 MCQ, 90 minutes | Cost: approximately $195 USD
CKA — Certified Kubernetes Administrator
The natural next step after mastering Docker. Kubernetes is Docker at scale, and your Docker knowledge maps directly to Kubernetes concepts.
Format: Hands-on practical exam, 2 hours | Cost: approximately $395 USD
AWS Solutions Architect
Includes deep coverage of ECS, Fargate, ECR, and EKS. Highly valued in the job market for senior DevOps and cloud engineering roles.
Format: 65 MCQ, 130 minutes | Cost: approximately $150 USD
Build these five projects to make your GitHub profile stand out to technical recruiters:
Dockerized MERN Stack — Full MERN (MongoDB, Express, React, Node) stack with Docker Compose. Proper health checks, named volumes, multi-stage build for React. Include a detailed README with an architecture diagram.
CI/CD Pipeline — GitHub Actions workflow that automatically builds an image on push to main, runs tests, pushes to ECR with a version tag, and deploys to ECS/Fargate. Include a rollback workflow.
Microservices Application — Three or more services (auth, products, orders) each in separate containers with an API gateway. Service discovery via Docker DNS, isolated databases, event-driven communication.
Security-Hardened Image — Dockerfile that passes a trivy scan with zero HIGH/CRITICAL CVEs, passes hadolint with zero warnings, runs as a non-root user, and uses multi-stage builds. Include scan reports in the README.
Monitoring Stack — Application with Prometheus, Grafana, and Alertmanager in Docker Compose. Custom dashboards showing container metrics, application metrics, and alerting rules.
Moving Forward
Completing this guide means you can confidently write production-grade Dockerfiles, deploy multi-container applications with Compose, use AWS ECR and ECS with Fargate, follow Docker security best practices, and tackle DevOps interviews with confidence.
Your Docker knowledge is the foundation for everything in modern DevOps. The concepts of containers, immutable images, service isolation, and declarative configuration carry directly into the Kubernetes ecosystem.
Note: Prefer a day-by-day path? This is covered in Mission 90 Days 22–30 — a free 90-day guided DevOps program with browser terminal missions.
What is the difference between a Docker image and a container?
An image is a read-only template built from a Dockerfile; a container is a running (or stopped) instance of an image with a writable layer on top. One image can spawn many containers.
What are Docker image layers and how does layer caching work?
Each Dockerfile instruction creates a layer. Docker caches unchanged layers and rebuilds only from the first changed instruction down, so ordering stable steps before volatile ones speeds up builds.
What is a multi-stage build and when should you use one?
A multi-stage build uses multiple FROM stages so build tools stay in an early stage and only the compiled artifacts are copied into a small final image — the standard way to shrink production images.
How do you reduce Docker image size?
Use a slim or distroless base, multi-stage builds, combine and clean up RUN layers, add a .dockerignore, and avoid installing build tools or caches in the final stage.
What is the difference between CMD and ENTRYPOINT?
ENTRYPOINT sets the fixed executable for the container; CMD provides default arguments that are easy to override at run time. Used together, ENTRYPOINT is the command and CMD is its default args.
When should you use a volume vs a bind mount?
Use named volumes for persistent data managed by Docker (databases, app state) and bind mounts for mounting host source code during local development.