Observability 1 — Prometheus & Grafana fundamentals

Prometheus & Grafana: scrape, store, query, visualise

Section 1 of 5 · ~3 min

Everything you’ve built so far — servers, containers, cloud resources — needs one more thing before anyone will trust it in production: you have to know it’s healthy without logging in to check. That’s observability, and metrics are its first pillar. A metric is a number sampled over time — requests per second, memory in use, error count — and the tool that has become the default for collecting them is Prometheus.

Prometheus pulls. It doesn’t wait for your apps to send it data; on a fixed interval it scrapes an HTTP endpoint — conventionally /metrics — on every target it’s told to watch, reads the numbers, and stores each one as a time series: a stream of timestamped values tagged with labels like job="api" or instance="web-1". Pull means Prometheus always knows what should be up, so a target that stops answering is itself an alert — the built-in up metric drops to 0.

Most software doesn’t expose /metrics on its own, so you run an exporter — a small process that translates a system’s stats into Prometheus format. node_exporter is the canonical one: point it at a Linux host and it publishes CPU, memory, disk, filesystem and network metrics — the machine’s vital signs.

Once the data is in, you query it with PromQL. Two patterns cover most of what you’ll write early on:

• rate() turns a counter into a per-second rate. Counters only climb, so their raw value is meaningless; rate(node_network_receive_bytes_total[5m]) gives bytes-per-second averaged over five minutes — and it handles counter resets for you. Never use rate() on a gauge.
• sum by () aggregates series that share a label. sum by (mode) (rate(node_cpu_seconds_total[5m])) collapses per-CPU-core counters into one rate per CPU mode (idle, user, system), so you see the whole box, not 8 separate cores.

Prometheus ships a basic expression browser, but you visualise with Grafana — dashboards of panels, each panel a PromQL query drawn as a graph, gauge or table. Grafana queries Prometheus live; it stores no metrics itself. The division of labour is clean: exporters expose, Prometheus scrapes and stores, Grafana draws.

Real world: Prometheus is a meter reader on a fixed round. It doesn’t wait for each house to phone in its usage — it walks the route every 15 seconds, reads each meter (/metrics), and writes the number in its ledger with the address and timestamp. A house that’s gone dark when the reader knocks is noted too — that silence is information. Grafana is the wall chart back at the office that turns the ledger into trend lines anyone can read at a glance.

Prometheus was created at SoundCloud in 2012 to monitor a fast-growing microservice fleet, then donated to the Cloud Native Computing Foundation, where it became the second graduated project after Kubernetes. That pairing is why nearly every Kubernetes cluster you’ll meet ships Prometheus-style metrics out of the box — the model you learn locally today is the same one running under the biggest platforms in the industry.

Today you run all three locally with docker compose — no cloud bill — scrape your own machine through node_exporter, write your first PromQL, and draw it in Grafana.

Hands-On Lab

Section 2 of 5 · ~4 min

Budget about 30 minutes. This whole stack runs locally on your machine with docker compose — Prometheus, node_exporter and Grafana in three containers, no cloud account and no bill. Open your WSL2 Ubuntu 24.04 terminal with Docker 27+ and jq installed (sudo apt-get install -y jq). Container IDs, IP addresses and every metric value are unique to your machine and moment — yours will differ from the samples below.

# 1. Make a project dir and write Prometheus's config: scrape itself and node_exporter every 15s.
mkdir -p ~/m90-observability && cd ~/m90-observability
cat > prometheus.yml <<'EOF'
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
  - job_name: node
    static_configs:
      - targets: ['node-exporter:9100']
EOF
cat prometheus.yml
# Output (the two scrape jobs Prometheus will pull metrics from):
# global:
#   scrape_interval: 15s
# scrape_configs:
#   - job_name: prometheus
# ...

# 2. Write the compose file: Prometheus, node_exporter and Grafana as three local containers.
cat > docker-compose.yml <<'EOF'
services:
  prometheus:
    image: prom/prometheus:v3.1.0
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
  node-exporter:
    image: quay.io/prometheus/node-exporter:v1.8.2
    ports: ["9100:9100"]
  grafana:
    image: grafana/grafana:11.4.0
    ports: ["3000:3000"]
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - ./grafana-ds.yml:/etc/grafana/provisioning/datasources/ds.yml
EOF
echo "wrote docker-compose.yml"
# Output:
# wrote docker-compose.yml

# 3. Provision Grafana's data source so it auto-connects to Prometheus (service name, not localhost).
cat > grafana-ds.yml <<'EOF'
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
EOF
echo "wrote grafana-ds.yml"
# Output:
# wrote grafana-ds.yml

# 4. Pull the images and start all three containers in the background.
docker compose up -d
# Output (image pulls happen on first run — yours will differ):
# [+] Running 4/4
#  ✔ Network m90-observability_default           Created
#  ✔ Container m90-observability-prometheus-1     Started
#  ✔ Container m90-observability-node-exporter-1  Started
#  ✔ Container m90-observability-grafana-1        Started

# 5. Confirm all three are up.
docker compose ps
# Output (STATUS shows Up for each — names/ports yours will differ):
# NAME                                 IMAGE                                     STATUS         PORTS
# m90-observability-grafana-1          grafana/grafana:11.4.0                    Up 12 seconds  0.0.0.0:3000->3000/tcp
# m90-observability-node-exporter-1    quay.io/prometheus/node-exporter:v1.8.2   Up 12 seconds  0.0.0.0:9100->9100/tcp
# m90-observability-prometheus-1       prom/prometheus:v3.1.0                    Up 12 seconds  0.0.0.0:9090->9090/tcp

# 6. Look at raw metrics straight from node_exporter — this is the /metrics text Prometheus scrapes.
curl -s http://localhost:9100/metrics | grep -m3 node_cpu_seconds_total
# Output (per-CPU, per-mode counters in seconds — values yours will differ):
# # HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# # TYPE node_cpu_seconds_total counter
# node_cpu_seconds_total{cpu="0",mode="idle"} 1423.55

# 7. Ask Prometheus whether it's scraping both targets (give it ~15s after startup).
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
# Output (both healthy = both targets are being scraped):
# {
#   "job": "prometheus",
#   "health": "up"
# }
# {
#   "job": "node",
#   "health": "up"
# }

# 8. Your first PromQL: the built-in `up` metric — 1 means the last scrape of a target succeeded.
curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=up' \
  | jq '.data.result[] | {job: .metric.job, up: .value[1]}'
# Output (1 for each target Prometheus can reach):
# {
#   "job": "prometheus",
#   "up": "1"
# }
# {
#   "job": "node",
#   "up": "1"
# }

# 9. rate() + sum by(): per-second CPU seconds, collapsed to one rate per mode. Wait ~1 min first.
curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum by (mode) (rate(node_cpu_seconds_total[1m]))' \
  | jq '.data.result[] | {mode: .metric.mode, rate: .value[1]}'
# Output (idle dominates on an idle box — your values will differ):
# {
#   "mode": "idle",
#   "rate": "3.86"
# }
# {
#   "mode": "system",
#   "rate": "0.07"
# }
# {
#   "mode": "user",
#   "rate": "0.05"
# }
# {
#   "mode": "iowait",
#   "rate": "0.01"
# }

# 10. A GAUGE, by contrast: available memory right now — read its value directly, never with rate().
curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=node_memory_MemAvailable_bytes' \
  | jq '.data.result[0].value[1]'
# Output (bytes available — divide by 1024^3 for GiB; yours will differ):
# "6543210496"

# 11. Confirm Grafana auto-connected to Prometheus, then open the UI to draw a panel.
curl -s http://localhost:3000/api/datasources | jq '.[] | {name, type, url}'
# Output (the provisioned data source — then browse to http://localhost:3000):
# {
#   "name": "Prometheus",
#   "type": "prometheus",
#   "url": "http://prometheus:9090"
# }
# In the browser: Explore → run  sum by (mode) (rate(node_cpu_seconds_total[5m]))  → Add to dashboard.

# 12. Stop and remove everything — containers, network and volumes — so nothing keeps running.
docker compose down -v
# Output:
# [+] Running 4/4
#  ✔ Container m90-observability-grafana-1        Removed
#  ✔ Container m90-observability-node-exporter-1  Removed
#  ✔ Container m90-observability-prometheus-1     Removed
#  ✔ Network m90-observability_default            Removed

Read the results back: up at 1 in step 8 proved Prometheus was scraping both targets; the rate() + sum by in step 9 turned raw ever-climbing counters into a live per-mode CPU rate; and step 10 showed a gauge you read directly, never through rate(). Step 12’s teardown keeps your laptop clean — the whole stack was disposable.

Common Errors & Fixes

Section 3 of 5 · ~2 min

These three catch almost everyone standing up their first Prometheus stack. Read the error text slowly — parsing it is the actual skill.

Common error: Calling rate() on an instant vector — forgetting the range window — in a query or Grafana panel, e.g. rate(node_cpu_seconds_total):

Error executing query: 1:6: parse error: expected type range vector in call to function "rate", got instant vector

Why: rate() measures change over a window, so it needs a range vector — a selector with a duration like [5m]. A bare metric name is an instant vector (one sample per series), which has no window to compute a rate across, so PromQL rejects it before it even runs.

Fix: Add a range selector sized to at least four scrape intervals: rate(node_cpu_seconds_total[5m]). The same rule applies to increase() and irate() — all three take a range vector.

How you’d spot it in prod: A Grafana panel showing a red parse error under the graph, or an alert rule that never evaluates, is almost always a missing [window] on a rate/increase call — the message names the exact function and the type mismatch.

Common error: The node target shows as down on Prometheus’s Targets page, with a scrape error, because prometheus.yml pointed at localhost:9100 instead of the compose service name:

Get "http://localhost:9100/metrics": dial tcp 127.0.0.1:9100: connect: connection refused

Why: Inside the Prometheus container, localhost is the Prometheus container itself — not the node_exporter container. Nothing listens on 9100 there, so the scrape is refused. Compose puts every service on a shared network where each is reachable by its service name.

Fix: Target the service name, as in step 1: targets: ['node-exporter:9100']. Reload with docker compose restart prometheus, then recheck the Targets page.

How you’d spot it in prod: A target stuck DOWN with connection refused is the classic container-networking mistake — localhost inside a container means that container. The same trap bites in Kubernetes, where you address pods by Service DNS, never localhost.

Common error: A Grafana panel using rate() with a window as short as the scrape interval shows No data, even though the metric exists:

(panel renders empty; the query rate(node_cpu_seconds_total[15s]) returns no samples)

Why: rate() needs at least two samples inside its window to measure a change. With a 15s scrape interval and a [15s] window, a given window often contains only one sample, so rate() has nothing to compute and returns an empty result — which Grafana renders as “No data”.

Fix: Make the range at least four times the scrape interval — [1m] or [5m] for a 15s scrape. As a rule, size rate windows generously; longer windows also smooth out spikes.

How you’d spot it in prod: Panels that flicker between a value and “No data”, or dashboards that look empty right after a metric appears, usually have a rate window too tight for the scrape interval — widen the window before suspecting the exporter.

Prometheus & Grafana Interview Questions

Section 4 of 5 · ~1 min

Cover the answers below and say your own version out loud first — explain pull vs push, and what rate() does to a counter, before you reveal each answer. Recalling before revealing is what makes these stick when an interviewer asks them cold. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 5 min — Open the Prometheus expression browser at http://localhost:9090/graph, run up and node_filesystem_avail_bytes, and toggle between the Table and Graph tabs to feel the difference between an instant value and a series over time.
• 10 min — Paste today’s queries into OpsCanopy’s PromQL Explainer to see rate() and sum by () broken down clause by clause in plain English — the fastest way to make PromQL syntax click.
• 10 min — Skim the LogQL ↔ PromQL Helper to preview how tomorrow’s Loki log queries mirror the PromQL you learned today — the same rate() and sum by shapes, applied to log lines instead of metrics.
• 5 min — In Grafana, add a second panel for node_memory_MemAvailable_bytes and set its unit to bytes, so you have a two-panel dashboard: one counter rate, one gauge. Screenshot it — it’s your first real dashboard.