Observability 2 — alerts, SLOs & log aggregation (Loki)

From signals to action: alerts, SLOs, and log aggregation

Section 1 of 5 · ~3 min

Day 60 gave you metrics and dashboards. But nobody watches a dashboard at 3am. The point of observability is that the system tells you when something is wrong and lets you prove how wrong. That is what alerting, SLOs, and logs add.

Alerting rules live in Prometheus: a PromQL expression plus a duration. The expr is evaluated every scrape, and when it stays true for the for: window the alert moves from pending to firing. Prometheus doesn’t notify anyone itself — it hands firing alerts to Alertmanager, which does the human-facing work. Alertmanager groups related alerts into one message, routes each to a receiver by matching labels (severity="critical" to PagerDuty, warnings to Slack), and can silence or inhibit noise during a known outage. A routing tree is match rules read top-down, so label matchers decide who gets paged.

SLIs, SLOs, and error budgets stop you alerting on everything. An SLI is a measured ratio — the fraction of requests served under 300ms. An SLO is the target you promise: 99.9% over 30 days. The gap to 100% is the error budget — the failure you’re allowed, about 43 minutes a month at 99.9%. You alert when you’re burning that budget fast, not on every blip. Google’s SRE teams pioneered this: budget spent, you stop shipping features and fix reliability; budget healthy, you can take risks. It turns “is it broken?” into a number engineers and product both act on.

Real world: An error budget is your monthly mobile-data allowance. 99.9% uptime is like 20 GB — a little failure is expected, that’s what the budget is for. A 2am page for one slow request is a buzz every time you open an app; you’d mute it. You only want the alert that says “at this rate you’ll blow the whole allowance by Thursday” — that’s burn-rate alerting.

Log aggregation with Loki + LogQL

Metrics tell you that something broke; logs tell you why. Loki is a log store built to sit beside Prometheus. Instead of indexing the full text of every line (expensive, like Elasticsearch), it indexes only a few labels — {container="api", level="error"} — and keeps the raw log compressed. That makes it cheap to run and fast to filter. An agent ships the logs: Grafana Alloy (the current collector, which replaced Promtail) tails container and file logs and pushes them to Loki’s HTTP API.

You query Loki with LogQL, which mirrors PromQL. A query starts with a label selector — {container="api"} — then pipes filters: |= "error" != "healthcheck". You can even turn logs into metrics: sum(rate({container="api"} |= "error" [5m])) graphs the error rate from log lines. Because metrics and logs share the same labels and the same Grafana UI, you pivot from a spiking metric to the exact logs behind it without leaving the page.

Today you wire it all locally with docker compose: Prometheus firing an alert into Alertmanager, Alloy shipping logs into Loki, and Grafana querying both.

Hands-On Lab

Section 2 of 5 · ~4 min

Budget about 30 minutes. This runs entirely on your laptop via docker compose — no cloud account, no bill. Open your WSL2 Ubuntu 24.04 terminal with Docker 27+ and jq installed (sudo apt-get install -y jq). Everything binds to localhost; nothing is exposed to the internet. Container IDs, timestamps and log lines are unique to each run — yours will differ from the samples.

# 1. Make a project folder and write the compose file — five local services.
mkdir -p ~/m90-obs && cd ~/m90-obs
cat > docker-compose.yml <<'EOF'
services:
  prometheus:
    image: prom/prometheus:v3.1.0
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./rules.yml:/etc/prometheus/rules.yml
  alertmanager:
    image: prom/alertmanager:v0.28.0
    ports: ["9093:9093"]
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
  loki:
    image: grafana/loki:3.3.2
    ports: ["3100:3100"]
  alloy:
    image: grafana/alloy:v1.5.1
    command: run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy
    volumes:
      - ./config.alloy:/etc/alloy/config.alloy
      - /var/run/docker.sock:/var/run/docker.sock
  grafana:
    image: grafana/grafana:11.4.0
    ports: ["3000:3000"]
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/ds.yml
EOF
docker compose config -q && echo "compose valid"
# Output:
# compose valid

# 2. Prometheus config: scrape itself, evaluate rules, send alerts to Alertmanager.
cat > prometheus.yml <<'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s
rule_files:
  - /etc/prometheus/rules.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
EOF
echo "wrote prometheus.yml"
# Output:
# wrote prometheus.yml

# 3. Alerting rules: one demo alert that always fires, one real "target down" rule.
cat > rules.yml <<'EOF'
groups:
  - name: demo
    rules:
      - alert: DemoAlwaysFiring
        expr: vector(1)
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "Demo alert — proves the rule to Alertmanager path works"
      - alert: TargetDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "A scrape target has been down for 1 minute"
EOF
echo "wrote rules.yml"
# Output:
# wrote rules.yml

# 4. Alertmanager: route critical alerts to one receiver, everything else to default.
cat > alertmanager.yml <<'EOF'
route:
  receiver: default
  group_by: [alertname]
  group_wait: 5s
  routes:
    - matchers:
        - severity="critical"
      receiver: critical
receivers:
  - name: default
  - name: critical
EOF
echo "wrote alertmanager.yml"
# Output:
# wrote alertmanager.yml

# 5. Alloy: discover running containers over the Docker socket and ship their logs to Loki.
cat > config.alloy <<'EOF'
discovery.docker "all" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "add_name" {
  targets = discovery.docker.all.targets
  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label  = "container"
  }
}

loki.source.docker "default" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.add_name.output
  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}
EOF
echo "wrote config.alloy"
# Output:
# wrote config.alloy

# 6. Grafana: provision the Prometheus and Loki data sources so both are ready on boot.
cat > grafana-datasources.yml <<'EOF'
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
EOF
echo "wrote grafana-datasources.yml"
# Output:
# wrote grafana-datasources.yml

# 7. Start everything locally. The first run pulls images; give it a minute.
docker compose up -d
docker compose ps
# Output (all five services Up — your container IDs and uptimes will differ):
# NAME                     IMAGE                    STATUS         PORTS
# m90-obs-alertmanager-1   prom/alertmanager:...    Up 20 seconds  0.0.0.0:9093->9093/tcp
# m90-obs-alloy-1          grafana/alloy:...        Up 20 seconds  12345/tcp
# m90-obs-grafana-1        grafana/grafana:...      Up 20 seconds  0.0.0.0:3000->3000/tcp
# m90-obs-loki-1           grafana/loki:...         Up 20 seconds  0.0.0.0:3100->3100/tcp
# m90-obs-prometheus-1     prom/prometheus:...      Up 20 seconds  0.0.0.0:9090->9090/tcp

# 8. Ask Prometheus which alerts are active — the demo rule fires within a scrape or two.
curl -s localhost:9090/api/v1/alerts \
  | jq '.data.alerts[] | {alertname: .labels.alertname, severity: .labels.severity, state: .state}'
# Output (DemoAlwaysFiring is firing; TargetDown stays inactive while the target is up):
# {
#   "alertname": "DemoAlwaysFiring",
#   "severity": "warning",
#   "state": "firing"
# }

# 9. Confirm Alertmanager received and grouped the alert, routed by its severity label.
curl -s localhost:9093/api/v2/alerts \
  | jq '.[] | {alertname: .labels.alertname, receivers: [.receivers[].name], state: .status.state}'
# Output (the warning landed in the default receiver; may take a scrape or two to appear):
# {
#   "alertname": "DemoAlwaysFiring",
#   "receivers": ["default"],
#   "state": "active"
# }

# 10. Query Loki with LogQL for recent Prometheus-container lines (Alloy shipped them).
curl -s -G localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={container="m90-obs-prometheus-1"} |= "level"' \
  --data-urlencode 'limit=2' | jq -r '.data.result[].values[][1]'
# Output (raw log lines; timestamps and messages will differ):
# time=2026-07-11T09:22:14.501Z level=INFO source=main.go:1213 msg="Server is ready to receive web and API requests."
# time=2026-07-11T09:22:14.498Z level=INFO source=head.go:723 msg="Replaying on-disk memory mappable chunks if any"

Now open http://localhost:3000 — anonymous admin is on, so you land straight in. Go to Explore, pick the Loki data source, and paste the same LogQL: {container=~"m90-obs-.+"} |= "level". Switch to the Prometheus source and run vector(1) or up to see the metric side. Same UI, same label model, both signals in one place. Then tear it down:

# 11. Tear it all down — containers, network, and Loki's volume — so nothing lingers.
docker compose down -v
# Output:
# [+] Running 6/6
#  ✓ Container m90-obs-grafana-1       Removed
#  ✓ Container m90-obs-alloy-1         Removed
#  ✓ Container m90-obs-loki-1          Removed
#  ✓ Container m90-obs-alertmanager-1  Removed
#  ✓ Container m90-obs-prometheus-1    Removed
#  ✓ Network m90-obs_default           Removed

Read it back: the rule fired in Prometheus, Alertmanager grouped and routed it by label, and the same log lines you pulled over the API showed up in Grafana next to the metrics — detection, notification, and the “why” all in one local stack.

Common Errors & Fixes

Section 3 of 5 · ~2 min

These three catch almost everyone wiring alerts and logs for the first time. Read the error text slowly — parsing it is the actual skill.

Common error: Alloy starts but no container logs ever reach Loki, and Alloy’s own log shows:

level=error msg="error creating docker client" component=loki.source.docker.default err="permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock"

Why: Alloy discovers and tails other containers’ logs through the Docker socket. If the socket isn’t mounted into the Alloy container — or the Alloy process can’t read it — discovery finds nothing and no logs flow to Loki.

Fix: Mount the socket and confirm the path: - /var/run/docker.sock:/var/run/docker.sock on the alloy service (it’s already in the compose above). On a locked-down host you may need to grant Alloy a group that can read the socket, or run rootless Docker’s socket path instead.

How you’d spot it in prod: Metrics are flowing but logs are silent for a whole node — suspect the log agent, not the app. Check the agent’s own logs first; a socket or file-permission error there explains an empty Loki.

Common error: Running a LogQL query with an empty or unbounded selector — {} in Grafana Explore or over the API:

parse error at line 1, col 1: queries require at least one regexp or equality matcher that does not have an empty-compatible value

Why: Loki refuses a stream selector that would match every stream, because with no index-narrowing label it would have to scan all data at once. At least one label matcher must pin the query to a real, non-empty value.

Fix: Add a concrete matcher — {container="m90-obs-prometheus-1"} — or a bounded regex like {container=~"m90-obs-.+"}, then chain line filters (|= "error") to narrow further within that stream.

How you’d spot it in prod: A dashboard panel or log alert that errors with “at least one matcher” is a LogQL query missing its label selector — usually someone pasted only the line-filter half. Put the label selector back at the front.

Common error: Alertmanager exits immediately and its container falls into a restart loop after a config edit:

level=error msg="Loading configuration file failed" err="undefined receiver \"critical\" used in route"

Why: The routing tree sends severity="critical" alerts to a receiver named critical, but no receiver with that name exists under receivers:. Alertmanager validates that every receiver a route references is defined, and refuses to start on a mismatch.

Fix: Add the missing receiver under receivers: — even an empty - name: critical is valid — then restart the service. Receiver names are case-sensitive and must match the route’s receiver: exactly.

How you’d spot it in prod: Alertmanager crash-looping right after a config change is almost always a validation error, and the message names the exact problem — a receiver, template, or matcher that doesn’t line up. Run amtool check-config before you deploy to catch it first.

Alerting & Observability Interview Questions

Section 4 of 5 · ~1 min

Cover the answers below and say your own version out loud first — name what an alerting rule does versus Alertmanager, and what an error budget buys you, before you reveal each answer. Recalling before revealing is what makes these stick when an interviewer asks them cold. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 5 min — Paste your alertmanager.yml route tree and a test alert’s labels into the Alertmanager route tester to see exactly which receiver each alert lands in.
• 10 min — Use the LogQL ↔ PromQL helper to translate a log-to-metric LogQL query into its PromQL equivalent and back, so the shared label model clicks.
• 5 min — Drop your alert rule’s expr into the PromQL explainer for a plain-English breakdown of exactly what it matches.
• 10 min — Read Google’s SRE Workbook chapter Implementing SLOs for how real teams pick objectives and turn error budgets into burn-rate alerts.