Writing
Notes from the canopy.
Observations on DevOps tooling, observability, and the small gaps in the ecosystem worth filling — written for the engineers who hit them.
23posts and counting.
OpsCanopy7 min read
failed to solve: did not complete successfully: exit code 1
BuildKit tells you which line failed but not why. How to read the error, get the real output back, and fix the four structural causes behind most of them.
- docker
- buildkit
- ci
- dockerfile
OpsCanopy6 min read
OOMKilled and exit code 137: why Kubernetes killed your pod
Exit code 137 means the kernel killed your container for exceeding its memory limit. How to confirm it, and why raising the limit is usually the wrong fix.
- kubernetes
- memory
- limits
- debugging
OpsCanopy9 min read
x509: certificate signed by unknown authority — the fix
What the error actually means, the four causes ranked by how often they bite, and the one-line openssl check that shows whether an intermediate is missing.
- tls
- certificates
- docker
- go
- security
OpsCanopy7 min read
Learn DevOps in 90 days: the incident-first path
A free day-by-day path from Linux to Kubernetes with playable incident missions. The full plan, the design decisions, and the honest tradeoffs.
- devops
- career
- learning
OpsCanopy12 min read
Why isn't my alert reaching the right receiver?
Alerts going to the wrong receiver, or none at all? Debug Alertmanager routing — first-match-wins, missing continue, matcher regex and catch-all defaults.
- alertmanager
- observability
- alerting
OpsCanopy10 min read
Alertmanager matchers, grouping and route inheritance
Alertmanager route config reference: the matchers, match and match_re forms, anchored regexes, the four grouping fields, and what a child route inherits.
- alertmanager
- observability
- alerting
OpsCanopy9 min read
Why Did Prometheus Drop My Target? Debugging relabel_configs
A target vanished or a label disappeared after relabeling. Debug Prometheus relabel_configs vs metric_relabel_configs, regex anchoring and keep/drop logic.
- prometheus
- observability
- relabeling
OpsCanopy8 min read
Why your GitHub Actions "if" condition always runs
GitHub Actions if condition always running true? It is the literal-text footgun: any text outside ${{ }} coerces to a truthy string. The cause and the fix.
- github-actions
- ci-cd
OpsCanopy7 min read
Why your GitHub Actions workflow didn't trigger
Branch name mismatches, the AND-semantics of branches + paths, the ** glob requirement and paths-ignore on pull_request — with the fix for each.
- github-actions
- ci-cd
- debugging
OpsCanopy10 min read
Prometheus Relabel Actions: keep, labelmap, hashmod
A field reference for all 11 Prometheus relabel actions — replace, keep, drop, labelmap, labeldrop, hashmod, keepequal, lowercase — each with an example.
- prometheus
- observability
- relabeling
OpsCanopy9 min read
7 Common .gitlab-ci.yml Mistakes (and How to Catch Them)
The .gitlab-ci.yml mistakes that turn pipelines red: undefined stages, jobs without scripts, broken needs and rules, anchor misuse — each with a fix.
- gitlab-ci
- ci-cd
- yaml
OpsCanopy9 min read
How to Validate .gitlab-ci.yml Before You Push
Stop pushing broken pipelines. Validate your .gitlab-ci.yml for YAML and structural errors in your browser — before the commit, not after the red pipeline.
- gitlab-ci
- ci-cd
- yaml
OpsCanopy10 min read
When to Use Docker Compose Instead of docker run
Which one to reach for, and what actually changes when you switch — default network, restart policy, detached mode — plus Compose back to a run line.
- docker
- docker-compose
- containers
OpsCanopy10 min read
How to Convert a docker run Command to docker-compose.yml
Convert any docker run command to a docker-compose.yml service, flag by flag — ports, volumes, environment, restart and more. A practical, copy-paste guide.
- docker
- docker-compose
- containers
OpsCanopy5 min read
How to read a PromQL query
A PromQL query reads inside out, not left to right. The four layers — selectors, ranges, functions and aggregations — so you can decode any expression fast.
- promql
- prometheus
- observability
OpsCanopy5 min read
LogQL vs PromQL: the same query in both languages
LogQL borrows PromQL’s shape but starts from log lines, not metrics. How the two query languages line up, where they translate cleanly, and where they don’t.
- logql
- promql
- observability
OpsCanopy6 min read
Stop shipping a stale .env.example
Your .env.example is documentation that silently rots. Why env drift breaks onboarding and deploys, how to detect missing and unused keys, and how to fix it.
- configuration
- developer-experience
- twelve-factor
OpsCanopy5 min read
Writing robust regular expressions for log lines
Building regexes that parse log lines reliably — anchoring, capture groups, escaping, greediness, and the failure modes that bite you in production.
- regex
- logs
- parsing
OpsCanopy5 min read
Migrating from cron to systemd timers
Converting crontab entries into systemd .timer and .service units — OnCalendar syntax, logging, randomized delays, catch-up runs and the migration gotchas.
- systemd
- cron
- linux
OpsCanopy6 min read
Reading cron expressions: a field-by-field guide
A field-by-field guide to reading cron expressions — the five time fields, ranges, steps, lists and @macros — plus the gotchas that misfire schedules.
- cron
- scheduling
- devops
OpsCanopy6 min read
The GitHub Actions security mistakes linters miss
YAML validators catch syntax, not exposure. Five high-impact GitHub Actions misconfigurations — pull_request_target, script injection and more — with fixes.
- github-actions
- security
- ci-cd
OpsCanopy5 min read
One CVE, four ignore files: Trivy, Grype, Snyk, osv-scanner
One triaged suppression, re-encoded in .trivyignore, .grype.yaml, .snyk and osv-scanner.toml — what maps cleanly between them, and what is lossy.
- security
- vulnerability-management
- devops
OpsCanopy5 min read
Unit Testing Loki Alert Rules: the gap promtool leaves
Prometheus has promtool test rules. Loki has nothing equivalent. Why testing LogQL alert rules matters, what a Loki rule unit test looks like, and how to do it.
- loki
- observability
- testing