Pipeline security — secrets management, image scanning (Trivy), supply-chain basics

Keeping secrets out and scanning what you ship

Section 1 of 5 · ~3 min

For four days your pipeline has built, tested and shipped images. It works — but right now it probably trusts too much and checks too little. Today closes both gaps: keep secrets out of the pipeline, and scan what the pipeline ships.

Secrets never live in the file. A workflow is committed to the repo, so anything you type into it — an API key, a cloud password, a token — lands in git history forever, readable by anyone who can see the repo or a fork, and deleting it later doesn’t erase the history. The fix is GitHub Actions secrets: encrypted values stored in repo settings, injected at run time as ${{ secrets.NAME }} and masked if they ever hit a log. Better still, for cloud access don’t store a key at all. OIDC (OpenID Connect) lets the runner prove its identity to the cloud and receive a short-lived token for that one run — no long-lived AWS key sitting in settings to leak or rotate. Same lesson as yesterday’s GITHUB_TOKEN: short-lived beats long-lived.

Two habits harden the pipeline itself. Least privilege: set permissions: explicitly (contents: read, plus only what a job truly needs) so a compromised step can’t do more than its task. And pin your actions — uses: actions/checkout@v4 follows a moving tag, so whoever controls that tag controls code running with your secrets. Pinning to a full commit SHA freezes exactly the code you reviewed.

Scan what you ship: images and the supply chain

Your image is a base image plus dependencies you didn’t write, and those carry known vulnerabilities (CVEs). Trivy, the open-source scanner from Aqua Security, reads an image and reports every known CVE in its OS packages and app libraries, each with a severity and the version that fixes it. Run trivy image myapp:tag locally, or add it as a CI step that fails the build on HIGH/CRITICAL findings — a quality gate for security, exactly like the test gate from Day 36.

Two supply-chain basics finish the job. An SBOM (Software Bill of Materials) is a machine-readable inventory of everything in your image, so when the next Log4j-style CVE drops you can answer “are we affected?” in seconds. And dependency pinning — exact versions in requirements.txt, digests for base images — makes builds reproducible and stops an attacker slipping a new version in under a floating range.

Real world: Secrets management is a hotel key card, not a house key. The house key — a long-lived cloud key — opens everything forever, and losing it means re-keying the whole house. The card — OIDC or GITHUB_TOKEN — is minted at check-in, opens only your room, and dies at checkout; lose it and it’s already worthless. Trivy is the metal detector at the door: it doesn’t assume the bags are clean because they came from a known supplier, it scans every one.

Put it together: nothing secret in the file, short-lived tokens over stored keys, pinned actions with least-privilege permissions, and every image scanned before it ships. That’s a pipeline you can hand production.

Hands-On Lab

Section 2 of 5 · ~3 min

Budget about 25 minutes. Work in your WSL2 Ubuntu 24.04 terminal, in the ~/dockerfile-lab repo from Day 34 with gh authenticated. You’ll store a secret without ever typing it into a file, scan an image with Trivy and turn the scan into a gate, then pin an action and your dependencies. Scanner patch versions, CVE IDs, counts and component totals move as the vulnerability DB updates — yours will differ from the samples below.

# 1. Store a secret in GitHub — encrypted, never typed into the workflow file.
cd ~/dockerfile-lab
echo -n "super-secret-value" | gh secret set MY_API_KEY
gh secret list
# Output (the value is never shown again — the timestamp is yours-will-differ):
# NAME        UPDATED
# MY_API_KEY  about 1 minute ago

Save this as .github/workflows/secure.yml. The secret is injected at run time, the token is scoped down, and the action is pinned to a full commit SHA — not a moving @v4 tag:

# .github/workflows/secure.yml
name: secure
on:
  push:
    branches: [main]
permissions:
  contents: read            # least privilege — only what this job needs
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      # pinned to a commit SHA (the comment records which version it is)
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - name: Use the secret without printing it
        run: curl -sS -H "Authorization: Bearer $API_KEY" https://api.example.com/ping
        env:
          API_KEY: ${{ secrets.MY_API_KEY }}

# 2. Install Trivy from Aqua's script, then confirm the version.
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sudo sh -s -- -b /usr/local/bin
trivy --version
# Output (patch version and DB timestamp are yours-will-differ):
# Version: 0.58.1

# 3. Scan a public image for known CVEs. The first run downloads the vuln DB.
trivy image --severity HIGH,CRITICAL python:3.12-slim
# Output (CVE IDs, counts and the fixed-in versions are yours-will-differ):
# python:3.12-slim (debian 12.x)
# Total: 4 (HIGH: 3, CRITICAL: 1)
# ┌──────────┬──────────────────┬──────────┬───────────┬───────────┐
# │ Library  │ Vulnerability    │ Severity │ Installed │ Fixed In  │
# ├──────────┼──────────────────┼──────────┼───────────┼───────────┤
# │ libssl3  │ CVE-20XX-XXXXX   │ CRITICAL │ 3.0.14-1  │ 3.0.15-1  │
# └──────────┴──────────────────┴──────────┴───────────┴───────────┘

# 4. Turn the scan into a gate: exit 1 if any HIGH/CRITICAL is found.
trivy image --severity HIGH,CRITICAL --exit-code 1 --quiet python:3.12-slim
echo "exit code: $?"
# Output (non-zero while findings exist — the count is yours-will-differ):
# exit code: 1

Add the scan to CI so a vulnerable image can never ship. Save this as .github/workflows/scan.yml:

# .github/workflows/scan.yml
name: scan
on: [push]
permissions:
  contents: read
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - name: Build image
        run: docker build -t app:${{ github.sha }} .
      - name: Trivy scan (gate)
        uses: aquasecurity/trivy-action@6c175e9c4083a92bbca2f9724c8a5e33bc2d97a5  # 0.30.0
        with:
          image-ref: app:${{ github.sha }}
          severity: HIGH,CRITICAL
          exit-code: '1'          # without this the scan reports but never fails

# 5. Generate an SBOM — a full inventory of what's inside the image.
trivy image --format cyclonedx --output sbom.json python:3.12-slim
jq '.components | length' sbom.json
# Output (component count is yours-will-differ):
# 142

# 6. Pin your app dependencies to exact versions so builds are reproducible.
pip freeze > requirements.txt
cat requirements.txt
# Output (your versions differ — note the == exact pins, never >= ranges):
# flask==3.1.0
# gunicorn==23.0.0
# werkzeug==3.1.3

# 7. Commit the hardened workflows and the pinned deps together.
git add .github/workflows/secure.yml .github/workflows/scan.yml requirements.txt
git commit -q -m "ci: secrets, Trivy scan gate, pinned action + deps"
git log --oneline -1
# Output (your commit SHA will differ):
# 7d2e4f1 (HEAD -> main) ci: secrets, Trivy scan gate, pinned action + deps

Read the last steps back: you set a secret GitHub encrypts and masks, referenced it without ever writing the value in a file, pinned the checkout action to an exact commit and scoped the token to read-only, then scanned an image, turned that scan into a build-failing gate, produced an SBOM, and pinned every dependency to an exact version. Nothing secret in the repo, and nothing ships that a scanner hasn’t cleared — that’s the whole of pipeline security in one commit.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These three are the ones almost everyone hits the first time they harden a pipeline. Read the error text slowly — parsing it is the actual skill.

Common error: Committing a real key straight into a workflow (or any file) and having the push rejected by secret scanning:

remote: error: GH013: Repository rule violations found for refs/heads/main.
remote: - Push cannot contain secrets
remote:     —— AWS Access Key ID ————————————————————
remote:      locations: commit abc1234 path: .github/workflows/deploy.yml:12

Why: GitHub’s push protection scans commits for credential patterns and blocks the push when it recognizes one. The key was hardcoded instead of stored as a secret — and even if the push had gone through, the value would live in git history forever, readable from every fork and clone.

Fix: Remove the literal value, store it with gh secret set NAME (or Settings → Secrets), and reference it as ${{ secrets.NAME }}. Because it already reached your local commit, treat the key as leaked and rotate it at the provider before doing anything else — rewriting history is not enough.

How you’d spot it in prod: A push suddenly rejected with GH013 ... Push cannot contain secrets, or a “secret detected” alert on the Security tab, means a credential is in the diff. Rotate the key first, then fix the reference — don’t just force-push the history away.

Common error: A Trivy step failing to start because it can’t download the vulnerability database:

FATAL	init error: DB error: failed to download vulnerability DB:
  OCI repository error: GET https://ghcr.io/v2/aquasecurity/trivy-db/...:
  TOOMANYREQUESTS: retry-after: 300, allowed: 100/minute

Why: Trivy pulls its CVE database from a registry (ghcr.io), and unauthenticated pulls share a rate limit. On a busy CI account or a burst of parallel jobs, the anonymous quota runs out and the download is throttled — the scanner never even gets to your image, so the step dies before finding a single CVE.

Fix: Authenticate the DB pull (pass a GITHUB_TOKEN so the request isn’t anonymous), and cache the downloaded DB between runs so most jobs don’t re-fetch it. Adding TRIVY_DB_REPOSITORY mirrors or the action’s cache: true both cut the pull count sharply.

How you’d spot it in prod: A scan step that fails intermittently with TOOMANYREQUESTS — green on a quiet afternoon, red during a release rush — is a rate-limit, not a vulnerability. The tell is that it fails in init/DB download, before any findings are printed.

Common error: A Trivy scan that reports critical CVEs but leaves the job green, so vulnerable images keep shipping:

# the trivy-action step shows a ✓, yet its log ends with:
Total: 7 (HIGH: 6, CRITICAL: 1)

Why: By default Trivy reports findings and exits 0 — it does not fail the build unless you tell it to. Omit exit-code: '1' (or leave severity unfiltered) and the step succeeds no matter what it finds, so the “security scan” is decorative: everyone trusts the green check while criticals sail through to the registry.

Fix: Set exit-code: '1' together with severity: HIGH,CRITICAL so the job fails on real risk, and confirm by pushing an image you know is vulnerable and watching the run go red. A gate you haven’t seen fail isn’t a gate yet.

How you’d spot it in prod: A pipeline with a “passing” scan step whose logs still list CRITICAL CVEs — or vulnerable images reaching production despite a green scan — is a scan running without an exit code. Read the step’s log, don’t trust the check mark.

Pipeline Security Interview Questions

Section 4 of 5 · ~1 min

Secrets, scanning and supply-chain hardening are where CI/CD screening turns into security screening — a calm answer that explains why short-lived beats long-lived and why a green scan can still be unsafe beats reciting flags. Cover each answer, say your own version out loud first, then compare — recalling before revealing is what makes it stick for interview day. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 5 min — Skim GitHub’s encrypted secrets guide and note the masking rule and why secrets aren’t passed to workflows triggered from a fork.
• 10 min — Run trivy fs . on a project to scan its dependencies and misconfigurations (not just an image), then skim the Trivy docs for the config and sbom scanners.
• 15 min — Read GitHub’s OIDC hardening overview, then enable Dependabot for the github-actions ecosystem so your pinned SHAs get reviewed bump PRs automatically.