Testing in pipelines — lint, unit tests, quality gates

Why tests gate a pipeline: lint, unit tests, and quality gates

Section 1 of 5 · ~3 min

Yesterday you hardened a pipeline that builds and ships. Today it earns the right to say no. A pipeline that only builds tells you the code compiles; a pipeline that tests tells you the code still works — and, wired to branch protection, it refuses to let broken code merge. That refusal is the point: a check nobody can quietly skip is worth far more than a test suite everyone forgets to run.

Three checks run before a merge, cheapest first:

• Lint — reads your code without running it and flags style slips and likely bugs: unused imports, undefined names, unreachable branches. ruff is the current Python linter — one Rust binary that checks a large repo in milliseconds; eslint is its JavaScript counterpart.
• Unit tests — actually run small pieces of your code and assert they behave: add(2, 2) == 4. pytest runs them for Python, vitest for JS/TS. A failing assertion exits non-zero.
• Coverage — measures how much of your code the tests exercised, as a percentage of lines run.

Each check is just a command, and the rule that makes CI work is blunt: a command that exits non-zero fails the step, the step fails the job, and the job paints a red X. ruff check . exits 1 when it finds a problem; pytest exits 1 when a test fails. You never write “if failed, stop” — a non-zero exit is the stop signal.

Real world: A quality gate is the airport metal detector, not the sign asking you to empty your pockets. The sign — a linter you could run locally — is advisory, and tired travellers walk straight past it. The detector is the gate: it beeps, the belt stops, and you do not reach the plane until you pass. CI is that detector for code — every commit walks through it, the same way, and a beep blocks boarding.

Quality gates that fail the build

A quality gate turns a measurement into a must. Coverage on its own is a number in a report; add pytest --cov=calc --cov-fail-under=80 and it becomes a gate — if coverage lands below 80%, pytest exits non-zero and the build goes red, exactly like a failed test. Codecov is a hosted product built entirely around this idea: it tracks coverage per pull request and fails a status check when a change drops it.

But a red X in the Actions tab doesn’t block anything on its own — anyone can still click merge. The last link is branch protection: in the repo settings you mark the CI check as a required status check on the default branch. Now GitHub greys out the merge button until that check is green. Red genuinely blocks. That combination — a command that exits non-zero on a real problem, plus a branch rule that requires it — is what turns “we have tests” into “broken code cannot reach main.”

One test job, wired to branch protection, is the whole difference between hoping the team runs tests and knowing main is always green.

Hands-On Lab

Section 2 of 5 · ~4 min

Budget about 25 minutes. You need a GitHub account, the gh CLI authenticated (gh auth login), and Python 3.12 with pip in your WSL2 Ubuntu 24.04 terminal. You’ll build a tiny project, run lint and tests locally, wire the same checks into a workflow, then open a pull request whose broken code turns the check red. Run IDs, SHAs, tool patch versions and timings are unique to each run — yours will differ from the samples.

# 1. Make a tiny Python project with one module and one test file.
mkdir -p ~/ci-tests && cd ~/ci-tests
git init -q -b main
cat > calc.py <<'EOF'
def add(a, b):
    return a + b

def is_even(n):
    return n % 2 == 0
EOF
cat > test_calc.py <<'EOF'
from calc import add, is_even

def test_add():
    assert add(2, 2) == 4

def test_is_even():
    assert is_even(4)
    assert not is_even(3)
EOF
ls
# Output:
# calc.py  test_calc.py

# 2. Pin the dev tools the pipeline will use, then install them.
cat > requirements-dev.txt <<'EOF'
ruff
pytest
pytest-cov
EOF
# Ubuntu 24.04's system Python is externally-managed — a bare pip install
# errors with "externally-managed-environment". Work inside a venv instead:
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt
# Output (tool patch versions are yours-will-differ):
# Successfully installed pytest-8.x.x pytest-cov-5.x.x ruff-0.x.x ...

# 3. Lint first — the cheapest check. A clean run says little and exits 0.
ruff check .
# Output:
# All checks passed!

# 4. Run the tests with a coverage gate: fail if under 80% of calc.py runs.
pytest --cov=calc --cov-fail-under=80
# Output (durations and exact percent are yours-will-differ):
# test_calc.py ..                                          [100%]
# ---------- coverage: platform linux, python 3.12 ----------
# Name       Stmts   Miss  Cover
# ------------------------------
# calc.py        4      0   100%
# ------------------------------
# Required test coverage of 80% reached. Total coverage: 100.00%
# ======================= 2 passed in 0.03s =======================

Save this workflow as .github/workflows/test.yml. It runs the same two commands you just ran, in order — lint, then the tests behind the coverage gate:

# .github/workflows/test.yml
name: test
on:
  push:
  pull_request:
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - name: Install dev dependencies
        run: pip install -r requirements-dev.txt
      - name: Lint with ruff
        run: ruff check .
      - name: Run tests with coverage gate
        run: pytest --cov=calc --cov-fail-under=80

# 5. Commit everything and publish the repo — the push fires the test workflow.
mkdir -p .github/workflows   # (save the YAML above into it first)
git add .
git commit -q -m "Add calc + CI test job"
gh repo create ci-tests --public --source=. --push
# Output (your account name replaces "pushkar"):
# ✓ Created repository pushkar/ci-tests on github.com
# ✓ Pushed commits to https://github.com/pushkar/ci-tests.git

# 6. Watch the run go green — lint, then the tests, then the coverage gate.
gh run watch
# Output (run/job IDs and timings are yours-will-differ):
# ✓ main test · 17420993312
# Triggered via push about 15 seconds ago
#
# JOBS
# ✓ test in 23s
#   ✓ Set up job
#   ✓ actions/checkout@v4
#   ✓ Set up Python
#   ✓ Install dev dependencies
#   ✓ Lint with ruff
#   ✓ Run tests with coverage gate
#
# ✓ Run test (17420993312) completed with 'success'

# 7. On a branch, break a test on purpose, then open a pull request.
git checkout -q -b bad-change
sed -i 's/return a + b/return a - b/' calc.py   # add() now subtracts
git commit -qam "Change add()"
git push -q -u origin bad-change
gh pr create --fill
# Output:
# https://github.com/pushkar/ci-tests/pull/1

# 8. The PR ran the same job — the broken assertion turns the check red.
gh pr checks
# Output (the test check is failing; yours-will-differ IDs shortened):
# test    fail    12s    https://github.com/pushkar/ci-tests/actions/runs/...

# 9. Read WHY it's red — pytest prints the exact failing assertion.
gh run view $(gh run list -L1 --json databaseId -q '.[0].databaseId') --log-failed | grep -A3 "test_add"
# Output:
# >       assert add(2, 2) == 4
# E       assert 0 == 4
# E        +  where 0 = add(2, 2)

Read the last steps back to yourself: you wrote one test job, and on the green push it ran lint, tests and a coverage gate untouched by hand. Then a pull request with a one-line bug turned that same job red — and the log named the failing assertion. Add a required status check on main (Go Deeper below) and that red X greys out the merge button: broken code physically cannot land. That is the whole job of tests in a pipeline — not to run, but to block.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These three catch nearly everyone the first time a workflow runs lint and tests. Read the error text slowly — parsing it is the real skill.

Common error: The lint or test step dies immediately because the tool was never installed on the runner:

Run ruff check .
/home/runner/work/_temp/abc123.sh: line 1: ruff: command not found
Error: Process completed with exit code 127.

Why: A hosted runner starts with only the base image plus whatever setup-python added — it does not have ruff, pytest, or any pip package until a step installs them. exit code 127 is the shell’s specific signal for “command not found,” distinct from a test that ran and failed (which is exit 1).

Fix: Add an install step — run: pip install -r requirements-dev.txt — before the lint and test steps, and make sure actions/setup-python runs first so pip points at the right interpreter. Re-push; the tools are now on PATH.

How you’d spot it in prod: A brand-new or just-edited test job that fails on its first tool invocation with command not found / exit 127 — while the same command works on your laptop — is a missing install step on the clean runner, not a bug in the code.

Common error: Every test passes, yet the build still goes red because the coverage gate wasn’t met:

======================= 2 passed in 0.04s =======================
FAIL Required test coverage of 80% not reached. Total coverage: 62.50%
Error: Process completed with exit code 1.

Why: --cov-fail-under=80 is a quality gate, not a test. When measured coverage falls below the threshold, pytest exits non-zero even though every assertion passed — the build is red because of the gate, not a broken test. The “2 passed” line right above the failure is exactly what sends people hunting for a failing test that doesn’t exist.

Fix: Add tests for the untested lines until you clear the bar (the coverage report’s Miss column shows which lines never ran). If the threshold is genuinely too high for now, lower --cov-fail-under deliberately and agree it as a team — don’t delete the gate to make the red go away.

How you’d spot it in prod: A pipeline that reports “all tests passed” and then fails with “coverage not reached” is the coverage gate doing its job. Read the coverage line, not just the pass/fail counts, before you assume a test broke.

Common error: A pull request shows a failing check, but the merge button is still green and broken code lands on main:

$ gh pr checks
test    fail    12s    https://github.com/pushkar/ci-tests/actions/runs/...
# ...yet the PR page still offers "Merge pull request"

Why: A failing CI run only publishes a status — it does not block a merge by itself. Unless the check is added as a required status check under branch protection for the target branch, GitHub lets a maintainer merge straight over the red X. The pipeline is advisory until a branch rule makes it mandatory.

Fix: In Settings → Branches, add a protection rule on main that requires the test status check to pass before merging. Once required, the merge button is disabled until the check is green — the red X finally blocks.

How you’d spot it in prod: Broken code reaching main even though CI clearly failed on the PR. The tell is a merge that completed while a check was red — the check was never actually marked required, so nothing enforced it.

Testing in Pipelines Interview Questions

Section 4 of 5 · ~1 min

These four are what an interviewer asks to check you grasp that a pipeline’s real job is to say no, not just to run — cover each answer, say your own version out loud first, then compare, because recalling before revealing is what makes it stick for interview day. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 5 min — Open the failing PR’s Checks tab in the browser, expand the red Run tests step, and read the same pytest failure gh printed — one click from the merge button it should be blocking.
• 10 min — Skim the ruff rules reference, add a [tool.ruff] block to a pyproject.toml selecting a rule set, and re-run ruff check . to see the gate tighten.
• 15 min — Add a branch protection rule on main requiring the test check, then try to merge the bad PR and watch the merge button stay disabled until you push a fix that turns the check green.