Phase 2 · CONTAINERS & CI/CD
Week 3 review + The Locked File
By the end of today
- Consolidate week 3 into one write, version, schedule automation loop
- Debug a silently failing scheduled job across permissions, git and cron
- Explain why chmod +x and cron environments trip up deploys
The Week-3 muscle: script it, version it, schedule it
Week 1 was one box. Week 2 was the network between boxes. Week 3 taught you to make a box do work without you standing over it — and today those six days snap into one reflex you’ll lean on for the rest of Phase 2.
The automation chain has three links, and you now own each one:
Write it. Days 15–16 turned your one-liners into real scripts: variables, conditionals and loops, then functions and exit codes wrapped around a working backup script. A script is just a file of commands the shell runs top to bottom — but it only runs as a program if its execute bit is set (Day 5’s chmod +x).
Version it. Days 17–18 put that script under git: branches, the merge-vs-rebase decision, then pull requests, reviews, tags and releases. Git doesn’t only track what the script says — it tracks whether the file is executable, and it remembers who changed either one.
Schedule it. Day 19 handed the script to a scheduler so it runs unattended — the classic five-field crontab, and the modern systemd timer that logs to journalctl and survives reboots. Day 20 then drilled all of Phase 1 for interviews.
Put the three together and you’ve built what every deploy pipeline is underneath: a versioned script, fired on a schedule, running with nobody watching. Which is exactly why it fails in ways a command you type never does. When an unattended job silently stops working, you don’t guess — you walk its chain of custody and stop at the first link that’s broken:
Real world: A night delivery is scheduled to reach the warehouse at 3 a.m. every day. The van shows up on time, the cargo is fine — but the driver’s gate badge was quietly deactivated during a security review. Nothing is “broken”: the schedule fired, the van ran, the goods exist. The gate just won’t open, and the only trace is a one-line “access denied” in a log nobody reads until the shelves are empty. A lost execute bit on a deploy script is exactly that badge.
A named example makes it concrete. The five-field cron syntax from Day 19 didn’t stay on the server: GitHub Actions reuses the exact same expression in a workflow’s on: schedule trigger, so 0 3 * * * means 3 a.m. daily in your crontab and in the cloud. And the failures rhyme — a scheduled workflow that never fires, or dies on a missing secret, gets debugged the same three ways: did it run, did it have permission, and what changed in git since it last worked. Learn the walk on a laptop cron job today and it transfers straight to the pipelines you build for the rest of this phase.
That chain of custody is why today is a mission, not a lecture. Reading about a broken deploy doesn’t build the reflex — chasing one does. The Locked File hands you a deploy script that simply won’t run, three suspects with alibis, and the week-3 tools to corner the real culprit.
Hands-On Lab
Today the lab is the mission. No WSL2, no setup, nothing to paste — The Locked File runs entirely in your browser.
This is your Week-3 boss fight, and the opening case of Phase 2. The nightly deploy hasn’t shipped for two days. The cron entry is still there, the script is still on disk, and everyone swears they never touched it — permissions, git and cron each have an alibi, and exactly one of them is lying. Everything you need is something you already learned in days 15–20, so play it as the walk from the diagram:
- Run it by hand first — does
./deploy.sheven execute, or does the shell refuse it? The error text tells you which link to suspect (days 15–16). - Check permissions —
ls -lthe script and read the mode bits. A missing execute bit is aPermission denied, not a broken script;chmod +xis the fix (day 5’s rule, day 15’s scripts). - Check git history —
git logandgit blamethe deploy script. Git records the execute bit, so a mode-only change points straight at the commit that locked the file (days 17–18). - Check the schedule and its environment — confirm the
crontabentry fires and that it runs with the PATH and working directory the script assumes, not your interactive shell’s (day 19).
Type help in the terminal to see the supported commands, and hint if you stall — it nudges without solving. There’s no penalty for poking around; the whole point is to run the run → permissions → git → schedule walk with your hands until it’s reflex, and to feel why a job that “worked yesterday” can die with nothing in the code changed. Get the deploy shipping again — then, if you want the bragging rights, replay it and try to corner the culprit in fewer commands.
When the deploy goes green, come back and note below which alibi fooled you — that reflection is where the lesson sticks.
Common Errors & Fixes
These are the mistakes that trip people up when they run the week-3 automation chain for real on Ubuntu 24.04 — the same links the mission rehearses. Read the error text slowly; parsing it is the skill.
Common error: Running a script directly after its execute bit was lost —
./deploy.sh— on a file you can still open and read:-bash: ./deploy.sh: Permission deniedWhy: The kernel only runs a file directly when its execute bit is set. Without it the mode is
-rw-r--r--(644): you can read and edit the script, but you cannot execute it, so the shell reportsPermission deniedrather thancommand not found— the file is right there, it just isn’t runnable.Fix: Set the bit with
chmod +x deploy.sh, then./deploy.shworks;ls -lshould show-rwxr-xr-x. As a one-off you can also runbash deploy.sh, which ignores the bit because you’re handing the file to an interpreter explicitly.How you’d spot it in prod: A deploy or cron log showing
Permission deniedon a script that ran fine yesterday is almost never a broken script — it’s a lost execute bit. Checkls -lbefore you touch a single line of the code.
Common error: A deploy script that “broke” with no code change —
git diffon it shows only a mode flip:diff --git a/deploy.sh b/deploy.sh old mode 100755 new mode 100644Why: Git stores each file’s mode and records the execute bit —
100755for executable,100644for not. An editor, a copy through a Windows filesystem, or an unzip can quietly strip the bit; git faithfully commits the change as a mode-only diff, so the file checks out non-executable for everyone who pulls it.Fix: Find the commit with
git log --oneline -- deploy.shandgit blame deploy.sh, then restore the bit in the repo:chmod +x deploy.sh && git commit -am "restore +x on deploy.sh", or set it directly withgit update-index --chmod=+x deploy.sh.How you’d spot it in prod: When a deploy starts failing after a merge but the code looks identical,
git logthe script and look for a mode-only change. The blame line names who dropped the bit and when — that’s the alibi that doesn’t hold up.
Common error: A job that works when you type it but fails the moment cron runs it — the cron log shows:
/home/pushkar/deploy.sh: line 4: docker: command not foundWhy: Cron runs jobs with a minimal environment.
PATHis typically only/usr/bin:/bin, your login profile (.bashrc/.profile) is never sourced, and the working directory is the user’s home. So a tool in/usr/local/bin, or one managed by a version manager, isn’t on the path even though it’s fine in your interactive shell.Fix: Make the script depend on nothing it isn’t given: use absolute paths (
/usr/local/bin/docker— where a manually installed binary or the standalone Compose plugin lands, off cron’s/usr/bin:/bin), set an explicitPATH=at the top, andcdinto the directory it needs. Better, move the job to asystemdtimer, where you declare the environment and read failures withjournalctl -u <unit>.How you’d spot it in prod: A scheduled job that succeeds by hand but logs
command not foundon schedule is an environment gap, not a broken script. Run it once with a stripped env to reproduce cron’s view before changing anything.
Automation & Debugging Interview Questions
The “works by hand but not on schedule” walk-through and the permissions-and-git basics below are among the most common Phase-1-into-Phase-2 screening questions — a calm, layered answer beats a clever one every time. The answer bank renders right after this note. Cover each answer, say your own version out loud first, then compare — recalling before revealing is what makes it stick for interview day.
Go Deeper
Optional extras if you have ~40 more minutes:
- 5 min — Replay The Locked File and try to corner the culprit in the optimal command count — speed here is just knowing the run → permissions → git → schedule walk cold.
- 10 min — On a throwaway script, run
chmod -x, try./it.shand read thePermission denied; thengit init, commit it,chmod +x, and watchgit diffshow theold mode/new modeflip. Feel the two places the execute bit lives. - 10 min — Run
systemctl list-timersand skimman systemd.timerto see the modern replacement for the crontab you’d otherwise reach for — the same schedule, with real logs. - 15 min — Reread the permissions and bash-scripting sections of the Linux for DevOps guide now that scripts, git and cron connect into one automation chain — the second pass is where it consolidates.
A cron job runs fine by hand but fails on schedule — why? Both
Almost always the environment, not the script. Cron runs jobs with a stripped-down environment: PATH is usually just /usr/bin:/bin, your login profile is never sourced, and the working directory is the user's home, not wherever you happened to be. So a tool in /usr/local/bin or under a version manager isn't found, a relative path resolves somewhere unexpected, and a variable you set in .bashrc is simply absent. My first move is to reproduce it the way cron sees it — run with a clean env — then fix the script to depend on nothing it isn't given: absolute paths, an explicit PATH at the top, and cd into the right directory. Modern systemd timers make this cleaner because you declare the environment.
How do you make a shell script executable, and what does the execute bit actually do? Product
chmod +x deploy.sh sets the execute bit, and after that ./deploy.sh runs the file as a program. The execute bit is one of the permission bits per class — owner, group, other — from Day 5: without it the kernel refuses to run the file even though you can still read it, which is why the error is Permission denied, not command not found. Two details I always mention: the shebang line, like #!/usr/bin/env bash, tells the kernel which interpreter to use, and bash deploy.sh runs the file regardless of the bit because you're handing it to bash explicitly. The execute bit only matters when you run the file directly.
Does git track whether a file is executable? Both
Yes — git stores a file's mode, and the only variation it records is the execute bit: 100644 for a normal file, 100755 for an executable one. So if someone edits a script through a tool that drops the bit, git diff shows an old mode 100755 / new mode 100644 change with no content diff at all, and git blame points straight at the commit that did it. That's why a deploy script can 'break' with nothing in the code changing — the permission changed. To fix it in the repo you chmod +x and commit, or use git update-index --chmod=+x so the executable mode is what everyone checks out.
Cron or systemd timers — which would you reach for today? Product
For anything on a modern Linux box I default to systemd timers. You still write the schedule in cron-style terms, but you get real logging through journalctl, the job runs as a proper unit you can start and inspect, you can express dependencies on other services, and OnCalendar plus Persistent will catch up a run the machine missed while it was off. Plain crontab is simpler and I'll still use it for a quick personal job, and you'll see it everywhere in older systems and in CI — GitHub Actions schedules workflows with the same five-field syntax. But when reliability and debuggability matter, a timer beats a crontab line whose only trace of failure is an email nobody reads.
Mark Day 21 complete
Tomorrow you leave the single box behind — containers explain what Docker actually does and why code runs the same everywhere.
Mission unlocked: The Locked File — you have the skills now.
Play (15–20 min)Stuck on today’s lab? Ask in Mission 90 Q&A