Build Automation & Reproducibility

Build Automation & Reproducibility

A CI pipeline is only as trustworthy as the build it runs. Lesson 2 showed you the anatomy of a pipeline — stages, runners, triggers. This lesson goes one level deeper: how do you write the build scripts that those stages execute, pin dependencies so they never drift, and structure a build so it produces the exact same binary on a developer laptop, a GitHub Actions runner, and a production server? These three properties — automated builds, pinned dependencies, and hermetic builds — are the foundation that separates a hobby CI setup from a Google-grade one.

Build Scripts: The Contract Between Code and Pipeline

Every CI stage calls a script. That script is the authoritative, executable description of how your software is built. The golden rule: if a human runs it once manually, it must be expressible as a script that the pipeline runs automatically — with no interactive prompts, no ambient environment variables, and no implicit system-level dependencies.

Write your build scripts with these properties:

• Exit on first failure: Use set -euo pipefail in Bash. Without it, a failing npm install is silently ignored and the pipeline reports green. -e exits on error, -u treats unset variables as errors, -o pipefail catches failures inside pipes.
• Explicit tool versions: Never call node and hope for the best. Pin the version with .nvmrc, .tool-versions (asdf), or a Docker image tag. The runner that installs Node 18 today may upgrade to Node 22 next quarter.
• Offline-first installs: Use npm ci instead of npm install. Use pip install --no-index when a mirror is available. Use go mod download with a module proxy. Network failures in CI are a top source of flaky builds.
• Idempotent steps: Each step should be safe to re-run. Avoid side effects like appending to files or mutating shared state between steps.

Pinned Dependencies: No Surprises in Production

Unpinned dependencies are the most common source of "it worked last week" failures. The pattern repeats across every ecosystem: a transitive package releases a patch, your lock file is absent or stale, and suddenly your build fails on a perfectly unchanged codebase. At big tech, every dependency — direct and transitive — is pinned exactly.

What pinning means per ecosystem:

• Node.js: Commit package-lock.json or yarn.lock. Install with npm ci (errors if lockfile is out of sync with package.json). Never commit node_modules/.
• Python: Commit requirements.txt generated by pip-compile (from pip-tools), which resolves and pins all transitive dependencies. Or use poetry.lock. Never use requirements.txt with open version ranges like requests>=2.0.
• Go: Commit go.sum. The Go toolchain verifies checksums cryptographically — any tampered dependency fails the build. Run go mod tidy before committing to keep it clean.
• Docker base images: Never use FROM ubuntu:latest. Pin to a digest: FROM ubuntu:24.04@sha256:abc123.... Image tags are mutable — the same tag can point to a different layer set tomorrow.
• CI actions: In GitHub Actions, pin third-party actions to a commit SHA, not a tag. uses: actions/checkout@v4 is mutable; uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 is pinned forever.

Hermetic Builds: Isolating Every Variable

A hermetic build is one that is fully isolated from the ambient environment. Given the same source code and the same inputs, a hermetic build produces bit-for-bit identical output regardless of which machine runs it, what is installed on that machine, or what time of day it is. This is the gold standard. Google's Bazel and Meta's Buck2 are purpose-built hermetic build systems that enforce this at the toolchain level.

You do not need Bazel to achieve most of the benefit. The practical hermetic build checklist:

• Run inside a container: Build inside a Docker image that contains every tool at a pinned version. The host OS becomes irrelevant.
• No network during build: All dependencies must be present in the image or in a cache layer before the compile step. A build that fetches from the internet mid-compile is non-hermetic — the internet changes.
• Eliminate timestamps and random seeds: Many compilers embed build timestamps. Use SOURCE_DATE_EPOCH (a standard environment variable) to freeze the timestamp. Use a fixed random seed in any generation step.
• Cache inputs, not outputs: Cache the downloaded dependency layer keyed on the lockfile hash. Never cache built artifacts across branches — stale cache is a common source of ghost failures.

Caching: The Speed Multiplier That Must Not Corrupt

Build caches are essential for pipeline speed — a 3-minute Node.js install becomes 5 seconds when the cache hits. But a corrupt cache can mask real failures for weeks. Apply these rules:

• Key on inputs, not time: The cache key must include the lockfile hash (hashFiles('**/package-lock.json')), the OS, and the Node version. A cache keyed only on a branch name will serve a stale hit after dependency changes.
• Restore but never trust fully: After restoring a cache, still run npm ci — the --prefer-offline flag uses the cache but verifies integrity. Never skip the install step just because the cache hit.
• Separate build cache from test cache: Store compiled output separately from downloaded packages. A bad compile artifact in cache causes ghost failures that are extremely hard to diagnose.
• Bust aggressively on major changes: Prefix cache keys with a manually bumped version (v2-deps-...). Bump the prefix whenever you suspect cache corruption — one cache bust is faster than hours of investigation.

Makefile and Task Runners: The Local-CI Bridge

The highest-leverage pattern for reproducibility is having a single command that does exactly what CI does. A Makefile (or Taskfile.yml for the Go/YAML ecosystem) serves as the standard interface: make build, make test, make lint — same targets, same commands, whether run by a human or a pipeline.

Common Failure Modes in Production

Even well-intentioned teams encounter these recurring problems:

• Floating base image tags: FROM node:lts pointed at Node 18 last year; today it points at Node 22. All your tests now run on a different runtime than production.
• Implicit system tools: A build script calls jq or yq that happens to be installed on the developer's laptop but is absent on a fresh runner. The build fails in CI with a cryptic "command not found."
• Timezone-sensitive tests: A test that asserts new Date().toLocaleDateString() === "1/1/2025" passes in UTC-0 and fails in UTC+3. The CI runner uses UTC; developer machines do not.
• Race conditions in parallel steps: Two build steps write to the same output directory. On a fast runner they collide; on a slow one they do not. The failure is non-deterministic and intermittent.

Each of these is a reproducibility violation. The solution in every case is the same: eliminate the ambient assumption that caused it — pin the image, declare the tool in the Dockerfile, freeze the timezone with TZ=UTC, and serialize conflicting steps.