GitOps with ArgoCD & Flux

Environment Promotion

18 min Lesson 7 of 30

Environment Promotion

Environment promotion is the controlled process of moving a verified artifact — a container image at a specific digest — from a lower-trust environment (dev) through a staging environment and finally into production, with a Git commit as the immutable audit record at each step. In GitOps, promotion is not a button in a UI or a flag in a shell script. It is a deliberate change to the desired state stored in Git. This distinction is the entire point.

The promotion model used at companies like Shopify, Stripe, and Weaveworks treats environments as separate folders (or branches) in the config repo, each reconciled independently. You never promote "code" — you promote a specific image tag or digest that was already built, scanned, and tested by CI. The only question Git has to answer is: which version of that artifact is each environment allowed to run?

The Anatomy of a Promotion Flow

A three-environment promotion path looks like this in Git terms:

dev: image tag updated automatically by CI on every merge to main. The GitOps agent reconciles within seconds. No human approval.
staging: image tag updated by a promotion script or a human PR. May require automated smoke tests to pass first. Gated by a branch protection rule or a required status check.
production: image tag updated only after a human approves a PR (or a policy engine like OPA Gatekeeper approves automatically). Change-freeze windows enforced via branch protection rules on the config repo.

End-to-end GitOps promotion flow: CI builds an image, commits the digest to the dev environment folder, and a controlled PR chain promotes it to staging then production. Each environment has its own GitOps agent watching its own folder.

Repository Layout for Multi-Environment GitOps

The canonical structure that scales to large orgs uses environment folders inside a single config repo (or separate repos per environment for strictest isolation). Here is a real Kustomize-based layout:

# gitops-config/
apps/
  api-service/
    base/
      deployment.yaml        # image: ghcr.io/myorg/api:SHA  (placeholder)
      service.yaml
      kustomization.yaml
    overlays/
      dev/
        kustomization.yaml   # sets image tag, replicas=1
      staging/
        kustomization.yaml   # sets image tag, replicas=2, resource limits
      production/
        kustomization.yaml   # sets image tag, replicas=6, HPA, PDB

# overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: ghcr.io/myorg/api
    newTag: sha-abc1234   # <-- THIS line is what CI/promotion scripts update
patches:
  - patch: |-
      - op: replace
        path: /spec/replicas
        value: 1
    target:
      kind: Deployment
      name: api-service

Always pin to image digest in production, not a mutable tag. Tags like v1.2.3 can be overwritten by a docker push. A digest like sha256:a3f7... is cryptographically immutable. Use kustomize edit set image with the full digest in your promotion script. ArgoCD Image Updater and Flux's ImagePolicy can automate digest pinning in dev/staging while leaving production pinned to a human-approved digest.

Automating Dev Promotion from CI

After a successful CI build, the pipeline updates the dev overlay automatically. This is the only environment where CI writes directly to the config repo. The pattern uses a bot token with write access scoped only to the config repo:

# .github/workflows/ci.yml  (app repo)
name: Build and Promote to Dev
on:
  push:
    branches: [main]

jobs:
  build-and-promote:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build and push image
        id: build
        uses: docker/build-push-action@v5
        with:
          push: true
          tags: ghcr.io/myorg/api-service:sha-${{ github.sha }}
          # digest is available as steps.build.outputs.digest

      - name: Update dev overlay in config repo
        env:
          CONFIG_REPO_TOKEN: ${{ secrets.CONFIG_REPO_PAT }}
          IMAGE_DIGEST: ${{ steps.build.outputs.digest }}
        run: |
          git clone https://x-token:${CONFIG_REPO_TOKEN}@github.com/myorg/gitops-config.git
          cd gitops-config

          cd apps/api-service/overlays/dev
          kustomize edit set image \
            ghcr.io/myorg/api-service@${IMAGE_DIGEST}

          git config user.email "ci-bot@myorg.com"
          git config user.name "CI Bot"
          git add kustomization.yaml
          git commit -m "chore(dev): promote api-service to ${IMAGE_DIGEST::19}"
          git push

Promoting to Staging: Script-Driven PR

Staging promotion is typically triggered manually (a developer runs a script or clicks a UI button after monitoring dev) or automatically after a time-based soak period in dev. The script opens a pull request; human merge is required. Required status checks on the PR (smoke tests, security scans) act as the gate.

#!/usr/bin/env bash
# scripts/promote.sh — run by a developer: ./promote.sh staging sha-abc1234
set -euo pipefail

ENV=$1        # staging | production
DIGEST=$2     # sha256:... or short sha tag

REPO="myorg/gitops-config"
BRANCH="promote/${ENV}-$(date +%Y%m%d-%H%M%S)"

gh repo clone ${REPO} /tmp/promote-clone -- --depth 1
cd /tmp/promote-clone

git checkout -b ${BRANCH}
cd apps/api-service/overlays/${ENV}
kustomize edit set image ghcr.io/myorg/api-service@${DIGEST}

git add kustomization.yaml
git commit -m "chore(${ENV}): promote api-service to ${DIGEST::19}"
git push origin ${BRANCH}

gh pr create \
  --repo ${REPO} \
  --title "Promote api-service to ${ENV}: ${DIGEST::19}" \
  --body "Automated promotion PR. Merge after CI checks pass." \
  --base main \
  --head ${BRANCH}

Flux ImageUpdateAutomation vs. manual scripts: Flux has a native ImageUpdateAutomation CRD that can watch a container registry and commit updated tags to Git automatically, with policies controlling which semver ranges are eligible (e.g. semver: range: '>=1.0.0 <2.0.0'). This is excellent for dev and staging. For production, disable automation and keep the human PR gate. ArgoCD delegates to external tools like Renovate or the ArgoCD Image Updater add-on for similar automation.

Promoting to Production: The Human Gate

Production promotion follows the same pattern as staging, but the PR requires explicit human review. At big-tech scale this is enforced structurally, not by convention:

CODEOWNERS file: apps/api-service/overlays/production/ @myorg/oncall-approvers — GitHub/GitLab automatically requests review from this team and blocks merge until one of them approves.
Branch protection rules: require 2 approvals, dismiss stale reviews on new commits, require all status checks to pass (including a custom "change-freeze" check that fails during declared freeze windows).
Policy engines: OPA Conftest or Kyverno policies run in the CI check on the config repo PR, validating that the image has passed the required security scan, the digest matches what was promoted through staging, and no resource limits were accidentally removed.

# .github/CODEOWNERS  (in the config repo)
# Production overlays require explicit oncall team approval
/apps/*/overlays/production/     @myorg/oncall-approvers
/apps/*/overlays/staging/        @myorg/senior-engineers

# Cluster-level config is platform-team only
/clusters/                       @myorg/platform-team

Production Failure Modes and How to Avoid Them

Promotions fail in predictable ways. Knowing these patterns lets you build guardrails before they hit production:

Promoting a mutable tag: CI builds :latest, staging passes, but by the time production is promoted, :latest points to a different build. Always promote image digests (sha256:...), never mutable tags.
Config drift between overlays: a developer manually edits the staging overlay to test something, forgets to remove it, and the staging-to-prod promotion carries the change silently. Use kustomize build in CI to diff the rendered output of staging vs prod before merging.
Race conditions in multi-service promotions: when service A depends on service B's new API, promoting A before B causes errors. Use ArgoCD sync waves (argocd.argoproj.io/sync-wave annotation) or Flux dependsOn to sequence deployments.
Forgetting to update environment-specific secrets: a new environment variable is added in dev but the Secret or ExternalSecret is not added to the prod overlay. The app deploys but crashes. Always validate rendered manifests against a schema.

Never let a CI bot write directly to production config paths. The bot must only have write access to dev overlays. Staging and production paths must require a human-authored PR. This is the single most important access control in your GitOps setup. A compromised CI secret that can write to production config is a supply-chain attack surface — it can deploy arbitrary code to your production cluster without anyone reviewing it.

Tracking Promotion State

At the end of a promotion, you should be able to answer from Git alone: which image digest is currently running in each environment? A simple convention is a VERSIONS.md or a structured versions.json in the repo root updated by the promotion script. Better yet, query ArgoCD or Flux directly:

# Query ArgoCD for the current synced revision in each environment
argocd app get api-service-dev   --output json | jq '.status.sync.revision'
argocd app get api-service-staging --output json | jq '.status.sync.revision'
argocd app get api-service-prod    --output json | jq '.status.sync.revision'

# With Flux, check the ImagePolicy to see what tag each env is pinned to
kubectl get imagepolicy -n flux-system -o wide

# Or just look at the git log for each overlay directory
git log --oneline -- apps/api-service/overlays/production/kustomization.yaml
# a3f91b2 chore(prod): promote api-service to sha256:a3f7... (2025-11-14)
# 9e21c45 chore(prod): promote api-service to sha256:9e21... (2025-11-07)

Environment promotion done well is the difference between a deployment process you can trust at 2 AM and one you dread every Friday afternoon. The discipline of "every change to every environment is a Git commit with a human approval chain" is what makes GitOps the foundation of safe, auditable delivery at scale.