Helm & Kubernetes Packaging

Helm in CI/CD & Alternatives

18 min Lesson 9 of 28

Helm in CI/CD & Alternatives

Running helm upgrade manually from a developer laptop is fine for learning, but in production every release must be automated, auditable, and safe. This lesson covers how top-tier engineering teams wire Helm into CI/CD pipelines, how to diff proposed changes before applying them, and when to reach for Kustomize instead of Helm.

Why Pipelines Own the Helm Upgrade

Allowing engineers to run helm upgrade from their workstations is a governance anti-pattern. It means:

Releases are not reproducible — the chart rendered on Alice's machine may differ from Bob's because of local plugin versions or uncommitted values files.
There is no audit log in your version control system. You discover what changed only by diffing Helm release secrets after the fact.
Secrets and credentials leak onto developer machines because the pipeline service account is the right place to hold cluster credentials.

The correct model: git is the single source of truth, and the pipeline is the only actor that calls helm upgrade. A push or merge triggers the pipeline; the pipeline lints, diffs, tests, and upgrades; humans review — they never touch the cluster directly.

A production Helm CI/CD pipeline: lint and validate, diff for human review, then upgrade atomically.

Pipeline Implementation: GitHub Actions

The workflow below is a production-grade pattern used at scale. It runs on every pull request to produce a diff, and again on merge to main to upgrade the cluster. Key flags explained inline.

# .github/workflows/helm-deploy.yaml
name: Helm Deploy

on:
  pull_request:
    paths:
      - 'charts/**'
      - 'values/**'
  push:
    branches: [main]
    paths:
      - 'charts/**'
      - 'values/**'

env:
  RELEASE_NAME: myapp
  NAMESPACE: production
  CHART_PATH: charts/myapp
  VALUES_FILE: values/production.yaml

jobs:
  lint-and-validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Helm
        uses: azure/setup-helm@v4
        with:
          version: '3.15.2'

      - name: Helm lint
        run: helm lint $CHART_PATH --values $VALUES_FILE --strict

      - name: Render templates
        run: |
          helm template $RELEASE_NAME $CHART_PATH \
            --values $VALUES_FILE \
            --namespace $NAMESPACE \
            --output-dir /tmp/rendered

      - name: Validate with kubeconform
        run: |
          curl -sL https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz \
            | tar xz -C /usr/local/bin
          kubeconform -strict -summary /tmp/rendered/myapp/templates/

  diff:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    needs: lint-and-validate
    steps:
      - uses: actions/checkout@v4

      - name: Configure kubeconfig
        run: |
          mkdir -p ~/.kube
          echo "${{ secrets.KUBECONFIG_B64 }}" | base64 -d > ~/.kube/config
          chmod 600 ~/.kube/config

      - name: Install helm-diff plugin
        run: helm plugin install https://github.com/databus23/helm-diff

      - name: Compute diff
        id: diff
        run: |
          DIFF=$(helm diff upgrade $RELEASE_NAME $CHART_PATH \
            --values $VALUES_FILE \
            --namespace $NAMESPACE \
            --allow-unreleased \
            --no-hooks \
            --color 2>&1 || true)
          echo "diff_output<<EOF" >> $GITHUB_OUTPUT
          echo "$DIFF" >> $GITHUB_OUTPUT
          echo "EOF" >> $GITHUB_OUTPUT

      - name: Post diff as PR comment
        uses: actions/github-script@v7
        with:
          script: |
            const diff = `${{ steps.diff.outputs.diff_output }}`;
            const body = diff.length > 0
              ? `## Helm Diff\n\`\`\`diff\n${diff}\n\`\`\``
              : '## Helm Diff\nNo changes detected.';
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body,
            });

  deploy:
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    needs: lint-and-validate
    environment: production          # requires manual approval in GitHub UI
    steps:
      - uses: actions/checkout@v4

      - uses: azure/setup-helm@v4
        with:
          version: '3.15.2'

      - name: Configure kubeconfig
        run: |
          mkdir -p ~/.kube
          echo "${{ secrets.KUBECONFIG_B64 }}" | base64 -d > ~/.kube/config
          chmod 600 ~/.kube/config

      - name: Helm upgrade
        run: |
          helm upgrade $RELEASE_NAME $CHART_PATH \
            --values $VALUES_FILE \
            --namespace $NAMESPACE \
            --create-namespace \
            --atomic \
            --wait \
            --timeout 10m \
            --history-max 10 \
            --set image.tag=${{ github.sha }}

Use GitHub Environments for prod gates. Adding environment: production to the deploy job means GitHub requires a named reviewer to approve before the job runs. You get a free, auditable approval gate without building one yourself. Every Fortune-500 Kubernetes shop uses this pattern or an equivalent (ArgoCD sync approval, Atlantis PR approval for Terraform).

The helm-diff Plugin

helm-diff is the single most important Helm plugin for safe production operations. It performs a three-way merge diff: current live state vs proposed rendered manifests, showing exactly which fields change before a single byte touches the cluster.

# Install the plugin (one-time per machine / CI runner image)
helm plugin install https://github.com/databus23/helm-diff

# Diff an upgrade — outputs a colour-coded unified diff
helm diff upgrade myapp charts/myapp \
  --values values/production.yaml \
  --namespace production \
  --allow-unreleased    # treat as install if release does not exist yet

# Diff against a specific revision stored in Helm history
helm diff revision myapp 4 5 --namespace production

# Useful flags
#   --suppress-secrets   hide Secret data values in the diff output
#   --three-way-merge    the default; compares live object, last applied, and new
#   --context 5          show 5 lines of context around each change

Production pitfall — diff without --suppress-secrets: By default helm diff prints Secret values in plaintext. If you post the diff output to a GitHub PR comment (a common pipeline practice), every Secret change is visible to anyone with repo read access. Always pass --suppress-secrets in pipeline steps that post output publicly, and ensure your pipeline logs are access-controlled.

Kustomize vs. Helm: Choosing the Right Tool

Kustomize ships inside kubectl since v1.14. Understanding when to prefer it over Helm is a critical production judgement call.

Kustomize works by overlaying patches onto base YAML. There is no templating language — instead you compose and patch. This is powerful for environments where you do not control the base manifests (e.g., you are layering company policy onto upstream vendor YAML you cannot modify).

# Kustomize directory layout
k8s/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   └── service.yaml
└── overlays/
    ├── staging/
    │   ├── kustomization.yaml          # references base, applies patches
    │   └── replica-patch.yaml
    └── production/
        ├── kustomization.yaml
        ├── replica-patch.yaml
        └── resources-patch.yaml

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
commonLabels:
  app: myapp

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../../base
namePrefix: prod-
replicas:
  - name: myapp
    count: 5
patches:
  - path: resources-patch.yaml

# Apply to cluster
kubectl apply -k overlays/production/

# Preview rendered output (equivalent to helm template)
kubectl kustomize overlays/production/

The decision framework used at scale:

Use Helm when your application chart will be shared across teams or published (internal chart library, OSS chart), when you need rich conditional logic ({{- if .Values.ingress.enabled }}), when you want release history and one-command rollbacks, or when you are installing community charts (Prometheus, cert-manager).
Use Kustomize when you are layering company policy onto upstream vendor YAML you cannot fork, when the team is allergic to Go template syntax and the configuration changes are purely additive patches, or when you want zero-dependency manifest management inside kubectl itself.
Use both together — a common big-tech pattern is Helm for application packaging (producing a base rendered set) and Kustomize to apply cluster-level or team-level patches on top, especially in GitOps workflows with ArgoCD (which natively supports both helm and kustomize application types).

ArgoCD bridges both worlds. An ArgoCD Application manifest can point at a Helm chart (with values overrides) or a Kustomize directory. Many large organisations use Helm for app owners and Kustomize for the platform team to apply mesh/policy patches on top — no fork of the app chart required. This is the GitOps model covered in the preceding tutorial.

Helmfile: Orchestrating Multiple Releases

When you have ten Helm releases that must deploy in a specific order with shared values, helmfile is the tool that composes them declaratively. It is particularly common in platform teams managing full-stack infrastructure (cert-manager before ingress-nginx before application charts).

# helmfile.yaml — orchestrate multiple Helm releases declaratively
repositories:
  - name: ingress-nginx
    url: https://kubernetes.github.io/ingress-nginx
  - name: jetstack
    url: https://charts.jetstack.io
  - name: internal
    url: oci://registry.company.com/helm-charts

releases:
  - name: cert-manager
    namespace: cert-manager
    chart: jetstack/cert-manager
    version: v1.15.0
    values:
      - installCRDs: true

  - name: ingress-nginx
    namespace: ingress-nginx
    chart: ingress-nginx/ingress-nginx
    version: 4.11.0
    needs:
      - cert-manager/cert-manager   # wait for cert-manager to be ready first

  - name: myapp
    namespace: production
    chart: internal/myapp
    version: "{{ requiredEnv \"APP_VERSION\" }}"
    values:
      - values/production.yaml.gotmpl   # .gotmpl files support Go templating
    needs:
      - ingress-nginx/ingress-nginx

# Deploy all releases in dependency order
helmfile sync

# Diff all releases
helmfile diff

# Destroy all releases (destructive — use carefully)
helmfile destroy

Secrets Management in Helm Pipelines

Never store plaintext secrets in values files committed to git. The three accepted patterns at production scale:

External Secrets Operator (ESO) — The recommended modern approach. ESO pulls secrets from AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault and creates Kubernetes Secret objects. Your Helm values file references the secret name, not the value. The secret never touches git or pipeline logs.
helm-secrets plugin + SOPS — Encrypts specific values files with AWS KMS, GCP KMS, or age keys. The encrypted file is committed to git; the pipeline decrypts it at deploy time using an IAM role or KMS key policy. Simple to adopt, no cluster components required.
Sealed Secrets — A Kubernetes controller generates a public key; you encrypt secrets with it locally (kubeseal); the encrypted SealedSecret CRD is committed to git. Only the controller can decrypt it. Excellent for pure GitOps environments.

Big-tech default: At AWS-native shops, the standard stack is External Secrets Operator pulling from AWS Secrets Manager + IRSA (IAM Roles for Service Accounts) so the ESO pod authenticates via the pod identity without any long-lived credentials. The Helm chart only knows the secret name (myapp/production/db-password), not its value. This keeps secrets out of every layer: git, pipeline logs, Helm release secrets, and ConfigMaps.

With Helm fully wired into your CI/CD pipeline — linting, diffing, environment-gated upgrades, and secrets handled out-of-band — your delivery process is repeatable, auditable, and safe at any scale. The next lesson wraps the tutorial with a complete production chart for a real-world application, applying every pattern from lessons 1 through 9.