GitOps with ArgoCD & Flux

App-of-Apps & ApplicationSets

18 min Lesson 5 of 30

App-of-Apps & ApplicationSets

A single ArgoCD Application resource works well for one service. A real platform team manages hundreds of services across dozens of clusters. Manually creating and maintaining one Application YAML per microservice — and then doing that again for staging, production, and each region — does not scale. ArgoCD solves this with two complementary patterns: App-of-Apps and ApplicationSets.

The App-of-Apps Pattern

The idea is simple: a root ArgoCD Application points at a Git directory that contains other ArgoCD Application manifests. When ArgoCD syncs the root, it discovers and creates the child applications. Those children are themselves full Application objects — they each point at their own Git source and each have their own sync policy.

App-of-Apps reconciliation flow Git Repo apps/root/ Root App ArgoCD Application sync Child: api Application Child: frontend Application Child: worker Application creates Cluster prod-us-east Cluster prod-us-east Cluster prod-us-east
The root Application syncs child Application manifests from Git; each child manages its own workload on the target cluster.

A minimal root app manifest looks like this:

# apps/root/api.yaml (this file lives in your gitops repo) apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: api namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io # cascade-delete resources on app deletion spec: project: default source: repoURL: https://github.com/acme/gitops targetRevision: HEAD path: envs/prod/api destination: server: https://kubernetes.default.svc namespace: api syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true
The finalizer resources-finalizer.argocd.argoproj.io is critical. Without it, deleting the Application object in ArgoCD leaves all the Kubernetes resources (Deployments, Services, PVCs) orphaned in the cluster. Always include it on every child application.

The root application itself is seeded once — typically via kubectl apply or ArgoCD's own install chart. After that, adding a new service means committing a new YAML file to the apps/root/ directory; ArgoCD picks it up automatically on the next sync.

ApplicationSets: Generator-Driven Scaling

App-of-Apps works well but still requires you to write one YAML per service per environment. ApplicationSets (now stable, part of ArgoCD core since v2.3) go further: a single ApplicationSet resource uses generators to produce many Application objects from a template. No copy-paste, no manual files per cluster.

The most important generators are:

  • List — inline list of key-value maps; the simplest generator for small, static sets.
  • Git — auto-discovers apps by scanning Git directories or files; great for monorepos.
  • Matrix — Cartesian product of two generators; e.g., services × environments.
  • Cluster — iterates over clusters registered in ArgoCD; deploy to every cluster automatically.
  • SCM Provider — scans all repos in a GitHub org; good for platform-wide enforcement.
  • Pull Request — creates a temporary Application for each open PR; enables preview environments.

Matrix Generator: Services × Environments

This is the production workhorse. The outer generator yields environments, the inner yields services; ArgoCD renders one Application per pair:

apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: microservices namespace: argocd spec: goTemplate: true # enables Go template syntax in the template block goTemplateOptions: ["missingkey=error"] generators: - matrix: generators: - list: elements: - env: staging cluster: https://k8s-staging.acme.internal valuesFile: values-staging.yaml - env: production cluster: https://k8s-prod-us-east.acme.internal valuesFile: values-prod.yaml - git: repoURL: https://github.com/acme/gitops revision: HEAD directories: - path: services/* # discovers services/api, services/frontend, etc. template: metadata: name: "{{.path.basenameNormalized}}-{{.env}}" labels: app.kubernetes.io/managed-by: argocd env: "{{.env}}" spec: project: default source: repoURL: https://github.com/acme/gitops targetRevision: HEAD path: "services/{{.path.basename}}" helm: valueFiles: - "../../envs/{{.env}}/{{.valuesFile}}" destination: server: "{{.cluster}}" namespace: "{{.path.basename}}" syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true - ServerSideApply=true
Enable goTemplate: true plus goTemplateOptions: ["missingkey=error"] on every ApplicationSet. The older {{env}} fasttemplate syntax silently renders undefined variables as empty strings, hiding configuration bugs. Go templates fail loudly on missing keys, which is exactly what you want in production.

Git Generator: Auto-Discovery in a Monorepo

When your monorepo uses a convention like services/<name>/kustomization.yaml, the Git file generator discovers new services automatically — no ApplicationSet change required when a developer adds a new directory:

apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: platform-services namespace: argocd spec: generators: - git: repoURL: https://github.com/acme/platform revision: HEAD files: - path: "services/**/app-config.json" # discovers any app-config.json under services/ template: metadata: name: "{{path.basenameNormalized}}" spec: project: platform source: repoURL: https://github.com/acme/platform targetRevision: HEAD path: "{{path}}" destination: server: https://kubernetes.default.svc namespace: "{{path.basename}}" syncPolicy: automated: prune: true selfHeal: true

ApplicationSet Sync Policy and Deletion Protection

ApplicationSets introduce an important risk: a generator misconfiguration (a typo in a directory glob, for example) can produce zero results, causing ArgoCD to delete all child applications and prune every resource in the cluster. Protect against this with the preserveResourcesOnDeletion policy:

spec: syncPolicy: preserveResourcesOnDeletion: true # do NOT cascade-delete cluster resources if the AppSet is deleted # Also consider: ignoreApplicationDifferences: - jsonPointers: - /spec/source/targetRevision # allow manual overrides without triggering a diff
Never run ApplicationSets with prune: true on the generated child apps without also setting preserveResourcesOnDeletion: true on the ApplicationSet itself. A bad generator update wiped a full production environment at a major fintech in 2023 because the directory glob returned zero matches — ArgoCD pruned all apps, which pruned all workloads.

Project Isolation

At scale you will have multiple teams. ArgoCD Projects enforce RBAC boundaries: which source repos a team may use, which clusters and namespaces they may deploy to, and which resource kinds they are allowed to create. ApplicationSets should always target a team-specific project rather than default:

# Define the project once apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: payments-team namespace: argocd spec: description: "Payments microservices — owned by payments-eng" sourceRepos: - "https://github.com/acme/payments-*" destinations: - server: "https://k8s-prod-us-east.acme.internal" namespace: "payments-*" clusterResourceWhitelist: - group: "" kind: Namespace namespaceResourceBlacklist: - group: "" kind: ResourceQuota # platform team owns quotas, not app teams roles: - name: deployer policies: - p, proj:payments-team:deployer, applications, sync, payments-team/*, allow groups: - acme:payments-eng

Operational Best Practices

  • One ApplicationSet per team, not per service. Fewer objects to reason about; generators handle the fan-out.
  • Use ServerSideApply=true in syncOptions when multiple controllers manage the same object (e.g., HPA + ArgoCD). It eliminates field-manager conflicts.
  • Test generators locally with argocd appset generate ./my-appset.yaml before applying — see what Applications would be created without touching the cluster.
  • Cap concurrent reconciliation with --application-set-concurrent-reconciliations=20 on the applicationset-controller; an uncapped ApplicationSet managing 500 apps can spike API server load on sync.
  • Tag child apps with labels (env, team, region) so you can filter in the ArgoCD UI and write targeted RBAC policies.
The App-of-Apps pattern gives you explicit, auditable child Application manifests committed to Git — easy to review in PRs. ApplicationSets give you generator-driven automation with zero per-app YAML. Big-tech platforms typically use both: ApplicationSets for homogeneous workloads (all microservices), and App-of-Apps for heterogeneous infrastructure bootstrapping (cert-manager, ingress, monitoring stack) where each app needs unique configuration that does not fit a template.

ES
Edrees Salih
1 hour ago

We are still cooking the magic in the way!