Platform Engineering & Developer Experience

Multi-Tenancy & Guardrails in Platforms

18 min Lesson 8 of 28

Multi-Tenancy & Guardrails in Platforms

When a platform team serves dozens or hundreds of product teams from shared infrastructure, every design decision about tenant isolation, quotas, and secure defaults has blast-radius implications. A misconfigured namespace can starve a revenue-critical service of CPU; a missing NetworkPolicy lets a compromised pod exfiltrate secrets across team boundaries; an absent LimitRange allows a runaway container to OOM the entire node. At Google scale, Borg enforces per-cell quota through the Borg Master; at Stripe, every microservice inherits a Kubernetes Namespace with pre-baked RBAC, ResourceQuotas, and NetworkPolicies generated by their internal "service provisioner." This lesson covers how to build that discipline into your own platform.

Tenant Isolation Models

Platform tenants are typically mapped to one of three boundaries, each offering a different isolation/cost trade-off:

Namespace-per-team — the most common Kubernetes model. Teams share a cluster but are separated by RBAC, NetworkPolicy, and ResourceQuota. Cost-efficient; isolation is logical, not physical. A node compromise crosses namespace lines.
Node-pool-per-tenant — workloads for sensitive teams land on dedicated node pools via nodeSelector or nodeAffinity combined with Taint/Toleration. Noisy-neighbour risk is gone; node cost is higher. Common for PCI/HIPAA tenants.
Cluster-per-tenant — full isolation. Used by hyperscalers offering managed Kubernetes (EKS, GKE) to external customers, or internally for compliance domains. vCluster and Loft bring this model back inside a single host cluster at near-namespace cost.

The right model is determined by your threat model, not convenience. A platform serving only internal teams with uniform compliance requirements can operate namespace isolation safely. A platform that hosts regulated workloads alongside general-purpose services must physically separate them — logical isolation is not sufficient for PCI DSS or HIPAA auditors.

Kubernetes ResourceQuota and LimitRange

Every namespace a platform provisions should receive both a ResourceQuota (cluster-wide hard caps) and a LimitRange (per-container defaults and maximums). Omitting LimitRange means a pod without explicit requests/limits runs with unbounded CPU/memory — the most common cause of noisy-neighbour incidents on shared clusters.

# namespace-quota.yaml — generated by the platform provisioner per team
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-payments
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.cpu: "16"
    limits.memory: 32Gi
    pods: "60"
    services: "20"
    persistentvolumeclaims: "10"
    count/deployments.apps: "30"
    count/jobs.batch: "20"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: team-limitrange
  namespace: team-payments
spec:
  limits:
  - type: Container
    default:          # injected when a container omits limits
      cpu: "500m"
      memory: 256Mi
    defaultRequest:   # injected when a container omits requests
      cpu: "100m"
      memory: 128Mi
    max:
      cpu: "4"
      memory: 4Gi
    min:
      cpu: "50m"
      memory: 64Mi

Enforce quota exhaustion alerting: a Prometheus rule on kube_resourcequota{type="used"} / kube_resourcequota{type="hard"} > 0.85 fires a warning before the namespace is full and new pods start Pending. Without this alert, engineers discover the quota limit only when their deployment silently fails during an incident.

Network Isolation with NetworkPolicy

By default, Kubernetes allows all pod-to-pod traffic. In a multi-tenant platform you must invert this: deny all by default, then add explicit allow rules. The baseline policy below is templated into every new namespace by the platform provisioner.

# default-deny.yaml — applied to every platform-provisioned namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: team-payments
spec:
  podSelector: {}      # selects ALL pods in the namespace
  policyTypes:
  - Ingress
  - Egress
---
# allow-same-namespace.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-intra-namespace
  namespace: team-payments
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector: {}  # any pod in the same namespace
  egress:
  - to:
    - podSelector: {}
  - to:                # DNS resolution (kube-dns in kube-system)
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP

CNI compatibility: NetworkPolicy objects are only enforced if your CNI plugin implements the spec. Flannel does not. Calico, Cilium, and Weave Net do. On a cluster where the CNI silently ignores NetworkPolicy, tenants believe they are isolated when they are not. Validate enforcement during platform bootstrap with a netpol-tester pod that probes cross-namespace connectivity and asserts it is blocked.

Admission Control: Policy as Guardrails

ResourceQuota and NetworkPolicy are reactive — they constrain what already exists. Admission policies are proactive: they block or mutate workloads at creation time. A mature platform uses Kyverno or OPA/Gatekeeper to enforce the rules that would otherwise require human code review at scale.

Critical policies every platform should ship out of the box:

Require non-root containers — deny any pod where securityContext.runAsNonRoot != true.
Disallow privileged containers — deny securityContext.privileged: true and allowPrivilegeEscalation: true.
Require image digest or allowed registry — deny latest tag; only allow pulls from your internal registry or a curated allowlist.
Require resource requests and limits — deny pods that would bypass your LimitRange defaults (belt-and-suspenders).
Require team labels — deny any workload missing app.kubernetes.io/team and app.kubernetes.io/service labels (enables cost attribution and on-call routing).

# kyverno-require-labels.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-team-labels
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-team-label
    match:
      any:
      - resources:
          kinds: [Deployment, StatefulSet, DaemonSet]
    validate:
      message: "Workloads must carry app.kubernetes.io/team and app.kubernetes.io/service labels."
      pattern:
        metadata:
          labels:
            app.kubernetes.io/team: "?*"
            app.kubernetes.io/service: "?*"

Secure Defaults: The Baseline Security Context

Rather than relying on developers to write correct security contexts, the platform should mutate pod specs at admission time to inject secure defaults. A Kyverno mutate rule or a Kubernetes PodAdmission Standard (restricted profile at the namespace level via pod-security.kubernetes.io/enforce: restricted label) provides this automatically.

At minimum, every workload baseline should include:

runAsNonRoot: true
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
seccompProfile.type: RuntimeDefault (uses the container runtime's default seccomp filter)
capabilities.drop: [ALL] — add back only what is explicitly required (e.g., NET_BIND_SERVICE for port 80)

Guardrails stack: per-namespace quotas, network isolation, and cluster-wide admission policies enforce tenant boundaries.

Cost Attribution and Chargeback

Multi-tenancy without cost visibility creates a tragedy of the commons: teams over-provision because they do not see the bill. The platform should emit per-namespace cost data. OpenCost (CNCF) or Kubecost can be deployed as a platform service to produce per-team spend reports ingested into your internal cost dashboard. The mandatory app.kubernetes.io/team label enforced by your Kyverno policy is the join key.

At Amazon, every service has a mandatory cost tag that gates deployment — a workload without a valid cost centre cannot be deployed to production. Your admission policy enforcing team labels is the Kubernetes equivalent of that gate.

Quota headroom strategy: set ResourceQuota limits at 1.5x the team's P95 historical usage, not at their theoretical maximum. Provisioning 128 GiB for a team that uses 8 GiB P95 wastes cluster capacity. Review quotas quarterly in partnership with teams during capacity planning cycles (see the Capacity Planning tutorial). Alert at 85% utilisation so teams can request increases before hitting the wall during a traffic spike.

vCluster for Stronger Isolation Without Full Cluster Cost

When a team needs cluster-level isolation — separate API server, separate RBAC universe, separate admission webhooks — but provisioning a full EKS/GKE cluster per team is prohibitively expensive, vcluster (by Loft Labs) virtualises a Kubernetes control plane inside a namespace of the host cluster. The tenant sees a real API server; the host cluster sees only pods. This is the pattern Datadog and several large SaaS platforms use for their own multi-tenant Kubernetes offerings.

# Install vcluster CLI, then create an isolated virtual cluster for a regulated tenant
vcluster create team-finserv \
  --namespace vcluster-team-finserv \
  --values - <<EOF
syncer:
  extraArgs:
    - --enforce-node-selector=true
    - --node-selector=pool=finserv-dedicated
isolation:
  enabled: true
  resourceQuota:
    enabled: true
    quota:
      requests.cpu: "8"
      requests.memory: 16Gi
  limitRange:
    enabled: true
    default:
      cpu: "500m"
      memory: 256Mi
    defaultRequest:
      cpu: "100m"
      memory: 128Mi
  networkPolicy:
    enabled: true
EOF

# The tenant can now kubectl into their vcluster as cluster-admin
# without touching any other tenant's workloads
vcluster connect team-finserv --namespace vcluster-team-finserv

Guardrail Governance: Audit, Override, and Escape Hatches

Rigid policies that block legitimate work generate shadow-IT: engineers work around the platform instead of with it. Every guardrail should have a documented, audited escape hatch. In Kyverno, policies can run in Audit mode (violations logged, not blocked) before promotion to Enforce. A platform namespace annotation — platform.io/policy-exception: approved-by=security-team,ticket=SEC-1234 — combined with a Kyverno PolicyException object provides an audited override without removing the policy cluster-wide.

The goal of platform guardrails is not to enforce compliance by making non-compliance impossible — it is to make the correct path easier than the incorrect path. When the guardrails are too tight without an escape mechanism, the blast radius of the workarounds is worse than the risk the guardrail was designed to prevent. Design for 95% automation and 5% audited override.