Signing & Verification

Signing & Verification

Every modern supply-chain attack exploits the same trust gap: a consumer — a Kubernetes cluster, a CI runner, a developer's workstation — fetches an artifact and has no way to prove it came from the source it claims. The artifact could have been swapped in transit, on the registry, or inside the build system itself. Artifact signing closes this gap by binding a cryptographic signature to the artifact at build time and verifying that signature before any consumption happens. This lesson covers the state-of-the-art open-source tooling — Sigstore / Cosign — and how to enforce verification at the Kubernetes admission layer so that unsigned or unverifiable images can never run in production.

Why Traditional Key Management Fails at Scale

Before Sigstore, signing artifacts meant managing long-lived GPG or OpenSSL private keys. At big-tech scale this creates three compounding problems. First, key distribution: every consumer must receive and trust the public key out-of-band — a bootstrapping problem with no clean solution. Second, key rotation: a compromised signing key invalidates all previous signatures, but revoking and redistributing keys across hundreds of teams is operationally painful. Third, attribution is opaque: a signature proves possession of a private key, not the identity of the human or CI job that actually triggered the build. If the key leaks, there is no way to distinguish legitimate from malicious signatures.

Sigstore Architecture: Fulcio, Rekor, and Cosign

Signing a Container Image with Cosign (keyless mode)

In GitHub Actions, the workflow runs with an OIDC token by default. Cosign detects this and performs the entire Fulcio/Rekor dance transparently. You need zero key management:

The --yes flag skips the interactive confirmation prompt, which is required in CI. COSIGN_EXPERIMENTAL=1 enables keyless mode (now the default in Cosign v2, but still a common pattern in older pipelines). After this step, the signature is stored as a separate OCI artifact in the same registry namespace — Cosign uses the convention sha256-<digest>.sig.

To verify a signature locally or in a script:

Enforcing Signatures at Kubernetes Admission Control

Signing without mandatory verification at the cluster boundary is security theater. The enforcement layer is a Kubernetes admission webhook that intercepts every Pod create/update request and rejects it if the image lacks a valid Cosign signature. Two mature options exist at big-tech scale:

• Policy Controller (from Sigstore) — a standalone admission webhook that reads ClusterImagePolicy CRDs. The recommended path for pure Sigstore setups.
• Kyverno — a general Kubernetes policy engine with built-in Cosign verification; preferred when you already use Kyverno for other policies.

The following example uses the Sigstore Policy Controller:

With this policy applied to the prod-namespace, any Pod referencing an image from ghcr.io/my-org/** that does NOT have a valid signature from the specified CI workflow is rejected with a 403 Forbidden before any container process starts. Namespaces not labelled for enforcement are unaffected — a common pattern is to enforce in prod first, then roll out to staging.

Verifying Attestations: Going Beyond the Signature

Cosign can also attach and verify structured attestations — signed metadata about the image such as SBOM documents, SLSA provenance records, or vulnerability scan results. This is how big-tech companies prove not just "this image was signed" but "this image was built from commit X, passed SAST with zero criticals, and its SBOM is attached." The cosign attest and cosign verify-attestation commands handle this workflow using the in-toto attestation format:

Failure Modes and Operational Concerns

Signature verification adds latency to pod scheduling (the admission webhook must reach the registry and optionally Rekor). At Google-scale this is typically 50–200 ms per unique image digest, cached aggressively by the Policy Controller. Three operational concerns to plan for:

• Webhook availability: If the Policy Controller pod is down, pods will fail to schedule (fail-closed behavior). Deploy it with at least two replicas, a PodDisruptionBudget, and a node affinity that keeps it on control-plane nodes or dedicated system nodes.
• Registry air-gap: In air-gapped environments, Rekor and Fulcio are unavailable. Either run a self-hosted Sigstore stack (sigstore-the-hard-way or the sigstore/scaffolding Helm chart), or use key-based signing with a Hardware Security Module (HSM) instead of keyless mode.
• Bootstrap exception: The Policy Controller image itself must be excluded from its own policy to avoid a circular dependency during cluster bootstrapping. Always add an explicit namespace or image exception for cosign-system.