HashiCorp Vault Architecture
HashiCorp Vault Architecture
HashiCorp Vault is the de facto secrets management platform across the industry — used by Google, Netflix, GitHub, and thousands of other engineering organisations. Before you run a single vault kv get, you need to understand what Vault is doing internally. The architectural model — seal/unseal, auth methods, secret engines, and policies — directly determines your threat model, your blast radius if something goes wrong, and your operational runbook. This lesson tears open the internals.
The Storage Backend and the Encryption Barrier
Vault itself stores nothing in plaintext. Every piece of data — secrets, tokens, leases, auth configuration — is encrypted before it hits the storage backend. The storage backend (Raft integrated storage in modern deployments, or Consul/DynamoDB in older ones) is intentionally kept dumb: it only stores opaque encrypted blobs. This separation is the foundation of Vault's security model. Even if your Raft snapshot leaks, an attacker gets encrypted ciphertext, not secrets.
The key that encrypts all data is called the encryption key. That encryption key itself is encrypted by a master key. The master key is split via Shamir's Secret Sharing into N shards (default: 5), of which M are required to reconstruct it (default: 3). Those shards are distributed to operators — this is the unseal process.
Seal and Unseal
When Vault starts (or after a restart, crash, or snapshot restore), it is in a sealed state. In this state, Vault can talk to its storage backend, but it cannot decrypt any data. It does not know the master key. It refuses all API requests. A sealed Vault is cryptographically locked.
Unsealing is the process of feeding enough key shards to Vault so it can reconstruct the master key, decrypt the encryption key, and begin serving requests. With the default Shamir setup, three of five operators each provide their shard — the master key is never assembled in full outside Vault's memory. Once unsealed, Vault holds the encryption key in memory only; it is never persisted to disk in plaintext.
vault operator init prints a root token. Root tokens bypass all policies — they are the equivalent of the UNIX root account. Store the root token in a hardware-backed secret store immediately after init, revoke it as soon as you have configured operational auth methods (vault token revoke <root-token>), and regenerate it only for emergency break-glass scenarios using vault operator generate-root with quorum. Running day-to-day operations with a root token is an audit finding at every major company.Auth Methods
Auth methods answer the question: How does a caller prove its identity to Vault? Vault is entirely agnostic about where callers come from — it supports pluggable auth methods that map external identities to Vault policies. The output of any successful auth method is a short-lived Vault token with attached policies. Once a caller has that token, the auth method is no longer in the picture.
The most important auth methods in production engineering:
- AppRole — designed for machines and CI pipelines. A Role ID (non-secret, embedded in config) plus a Secret ID (secret, injected at runtime by a trusted orchestrator) together prove identity. Netflix popularised this pattern for service-to-service secrets bootstrap.
- Kubernetes — the service account JWT from a Pod is presented to Vault. Vault calls the Kubernetes API to verify the JWT, checks it matches the expected namespace/service account, and issues a token. Zero static credentials required in the Pod.
- AWS IAM — a signed
GetCallerIdentityrequest proves the caller is a specific IAM role or user. Works for EC2, Lambda, ECS, and anywhere IAM credentials exist. No secrets in your AMI or container image. - OIDC / JWT — used to authenticate GitHub Actions, GitLab CI, and any OIDC-capable identity provider. The CI job's OIDC token is validated against the provider's JWKS endpoint.
- Token — the built-in method. Every Vault token can create child tokens. Used for bootstrapping and humans via the CLI. Not for production workloads.
Secret Engines
Secret engines are Vault's pluggable backends for generating and managing secrets. This is the most powerful aspect of Vault: rather than being a static key-value store, Vault can dynamically generate secrets that expire automatically. The secret engine mounted at a given path handles all operations at that path.
Core secret engines you will use in production:
- KV v2 (
kv-v2) — versioned key-value store. Every write creates a new version. Supports check-and-set to prevent write conflicts. Used for static secrets like API keys, OAuth client secrets, and third-party credentials that cannot be dynamically generated. - Database — connects to your database and dynamically generates short-lived credentials (username + password) for each caller. The credentials are automatically revoked when the lease expires. Works with PostgreSQL, MySQL, MongoDB, Redis, Elasticsearch, and more. Eliminates shared database passwords entirely.
- AWS — generates short-lived AWS IAM access keys, or assumes roles and returns STS credentials. A Lambda function gets a unique AWS key pair that auto-expires in 15 minutes, rather than sharing a long-lived key embedded in environment variables.
- PKI — a full certificate authority built into Vault. Issues X.509 certificates on demand. Each service gets its own short-lived TLS cert (minutes to hours), making certificate compromise nearly irrelevant because they expire before an attacker can use them. This is the basis of Lesson 8.
- Transit — encryption-as-a-service. Vault holds the key and never returns it; callers send plaintext and receive ciphertext (or vice versa). Used to encrypt database columns, message payloads, or PII without distributing the encryption key to every service.
Policies
Vault policies are the authorisation layer — they define what an authenticated identity can do. Policies are written in HCL (or JSON) and express path-based access control. Every capability (read, write, delete, list, create, update, patch, sudo) must be explicitly granted; Vault denies by default.
Policies are attached to tokens at auth time. A token can have multiple policies — the effective permission set is the union of all policies. The root policy is implicitly attached to root tokens only and bypasses all checks.
vault policy fmt lint check. Never edit policies directly in the Vault UI in production.Lease, Renewal, and Revocation
Every secret returned by a dynamic secret engine comes with a lease: a TTL after which Vault automatically revokes the credential at the source (e.g., drops the database user). This is the core operational advantage over static secrets — even if a credential leaks, it expires in minutes or hours rather than living forever until someone manually rotates it.
Applications that need secrets beyond the initial TTL must renew the lease before it expires. Vault Agent (a sidecar process) handles renewal transparently. When a service is decommissioned, an operator can immediately revoke all leases issued under a specific auth token or role, instantly invalidating every database credential and API key that service held — impossible with static secrets.
tmpfs volume as files, and keeps them renewed. Your application reads a file — no Vault dependency in application code, no secrets in environment variables, no secrets in image layers.High Availability and Raft
Production Vault runs as a cluster. The recommended storage backend since Vault 1.4 is integrated Raft storage: a built-in Raft consensus protocol that replicates data across cluster nodes. A typical production deployment is 3 or 5 nodes behind an internal load balancer. Only the active node handles writes; standby nodes proxy read/write requests to the active node or (with performance standbys) serve reads locally.
Raft snapshots are the disaster recovery mechanism. Automate a daily vault operator raft snapshot save to encrypted S3 or GCS. When a cluster is lost, bootstrap a new cluster, restore the snapshot, and unseal. With Auto Unseal via KMS, this process takes under 10 minutes.