Jenkins & Enterprise CI/CD

Credentials & Secrets in Jenkins

18 min Lesson 7 of 28

Credentials & Secrets in Jenkins

Secrets are the single most dangerous artifact in any CI/CD system. A leaked credential in a build log, a hardcoded API key in a Jenkinsfile, or a broadly-scoped token shared across every pipeline has ended careers and triggered breach disclosures at companies of every size. Jenkins ships with a mature credentials subsystem — but using it safely at enterprise scale requires understanding how it works, where it stores things, and which patterns lead to silent data leakage.

The Credentials Store

Jenkins stores credentials in an encrypted key-value store backed by the Credentials Plugin (bundled since Jenkins LTS 2.x). The master encryption key lives in $JENKINS_HOME/secrets/master.key; individual credential ciphertexts are stored as XML files under $JENKINS_HOME/credentials.xml (global) or inside each $JENKINS_HOME/jobs/<name>/config.xml (folder/item scope).

The encryption envelope is two-tier: the master key is used to derive a per-credential AES-128 key via a secret seed file (hudson.util.Secret). This means backing up Jenkins without its secrets/ directory renders credentials unrecoverable — and conversely, copying just credentials.xml without the master key is safe as ciphertext.

Credential scoping in Jenkins: System → Global → Folder → Item. Narrower scope = smaller blast radius on compromise.

Credential Types

The Credentials Plugin ships several built-in types; enterprise plugins add more:

Username with password — the most common; bound to usernameVariable and passwordVariable.
Secret text — single opaque string (API tokens, signing keys).
SSH Username with private key — key entered directly or loaded from a file path on the agent; passphrase stored separately.
Certificate (PKCS#12) — mutual TLS, code signing. Bound via KEYSTORE_FILE / KEYSTORE_PASSWORD.
Secret file — arbitrary binary blob (kubeconfig, GCP JSON key). Materialised as a temp file on the agent's workspace; Jenkins cleans it after the step.
Vault: AppRole / AWS IAM / GCP Workload Identity — provided by the HashiCorp Vault Plugin, AWS Credentials Plugin, etc. No static secret ever stored in Jenkins.

Using Credentials in a Declarative Pipeline

The canonical way to expose a credential inside a pipeline step is the withCredentials block. Jenkins auto-masks any variable bound this way: if the raw value appears in any log line, it is replaced with ****.

// Declarative Pipeline — binding multiple credential types
pipeline {
  agent any
  environment {
    // shorthand: binds DOCKER_HUB_CREDS_USR + DOCKER_HUB_CREDS_PSW
    DOCKER_HUB_CREDS = credentials('dockerhub-prod')
  }
  stages {
    stage('Push Image') {
      steps {
        // explicit binding — preferred for non-usernamePassword types
        withCredentials([
          usernamePassword(credentialsId: 'dockerhub-prod',
                           usernameVariable: 'DHUB_USER',
                           passwordVariable: 'DHUB_PASS'),
          string(credentialsId: 'snyk-token',
                 variable: 'SNYK_TOKEN'),
          sshUserPrivateKey(credentialsId: 'deploy-key',
                            keyFileVariable: 'SSH_KEY')
        ]) {
          sh '''
            echo "$DHUB_PASS" | docker login -u "$DHUB_USER" --password-stdin
            docker push "$DHUB_USER/myapp:${BUILD_NUMBER}"
            snyk container test "$DHUB_USER/myapp:${BUILD_NUMBER}" \
              --token="$SNYK_TOKEN"
            ssh -i "$SSH_KEY" deploy@prod.internal \
              "docker pull $DHUB_USER/myapp:${BUILD_NUMBER} && systemctl restart app"
          '''
        }
      }
    }
  }
}

Echo leakage: Never run echo $SECRET or sh 'printenv' inside a withCredentials block. Jenkins masks exact matches, but base64-encoded or URL-encoded forms of the secret are not masked. Any transformation of the raw value can escape into logs unredacted.

Credential Scoping in Practice

Jenkins supports three meaningful scopes:

System — available only to Jenkins internals (node connections, mail server). Not usable inside pipeline steps. Use for infra-level credentials.
Global (domain: Jenkins) — visible to all jobs on the controller. Use only for genuinely shared, low-value credentials (e.g., a read-only Nexus mirror account).
Folder / Item — recommended for anything sensitive. A credential in /team-payments/ is invisible to jobs in /team-platform/. This is the primary blast-radius control in a multi-team Jenkins instance.

Role-Based Access Control (RBAC): Scope alone is not enough. Install the Role Strategy Plugin and grant Credentials/View only to the pipeline owner role. Without RBAC, any user who can configure a job can write a one-liner pipeline that prints every credential in scope to the build log.

Integrating with External Vaults

At scale, storing secrets inside Jenkins is an anti-pattern. The accepted enterprise approach is to treat Jenkins as a consumer, not a store. Secrets live in HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault; Jenkins retrieves them at runtime using short-lived tokens:

// HashiCorp Vault Plugin — Vault AppRole binding
// Jenkins credential: 'vault-approle' (Vault AppRole type)
pipeline {
  agent any
  stages {
    stage('Deploy') {
      steps {
        withVault(
          configuration: [
            vaultUrl: 'https://vault.internal:8200',
            vaultCredentialId: 'vault-approle',
            engineVersion: 2
          ],
          vaultSecrets: [
            [path: 'secret/data/payments/stripe',
             secretValues: [
               [envVar: 'STRIPE_KEY', vaultKey: 'api_key'],
               [envVar: 'STRIPE_WEBHOOK', vaultKey: 'webhook_secret']
             ]
            ]
          ]
        ) {
          sh 'node deploy-payment-service.js'
        }  // STRIPE_KEY and STRIPE_WEBHOOK vanish here
      }
    }
  }
}

With this pattern, secrets are never persisted in Jenkins: no XML on disk, no database row, no config.xml. Vault enforces TTL, audit logging, and dynamic rotation independently of Jenkins.

Masking Behaviour and Its Limits

Jenkins masking intercepts each println / log write and performs a literal string replacement. Understand its limits in production:

Masking is per-value, not per-byte-sequence. A 3-character secret (abc) matches trivially in log noise — avoid short secrets.
Multi-line secrets (PEM keys) are masked line by line only if the Mask Passwords Plugin is active. The built-in masking covers only the exact bound string.
Archived test reports, JUnit XML, Jacoco HTML, and other artifacts are never masked. A stack trace can contain a secret passed as an argument.
Agent console output forwarded to external log aggregators (Splunk, ELK) passes through Jenkins masking before transmission — but only if the Pipeline Logging Plugin or equivalent interceptor is in the path.

Audit-log every credential use. Enable the Audit Trail Plugin + configure it to write to a write-once S3 bucket or a SIEM. You want an immutable record of which pipeline used which credential ID and when — essential for breach investigation and SOC 2 / ISO 27001 compliance.

Credential Rotation without Downtime

The safest rotation strategy is: create the new credential with a new ID, update all pipeline references in source control (Jenkinsfile changes go through PR review), deploy the updated pipelines, then delete the old credential. Never overwrite a credential in-place mid-pipeline-run — in-flight builds hold a reference to the credential ID and will re-read the value; if the old value is now invalid, those builds fail mid-flight in a non-deterministic way.

Least-Privilege Checklist

Use folder-scoped credentials for every team or project boundary.
Grant pipelines the narrowest token scope possible (read-only deploy token vs. repo admin token).
Prefer short-lived dynamic secrets from Vault or cloud IAM over long-lived static tokens.
Never echo or log secrets, even in a catch block.
Rotate all credentials on any engineer offboarding or breach suspicion.
Back up $JENKINS_HOME/secrets/ separately, encrypted, with access auditing.