Jenkins & Enterprise CI/CD

Jenkins Architecture & Setup

18 min Lesson 1 of 28

Jenkins Architecture & Setup

Jenkins is the industry's most widely deployed automation server. Built on Java, it has powered CI/CD pipelines at companies of every size since 2011 — and despite the rise of hosted alternatives, it remains the go-to solution wherever teams need deep customization, air-gapped operation, or the ability to orchestrate work across hundreds of heterogeneous machines. Before you write a single line of Jenkinsfile, you need a precise mental model of how Jenkins is structured and why that structure exists.

Controller / Agent Architecture

Jenkins is built around a clear two-tier model. The controller (historically called the "master") is the brain: it hosts the web UI, stores configuration and build history, reads pipeline definitions, schedules jobs, and dispatches work to agents (historically called "slaves"). Agents are the muscle: dumb workers that run the actual build commands and report results back to the controller.

This separation exists for three production-critical reasons:

Isolation: Build processes execute untrusted code, consume significant CPU/memory, and can leave artifacts on disk. Keeping that off the controller protects the scheduling engine from runaway builds and supply-chain attacks.
Scale: A single controller can manage hundreds of agents. Horizontal agent scaling is independent of controller capacity — adding build throughput never requires touching the controller.
Heterogeneity: Different jobs need different environments. An iOS build needs macOS; a Windows .NET job needs Visual Studio; a GPU job needs CUDA drivers. Each agent can carry a different OS, toolchain, or hardware.

Jenkins controller / agent topology. The controller schedules and dispatches; agents execute. Each agent type carries a label that pipeline stages target.

How Agents Connect

Agents connect to the controller in one of two modes:

SSH (inbound from controller): The controller SSHes into the agent, copies the agent.jar (remoting library), and starts it. The agent machine needs an SSH server and a reachable address. This is the traditional mode for persistent VMs.
JNLP / WebSocket (outbound from agent): The agent initiates the connection to the controller on port 50000 (or via WebSocket on port 8080). This is mandatory when agents are behind NAT, inside Kubernetes pods, or on ephemeral cloud VMs. The Kubernetes plugin exclusively uses this mode — pods connect out, the controller never reaches in.

Key idea: Prefer the WebSocket transport in new installations. It multiplexes over the same HTTPS port (8080 or 443 behind a reverse proxy), eliminating the need to open port 50000 through firewalls. Enable it under Manage Jenkins → Security → Agents → WebSocket.

Installing Jenkins (Production-Grade)

The fastest path to a durable controller on Ubuntu is the official APT repository, which gives you signed packages and clean upgrade paths. Below is the standard 2025 installation sequence.

# --- 1. Prerequisites ---
sudo apt-get update
sudo apt-get install -y fontconfig openjdk-21-jdk

# Verify Java
java -version
# openjdk version "21.0.x" 2024-...

# --- 2. Add Jenkins APT repository ---
sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key

echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian-stable binary/" \
  | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null

# --- 3. Install ---
sudo apt-get update
sudo apt-get install -y jenkins

# --- 4. Start & enable ---
sudo systemctl enable --now jenkins

# --- 5. Confirm status ---
sudo systemctl status jenkins

# Retrieve the initial admin password
sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Jenkins is now listening on http://<server-ip>:8080. Open it in a browser and paste the initial admin password to start the setup wizard. Choose Install suggested plugins for a baseline that includes Git, Pipeline, Blue Ocean, and Credentials plugins.

Initial Security Hardening

A freshly installed Jenkins with default settings is not production-safe. The following steps are non-negotiable before exposing the controller to the network.

# --- 6. Reverse proxy with TLS (nginx example) ---
# Install nginx and certbot
sudo apt-get install -y nginx certbot python3-certbot-nginx

# /etc/nginx/sites-available/jenkins
server {
    listen 80;
    server_name jenkins.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name jenkins.example.com;

    ssl_certificate     /etc/letsencrypt/live/jenkins.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/jenkins.example.com/privkey.pem;

    # Security headers
    add_header Strict-Transport-Security "max-age=63072000" always;
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;

    location / {
        proxy_pass         http://127.0.0.1:8080;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto https;

        # Required for WebSocket agent transport
        proxy_http_version 1.1;
        proxy_set_header   Upgrade    $http_upgrade;
        proxy_set_header   Connection "upgrade";
    }
}

# Obtain certificate and reload nginx
sudo certbot --nginx -d jenkins.example.com
sudo systemctl reload nginx

Security Model: What to Configure Immediately

After the setup wizard, navigate to Manage Jenkins → Security and lock down the following areas:

Authentication: Switch from "Jenkins own user database" to your corporate SSO as soon as possible. Jenkins supports LDAP, SAML 2.0 (via plugin), and GitHub OAuth. Single-sign-on means no separate Jenkins password rotation burden and immediate deprovisioning when employees leave.
Authorization: Set the strategy to Role-Based Access Control (RBAC) using the Role Strategy Plugin. At minimum, define three roles: admin (full access), developer (read + build + cancel own builds), and read-only (view logs, no trigger). Never run in "Anyone can do anything" mode on a real controller.
Agent security: Enable Agent → Controller Security (Manage Jenkins → Security → Agent Protocols). This prevents a compromised agent from reading arbitrary controller files or manipulating the job queue. It is off by default in some older installs.
Script Security: The Script Security Plugin restricts what Groovy code in pipelines can execute. Groovy sandbox mode is mandatory — disable it only for specific trusted shared libraries and only with code review.

Pro practice: Immediately after installation, go to Manage Jenkins → Plugins → Available and install the OWASP Dependency-Check, Warnings Next Generation, and Job DSL plugins. These three give you vulnerability scanning, aggregated static analysis, and pipeline-as-code job definition — the three capabilities you will need within your first week of real usage.

Jenkins Home Directory

Everything Jenkins owns lives under JENKINS_HOME — by default /var/lib/jenkins. Understanding its layout is critical for backups and disaster recovery:

config.xml — global configuration: security settings, SCM tools, JDK/Maven/Node tool definitions.
credentials.xml — encrypted credential store. Guard this file; losing it means losing the ability to decrypt stored secrets.
jobs/ — one sub-directory per job, containing config.xml (job definition) and builds/ (build history, logs, artifacts).
nodes/ — agent node configuration files.
plugins/ — installed plugin JARs and their data.
secrets/ — master encryption key and secret files. Mode 0700; never back up to a public location.
workspace/ — build workspaces. Do NOT back this up; it is ephemeral and often large.

Production pitfall: The single most common disaster is a full disk caused by JENKINS_HOME/jobs/<job>/builds/ growing unboundedly. Every Declarative Pipeline should include a buildDiscarder option: options { buildDiscarder(logRotator(numToKeepStr: '30', artifactNumToKeepStr: '5')) }. Set this globally in Manage Jenkins → System → Global Pipeline Libraries and locally in every Jenkinsfile. Controllers that run out of disk crash hard and can corrupt the job database.

Connecting Your First Agent

With the controller running and secured, add a persistent Linux agent via SSH to off-load build workloads immediately.

# On the AGENT machine — create a dedicated jenkins user
sudo useradd -m -s /bin/bash jenkins-agent
sudo mkdir -p /home/jenkins-agent/.ssh
sudo chmod 700 /home/jenkins-agent/.ssh

# On the CONTROLLER machine — generate an SSH key pair
ssh-keygen -t ed25519 -C "jenkins-controller" -f ~/.ssh/jenkins_agent_ed25519 -N ""

# Copy the public key to the agent
ssh-copy-id -i ~/.ssh/jenkins_agent_ed25519.pub jenkins-agent@<AGENT_IP>

# In Jenkins UI:
#  Manage Jenkins > Nodes > New Node
#    Name: linux-agent-01
#    Type: Permanent Agent
#    # of executors: (CPU core count - 1)
#    Remote root directory: /home/jenkins-agent
#    Labels: linux docker
#    Launch method: Launch agents via SSH
#      Host: <AGENT_IP>
#      Credentials: (add the private key from ~/.ssh/jenkins_agent_ed25519)
#      Host Key Verification: Known hosts file (add agent to ~/.ssh/known_hosts first)

# On the AGENT machine — install Docker so pipelines can run containers
sudo apt-get install -y docker.io
sudo usermod -aG docker jenkins-agent

What the Controller Should Never Do

A hardened Jenkins controller should have zero executors — navigate to Manage Jenkins → Nodes → Built-In Node → Configure and set Number of executors to 0. This is one of the most important production hardening steps and is consistently skipped by teams new to Jenkins.

Running builds on the controller exposes the entire JENKINS_HOME — including credentials, job secrets, and the encryption key — to arbitrary build code. If a pipeline checks out a repository with a malicious Jenkinsfile or a build script, it can read secrets/master.key directly. Off-loading all execution to agents bounds the blast radius of a compromised build to one agent workspace.

Architecture at Scale: What Google-Scale Jenkins Looks Like

Large organizations (Netflix, LinkedIn, Walmart Labs) run Jenkins in a controller-per-team or controller-per-domain topology rather than a single shared controller. A single controller has a practical ceiling around 200–300 concurrent builds before the JVM heap and filesystem I/O become bottlenecks. The Kubernetes plugin is the standard solution for elastic agent provisioning: each build stage spins up a fresh pod, runs, and is destroyed — eliminating agent drift and enabling per-job toolchain isolation without maintaining a fleet of persistent VMs.

The next lesson explores Freestyle jobs vs. Pipelines — the two job types Jenkins offers and why Declarative Pipelines have replaced Freestyle for every serious CI/CD use case.