Every Linux server you ship to production is a target the moment it acquires a public IP address. Automated scanners find open ports within minutes; credential-stuffing bots hammer SSH around the clock. Security hardening is not a one-time checkbox — it is a layered, disciplined practice of reducing attack surface, limiting blast radius, and detecting intrusions early. This lesson walks through the four pillars that every DevOps engineer must own: surface minimization, firewall management with ufw/firewalld, brute-force protection with fail2ban, and automated patching aligned with CIS Benchmarks.
Pillar 1 — Minimize Attack Surface
The fewest services, the fewest packages, the fewest open ports — that is the goal. Every additional daemon is an additional CVE waiting to be announced. On a fresh Ubuntu or RHEL node, audit what is running immediately:
# List every listening socket and the process that owns it
ss -tlnp
# Disable and mask services you will never need
systemctl disable --now avahi-daemon cups bluetooth ModemManager
systemctl mask avahi-daemon cups bluetooth ModemManager
# Remove packages that are not required (Ubuntu example)
apt-get purge -y telnet rsh-client rsh-redone-client nis talk
apt-get autoremove -y
# Verify no world-writable directories or SUID binaries were introduced
find / -xdev -perm -0002 -type d 2>/dev/null | sort
find / -xdev \( -perm -4000 -o -perm -2000 \) -type f 2>/dev/null | sort
Masking (not just disabling) a unit writes a symlink to /dev/null so neither humans nor scripts can accidentally start it. Use masking for services with no legitimate use on that host.
Pro practice: In AWS/GCP/Azure, choose minimal base images (Amazon Linux 2023 minimal, Ubuntu Minimal, RHEL minimal) and layer only what your workload needs. The smaller the image, the smaller the CVE surface — and the faster your CI security scans run.
Also harden kernel parameters via sysctl. The /etc/sysctl.d/99-hardening.conf file is the right place for persistent changes:
# /etc/sysctl.d/99-hardening.conf
# Disable IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
# Ignore ICMP broadcast requests (Smurf attack mitigation)
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Enable SYN cookies (SYN flood protection)
net.ipv4.tcp_syncookies = 1
# Disable packet forwarding unless this host is a router
net.ipv4.ip_forward = 0
net.ipv6.conf.all.forwarding = 0
# Restrict kernel pointer leaks via /proc
kernel.kptr_restrict = 2
kernel.dmesg_restrict = 1
# Prevent core dumps from setuid programs
fs.suid_dumpable = 0
Apply immediately with sysctl --system and verify: sysctl net.ipv4.tcp_syncookies.
Pillar 2 — Host Firewall with ufw / firewalld
Cloud security groups and VPC ACLs are your perimeter. The host firewall is your last line of defence — it limits damage when a misconfigured security group or a lateral-movement attack reaches the instance. Never rely on just one layer.
ufw (Uncomplicated Firewall) wraps iptables/nftables and is the standard on Ubuntu/Debian. firewalld is the standard on RHEL/CentOS/Fedora — both achieve the same goal:
### --- ufw (Ubuntu/Debian) ---
ufw default deny incoming
ufw default allow outgoing
# Allow only what this server actually needs
ufw allow 22/tcp # SSH — lock to a specific source IP in production
ufw allow 80/tcp
ufw allow 443/tcp
# Restrict SSH to a known management CIDR instead of the world
ufw delete allow 22/tcp
ufw allow from 10.0.0.0/8 to any port 22 proto tcp
ufw enable
ufw status verbose
### --- firewalld (RHEL/Fedora) ---
firewall-cmd --set-default-zone=drop # deny everything by default
firewall-cmd --zone=drop --add-service=ssh --permanent
firewall-cmd --zone=drop --add-service=http --permanent
firewall-cmd --zone=drop --add-service=https --permanent
firewall-cmd --reload
firewall-cmd --zone=drop --list-all
Production pitfall: Running ufw enable over an existing SSH session will lock you out if port 22 is not already allowed. Always add your SSH rule before enabling ufw. On cloud VMs, also verify the security group allows your source IP before making the host firewall active.
Pillar 3 — Brute-Force Protection with fail2ban
fail2ban tails log files, matches patterns via configurable regexes (called filters), and temporarily bans offending IPs using iptables/nftables rules. It ships with filters for SSH, nginx, Apache, Postfix, and dozens of other services out of the box.
# Install
apt-get install -y fail2ban # Debian/Ubuntu
dnf install -y fail2ban # RHEL/Fedora
# NEVER edit /etc/fail2ban/jail.conf — it is overwritten on upgrades.
# Instead, create /etc/fail2ban/jail.local for overrides:
cat > /etc/fail2ban/jail.local <<'EOF'
[DEFAULT]
# Ban for 1 hour after 5 failures within a 10-minute window
bantime = 3600
findtime = 600
maxretry = 5
# Use nftables backend on modern systems (or iptables on older)
banaction = nftables-multiport
banaction_allports = nftables-allports
[sshd]
enabled = true
port = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s
maxretry = 3
[nginx-http-auth]
enabled = true
[nginx-botsearch]
enabled = true
maxretry = 2
EOF
systemctl enable --now fail2ban
# Inspect bans
fail2ban-client status sshd
fail2ban-client status sshd | grep 'Banned IP'
# Manually unban an IP (e.g. after locking yourself out)
fail2ban-client set sshd unbanip 198.51.100.42
Pro practice: In large fleets, fail2ban is complemented by a WAF (AWS WAF, Cloudflare) upstream and a SIEM (Splunk, Elastic) downstream. fail2ban handles the per-host tactical response; the SIEM correlates patterns across hundreds of hosts to catch distributed low-and-slow attacks that never trigger a single node's threshold.
Pillar 4 — Patch Management and CIS-Style Hardening
Unpatched packages are the most common initial access vector. Automate security updates — do not rely on humans to remember to run apt upgrade. On Ubuntu, unattended-upgrades handles this. On RHEL, dnf-automatic does the same job.
# Ubuntu — enable automatic security-only updates
apt-get install -y unattended-upgrades
dpkg-reconfigure --priority=low unattended-upgrades
# Verify the config (should already be correct after reconfigure)
# /etc/apt/apt.conf.d/50unattended-upgrades:
# Unattended-Upgrade::Allowed-Origins {
# "${distro_id}:${distro_codename}-security";
# };
# Unattended-Upgrade::Automatic-Reboot "false"; # reboot in a maintenance window instead
# Unattended-Upgrade::Mail "ops@example.com";
# RHEL/Fedora — dnf-automatic
dnf install -y dnf-automatic
# /etc/dnf/automatic.conf: upgrade_type = security, apply_updates = yes
systemctl enable --now dnf-automatic-install.timer
# Audit which packages need security updates right now (Ubuntu)
apt-get -s upgrade 2>/dev/null | grep -i security
Beyond patching, the CIS Benchmarks (Center for Internet Security) are the industry-standard hardening guides for every major Linux distribution. Key CIS-aligned controls to apply immediately include:
SSH hardening — PermitRootLogin no, PasswordAuthentication no, MaxAuthTries 3, AllowUsers deploy, Protocol 2 in /etc/ssh/sshd_config.d/99-hardening.conf (a drop-in, not editing the main file)
Password policy — pam_pwquality or pam_cracklib for complexity; PASS_MAX_DAYS 90 in /etc/login.defs
Audit daemon — install auditd and load a ruleset based on CIS or STIG; this creates a tamper-evident log of privileged operations
AppArmor / SELinux — keep enabled in enforcing mode; never set SELINUX=disabled in production
Key idea: Use an automated scanner to measure your compliance. lynis is an open-source tool that audits a live system against CIS controls and produces a hardening index score with prioritized recommendations. Run lynis audit system after hardening to quantify your posture and track regressions over time.
In large organizations, these controls are codified in infrastructure-as-code — Ansible hardening roles (e.g. devsec.hardening), Terraform modules, or Packer AMI pipelines — so every new instance is born hardened rather than hardened post-facto. Manual hardening is a starting point; automation is the destination.