Networking Essentials for DevOps

Firewalls & Network Security

18 min Lesson 7 of 30

Firewalls & Network Security

A firewall is the first line of defence for every host and every network segment in your infrastructure. As a DevOps engineer you will configure and debug firewalls daily — on bare-metal servers, inside Kubernetes nodes, and through cloud-provider abstractions. Understanding what sits behind the GUI checkbox is the difference between shipping a working ruleset and opening an accidental hole in production.

Packet Filtering — The Core Mental Model

Every firewall applies a decision to each packet: ACCEPT, DROP, or REJECT. Rules are evaluated in order; the first match wins. At the end of every chain sits a default policy. The only production-safe defaults are DROP on INPUT and FORWARD, ACCEPT on OUTPUT. Starting from deny-all and allowing only what is needed is a security fundamental — it is the principle of least privilege applied to traffic.

Firewalls operate at different layers. A pure packet filter (Layer 3/4) inspects only source IP, destination IP, protocol, and port. A stateful firewall also tracks the state of connections — whether a packet is part of an established session, a new connection request, or an invalid out-of-context packet. Stateful filtering is what makes it safe to have a permissive OUTPUT policy: the firewall automatically allows return traffic for connections your host initiated without requiring explicit inbound rules.

Connection tracking states in a stateful firewall: NEW, ESTABLISHED, RELATED are allowed; INVALID packets are always dropped.

iptables — Linux Packet Filtering from First Principles

iptables is the traditional Linux firewall interface, still running on the majority of production Linux systems including Docker hosts and pre-1.29 Kubernetes nodes. It organises rules into tables (filter, nat, mangle, raw) and chains (INPUT, OUTPUT, FORWARD, PREROUTING, POSTROUTING). For host security the filter table is all you need.

# --- Minimal production-hardened iptables ruleset for an app server ---
# Run as root. Order matters: rules are evaluated top to bottom.

# 1. Flush existing rules and zero counters
iptables -F
iptables -X
iptables -Z

# 2. Set default policies to DROP on INPUT and FORWARD, ACCEPT on OUTPUT
iptables -P INPUT   DROP
iptables -P FORWARD DROP
iptables -P OUTPUT  ACCEPT

# 3. Allow loopback (required — many apps communicate over 127.0.0.1)
iptables -A INPUT -i lo -j ACCEPT

# 4. Allow return traffic for connections this host initiated (stateful)
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# 5. Drop invalid packets immediately (before any ACCEPT rule)
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

# 6. Allow SSH from a specific management CIDR only
iptables -A INPUT -p tcp --dport 22 -s 10.0.0.0/8 -m conntrack --ctstate NEW -j ACCEPT

# 7. Allow inbound HTTPS from anywhere
iptables -A INPUT -p tcp --dport 443 -m conntrack --ctstate NEW -j ACCEPT

# 8. Allow inbound HTTP (only if you redirect HTTP to HTTPS at the LB, not here)
# iptables -A INPUT -p tcp --dport 80 -m conntrack --ctstate NEW -j ACCEPT

# 9. Allow ICMP echo (ping) — useful for monitoring; rate-limit to avoid amplification
iptables -A INPUT -p icmp --icmp-type echo-request -m limit --limit 5/s --limit-burst 10 -j ACCEPT

# 10. Log and drop everything else (first 5/min so logs stay manageable)
iptables -A INPUT -m limit --limit 5/min --limit-burst 10 -j LOG --log-prefix "iptables-DROP: " --log-level 4
iptables -A INPUT -j DROP

# Persist across reboots (Debian/Ubuntu)
apt-get install -y iptables-persistent
netfilter-persistent save

Why DROP instead of REJECT? DROP silently discards the packet; the sender waits until it times out. REJECT sends an ICMP "port unreachable" immediately. On the internet, DROP is preferred for inbound because it gives attackers no confirmation that the port exists. Inside a trusted network, REJECT speeds up debugging by failing fast. Use REJECT in the FORWARD chain between internal segments so your developers get quick feedback on misconfigured service calls.

nftables — The Modern Replacement

nftables ships as the default on Debian 10+, RHEL 8+, and Ubuntu 22.04+. It replaces iptables, ip6tables, arptables, and ebtables with a single unified framework. The rule language is more expressive, evaluates faster using kernel-side maps and sets (no linear scan for IP allowlists), and the configuration is a single coherent file rather than a sequence of CLI commands.

# /etc/nftables.conf — equivalent ruleset to the iptables example above

flush ruleset

table inet filter {

    # Named set for trusted management IPs (O(1) lookup, no chain scan)
    set mgmt_cidrs {
        type ipv4_addr
        flags interval
        elements = { 10.0.0.0/8, 172.16.0.0/12 }
    }

    chain input {
        type filter hook input priority 0; policy drop;

        iif lo accept                                        comment "loopback always allowed"
        ct state established,related accept                  comment "return traffic"
        ct state invalid drop                                comment "drop malformed state"

        ip protocol icmp icmp type echo-request limit rate 5/second accept

        tcp dport 22 ip saddr @mgmt_cidrs ct state new accept  comment "SSH from mgmt only"
        tcp dport 443 ct state new accept                       comment "HTTPS from anywhere"

        limit rate 5/minute log prefix "nft-DROP: " drop    comment "log before final drop"
    }

    chain forward {
        type filter hook forward priority 0; policy drop;
    }

    chain output {
        type filter hook output priority 0; policy accept;
    }
}

Pro practice: At Google and similar companies, nftables sets are used to maintain dynamic IP blocklists (IP reputation lists) that are updated by automation without touching the chain rules themselves. The nft add element inet filter blocklist { 1.2.3.4 } command adds an IP to a set atomically with no rule flush — zero downtime, sub-millisecond effect.

Cloud Security Groups — Stateful Filtering as a Service

AWS Security Groups, GCP Firewall Rules, and Azure NSGs are the cloud-native equivalent of a stateful firewall. They are enforced at the hypervisor level — traffic that the security group blocks never reaches the instance, not even the NIC. This means they cannot be bypassed by misconfiguring iptables inside the VM.

Key properties of AWS Security Groups that trip up engineers:

Stateful by default. If you allow port 443 inbound, the return traffic is automatically permitted — you do not need an outbound rule for port 443 responses.
Allow-only. There is no deny rule in a security group. To block specific traffic you must use a Network ACL (NACL), which is stateless and does have explicit deny rules.
Attached to ENIs, not instances. An instance with two network interfaces can have different security groups on each interface — important for multi-homed bastion or proxy designs.
Source can be another security group ID. Instead of hard-coding IP ranges that change as you scale, reference the security group of the calling tier directly: sg-0abc123 (app-servers) → sg-0def456 (rds-mysql) on port 3306. Auto-scales correctly.

Three-tier AWS security group design: each tier only accepts traffic from the security group of the tier above it, not from raw IP ranges.

Stateful vs. Stateless Filtering — When Each Applies

Network ACLs (NACLs) in AWS are stateless: you must write explicit rules for both directions of a flow. They also evaluate rules in numeric order (100, 200, 300…) and stop at the first match, including explicit DENYs. NACLs are your last-resort blocklist — use them to deny specific CIDR ranges at the subnet boundary when a security group cannot express a deny. In practice, most teams use Security Groups for allow-list logic and NACLs only for broad IP-range blocking (known bad actor ASNs, geographic blocks).

Production pitfall — ephemeral ports: TCP connections use a server port (443) and a client-chosen ephemeral port (typically 1024–65535). Stateless firewalls (NACLs, some on-premise systems) must explicitly allow the ephemeral port range on return traffic. Forgetting this is one of the most common causes of "works outbound, fails on the way back" bugs. The fix is to allow TCP return traffic on ports 1024–65535 from the destination. Stateful firewalls handle this automatically — another reason to prefer them.

Common Failure Modes and How to Debug Them

Rule order error: A broad ACCEPT before a specific DROP means the DROP is never reached. Always put specific rules before general ones. On iptables, use iptables -L -n -v --line-numbers to inspect order and hit counters.
New chain never linked: Creating a custom chain with iptables -N my-chain does nothing unless you jump to it from INPUT with -j my-chain. Forgetting the jump is a silent no-op.
Security group too permissive: 0.0.0.0/0 on port 22 is the single most common cloud misconfiguration found in production breach post-mortems. Replace with the CIDR of your VPN or bastion host.
Docker overwriting iptables: Docker inserts rules into the FORWARD chain and the DOCKER and DOCKER-USER chains. If you flush iptables manually, Docker-managed containers lose network connectivity. Add your custom rules to the DOCKER-USER chain, which Docker never overwrites.

Pro practice: At scale, treat firewall rules as code. Store nftables.conf or Terraform security group resources in version control, enforce peer review on changes, and run automated compliance checks (AWS Config rules, opa policies) that alert when 0.0.0.0/0 appears on sensitive ports. Manual firewall changes in production are a change-management anti-pattern.