Kafka Architecture for Operators
Kafka Architecture for Operators
Apache Kafka is a distributed commit log masquerading as a message broker. Understanding its internal architecture — not just how to use it from a producer or consumer perspective, but how data is stored, replicated, and coordinated across a cluster — is what separates operators who can keep a Kafka cluster healthy at scale from those who get paged at 3 AM and have no idea where to look. This lesson maps the entire internal architecture: brokers, topics, partitions, replication, and the controller, at a level of detail that lets you reason about production failures before they happen.
Brokers: The Unit of Scale
A Kafka broker is an ordinary JVM process running on a host. Its job is to persist and serve log segments for the partitions it is assigned. A production cluster at big-tech companies runs anywhere from 12 to several hundred brokers, typically on dedicated hosts with local NVMe or high-throughput EBS-optimized instances. Each broker is identified by a numeric broker.id (or, in KRaft mode, a UUID that maps to the same concept).
The broker stores partition data on disk as immutable, append-only segment files. Each segment is a pair: a .log file containing raw message bytes and an .index file mapping offsets to byte positions. When a segment reaches the configured log.segment.bytes (default 1 GB), it is rolled to a new active segment. The broker never overwrites existing segments — it only appends and eventually deletes old segments via the retention policy. This is the core design that makes Kafka fast: sequential I/O is an order of magnitude faster than random I/O on both spinning disks and SSDs.
acks=all). The log-structured storage model means recovery after a crash is simply replaying from the last committed offset — no WAL replay, no B-tree reconstruction.
Key broker configuration knobs every operator must know:
Topics, Partitions, and the Log
A topic is a logical feed of records. It is a namespace, not a storage unit. All actual data lives in partitions. When you create a topic with --partitions 12, you create 12 independent, ordered, append-only logs that are distributed across the cluster. Each partition is stored on exactly one broker at any moment (the leader), with copies on additional brokers (followers/replicas).
The partition count determines your maximum parallelism for consumption: a consumer group can have at most one active consumer per partition. If you have 12 partitions and 20 consumers in a group, 8 consumers will be idle. Partition count is permanent in the upward direction — you can increase partitions but never decrease them, and increasing partitions breaks key-based ordering for existing consumers because the same key may now hash to a different partition. Size your partition count correctly at creation time.
Production guidance: at LinkedIn (Kafka's birthplace), the rule of thumb is that a single partition can handle roughly 10 MB/s of throughput on modern hardware. For a topic expected to carry 1 GB/s, that implies 100 partitions. Across a 10-broker cluster, that is 10 partition leaders per broker, which is manageable. Clusters with more than 4,000 partitions per broker begin to show ZooKeeper (or KRaft metadata) pressure and longer failover times.
Replication: ISR and the Durability Contract
Kafka replication is pull-based: follower replicas continuously fetch from the leader, not the other way around. Each partition has exactly one leader broker and zero or more follower brokers. Together these form the replica set. The subset of replicas that are currently caught up with the leader is called the In-Sync Replica set (ISR).
The ISR is the heart of Kafka's durability guarantee. When a producer sends with acks=all (also written acks=-1), the leader does not acknowledge the write until every broker in the ISR has persisted the message. If a follower falls behind by more than replica.lag.time.max.ms (default 30 seconds), it is removed from the ISR. This means a slow or crashed follower does not block producers — it just leaves the ISR, degrading your redundancy level silently until it catches up or is replaced.
The critical operator invariant: min.insync.replicas is the floor. If the ISR shrinks below this value, producers with acks=all will receive a NotEnoughReplicasException and the partition becomes read-only for writes. This is intentional: it is better to reject writes than to accept them with insufficient durability. At Google-scale operations, min.insync.replicas=2 with replication.factor=3 is the universal baseline — it tolerates one broker failure without any write disruption, and requires two simultaneous failures before writes are blocked.
The Controller: Cluster Brain
The controller is a single broker elected to manage cluster metadata and coordinate partition leadership. In the legacy ZooKeeper-based architecture (Kafka pre-3.3), the controller was elected via an ephemeral ZooKeeper node — whichever broker claimed /controller first became the controller. When the controller crashed or its ZooKeeper session expired, other brokers raced to claim the node and a new controller was elected within seconds.
In the modern KRaft mode (Kafka 3.3+ GA, mandatory in Kafka 4.0+), ZooKeeper is removed entirely. The controller is now a Raft-based consensus group: a small set of brokers (typically 3 or 5, separate from or co-located with the data brokers) forms a Raft quorum. One is the active controller, the others are followers. The metadata log — every topic creation, partition reassignment, broker registration, ISR change — is itself a Kafka partition replicated across this quorum. This means controller failover is now sub-second rather than the 15-30 second ZooKeeper session timeout window that defined legacy Kafka SLOs.
The controller's responsibilities are:
- Leader election: When a partition leader fails, the controller picks the next in-sync replica to promote and pushes the new leader epoch to all affected brokers.
- Broker registration: When a broker starts or crashes, it registers/deregisters with the controller. The controller then triggers partition reassignments as needed.
- ISR changes: Brokers report ISR shrink/expand events to the controller, which persists them in the metadata log and propagates them to other brokers.
- Topic and partition management: All administrative operations (create/delete topic, reassign partitions, alter configs) flow through the controller.
Partition Assignment and Rack Awareness
Kafka distributes partition replicas across brokers to balance leader load and tolerate broker failures. The default assignment algorithm spreads replicas in a round-robin manner, but in a multi-rack or multi-AZ deployment, you need rack awareness to ensure that each partition has replicas in different physical failure domains.
Configure broker.rack on each broker to match its AZ or physical rack label. When Kafka creates a topic, it will ensure that the replica set for each partition spans different racks. This means losing an entire AZ will not cause data loss for any partition (as long as the remaining replicas are in the ISR).
--throttle with a value set to 20-30% of your inter-broker bandwidth. Also watch the ISR during reassignment: the new replica starts as an observer, builds its log, then joins the ISR — until it joins, your partition has reduced redundancy.
Key Failure Modes Every Operator Must Know
Unclean leader election: If unclean.leader.election.enable=true and all ISR members are down, Kafka can elect an out-of-sync replica as leader. This replica may be missing the last committed messages, causing data loss. Set this to false in all production clusters. The trade-off is availability: if the entire ISR is lost, the partition will be unavailable until an ISR member recovers. This is the correct trade-off for a durable log system.
Controller storm: Rapid broker restarts can trigger repeated controller elections, causing a storm of leader epoch bumps and metadata propagation across all brokers. Symptom: producers and consumers see frequent NOT_LEADER_OR_FOLLOWER and LEADER_NOT_AVAILABLE errors. Mitigation: stagger broker restarts with at least 30-60 seconds between each, and watch kafka.controller:type=KafkaController,name=ActiveControllerCount — it must always be exactly 1 across the cluster.
Log directory failure: If a broker loses a disk, all partitions on that log directory are taken offline. The broker remains up and serves its other log directories. Kafka 1.1+ introduced JBOD (Just a Bunch of Disks) support: configure multiple log.dirs entries pointing to separate disks. A disk failure only offline the partitions on that disk, not the entire broker. Use this in production instead of RAID.
UnderReplicatedPartitions — must be 0 at steady state; any value above 0 means your cluster is degraded. (2) ActiveControllerCount — must be exactly 1. (3) OfflinePartitionsCount — must be 0; any offline partition is a complete data unavailability event. (4) RequestHandlerAvgIdlePercent — should stay above 30%; dropping below 20% signals that the broker is CPU-bound and about to start dropping requests. Wire these into your existing Prometheus alerting stack; they are the four-metric SLO for any Kafka cluster.
Summary
Kafka's architecture is a set of deliberate trade-offs: append-only segment files for throughput, pull-based ISR replication for durability without blocking producers, a centralized controller (now Raft-based in KRaft) for consistent metadata, and rack-aware partition assignment for fault isolation. Every production failure mode — unclean elections, ISR shrink, controller storms, log directory failures — flows directly from these design decisions. Operators who understand the architecture can read the failure from metrics and logs in seconds; those who do not spend hours in the dark. In the next lesson we move to the producer and consumer side: how writes are batched, compressed, and acknowledged, and how consumer groups coordinate partition assignment.