NestJS — Enterprise Node.js

Message Brokers: RabbitMQ & Kafka

16 min Lesson 38 of 48

Message Brokers: RabbitMQ & Kafka

When services need to communicate without tight coupling, a message broker sits between them: producers send messages to the broker, and consumers read from it at their own pace. NestJS ships two first-class transporters for the most widely used brokers — RabbitMQ (AMQP queues) and Apache Kafka (distributed log topics). Choosing the right one shapes your system's delivery guarantees, throughput, and operational complexity.

RabbitMQ — queue-based messaging

RabbitMQ implements the AMQP protocol. Producers publish messages to exchanges; the exchange routes them to one or more queues; consumers pull from a queue and send an acknowledgement (ack) once processing succeeds. Unacknowledged messages are redelivered.

npm install @nestjs/microservices amqplib amqp-connection-manager

import { NestFactory } from '@nestjs/core';
import { Transport, MicroserviceOptions } from '@nestjs/microservices';

async function bootstrap() {
  const app = await NestFactory.createMicroservice<MicroserviceOptions>(AppModule, {
    transport: Transport.RMQ,
    options: {
      urls: ['amqp://localhost:5672'],
      queue: 'orders_queue',
      queueOptions: { durable: true },   // survive broker restart
      noAck: false,                      // manual acknowledgements
    },
  });
  await app.listen();
}
bootstrap();

A consumer handler uses @MessagePattern (RPC) or @EventPattern (fire-and-forget):

import { Controller } from '@nestjs/common';
import { EventPattern, Payload, Ctx, RmqContext } from '@nestjs/microservices';

@Controller()
export class OrdersConsumer {
  @EventPattern('order_placed')
  async handleOrderPlaced(
    @Payload() data: { orderId: string; total: number },
    @Ctx() context: RmqContext,
  ) {
    console.log('Processing order', data.orderId);
    // --- do work ---
    const channel = context.getChannelRef();
    const originalMsg = context.getMessage();
    channel.ack(originalMsg);   // acknowledge only after successful processing
  }
}

Manual acks prevent message loss. With noAck: false, a message stays in the queue until your handler explicitly acks it. If the process crashes mid-flight, RabbitMQ redelivers to another consumer — ensuring at-least-once delivery.

Apache Kafka — topic-based log streaming

Kafka stores messages in ordered, immutable topic partitions (a distributed commit log). Consumers belong to a consumer group; each partition is assigned to exactly one member of a group, enabling horizontal scaling while guaranteeing order within a partition. Messages are retained for a configurable period — not deleted on consumption.

npm install @nestjs/microservices kafkajs

import { NestFactory } from '@nestjs/core';
import { Transport, MicroserviceOptions } from '@nestjs/microservices';

async function bootstrap() {
  const app = await NestFactory.createMicroservice<MicroserviceOptions>(AppModule, {
    transport: Transport.KAFKA,
    options: {
      client: {
        brokers: ['localhost:9092'],
      },
      consumer: {
        groupId: 'orders-consumer-group',   // all instances share one group
      },
    },
  });
  await app.listen();
}
bootstrap();

Subscribing to a Kafka topic uses the same @EventPattern decorator:

import { Controller } from '@nestjs/common';
import { EventPattern, Payload, KafkaContext, Ctx } from '@nestjs/microservices';

@Controller()
export class InventoryConsumer {
  @EventPattern('inventory.updated')
  async handleInventoryUpdate(
    @Payload() message: { sku: string; quantity: number },
    @Ctx() context: KafkaContext,
  ) {
    const { offset, partition, topic } = context.getMessage();
    console.log(`[${topic}/${partition}@${offset}]`, message.sku);
    // Kafka auto-commits offset after the handler resolves (commitOffsets: true by default)
  }
}

Consumer groups and scaling

In Kafka, all service replicas share the same groupId. Kafka assigns each partition to one replica — so adding replicas increases throughput up to the number of partitions. In RabbitMQ, multiple consumers on the same queue compete for messages in round-robin fashion — scaling is simpler but there are no ordered partitions.

Choose based on your guarantees. Need task queues with at-most or at-least-once delivery, dead-letter queues, and routing flexibility? Use RabbitMQ. Need high-throughput event streams, replay, audit logs, or fan-out to multiple independent consumer groups? Use Kafka. Many production systems use both.

Choosing between them

RabbitMQ: lower operational overhead, rich routing (direct, topic, fanout exchanges), mature dead-letter support, ideal for work queues and RPC patterns.
Kafka: extremely high throughput (millions msg/s), durable ordered log, message replay, multiple independent consumer groups reading the same stream.
Both: support manual acknowledgements and at-least-once delivery; both have NestJS first-class support with identical decorator API.

Do not conflate queues and topics. A RabbitMQ queue delivers each message to one consumer (competing consumers). A Kafka topic can be consumed by multiple independent consumer groups, each getting a full copy of every message. Using a queue where you need fan-out silently drops events for all but one consumer.

Summary

NestJS supports both RabbitMQ (AMQP, queue-based, ack-driven) and Kafka (topic partitions, consumer groups, durable log) through identical @EventPattern / @MessagePattern decorators. RabbitMQ excels at task queues and routing; Kafka shines at high-throughput event streaming and replay. Always use manual acknowledgements in RabbitMQ to guarantee at-least-once delivery, and group your Kafka consumers correctly to balance partitions across replicas.