Building Microservices with Spring Boot

Distributed Logging & Correlation IDs

18 min Lesson 8 of 12

Distributed Logging & Correlation IDs

When a single user request travels through an order service, an inventory service, and a notification service, three separate log files record three separate fragments of the same story. Without a shared identifier threading those fragments together, debugging a production failure means manually correlating timestamps across services — an error-prone exercise that can take hours. Correlation IDs solve this problem by attaching a unique identifier to every request at the edge and propagating it through every downstream call, so you can grep a single value and reassemble the full trace.

The Problem in Concrete Terms

Consider a request that enters your API gateway at 14:32:01.042. The gateway calls the order service, which calls inventory, which calls the warehouse adapter. Each service logs its own events. Without correlation:

You search order-service logs for the user email — you find three records that match different users.
You cannot tell which inventory call belongs to which order call.
A 503 buried in the warehouse adapter log is invisible unless you happen to look there.

With a correlation ID (X-Correlation-ID: a7f3c91b-4d2e-4f8a-b6c1-9e0d3a2b1f5c) present in every log line, a single search across your log aggregator (Splunk, Loki, CloudWatch Insights) surfaces every event in the entire chain in chronological order.

Micrometer Tracing + Brave: The Spring Boot 3 Approach

Spring Boot 3 ships with Micrometer Tracing as its built-in distributed tracing abstraction, replacing the deprecated Spring Cloud Sleuth. Micrometer Tracing wraps a pluggable tracer bridge — in most projects that bridge is Brave (from the Zipkin ecosystem). Add these to each microservice pom.xml:

<!-- Micrometer Tracing with Brave bridge -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
<!-- Zipkin reporter (optional — exports spans to Zipkin/Tempo) -->
<dependency>
    <groupId>io.zipkin.reporter2</groupId>
    <artifactId>zipkin-reporter-brave</artifactId>
</dependency>

With just these dependencies, Spring Boot auto-configures a Tracer. Every incoming HTTP request automatically receives a trace ID (identifies the entire distributed request tree) and a span ID (identifies the current unit of work within that tree). Both are injected into SLF4J MDC under the keys traceId and spanId, making them available to every log line automatically.

Trace ID vs Span ID: The trace ID stays constant across the entire distributed call chain. The span ID changes at every service boundary — each service creates a new child span. Think of the trace as the shipping tracking number and each span as one leg of the delivery journey.

Configuring the Log Pattern

Update application.yml in each service so both IDs appear in every log line:

# application.yml
spring:
  application:
    name: order-service

logging:
  pattern:
    console: "%d{HH:mm:ss.SSS} [%thread] %-5level [%X{traceId:-},%X{spanId:-}] %logger{36} - %msg%n"

management:
  tracing:
    sampling:
      probability: 1.0   # 100% in dev; drop to 0.1 or lower in production

The %X{traceId:-} syntax reads the MDC key traceId; the :- suffix means "empty string if absent" so log lines from background threads that have no active trace still render cleanly.

A log line now looks like:

14:32:01.055 [http-nio-8081-exec-3] INFO  [a7f3c91b4d2e4f8a,b6c19e0d3a2b1f5c] c.example.OrderService - Created order 9921 for user 42

Propagating the Trace to Downstream Services

Micrometer Tracing propagates trace context automatically when you use WebClient or RestClient, because Spring Boot registers a tracing exchange filter. For WebClient you must obtain the bean Spring Boot creates — do not build your own bare instance:

@Configuration
public class WebClientConfig {

    // Inject the auto-configured builder — it already has the tracing filter
    @Bean
    public WebClient inventoryClient(WebClient.Builder builder) {
        return builder
                .baseUrl("http://inventory-service")
                .build();
    }
}

When OrderService calls inventoryClient, Spring automatically adds the b3 (or W3C traceparent) header to the outgoing request. The inventory service reads that header, creates a child span, and logs with the same trace ID. The whole chain is linked.

Use W3C Trace Context headers in new projects. Configure management.tracing.propagation.type=W3C in all services. The W3C traceparent header is the IETF standard and is understood by AWS X-Ray, Azure Monitor, and Google Cloud Trace out of the box. The older B3 format (Zipkin-style) is still common but less universally supported.

Manual Span Creation for Business Operations

Auto-instrumented HTTP spans tell you that a call was made and how long it took, but they say nothing about what happened inside your business logic. Create explicit child spans for operations that matter:

@Service
@RequiredArgsConstructor
public class OrderService {

    private final Tracer tracer;
    private final InventoryClient inventoryClient;

    public Order placeOrder(OrderRequest request) {
        // Create a named child span for the reservation step
        Span reserveSpan = tracer.nextSpan().name("inventory.reserve").start();
        try (Tracer.SpanInScope ws = tracer.withSpan(reserveSpan)) {
            reserveSpan.tag("product.id", String.valueOf(request.getProductId()));
            reserveSpan.tag("quantity", String.valueOf(request.getQuantity()));

            boolean reserved = inventoryClient.reserve(request.getProductId(), request.getQuantity());
            if (!reserved) {
                reserveSpan.tag("reservation.result", "insufficient_stock");
                throw new InsufficientStockException(request.getProductId());
            }
            reserveSpan.tag("reservation.result", "success");
            return saveOrder(request);
        } catch (Exception e) {
            reserveSpan.error(e);
            throw e;
        } finally {
            reserveSpan.end();
        }
    }
}

The tag() calls attach searchable key-value metadata to the span. When you open this trace in Zipkin or Grafana Tempo you will see a visual timeline with inventory.reserve nested beneath the HTTP span, labelled with the product ID and result.

Adding a Custom Correlation ID Header

Sometimes you need a business-level correlation ID that your clients or partners supply — for example, a payment gateway sends a X-Payment-Reference that must appear in every log related to that payment. You can extract it and write it to MDC alongside the trace ID:

@Component
@Order(Ordered.HIGHEST_PRECEDENCE)
public class CorrelationIdFilter extends OncePerRequestFilter {

    private static final String HEADER = "X-Correlation-ID";
    private static final String MDC_KEY = "correlationId";

    @Override
    protected void doFilterInternal(HttpServletRequest request,
                                    HttpServletResponse response,
                                    FilterChain chain) throws ServletException, IOException {
        String correlationId = request.getHeader(HEADER);
        if (correlationId == null || correlationId.isBlank()) {
            correlationId = UUID.randomUUID().toString();
        }
        MDC.put(MDC_KEY, correlationId);
        response.setHeader(HEADER, correlationId);   // echo it back so clients can track
        try {
            chain.doFilter(request, response);
        } finally {
            MDC.remove(MDC_KEY);  // critical: clean up to avoid thread-pool pollution
        }
    }
}

Always remove MDC entries in a finally block. Servlet containers reuse threads. If you forget to call MDC.remove(), the correlation ID from request A leaks into request B on the same thread — producing silently incorrect logs that are harder to debug than no correlation at all.

Add %X{correlationId:-} to your log pattern and the business-level ID appears alongside the Micrometer trace ID, giving you two complementary lenses on every request.

Propagating Context to Async and Virtual Threads

MDC is stored in a ThreadLocal. When you submit work to an ExecutorService or use @Async, the new thread starts with an empty MDC. Micrometer Tracing solves this for its own spans via ContextSnapshot:

@Async
public CompletableFuture<Void> sendNotificationAsync(Order order) {
    // Micrometer propagates the active span automatically into @Async methods
    // when you use its TaskDecorator integration:
    //   spring.task.execution.pool.core-size=5
    // and the TraceAsyncAspect auto-config handles the rest.
    log.info("Sending notification for order {}", order.getId()); // traceId still present
    return CompletableFuture.completedFuture(null);
}

For manual thread pool usage, wrap your Runnable with a context snapshot so MDC is copied across:

ContextSnapshot snapshot = ContextSnapshotFactory.builder().build()
        .captureAll();

executor.execute(snapshot.wrap(() -> {
    log.info("Async work — traceId is still here");
}));

Security Consideration: Never Trust Incoming Correlation IDs Blindly

If you accept an X-Correlation-ID from an external client and log it verbatim, you open a log injection vector. A malicious value like abc\n14:32:00 WARN [attacker] Fake log line can forge entries in your log stream, corrupting your audit trail. Always sanitise the header before writing to MDC:

// Safe sanitisation — keep only alphanumeric, hyphens, and underscores
correlationId = correlationId.replaceAll("[^a-zA-Z0-9\\-_]", "").substring(0, Math.min(correlationId.length(), 64));

Summary

Distributed logging and correlation IDs are the foundation of microservice observability. Add micrometer-tracing-bridge-brave to every service and let Spring Boot auto-configure trace and span IDs in MDC. Use the WebClient.Builder bean to get automatic header propagation. Add explicit child spans for important business operations using Tracer. Supplement Micrometer traces with a custom OncePerRequestFilter for business-level correlation IDs — and always sanitise incoming header values and clean MDC in a finally block. With these pieces in place, a single trace ID unlocks the full story of any request across your entire system.