Distributed Tracing & OpenTelemetry

OpenTelemetry: The Standard

18 min Lesson 3 of 28

OpenTelemetry: The Standard

Before 2019, every observability vendor shipped its own agent, its own SDK, and its own wire format. Instrumenting an application meant picking a vendor up front and baking that choice into your source code. Switching later required ripping out thousands of lines of instrumentation calls. OpenTelemetry (OTel) was created to break that lock-in — once and for all.

What OpenTelemetry Actually Is

OpenTelemetry is a CNCF-graduated open standard and collection of libraries for generating, collecting, and exporting telemetry: traces, metrics, and logs. It is vendor-neutral by design. You instrument once; you route data to any backend — Jaeger, Datadog, Honeycomb, Grafana Tempo, AWS X-Ray — by changing a single config line, not your application code.

OTel defines three distinct layers:

API — Language-specific interfaces for creating spans, recording metrics, and emitting log records. The API is intentionally thin. It is safe to call from library code even if no SDK is present (calls become no-ops). This is what open-source libraries instrument against.
SDK — The concrete implementation of the API. The SDK handles sampling decisions, processor pipelines, and exporters. Application owners configure and provide the SDK at startup; library authors never touch it.
OTLP (OpenTelemetry Protocol) — The wire protocol. gRPC (port 4317) or HTTP/protobuf (port 4318). OTLP is now the lingua franca of telemetry: every serious backend speaks it natively. Sending OTLP means you are no longer coupled to Zipkin's JSON model or Jaeger's Thrift format.

Why the API/SDK split matters: A popular open-source HTTP library can safely ship OTel API calls to create spans without forcing any telemetry backend on its users. If the application owner installs an OTel SDK, those spans become real. If they do not, the calls cost essentially nothing. This is the pattern every major framework (Spring Boot, Express, Django, Laravel) is converging on.

The OTel Architecture at a Glance

OpenTelemetry architecture: the API/SDK sits inside your app, the Collector receives OTLP and fans out to multiple vendor backends simultaneously.

Vendor Neutrality in Practice

The concrete benefit: with OTel you can route the same trace data to Jaeger (self-hosted, free) for your on-call team and simultaneously to Honeycomb or Datadog for your SRE platform — without touching your application. The Collector's pipeline model makes this a config change, not a code change.

At Google, Uber, and other large-scale operators, the typical production setup is:

Applications export OTLP to a local Collector sidecar or DaemonSet (never directly to the backend — network blips in the app process would drop spans).
The Collector applies tail-based sampling, enriches spans with k8s metadata, and fans out to multiple backends.
Teams can swap or add backends with zero application deploys.

Auto-Instrumentation vs Manual Instrumentation

OTel ships auto-instrumentation for most major frameworks: the Java agent instruments Spring Boot, JDBC, gRPC, and HTTP clients by bytecode injection at JVM startup. Python's opentelemetry-instrument wraps Django/FastAPI/SQLAlchemy automatically. Node.js uses @opentelemetry/auto-instrumentations-node. You get spans for every inbound request, outbound HTTP call, and database query with zero code changes.

Manual instrumentation — using the OTel API directly — is reserved for business-level spans: "process payment", "evaluate risk score", "render recommendation feed". These are the spans that actually explain why latency happened, not just that an HTTP call was slow.

Big-tech pattern: Start with auto-instrumentation to get 80% coverage immediately. Then instrument your most critical business paths manually. Treat custom spans as a first-class API — name them by domain concept, not implementation detail (order.process, not OrderService.processOrder).

A Minimal OTel SDK Bootstrap (Python)

The following shows how to configure the SDK with an OTLP exporter — this is what runs at application startup, in your framework's initialization hook or in a dedicated tracing.py module:

# tracing.py — OTel SDK bootstrap (Python)
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "checkout-service",
    "service.version": "2.4.1",
    "deployment.environment": "production",
})

exporter = OTLPSpanExporter(
    endpoint="http://otel-collector:4317",   # gRPC, no TLS inside cluster
    # endpoint="https://collector.prod:4317",  # TLS for cross-cluster
)

provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

# In your request handler:
tracer = trace.get_tracer("checkout-service")

def process_order(order_id: str):
    with tracer.start_as_current_span("order.process") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("order.region", "eu-west-1")
        # ... business logic

Notice the resource: every span this SDK emits carries service.name, service.version, and deployment.environment. These are OTel semantic conventions — standardized attribute names that backends use to group and filter traces. Consistency across your fleet is critical: if five teams invent their own service name keys, no backend can correlate them.

OTLP: The Wire Protocol

OTLP is a Protocol Buffers-defined protocol over gRPC or HTTP. The gRPC transport (port 4317) is preferred for high-volume production workloads: it multiplexes over a single TCP connection, applies backpressure, and compresses efficiently. The HTTP transport (port 4318) is useful when gRPC is blocked by proxies or when sending from a browser.

A key operational detail: OTLP exporters batch spans by default. The BatchSpanProcessor buffers spans in memory and flushes on a schedule (default: every 5 seconds, max 512 spans per batch). This means:

Production pitfall — graceful shutdown: If your application crashes or is OOM-killed, buffered spans in the BatchSpanProcessor are lost. Always call provider.shutdown() in your SIGTERM handler and set max_export_batch_size conservatively. The Collector sidecar mitigates this — spans in the Collector's buffer survive an application pod restart.

Semantic Conventions

OTel ships a registry of standardized attribute names for HTTP, databases, messaging, RPC, and more. These are not suggestions — they are the shared vocabulary that makes cross-team, cross-language traces legible in any backend. Examples:

http.method, http.status_code, http.url — HTTP spans
db.system, db.statement, db.name — Database spans
messaging.system, messaging.destination — Kafka/RabbitMQ spans
rpc.system, rpc.service, rpc.method — gRPC spans

Adopting conventions up front means your Grafana dashboards and alert rules will work for every team's services without per-team customization. Skipping them means every team builds its own attribute schema and your platform team spends their time writing attribute-mapping transforms in the Collector — expensive technical debt at scale.

# Verify your Collector is receiving spans (check logs or use grpc_cli):
grpcurl -plaintext -d '{}' otel-collector:4317 opentelemetry.proto.collector.trace.v1.TraceService/Export

# Or check the Collector's health endpoint:
curl http://otel-collector:13133/

# View span counts from Collector Prometheus metrics:
curl http://otel-collector:8888/metrics | grep otelcol_receiver_accepted_spans

OpenTelemetry's guarantee is simple but profound: your observability investment is yours, not your vendor's. As backends and vendors evolve, your instrumentation stays constant. At big-tech scale — thousands of services, dozens of teams, multi-cloud deployments — that portability is not a nice-to-have. It is an architectural requirement.