The cross-service observability primitive that makes a request's full journey through a distributed system inspectable — recognising that without trace context propagation, logs and metrics are local signals that can't be assembled into the system-level story they imply, and that the trace itself is what turns "the checkout is slow somewhere" into "the checkout is slow because service C's database call is taking 1.4s in the cache-miss path."
A primitive approach to debugging a distributed system: when a request is slow, the engineer looks at logs from each service and tries to reconstruct what happened by correlating timestamps and request IDs by hand. This works at small scale and breaks down rapidly: services don't all use the same request ID format; clock skew makes timestamp correlation unreliable; the request's path through fan-out and fan-in patterns is opaque; and the engineer ends up reasoning about the system rather than reading a record of what actually happened.
A production distributed tracing architecture treats the request's journey as a first-class observable. Each service participating in handling a request creates a span — a record of the work it did, with start time, duration, attributes, and references to its parent span (the operation that called it) and child spans (the operations it called). Spans share a trace ID — a single identifier carried through the entire request, across every service it touches. The trace ID and span context propagate through HTTP headers, message queue metadata, and RPC envelopes following the W3C Trace Context standard. The collected spans are assembled into a trace: a tree (or DAG) of all the work that happened to fulfil the original request. The trace becomes the architectural artefact that lets engineers see what actually happened, in what order, with what timing, across services they may not even know about.
The architectural shift is not "we instrumented some services with traces." It is: distributed tracing is a cross-cutting architectural primitive whose value depends on context propagation working across every service boundary, on sampling preserving the rare events worth investigating, on retention and cardinality being designed for the questions you'll ask, and on correlation with logs and metrics being instrumented at the same time — and treating tracing as an optional add-on per service produces partial traces that hide more than they reveal.
The defining property of distributed tracing is context propagation: the trace ID and current span ID travel from one service to the next, encoded in the HTTP header, message queue metadata, RPC envelope, or whatever transport mechanism connects services. The receiving service reads the propagated context, creates a child span linked to the parent, and propagates the new context to anything it calls. If propagation works, the trace assembly produces a complete picture of the request's journey. If propagation breaks at any boundary, the trace splits into two unconnected fragments, and the assembly produces partial pictures that hide what actually happened. The architectural discipline is to use a propagation standard (W3C Trace Context is the industry default) consistently across every service, every transport, and every team — not as a per-service implementation choice but as a system-level invariant. A single service that drops propagation breaks tracing for every request that touches it.
traceparent, tracestate headers) is the propagation standard, used consistently across all services regardless of language, framework, or team.Pick a multi-service flow in your system. Start at the entry point, follow the trace through to a deep service. Is the trace complete — every service appearing as a span, parent-child relationships preserved, no orphan spans? If the trace is fragmented, propagation is breaking somewhere — and every investigation that depends on tracing through that boundary is currently working with partial information.
W3C Trace Context Recommendation is the canonical standard for context propagation; OpenTelemetry — Context Propagation covers the operational implementation across languages and transports. The cross-language consistency is what makes the standard valuable — adopted by Jaeger, Zipkin, Honeycomb, Datadog, and most cloud providers' tracing offerings.
A span is more than a timed operation; the parent-child relationship between spans encodes the system's call structure. When service A calls service B, which calls service C, the trace shows A's span as the root, B's span as A's child (with timing showing how long the call took including network), and C's span as B's child. When service A makes parallel calls to B and D, the trace shows both as children of A, with overlapping timings revealing the parallelism. When the request enters a fan-out pattern (one operation triggers many child operations), the trace shows the fan-out shape directly. The architectural reading: the trace IS a snapshot of the system's call graph for that request — every architectural decision about sync vs async, fan-out vs sequential, retry-on-failure vs fail-fast is visible in the span structure. The discipline is to instrument spans at the boundaries that matter (service entry, outbound calls, significant internal operations), with attributes (operation name, status, key parameters) that make the architecture readable from the trace.
db.statement, http.method, messaging.destination follow the OpenTelemetry Semantic Conventions.Pick a recent investigation that used traces. Did the trace's structure correctly represent the system's call graph — fan-out shown as multiple children, async work shown as separate roots or follows-from, parallelism visible from overlapping timings? If the trace structure was misleading or required interpretation, span instrumentation isn't tracking the architecture correctly — and traces are providing weaker investigation support than they should.
OpenTelemetry — Spans covers the span model and parent-child relationships; OpenTelemetry Semantic Conventions standardises attribute names so spans from different services and frameworks are mutually comparable. Distributed Tracing in Practice — Parker et al. treats span instrumentation patterns at practitioner depth.
At full instrumentation in a high-traffic system, every request produces a trace, and the resulting trace volume — millions or billions per day — exceeds practical storage and query capacity. Sampling is necessary; the strategy determines which traces survive and what they're useful for. Head sampling — the decision to keep or drop is made at the trace's start, propagated with the context, and inherited by every span — is simple and efficient: the sampling decision is made once, no trace is partially captured, and the cost is bounded by the sampling rate. The drawback: head sampling is blind to the trace's outcome, so a 1% head sample drops 99% of error traces just like it drops 99% of success traces. Tail sampling — the decision is made after the trace completes, based on its outcome (errors, slow tails, specific attributes) — preserves the rare-but-valuable traces. The drawback: tail sampling requires buffering complete traces before deciding, which is more expensive and operationally more complex. The architectural discipline is to choose deliberately based on what the traces are for: head sampling for general observability and statistical analysis; tail sampling for incident investigation where errors and slow tails are the queries that matter most; sometimes both, with head sampling for routine traffic and tail sampling on top to preserve outliers.
Pick the sampling strategy your tracing system uses. Is it head, tail, or a combination, with documented rationale? When an error happens in production, what's the probability that the relevant trace survived sampling and is queryable? If the answer is "1% — the head-sample rate," investigation is operating on luck.
OpenTelemetry — Sampling covers head and tail sampling primitives. Honeycomb — Dynamic Sampling by Example treats sampling at practitioner depth, with patterns for stratified sampling that preserve rare events.
Traces, like logs, can carry high-cardinality data — request IDs, user IDs, full URLs, query parameters. Like metrics, indexing and querying that data has cost that scales with cardinality. Like logs, retaining traces forever is wasteful for old traces nobody queries; tiered retention matches data value to cost. The architectural patterns transfer: indexed attributes are limited to those that matter for query (service, status, latency bucket); high-cardinality attributes (user ID, request ID) are preserved in the trace but indexed selectively or not at all; retention tiers (hot for recent, warm for last week, archive for compliance) reflect query patterns. The discipline is to recognise that traces are subject to the same data-architecture concerns as their sibling signals, and to design the architecture deliberately rather than letting it grow ad-hoc until the storage cost becomes the dominant infrastructure expense.
Pick the tracing backend your organisation uses. What's the trace retention period, and what proportion of traces are in hot vs warm vs archive storage? What's the indexed cardinality? If the answers are "we keep everything in hot storage forever," the cost is paying for data nobody queries — the next time volume doubles, the cost will too.
Honeycomb — Cardinality covers the architectural cost of high-cardinality observability data, applicable to traces specifically. Jaeger Documentation — Storage Backends covers retention and indexing patterns at the practitioner level.
When a request takes 2 seconds, the question "where did the time go?" has many possible answers — and the trace is the only signal that can answer it definitively. Critical path analysis — identifying the sequence of spans that determined the request's total duration — is what traces are uniquely good at. In a sequential call graph, the critical path is the chain from root to deepest span. In parallel patterns, the critical path runs through the slowest of the parallel branches; speeding up the others doesn't help. In retry patterns, the critical path includes the retried operation's full duration. The architectural pattern that makes critical path analysis tractable is good span instrumentation (the right boundaries are spanned), accurate timing (clock skew and propagation latency are accounted for), and tooling that highlights the critical path visually. The discipline is to use traces for critical-path investigation as the default approach to "this request is slow," rather than reasoning from logs or measurements about which service to suspect.
Pick a recent latency investigation. Did the team start with traces and identify the critical-path service, or did they start with logs and metrics and reason about which service to suspect? If the latter, the tracing capability either isn't trusted or isn't well-instrumented enough — and investigations are taking longer than they should.
Distributed Tracing in Practice — Parker et al. covers critical-path analysis at practitioner depth, with concrete patterns for using traces to identify performance problems. Jaeger — UI Documentation covers the visualisation primitives that make critical paths legible.
The three pillars of observability — logs, metrics, traces — are valuable independently and exponentially more valuable when correlated. The trace ID is the correlation primitive: every log line emitted while a trace is active should carry the trace ID; every metric data point should be correlatable to the traces that produced it (via exemplars or service-name correlation); every trace should link to logs and metrics from the same service in the same time window. With correlation in place, an investigation flows: a metric anomaly leads to a relevant trace, the trace's critical path identifies the implicated service, the service's logs for the trace's time window reveal the underlying cause. Without correlation, the same investigation requires manual stitching across three tools, three time-window translations, and a patient operator to reconstruct what the unified observability surface should have shown directly. The architectural discipline is to instrument correlation as a first-class concern, not as an afterthought: the logging library reads the active trace context and includes the trace_id and span_id in every emitted log; the metrics emission attaches trace exemplars where supported; the tracing UI links to logs and metrics for matching service and time window with one click.
Pick a recent latency or error investigation. From a metric anomaly, how many clicks does it take to find a relevant trace? From the trace's critical-path service, how many clicks to that service's logs for the same time window? If either is "we navigate manually across tools, copying IDs," the correlation is breaking down — and the time-to-diagnose is paying the cost.
OpenTelemetry — Trace and Log Correlation covers the trace_id propagation into logs as a first-class concern. OpenTelemetry — Exemplars covers metric-to-trace correlation. Distributed Systems Observability — Cindy Sridharan treats the three-pillar correlation as the architectural payoff that makes observability investigation tractable.
The diagram below shows the canonical distributed tracing architecture: services participating in a request flow, each creating spans linked to the propagated trace context (W3C Trace Context standard); span export to a collector with sampling decision (head and/or tail); trace storage with retention tiers; query and visualisation surface; correlation primitives (trace_id in logs, exemplars in metrics) linking the trace to the other observability signals.
Half the services emit spans; the other half don't. Traces fragment at the boundaries between them. Investigations that need to follow a request through the unspanned services hit a wall.
Tracing is treated as a system-level invariant, not a per-service choice. All services use the same propagation standard (W3C Trace Context). The framework or instrumentation library handles propagation; application code doesn't have to.
Synchronous calls between services have their context propagated correctly. Async work — message queues, event streams, scheduled jobs — drops the trace context. Traces fragment at every async boundary.
Async transport mechanisms propagate context through their own envelopes — message attributes, event metadata. The propagation works regardless of the synchrony of the call.
The sampling rate is whatever the default was when the tracing library was first installed. Errors are sampled at 1% along with everything else; investigation depends on luck.
Sampling strategy chosen deliberately. Head sampling for routine traffic; tail sampling for outliers and errors. Documented rationale; rate recorded in the trace. The strategy reflects what the traces are used for.
All traces in hot storage for a year. Cost grows linearly with traffic; query patterns don't justify the retention.
Retention tiers matched to query patterns. Hot for recent (24-72h); warm for last week or month; archive only where compliance requires. Trace storage as carefully designed as log storage.
The trace exists; the logs exist; the metrics exist. Correlating across them requires manual stitching by the operator. Time-to-diagnose is multiplied by the friction.
Trace_id propagated into every log statement during the trace. Trace exemplars attached to metric data points. UI supports click-through across signal types. The operator stays in flow.
Cross-language, cross-framework consistency. The standard is what makes tracing work; per-service propagation choices guarantee that traces fragment.
Application code that handles propagation manually gets it wrong inconsistently. The framework handles it; application code is unaware.
Without async propagation, every async pattern produces broken traces. Message attributes, event metadata, scheduled-job context — all carry trace context.
Service entry, outbound calls, significant internal operations. OpenTelemetry semantic conventions for attribute names. The trace structure encodes the architecture.
Head, tail, or combination. Rate recorded in trace. The strategy reflects what the traces are used for; defaults aren't accepted as the strategy.
Tail sampling or stratified head sampling that doesn't drop the genuinely-investigation-worthy traces. Routine traffic and outliers are sampled differently.
Hot for recent, warm for medium-term, archive for compliance only. High-cardinality attributes preserved but indexed selectively.
Investigators start with traces, not with logs and reasoning. Tracing UI highlights the critical path visually. The capability is trusted and well-instrumented.
Logging framework reads the active trace context. Application code is unaware. Logs and traces correlate without manual stitching.
"Why did this metric spike" becomes a one-click investigation through the linked exemplar trace. Correlation across the three pillars is operationally seamless.
Other substantive pages in the library that link here: