Skip to main content
INS // Insights

Distributed Tracing Implementation Guide

Updated July 2026 · 5 min read

Logs tell you something failed. Metrics tell you how often. Traces tell you where the time went across services — which is why this distributed tracing implementation guide focuses on what Rutagon actually ships in production stacks: OpenTelemetry instrumentation, sampling that does not bankrupt storage, and dashboards engineers open during incidents.

Buyer Pain: Latency Without a Culprit

Symptoms we walk into:

  • p99 spikes with “all green” service dashboards
  • Customer tickets that cannot be correlated to a single service log
  • Microservices blame games (“it’s the database” / “it’s the API gateway”)
  • Tracing pilots that died after cardinality exploded the bill

Related context: startup cloud architecture patterns and zero-downtime blue-green deployment AWS. Capability: AWS cloud infrastructure and full-stack development.

Distributed Tracing Implementation Guide: OTel Contract

We standardize on OpenTelemetry so you are not married to one vendor backend:

App SDKs (OTel)
    → Collector (tail sampling, scrubbing, export)
        → Backend (Tempo/Jaeger/X-Ray/Datadog/...)
            → Trace UI + exemplars linked from metrics

Instrumentation priorities (in order):

  1. Ingress (API gateway / load balancer correlation)
  2. Service-to-service HTTP/gRPC
  3. Datastores (SQL/Redis) with sanitized statements
  4. Async boundaries (queues) with context propagation
import { trace, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("checkout-service");

export async function chargeOrder(orderId: string) {
  return tracer.startActiveSpan("chargeOrder", async (span) => {
    span.setAttribute("order.id", orderId);
    try {
      const result = await paymentClient.charge(orderId);
      span.setAttribute("payment.status", result.status);
      return result;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

Sampling: The Make-or-Break Decision

Head sampling alone drops the interesting failures. We usually ship tail sampling in the collector for error and high-latency traces, with a lower baseline rate for healthy traffic.

Rules of thumb we implement:

  • Always keep errors and explicit “debug this request” flags
  • Keep a small percentage of success paths for baselines
  • Bound attributes — no unbounded user_email as high-cardinality labels
  • Scrub auth headers and PII at the collector

Propagation Across Queues

The silent killer of tracing programs: async workers that start new roots. We propagate traceparent through message attributes and continue spans in consumers. Without that, your “distributed” trace is a pile of orphans.

Dashboards Engineers Actually Use

We wire:

  • Service map with error rates
  • Trace exemplars from latency SLO burn alerts
  • Top slow DB spans by normalized query shape
  • Deployment markers correlated to trace regressions

Pair with cost discipline from Kubernetes cost optimization guide when the collector fleet itself gets expensive, and platform sequencing from startup platform engineering roadmap once published in-cluster.

Production Lessons

Lesson 1 — Tracing without ownership dies. Each service team owns span quality; platform owns collector defaults.

Lesson 2 — Cardinality is a product risk. Attribute allowlists beat “log everything as a span attribute.”

Lesson 3 — Start with critical user journeys. Checkout/login/provision — not every cron.

Lesson 4 — Connect to deploy pipelines. A trace spike without knowing what shipped is half an answer.

Ready to implement distributed tracing that survives production cardinality? Talk to Rutagon — contact@rutagon.com or 907-841-8407.

Start a Conversation →

Collector Configuration Essentials

The collector is where most pilots fail. We treat it as a production service:

  • Horizontal replicas behind a load balancer / gateway
  • Separate pipelines for app traces vs infra spans when volume differs
  • Memory limiter and load shedding before the backend melts
  • Config as code with progressive rollout

A minimal mental model:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors-keep
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: latency-keep
        type: latency
        latency: {threshold_ms: 2000}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

Tune numbers to your SLO burn rates — do not copy thresholds blindly.

Frontend and Mobile Correlation

Backend-only traces miss the user’s truth. Where product needs it, we connect RUM / mobile spans with backend traceparent so a slow checkout shows browser, API, and database in one view. This is optional phase-two work — only after the critical backend journey is solid.

Security Review Checklist

Before wide rollout we run:

  • Attribute allowlist reviewed for PII
  • Auth header scrubbing verified with a red-team sample payload
  • Access control on the trace UI (not “entire eng company by default” for regulated data)
  • Retention aligned to log retention so legal holds are coherent

Related hardening often sits beside security automation delivery.

Operating Model After Launch

Tracing is not a project that ends at “collector is green.” We leave:

  • A service instrumentation scorecard
  • On-call runbook: how to find a trace from a customer ID
  • A monthly cardinality/cost review
  • Ownership for semantic convention upgrades

Without that operating model, last quarter’s beautiful demo becomes this quarter’s ignored tab.

Sampling Without Losing the Plot

We use head-based sampling at 10% for healthy traffic and force-sample on error and high-latency traces. That kept APM cost predictable while still capturing the failures that matter. Trace IDs propagate through API Gateway, Lambda, and the worker queue so a single customer ticket maps to one timeline.

Frequently Asked Questions

What should a distributed tracing implementation guide cover first?

Instrumentation standards, context propagation (including queues), sampling policy, PII scrubbing, and one critical journey end-to-end before a company-wide rollout.

OpenTelemetry or vendor agents?

We prefer OTel SDKs + collector so backends can change. Vendor agents can be faster to start; we still push for OTel semantic conventions to avoid lock-in.

How do you control tracing cost?

Tail sampling, attribute allowlists, span name normalization, and collector autoscaling with budgets. Cost is part of the design, not a surprise month-two invoice.

Do we need traces if we already have logs and metrics?

If you have more than a few services or async workers, traces pay for themselves in incident time saved. Logs and metrics remain necessary — traces connect them.

How long does a pilot take?

A single critical journey with collector, backend, and one dashboard often lands in days to a couple of weeks depending on stack complexity. Fleet-wide coverage follows service by service.