Most engineering teams have monitoring. Few have observability. The difference: monitoring tells you that something is wrong; observability tells you why. A system with monitoring shows you "error rate is elevated." A system with observability shows you "error rate elevated in the checkout service for users in the EU, correlated with a 300ms latency spike in the payment service after the 14:32 deployment."
This guide covers the three pillars of observability — metrics, logs, and traces — and the practices that make them actionable rather than decorative.
The Three Pillars
Metrics
Metrics are time-series numerical measurements. They're efficient to store and query, making them ideal for dashboards, alerts, and trend analysis.
Choosing what to instrument:
The RED method (for services): - Rate: requests per second - Errors: error rate (5xx responses / total requests) - Duration: latency distribution (P50, P95, P99)
The USE method (for infrastructure): - Utilization: CPU, memory, disk utilization percentage - Saturation: queue depth, connection pool fullness - Errors: hardware errors, network packet errors
These two frameworks cover what matters for most systems. Every service should expose RED metrics; every piece of infrastructure should expose USE metrics.
Instrumentation approach:
For AWS services: CloudWatch Metrics provide native USE metrics for RDS, Lambda, ECS, and other services. No additional instrumentation needed.
For application code: Prometheus client libraries (available for every major language) expose custom metrics via HTTP endpoint. Prometheus scrapes these endpoints; metrics flow to your observability platform.
For OpenTelemetry: The emerging standard for portable instrumentation. OTel SDKs export metrics, traces, and logs in a vendor-neutral format to any compatible backend.
Logs
Logs are discrete events with structured context. They capture what happened and provide the "why" that metrics alone can't answer.
The structured logging imperative:
Unstructured logs ("user 12345 placed order") are human-readable but machine-unqueryable. Structured logs (JSON with consistent fields) are both:
{
"timestamp": "2026-07-08T14:32:15.234Z",
"level": "ERROR",
"service": "checkout-service",
"trace_id": "4bf92f3577b34da6",
"span_id": "00f067aa0ba902b7",
"user_id": "usr_12345",
"order_id": "ord_98765",
"event": "payment_failed",
"payment_processor": "stripe",
"error_code": "card_declined",
"duration_ms": 1243
}
Every log entry should include: timestamp, log level, service name, trace ID (critical for correlation), and relevant business context.
Log volume management:
Logging everything at DEBUG level is not observability — it's noise with storage costs. Use log levels correctly: - DEBUG: Temporary diagnostic information (disable in production) - INFO: Normal business events (order placed, user logged in) - WARN: Unusual but non-fatal conditions (retry succeeded, cache miss, deprecated API call) - ERROR: Failures that affect functionality (payment failed, database connection lost)
Set production log level to INFO. Errors and warnings surface automatically; debug-level noise stays off.
Sampling for high-volume systems:
At 100,000 requests/second, logging every request is expensive and produces too much data to analyze. Implement log sampling: log 100% of errors, 100% of slow requests (>P99 latency), and 1-5% of normal requests. This captures all anomalous events while controlling volume.
Distributed Traces
Traces are the defining observability capability for distributed systems. A trace follows a request through every service it touches, showing the full call chain with timing for each step.
Why traces are essential:
In a monolith, a slow endpoint is easily profiled. In microservices, a slow user-facing request might involve 5-10 services. Without tracing, you see the symptom (slow API call) but not the cause (database query in service C taking 2 seconds because it's missing an index). Tracing makes the invisible visible.
The trace ID propagation pattern:
Every trace starts with a unique trace ID generated at the entry point (API gateway or frontend). This ID is propagated through every downstream service call via HTTP headers:
W3C Trace Context (standard):
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
Every service in the call chain creates a span with its operation name, duration, and the parent trace ID. These spans are emitted to your tracing backend (Jaeger, Zipkin, Datadog APM, AWS X-Ray) which assembles the full trace.
Auto-instrumentation:
OpenTelemetry provides auto-instrumentation agents for Python, Java, Node.js, and .NET that automatically trace HTTP calls, database queries, and messaging operations without code changes. Deploy the OTel agent alongside your service and traces start flowing immediately.
Alert Design: Preventing Alert Fatigue
The most common observability failure: too many alerts, most of which are noise. Engineers learn to ignore alerts; real incidents go undetected in the noise.
The principles of actionable alerts:
Alert on symptoms, not causes: Don't alert on "CPU is at 80%." Alert on "latency P99 > 1 second" (the symptom). CPU elevation might cause it, or it might not. Symptom-based alerts have higher signal-to-noise ratio.
Alert on rates, not values: Don't alert on "there were 50 errors in the last minute." Alert on "error rate exceeded 1% for 3 consecutive minutes." Rate-based alerts are resilient to traffic volume changes.
Multi-window burn rate alerts: For SLO-based alerting, burn rate alerts detect when you're consuming your error budget too fast to sustain your SLO. A 1-hour burn rate of 14.4× and a 5-minute burn rate of 14.4× together indicate a fast-burn incident that will consume your monthly budget in hours.
Tiered severity: - Page (wake someone up): User-facing impact detected, SLO at risk - Ticket (address during business hours): Elevated error rate below SLO threshold, anomalous but not urgent - Log (informational): Unusual condition worth noting
Most alerts should be tickets, not pages. Over-paging burns out on-call engineers.
Alert ownership:
Every alert should have an owner (team or individual) who is responsible for either fixing the condition or acknowledging it's not actionable and removing/adjusting the alert. Alerts without owners become background noise.
Observability Platform Options
Datadog: Best-in-class integrated platform. Metrics, logs, traces, and APM in one tool with excellent UX. Expensive at scale but reduces tool proliferation cost.
Grafana + Prometheus + Loki + Tempo: Open-source stack that can match Datadog on capability at lower licensing cost, but higher operational overhead to run and maintain.
AWS-native (CloudWatch + X-Ray): Lower cost for AWS-centric workloads, less functionality than dedicated platforms. Works well for simpler systems.
New Relic, Dynatrace: Strong alternatives to Datadog with comparable capability and different pricing models.
OpenTelemetry as vendor hedge: Instrument with OTel and route to any backend. This prevents vendor lock-in for instrumentation — you can switch platforms without re-instrumenting your code.
Rutagon designs and implements observability systems for engineering teams running distributed workloads in production. Contact us to discuss your observability strategy.
Frequently Asked Questions
What's the most important observability to add first?
Add distributed tracing first if you don't have it — it provides the most diagnostic leverage for complex systems with minimal instrumentation effort using auto-instrumentation. Second priority: structured logging with trace ID correlation. Third: dashboards for your RED/USE metrics. Alerts built on top of these three pillars are far more actionable than alerts without the underlying observability.
How much does enterprise observability tooling cost?
Datadog costs approximately $15-25/host/month for infrastructure monitoring, plus APM costs that scale with trace volume. For a 50-host production environment, expect $750-$2,500/month in Datadog licensing plus usage-based costs. The Grafana open-source stack eliminates licensing but adds ~0.5-1 FTE of operational overhead. At 50-100 hosts, Datadog's TCO is often competitive with the operational cost of the open-source stack.
What's the difference between monitoring and observability?
Monitoring is watching known metrics and alerting when they exceed known thresholds. Observability is the ability to understand internal system state from external outputs — to answer novel questions about system behavior without adding new instrumentation. Observability enables you to investigate issues you didn't anticipate; monitoring only detects issues you thought to look for.
How do you correlate logs, metrics, and traces?
The correlation key is the trace ID, which appears in all three signals. When a trace shows elevated latency, you can pivot to logs for that trace ID to see detailed event context, and to metrics to see the aggregate pattern. A unified observability platform (Datadog, Grafana) provides this correlation automatically. Building it manually requires consistent trace ID propagation across all three signals.
What SLOs should engineering teams define?
Start with service-level objectives for user-facing services: availability (e.g., 99.9% of requests return a non-5xx response), latency (P99 < 1 second), and error rate (< 0.1% of requests return an error). Set SLOs based on what users actually notice — an SLO of 99.99% for a non-critical batch job is over-engineered. Start with 3-5 SLOs per service, measure them for 90 days, then refine.