Distributed Logging

In a monolith, debugging is straightforward — you open one log file and search for the error. In a microservices architecture with 50+ services running across hundreds of containers, a single user request might touch 10 different services. Finding the root cause of a failure requires aggregating, correlating, and searching logs from all those services in one place.

A distributed logging system collects logs from every service, transports them to a central store, indexes them for fast search, and provides dashboards for analysis.

1. The Logging Pipeline

Every distributed logging system follows the same fundamental pipeline:

2. Structured Logging

The single most impactful improvement you can make to your logging is switching from unstructured to structured logs.

Unstructured (Bad)

2024-01-15 14:30:22 ERROR Payment failed for user 123, order 456, amount $29.99, card declined

This is human-readable but machine-unfriendly. Parsing the user ID, order ID, or amount requires fragile regex patterns.

Structured (Good)

{
    "timestamp": "2024-01-15T14:30:22.456Z",
    "level": "ERROR",
    "service": "payment-service",
    "message": "Payment failed",
    "user_id": "123",
    "order_id": "456",
    "amount": 29.99,
    "currency": "USD",
    "error_code": "CARD_DECLINED",
    "trace_id": "abc-123-def-456",
    "span_id": "span-789"
}

With structured logs, you can query:

• service:payment-service AND level:ERROR — All payment errors.
• user_id:123 — Everything that happened for user 123.
• trace_id:abc-123-def-456 — The entire journey of a single request across all services.

3. Log Levels

Log levels indicate the severity of an event. They allow you to filter noise and focus on what matters.

Production Configuration

In production, set the log level to INFO or WARN. DEBUG and TRACE logs generate enormous volume and should only be enabled temporarily for specific services during active debugging.

Development:  log level = DEBUG  (verbose, see everything)
Staging:      log level = DEBUG  (mirror development for testing)
Production:   log level = INFO   (normal operations + errors)

4. Distributed Tracing and Correlation IDs

The most critical challenge in distributed logging is correlating logs across services. When a user's request flows through 10 services, how do you find all the log entries related to that single request?

Correlation IDs (Trace IDs)

When a request enters the system, the API Gateway generates a unique Trace ID and attaches it as an HTTP header (e.g., X-Trace-ID: abc-123). Every downstream service reads this header, includes it in all its log entries, and forwards it to the next service.

Now, searching trace_id:abc-123 in Kibana returns every log entry from every service that participated in that request — in chronological order.

OpenTelemetry

OpenTelemetry is the industry standard for distributed tracing. It provides:

• Trace ID: Identifies the entire request journey across all services.
• Span ID: Identifies a single operation within a service (e.g., "database query", "HTTP call to payment service").
• Parent Span ID: Links child spans to parent spans, creating a tree of operations.

5. The ELK and EFK Stacks

The two most popular logging stacks are:

ELK Stack (Elasticsearch + Logstash + Kibana)

EFK Stack (Elasticsearch + Fluentd + Kibana)

Replaces Logstash with Fluentd (or Fluent Bit), which is lighter and more popular in Kubernetes environments.

Grafana Loki (Alternative)

Loki is a cost-effective alternative to Elasticsearch designed by Grafana Labs. Unlike Elasticsearch, Loki does not index the full text of log lines. It only indexes labels (metadata like service, level, trace_id), storing the raw log text compressed on object storage (S3).

6. Log Retention and Cost Management

At scale, logging generates terabytes per day. Storing everything forever is prohibitively expensive.

Retention Strategy

Cost Optimization Techniques

1. Sampling: For high-volume endpoints (health checks, static assets), log only 1% of requests instead of 100%.
2. Hot-warm-cold architecture: Store recent logs on fast SSD nodes (hot), move older logs to cheaper HDD nodes (warm), and archive to S3 (cold).
3. Log aggregation: Instead of logging every individual request, aggregate metrics (e.g., "endpoint /api/users: 50,000 requests, avg latency 120ms, 5 errors") at the service level.
4. Drop verbose fields: Strip large request/response bodies from logs before indexing.

[!TIP] In a system design interview, always mention the logging pipeline (collection, transport, processing, storage, visualization), structured logging with trace IDs, and the cost trade-off between full indexing (Elasticsearch) and label-only indexing (Loki).