Distributed Tracing with Tempo or Jaeger

Distributed tracing gives you end-to-end visibility across microservices: which spans took too long, where errors propagated, and which downstream calls caused a cascade. This guide wires up a full tracing pipeline using the OpenTelemetry Collector as the ingestion layer and either Grafana Tempo or Jaeger as the backend store. You'll instrument a real service with an OTLP exporter and verify traces appear in the UI.

Architecture Overview

The pipeline has three stages:

• Instrumented application — generates spans using an OpenTelemetry SDK and ships them via OTLP (gRPC port 4317 or HTTP port 4318).
• OpenTelemetry Collector — receives, batches, and forwards spans to one or more backends. Running a collector means you can swap backends without redeploying every service.
• Backend (Tempo or Jaeger) — stores and queries traces.

Prerequisites

• Docker Engine 24+ and Docker Compose v2 (for the quick-start containers).
• A working service you can modify — this guide uses a Python Flask app as the example.
• Ports 4317, 4318, 3200 (Tempo), 16686 (Jaeger UI), and 14250 (Jaeger gRPC) free on the host.

Option A: Grafana Tempo Backend

1. Write the Tempo Compose Stack

Create a working directory and a minimal docker-compose.yml:

mkdir tracing && cd tracing

cat > docker-compose.yml <<'EOF'
version: "3.9"
services:
  tempo:
    image: grafana/tempo:2.4.1
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
      - tempo-data:/var/tempo
    ports:
      - "3200:3200"   # Tempo HTTP API / Grafana datasource
      - "4317:4317"   # OTLP gRPC (accept from collector)

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.99.0
    command: ["--config=/etc/otel-collector.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector.yaml
    ports:
      - "4317:4317"   # expose to instrumented apps on host
      - "4318:4318"
    depends_on:
      - tempo

  grafana:
    image: grafana/grafana:10.4.2
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/tempo.yaml
    depends_on:
      - tempo

volumes:
  tempo-data:
EOF

2. Configure Tempo

cat > tempo.yaml <<'EOF'
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317

ingester:
  trace_idle_period: 10s
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 48h

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/blocks
    wal:
      path: /var/tempo/wal
EOF

3. Provision the Grafana Datasource

cat > grafana-datasources.yaml <<'EOF'
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    url: http://tempo:3200
    isDefault: true
EOF

Option B: Jaeger Backend

Replace the tempo and grafana services with Jaeger all-in-one. Use this block instead of Option A's compose file:

cat > docker-compose.yml <<'EOF'
version: "3.9"
services:
  jaeger:
    image: jaegertracing/all-in-one:1.57
    ports:
      - "16686:16686"  # Jaeger UI
      - "14250:14250"  # gRPC model.proto
    environment:
      - COLLECTOR_OTLP_ENABLED=true

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.99.0
    command: ["--config=/etc/otel-collector.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector.yaml
    ports:
      - "4317:4317"
      - "4318:4318"
    depends_on:
      - jaeger
volumes: {}
EOF

Configure the OpenTelemetry Collector

This config applies to both options — just swap the exporter endpoint in the exporters block.

cat > otel-collector.yaml <<'EOF'
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 512

exporters:
  # --- For Tempo (Option A) ---
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

  # --- For Jaeger (Option B) ---
  # otlp/jaeger:
  #   endpoint: jaeger:4317
  #   tls:
  #     insecure: true

  logging:
    verbosity: normal

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo, logging]
      # exporters: [otlp/jaeger, logging]  # use this for Option B
EOF

Comment/uncomment the relevant exporter lines to match your chosen backend before starting.

Start the Stack

docker compose up -d
docker compose ps

All services should show running or healthy within 30 seconds. Check collector logs if not:

docker compose logs otel-collector --tail 30

Instrument a Python Service

Install the OpenTelemetry SDK and Flask auto-instrumentation packages. These commands apply on any distro with pip:

pip install opentelemetry-sdk \
            opentelemetry-exporter-otlp-proto-grpc \
            opentelemetry-instrumentation-flask \
            flask

Manual SDK Setup (minimal example)

cat > app.py <<'EOF'
from flask import Flask
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.resources import Resource

resource = Resource.create({"service.name": "my-flask-app"})
provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

@app.route("/")
def index():
    tracer = trace.get_tracer(__name__)
    with tracer.start_as_current_span("do-work"):
        return "Hello, traced world!"

if __name__ == "__main__":
    app.run(port=5000)
EOF

python app.py

Send a request to generate a trace:

curl http://localhost:5000/

Using the Auto-Instrumentation Agent (zero-code option)

If you don't want to modify the application source, the OTel agent handles injection:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

OTEL_SERVICE_NAME=my-flask-app \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_INSECURE=true \
opentelemetry-instrument python app.py

Verification

Tempo/Grafana: Open http://localhost:3000, navigate to Explore → Tempo datasource, click Search, and filter by service.name = my-flask-app. Traces should appear within a few seconds of the curl request.

Jaeger: Open http://localhost:16686, select my-flask-app from the Service dropdown, and click Find Traces.

In both UIs you should see the root span for the HTTP GET and the nested do-work child span with start time, duration, and any attributes attached.

Troubleshooting

• No traces appear: Confirm the collector received spans with docker compose logs otel-collector. Look for lines like TracesExporter {"#spans": 2}. If absent, the exporter endpoint or port may be wrong.
• Connection refused on 4317: The collector's gRPC endpoint must be 0.0.0.0:4317 inside the container, not localhost. Verify this in otel-collector.yaml.
• TLS errors: Set insecure: true on the exporter when using plain gRPC without a certificate. For production, provision TLS certs and remove that flag.
• Tempo returns 404 on trace lookup: The ingester holds recent traces in its WAL before flushing to blocks. Wait 15–30 seconds after sending spans before querying.
• High memory in collector: Reduce send_batch_size or add a memory_limiter processor before the batch processor in the pipeline.

Production Considerations

• Replace local Tempo storage with object storage (S3, GCS) via the backend: s3 block in tempo.yaml for durability beyond a single node.
• Run the collector as a DaemonSet on Kubernetes so each node ships its own spans, or as a central Deployment gateway — both patterns are valid depending on scale.
• Add a tail_sampling processor in the collector to keep 100% of error traces and a configurable percentage of healthy ones, reducing storage costs significantly.
• Propagate the W3C traceparent header between services so that spans from separate processes stitch into a single distributed trace tree.