highAdvancedUSE#opentelemetry#collector#traces#export#sampling

OpenTelemetry traces lost — collector overload or context canceled on export

Traces arrive at the collector but do not appear in the backend (Jaeger/Tempo)

Likely source:availabilityperformance

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Traces arrive at the collector but do not appear in the backend (Jaeger/Tempo)
  • Collector logs show 'context canceled' or 'context deadline exceeded' on export
  • Span count drops during traffic spikes — sampling rate seems to fluctuate
  • OTLP exporter queue overflows and spans are dropped

OpenTelemetry collector queue overflow is silent — spans are dropped without error. Always monitor otelcol_exporter_send_failed_spans and set an alert at > 0. Use batch processor with appropriate timeout and send_batch_size to smooth out spikes.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Collector queue size and dropped spans

    curl -s http://<OTEL-COLLECTOR>:8888/metrics | grep -E 'otelcol_processor_send_failed_spans|otelcol_exporter_send_failed_spans|queue_size'

  • Collector logs for export errors

    kubectl logs -n observability deploy/otel-collector | grep -i 'context canceled\|deadline exceeded\|send failed' | tail -10

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • otelcol_exporter_queue_size is at max (default 2000)
  • otelcol_exporter_send_failed_spans is climbing
  • Collector logs show 'context deadline exceeded' on export

Confirm before fixing

curl -s http://<OTEL-COLLECTOR>:8888/metrics | grep 'otelcol_exporter_queue_size'
READ

Check the queue size

Expected: Queue size at or near max

Fix

kubectl set env deployment/otel-collector -n observability OTEL_EXPORTER_OTLP_TIMEOUT=30000
CHANGE

Increase the exporter timeout to give the backend more time to accept spans

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
curl -s http://<OTEL-COLLECTOR>:8888/metrics | grep -E 'otelcol_exporter_send_failed_spans|otelcol_exporter_queue_size'
READ

Check for dropped spans and queue overflow

2
kubectl get pods -n observability -l app=otel-collector
READ

Check collector pod health and restart count

3
kubectl exec -n observability deploy/otel-collector -- curl -s http://<BACKEND>:<PORT>/ready
READ

Test backend connectivity from the collector

4
kubectl set env deployment/otel-collector -n observability OTEL_EXPORTER_OTLP_TIMEOUT=30000
CHANGE

Increase the export timeout

Prevention

  • Monitor otelcol_exporter_send_failed_spans and alert at > 0 for > 1 minute.
  • Set the batch processor's send_batch_size and timeout to balance latency and throughput.
  • Use the load balancing exporter for multi-collector deployments to avoid single-collector overload.
  • Size collector memory limits based on peak span throughput (1 span ≈ 1-5KB in queue).

Related runbooks

Same stack, same failure family — worth a skim.

OpenTelemetry collector fails to start — invalid YAML config or missing receivercritical

Collector pod shows CrashLoopBackOff after config change

OpenTelemetryIntermediate3 causes
Open runbook
OpenTelemetry traces not arriving at the collector — export fails or batch processor abortscritical

Application exports traces but nothing appears in Jaeger/Tempo

OpenTelemetryIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.