# OpenTelemetry

> Configure OpenTelemetry SDKs and Collector pipelines, propagate trace context, choose a sampling strategy and find where telemetry is lost.

Canonical: https://www.wiki.jodisand.me/opentelemetry/
Reviewed: 2026-09-24
Related: [Prometheus](https://www.wiki.jodisand.me/prometheus/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [HTTP and curl](https://www.wiki.jodisand.me/http/index.md), [Go](https://www.wiki.jodisand.me/go/index.md)


## Cheatsheet

| Task | Command or setting |
| --- | --- |
| Point an SDK at a Collector over gRPC | `OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 OTEL_EXPORTER_OTLP_PROTOCOL=grpc` |
| Point an SDK at a Collector over HTTP | `OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` |
| Name the service | `OTEL_SERVICE_NAME=my-app` |
| Sample 10% of new traces, follow the parent otherwise | `OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.1` |
| Turn off one signal | `OTEL_METRICS_EXPORTER=none` |
| Turn off the SDK entirely | `OTEL_SDK_DISABLED=true` |
| Validate Collector config | `otelcol validate --config=config.yaml` |
| Print what the Collector receives | add the `debug` exporter with `verbosity: detailed` |
| Collector's own metrics | `curl -s localhost:8888/metrics` |
| Health check extension | `curl -s localhost:13133` |
| Pipeline internals (zpages) | `curl -s localhost:55679/debug/tracez` |
| Trace header on the wire | `curl -v` and look for `traceparent` |

Replace `otelcol` with your distribution's binary (`otelcol-contrib`, `otelcol-k8s`). The OTLP defaults are port 4317 for gRPC and 4318 for HTTP.

## Signals, resources and context

Three signals share one resource and one context.

- A **trace** is a tree of spans with a shared 16-byte trace ID. Each span records one operation with start and end time, attributes, events and a status.
- A **metric** is an aggregated measurement: counter, up-down counter, gauge or histogram.
- A **log** is a timestamped record. When emitted inside an active span it carries that span's trace and span ID.

Trace IDs attached to logs and to metric exemplars are what let a backend jump from a latency spike to the trace behind it.

A **resource** describes the producer: `service.name`, `service.version`, `deployment.environment.name`, `k8s.pod.name`. It is attached to every signal from that process. Backends group by `service.name`, so a missing or wrong value makes data hard to find; SDKs fall back to `unknown_service` (plus the process name in some languages).

> [!NOTE]
> Semantic conventions 1.27 renamed `deployment.environment` to `deployment.environment.name`. Check which one your backend and dashboards expect. See [semantic conventions](https://opentelemetry.io/docs/specs/semconv/).

**Context propagation** carries the trace across process boundaries in the W3C `traceparent` header, with optional `tracestate` and `baggage` headers.

```text
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
             version-trace id (32 hex)-parent span id (16 hex)-flags (01 = sampled)
```

A trace breaks exactly where propagation breaks: an uninstrumented hop, a proxy or queue that drops headers, async work that loses the context object, or two services configured with different propagators (`tracecontext` against `b3`, for example).

## SDK configuration with environment variables

All official SDKs read the same environment variables. Set them in the deployment rather than in code so they can change without a rebuild.

```sh
OTEL_SERVICE_NAME=my-app
OTEL_RESOURCE_ATTRIBUTES=service.version=1.4.2,deployment.environment.name=prod
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1
OTEL_PROPAGATORS=tracecontext,baggage
```

Two details cause most "no data" reports:

- The specification default protocol is `http/protobuf`, but some SDKs default to `grpc`. Sending gRPC to 4318 or HTTP to 4317 fails. Set `OTEL_EXPORTER_OTLP_PROTOCOL` explicitly and match the port.
- With HTTP, `OTEL_EXPORTER_OTLP_ENDPOINT` is a base URL and the SDK appends `/v1/traces`, `/v1/metrics` or `/v1/logs`. The signal-specific variables such as `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` are used as-is, so they must include the path.

See the [OTLP exporter specification](https://opentelemetry.io/docs/specs/otel/protocol/exporter/) and [SDK environment variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/).

## Manual spans

Auto-instrumentation covers HTTP servers and clients, database drivers and messaging libraries. Add manual spans for business operations the framework cannot name.

```go
tr := otel.Tracer("checkout")
ctx, span := tr.Start(ctx, "reserve_inventory")
defer span.End()
span.SetAttributes(attribute.String("sku", sku), attribute.Int("qty", qty))
if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, "reservation failed")
}
```

Pass `ctx` onward. A child span started from a different context starts a new trace.

High-cardinality span attributes (user ID, order ID) are normal in traces; that is what traces are for. Do not copy them onto metric attributes, where each value creates a new series (see [Prometheus cardinality](https://www.wiki.jodisand.me/prometheus/#cardinality)). Never record credentials, tokens or personal data; redact in the Collector if the SDK cannot be changed.

## SDK setup in Go

There is no agent for Go; the SDK is wired in `main`. The exporter constructors read `OTEL_EXPORTER_OTLP_*` from the environment, `resource.Default()` reads `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES`, and the default sampler reads `OTEL_TRACES_SAMPLER`, so code only needs to choose the exporter package and register the provider. See [Go](https://www.wiki.jodisand.me/go/) for module conventions.

```go
import (
    "context"

    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" // otlptracegrpc for 4317
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func setupTracing(ctx context.Context) (func(context.Context) error, error) {
    exp, err := otlptracehttp.New(ctx)          // endpoint, headers and TLS from OTEL_EXPORTER_OTLP_* variables
    if err != nil {
        return nil, err
    }
    res, err := resource.Merge(resource.Default(), // env detector: OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES
        resource.NewWithAttributes(semconv.SchemaURL, semconv.ServiceVersion("1.4.2")))
    if err != nil {
        return nil, err
    }
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),               // BatchSpanProcessor; exports every 5s or 512 spans by default
        sdktrace.WithResource(res),
    )
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{}, propagation.Baggage{}))
    return tp.Shutdown, nil                      // call with a timeout on exit or the last batch is lost
}

// Server: one span per request, named after the route pattern when the mux provides one
mux := http.NewServeMux()
srv := &http.Server{Addr: ":8080", Handler: otelhttp.NewHandler(mux, "server")}

// Client: injects traceparent into every outbound request
client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
```

Two omissions account for most silent Go setups: forgetting `SetTextMapPropagator` (the default propagator is a no-op, so nothing is injected or extracted) and exiting without `Shutdown`, which drops whatever the batch processor still holds. `go.opentelemetry.io/contrib/exporters/autoexport` picks the exporter from `OTEL_TRACES_EXPORTER` (`otlp`, `console`, `none`) if you want that decided at deploy time.

## SDK setup in Python

Python has zero-code instrumentation through a launcher and a manual SDK for code you own. The launcher patches supported libraries at import time and configures exporters from the standard variables.

```sh
uv pip install opentelemetry-distro opentelemetry-exporter-otlp   # SDK, launcher and both OTLP exporters
opentelemetry-bootstrap -a install       # detects installed libraries (Flask, requests, psycopg, ...) and installs their instrumentations
OTEL_SERVICE_NAME=my-app OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
  opentelemetry-instrument gunicorn my_app:app   # wraps the interpreter; no code change
opentelemetry-instrument --traces_exporter console --metrics_exporter none python main.py   # spans to stdout for a local check
```

The Python `opentelemetry-exporter-otlp` package defaults to gRPC on 4317; set `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` to use 4318. `OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=redis,urllib3` turns off individual instrumentations, `OTEL_PYTHON_EXCLUDED_URLS=healthz,metrics` stops health probes creating a span per scrape, and `OTEL_PYTHON_LOG_CORRELATION=true` writes trace and span IDs into standard `logging` records. Frameworks that fork workers (gunicorn, uWSGI) need the provider created in the post-fork hook, otherwise every worker shares one exporter connection inherited from the parent.

```python
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(resource=Resource.create({"service.name": "my-app", "service.version": "1.4.2"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))   # endpoint from OTEL_EXPORTER_OTLP_ENDPOINT
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("checkout")
with tracer.start_as_current_span("reserve_inventory") as span:      # context manager ends the span and restores the parent
    span.set_attribute("sku", sku)
    try:
        reserve(sku)
    except ReservationError as exc:
        span.record_exception(exc)
        span.set_status(trace.StatusCode.ERROR, "reservation failed")
        raise
```

`set_tracer_provider` can be called once per process; later calls log a warning and are ignored, which is why a library must never call it. Threads started with `threading.Thread` do not inherit the active span; pass `context.get_current()` in and call `context.attach` in the worker.

## Semantic conventions

Conventions name attributes so that dashboards and backends work across languages. Stable HTTP spans use `http.request.method`, `url.path`, `url.scheme` and `http.route` on servers, `url.full`, `server.address` and `server.port` on clients, and `http.response.status_code`, `network.protocol.version` and `error.type` on both. The span name is `{method} {route}` when a low-cardinality route is known and `{method}` alone otherwise; a span named after the raw path with IDs in it creates one operation per user in every backend. Metrics follow the same pattern: `http.server.request.duration` is a histogram in seconds with `http.request.method`, `http.route` and `http.response.status_code` attributes.

Older instrumentations emit the pre-1.21 names (`http.method`, `http.status_code`, `net.peer.name`). Java, .NET and Python read `OTEL_SEMCONV_STABILITY_OPT_IN=http` to emit the stable names, or `http/dup` to emit both during a migration. Custom attributes belong under a namespace you own (`myorg.order.id`), never under `http.` or `db.` where a future convention can collide.

## Exporting to Jaeger, Tempo and Prometheus

Every current backend accepts OTLP directly, so the SDK-side exporter is always OTLP and the choice happens in the Collector. Jaeger removed its own exporter from the Collector once Jaeger accepted OTLP natively (Jaeger 1.35, and Jaeger 2 is itself built on the Collector).

```sh
# Jaeger all-in-one for local work: UI on 16686, OTLP on 4317 and 4318
podman run --rm -p 16686:16686 -p 4317:4317 -p 4318:4318 cr.jaegertracing.io/jaegertracing/jaeger:2.21.0
# Tempo's OTLP receiver listens on the same ports; grpc endpoints take host:port, HTTP takes a URL
```

```yaml
exporters:
  otlp_grpc/jaeger:
    endpoint: jaeger.observability:4317
    tls: { insecure: true }              # plaintext inside the cluster; omit for TLS
  otlp_http/tempo:
    endpoint: https://tempo.example.com   # /v1/traces is appended
    headers: { X-Scope-OrgID: my-tenant }
  prometheus:                            # pull: exposes /metrics for Prometheus to scrape
    endpoint: 0.0.0.0:9464
    metric_expiration: 5m                # series without updates disappear after this
    enable_open_metrics: true            # required for exemplars
  prometheus_remote_write:               # push: Prometheus with remote-write receiver, Mimir, Thanos
    endpoint: https://mimir.example.com/api/v1/push
    resource_to_telemetry_conversion: { enabled: true }   # copy resource attributes to labels
```

OTLP metrics carry resource attributes separately from data point attributes. Without `resource_to_telemetry_conversion`, `service.name` and `k8s.pod.name` end up in a `target_info` series rather than on every metric, and two pods reporting the same metric collide into one series. Dots become underscores in Prometheus (`http.server.request.duration` becomes `http_server_request_duration_seconds`), and delta-temporality counters must pass through the `deltatocumulative` processor before Prometheus will accept them. See [Prometheus](https://www.wiki.jodisand.me/prometheus/) for the scrape side.

## Dropping and sampling in the Collector

The `filter` processor removes whole records that match an OTTL condition; `transform` edits records in place. Both use the same expression language.

```yaml
processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - span.attributes["url.path"] == "/healthz"          # drop probe spans
      - resource.attributes["service.name"] == "load-test"
    log_conditions:
      - log.severity_number < SEVERITY_NUMBER_WARN            # keep warnings and above
    metric_conditions:
      - metric.name == "go_gc_duration_seconds"
  probabilistic_sampler:      # head-style sampling in the Collector, consistent by trace ID
    sampling_percentage: 15
    mode: proportional        # honours a previous sampler's decision recorded in tracestate
```

`probabilistic_sampler` hashes the trace ID, so every Collector instance with the same `hash_seed` makes the same decision and traces stay whole without trace-aware load balancing. It cannot keep errors preferentially; that needs `tail_sampling`. Dropping a span in the middle of a trace leaves its children orphaned in the backend, so filter on resource attributes or on leaf spans such as health checks.

## Collector configuration

The Collector wires receivers, processors and exporters into named pipelines per signal. A component is declared at the top level and does nothing until a pipeline under `service` references it. Processors run in the order listed in each pipeline.

```yaml
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }     # default is localhost; 0.0.0.0 is needed in a container
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:                 # first: refuses data before the process is OOM-killed
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  k8s_attributes:                 # adds pod, namespace, node and workload attributes
    auth_type: serviceAccount
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.name]
  resource_detection:
    detectors: [env, system]
  transform:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - delete_key(attributes, "http.request.header.authorization")
  batch:                          # after memory_limiter and any sampling
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp_http/tempo:
    endpoint: https://tempo.example.com
    retry_on_failure: { enabled: true, max_elapsed_time: 300s }
    sending_queue: { enabled: true, queue_size: 5000 }
  prometheus_remote_write:
    endpoint: https://mimir.example.com/api/v1/push
  debug:
    verbosity: detailed           # troubleshooting only: logs every record

extensions:
  health_check: { endpoint: 0.0.0.0:13133 }
  zpages: {}

service:
  extensions: [health_check, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8s_attributes, transform, batch]
      exporters: [otlp_http/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, k8s_attributes, batch]
      exporters: [prometheus_remote_write]
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus: { host: 0.0.0.0, port: 8888 }
```

> [!IMPORTANT] Component names changed in 2026
> Collector v0.144.0 renamed `otlp` (exporter) to `otlp_grpc` and `otlphttp` to `otlp_http`. Contrib components followed, for example `k8sattributes` to `k8s_attributes`, `resourcedetection` to `resource_detection` and `prometheusremotewrite` to `prometheus_remote_write`. The old names remain as deprecated aliases that log a warning. Older Collectors only accept the old names, so match the names to the version you run. The `otlp` receiver kept its name.

`service.telemetry.metrics.address` is ignored from v0.123.0; use `readers` as above. Without any reader configuration the Collector serves its metrics on `127.0.0.1:8888`.

Put `memory_limiter` first so it can refuse data at the receiver, which returns a retryable error to the sender. Put `batch` after `memory_limiter` and after any sampling processor, so data that will be dropped is not batched first. Recent Collectors can also batch in the exporter with `sending_queue.batch`; the `batch` processor is still supported.

The exporter `sending_queue` is in memory by default. Queued data is lost on restart unless the queue is given a `storage` extension such as `file_storage`.

## Deployment patterns

| Pattern | Use |
| --- | --- |
| Agent (DaemonSet) | Receives from pods on the node, adds node and pod attributes, forwards to a gateway |
| Gateway (Deployment) | Central processing: tail sampling, redaction, fan-out to several backends, backend credentials |
| Sidecar | A workload needs an isolated pipeline or its own credentials |

Agent plus gateway is the usual Kubernetes layout. The agent sees the pod's source IP, which `k8s_attributes` uses to look up metadata. The gateway holds backend credentials in one place and is where tail sampling can see a whole trace.

The OpenTelemetry Operator can inject auto-instrumentation without changing images:

```yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default
spec:
  exporter:
    endpoint: http://otel-collector:4318
  propagators: [tracecontext, baggage]
  sampler:
    type: parentbased_traceidratio
    argument: "0.1"
```

```yaml
# Pod template annotation; also -python, -nodejs, -dotnet, -go
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
```

Injection happens at pod creation, so existing pods need a restart. Check the injected environment with `kubectl get pod <pod> -o yaml | grep OTEL_` and see [Kubernetes](https://www.wiki.jodisand.me/kubernetes/#start-with-a-failing-workload) if the pod fails to start after injection.

## Sampling

Head sampling decides at the root span and is cheap, but it cannot know whether the request will fail. `parentbased_*` samplers make child services follow the root's decision (the `01` flag in `traceparent`), so a trace is either complete or absent.

Tail sampling buffers every span of a trace in the Collector and decides after `decision_wait`. It is the only way to keep every error and every slow request while dropping most healthy traffic.

```yaml
processors:
  tail_sampling:
    decision_wait: 10s        # default 30s
    num_traces: 100000        # traces held in memory; default 50000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }
```

Every span of a trace must reach the same Collector instance. Scale tail sampling with two tiers: a first tier using the load-balancing exporter routed by trace ID, and a second tier running `tail_sampling`. Plain round-robin load balancing splits traces and each instance decides on fragments.

`decision_wait` must exceed the duration of your longest normal trace. Spans that arrive after the decision are buffered and decided again on their own (unless a decision cache is configured), which produces traces with missing tails at the slowest service. See the [tail sampling README](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md).

## Sending a test span

```json
{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"curl-test"}}]},"scopeSpans":[{"scope":{"name":"manual"},"spans":[{"traceId":"5b8efff798038103d269b633813fc60c","spanId":"eee19b7ec3c1b174","name":"test-span","kind":1,"startTimeUnixNano":"1700000000000000000","endTimeUnixNano":"1700000001000000000"}]}]}]}
```

```sh
# Save the JSON above as span.json, then post it to the OTLP/HTTP receiver
curl -sS -w '\n%{http_code}\n' -X POST http://localhost:4318/v1/traces \
  -H 'Content-Type: application/json' -d @span.json
```

```text
{"partialSuccess":{}}
200
```

Some versions return `{}` instead; either body with status 200 means the receiver accepted the span. With the `debug` exporter in the pipeline, the span appears in the Collector log. If it appears there but not in the backend, the problem is on the export side.

## Collector checks

```sh
# Validate before rolling out
otelcol validate --config=config.yaml

# Data refused, failed to send, or failed to enqueue
curl -s localhost:8888/metrics | grep -E 'otelcol_(receiver_refused|exporter_send_failed|exporter_enqueue_failed)'

# Queue fill level against capacity
curl -s localhost:8888/metrics | grep -E 'otelcol_exporter_queue_(size|capacity)'

# Spans accepted per receiver
curl -s localhost:8888/metrics | grep otelcol_receiver_accepted_spans

# Collector memory
curl -s localhost:8888/metrics | grep -E 'otelcol_process_(memory_rss|runtime_heap_alloc_bytes)'

# Confirm a service forwards the trace header (look for traceparent on outbound calls in its logs or proxy)
curl -sv -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' https://api.example.com/healthz 2>&1 | grep -i traceparent

# Watch records flowing through the debug exporter
kubectl logs -n observability deploy/otel-collector -f | grep -m5 'InstrumentationScope'

# What an SDK process will read from its environment
env | grep ^OTEL_

# Which receivers, processors, exporters and extensions this binary contains
otelcol components | grep -A40 '^processors:'

# Validate with an inline override layered on the file (the yaml: provider)
otelcol validate --config=config.yaml --config='yaml:service::telemetry::logs::level: debug'

# Merge a local override on top of the deployed config (later files win)
otelcol --config=config.yaml --config=overrides.yaml

# Is the health check extension happy (200) or not (503)
curl -s -o /dev/null -w '%{http_code}\n' localhost:13133

# Spans dropped by tail sampling, per policy decision
curl -s localhost:8888/metrics | grep -E 'otelcol_processor_tail_sampling_(count_traces_sampled|sampling_decision_latency)'

# Data the memory_limiter refused
curl -s localhost:8888/metrics | grep otelcol_processor_refused

# Batch sizes actually sent, to tune send_batch_size
curl -s localhost:8888/metrics | grep otelcol_processor_batch_batch_send_size

# Export throughput per exporter and signal
curl -s localhost:8888/metrics | grep -E 'otelcol_exporter_sent_(spans|metric_points|log_records)'

# Late spans that arrived after a tail-sampling decision
curl -s localhost:8888/metrics | grep otelcol_processor_tail_sampling_late_span_age

# Follow sampled trace fragments through zpages for one span name
curl -s 'localhost:55679/debug/tracez?zspanname=GET%20/api&ztype=1&zsubtype=0' | sed 's/<[^>]*>//g' | grep -v '^\s*$' | head -40

# Post a test metric to the OTLP/HTTP receiver
curl -sS -X POST localhost:4318/v1/metrics -H 'Content-Type: application/json' -d '{"resourceMetrics":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"curl-test"}}]},"scopeMetrics":[{"metrics":[{"name":"test.counter","sum":{"aggregationTemporality":2,"isMonotonic":true,"dataPoints":[{"asInt":"1","timeUnixNano":"'$(date +%s%N)'"}]}}]}]}]}'

# Post a test log record with a trace ID so the backend can link it
curl -sS -X POST localhost:4318/v1/logs -H 'Content-Type: application/json' -d '{"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"curl-test"}}]},"scopeLogs":[{"logRecords":[{"timeUnixNano":"'$(date +%s%N)'","severityNumber":9,"severityText":"INFO","body":{"stringValue":"hello"},"traceId":"5b8efff798038103d269b633813fc60c","spanId":"eee19b7ec3c1b174"}]}]}]}'

# Check the gRPC receiver speaks OTLP without an SDK: any grpc-status header back (even an error) proves the service is there
curl -sS --http2-prior-knowledge -X POST -H 'content-type: application/grpc' -D - -o /dev/null http://localhost:4317/opentelemetry.proto.collector.trace.v1.TraceService/Export | grep -i grpc-status

# Generate a valid random traceparent for a manual request
printf 'traceparent: 00-%s-%s-01\n' "$(openssl rand -hex 16)" "$(openssl rand -hex 8)"

# Send a request with a fresh trace and print the IDs so you can search the backend for them
tp="00-$(openssl rand -hex 16)-$(openssl rand -hex 8)-01"; echo "$tp"; curl -sS -o /dev/null -H "traceparent: $tp" https://api.example.com/orders

# Print the spans the Go SDK would export, without a Collector
OTEL_TRACES_EXPORTER=console go run ./cmd/my-app

# Same for Python, one span per line
opentelemetry-instrument --traces_exporter console --metrics_exporter none --logs_exporter none python main.py

# Which instrumentations the Python launcher will activate
opentelemetry-bootstrap -a requirements

# Pods with auto-instrumentation injected by the Operator
kubectl get pods -A -o json | jq -r '.items[] | select(.metadata.annotations["instrumentation.opentelemetry.io/inject-java"]=="true" or .metadata.annotations["instrumentation.opentelemetry.io/inject-python"]=="true") | "\(.metadata.namespace)/\(.metadata.name)"'

# OTEL_ variables the Operator injected into a running pod
kubectl exec -n my-namespace my-pod -c app -- env | grep ^OTEL_ | sort

# Collector pods and their restart counts
kubectl get pods -n observability -l app.kubernetes.io/name=opentelemetry-collector -o custom-columns='POD:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,REASON:.status.containerStatuses[0].lastState.terminated.reason'

# Collector config as deployed, from its ConfigMap (the Helm chart stores it under the key "relay")
kubectl get cm -n observability otel-collector -o jsonpath='{.data.relay}' > deployed.yaml && otelcol validate --config=deployed.yaml

# Reload Collector pods after a ConfigMap change (they do not watch the file)
kubectl rollout restart deploy/otel-collector -n observability

# Errors and forbidden responses in the Collector log in the last 10 minutes
kubectl logs -n observability deploy/otel-collector --since=10m | grep -E '"level":"(error|warn)"' | head

# Search Tempo for the trace you just generated
curl -sS "https://tempo.example.com/api/traces/${tp:3:32}" -H "X-Scope-OrgID: my-tenant" | jq '.batches[].resource'

# Search Jaeger for recent traces from one service
curl -sS 'http://localhost:16686/api/traces?service=my-app&limit=5' | jq '.data[].traceID'
```

Internal metric names can carry a `_total` suffix depending on the Prometheus exporter settings, so match on the prefix as above. See [internal telemetry](https://opentelemetry.io/docs/collector/internal-telemetry/).

## Scripts

Post a test span and confirm the receiver counted it, so a "no data" report can be split into ingest and export halves from inside the cluster.

```sh
#!/usr/bin/env bash
# usage: otel-ingest-check.sh [collector-host] ; needs curl and the Collector's 8888 port reachable
set -euo pipefail
host=${1:-localhost}
accepted() { curl -sf "http://$host:8888/metrics" | awk '/^otelcol_receiver_accepted_spans/ {s += $2} END {print s + 0}'; }
before=$(accepted)
tid=$(openssl rand -hex 16); sid=$(openssl rand -hex 8); now=$(date +%s%N)
body=$(printf '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"ingest-check"}}]},"scopeSpans":[{"spans":[{"traceId":"%s","spanId":"%s","name":"ingest-check","kind":1,"startTimeUnixNano":"%s","endTimeUnixNano":"%s"}]}]}]}' "$tid" "$sid" "$now" "$now")
code=$(curl -sS -o /dev/null -w '%{http_code}' -X POST "http://$host:4318/v1/traces" -H 'Content-Type: application/json' -d "$body")
[[ $code == 200 ]] || { printf 'receiver returned %s\n' "$code" >&2; exit 1; }
sleep 2
after=$(accepted)
printf 'trace %s accepted; receiver counter %s -> %s\n' "$tid" "$before" "$after"
(( after > before )) || { printf 'counter did not move: is 8888 the right Collector?\n' >&2; exit 1; }
```

Summarise a Collector's health from its internal metrics: refused, failed and queued data per component, so the failing stage is obvious before reading logs.

```sh
#!/usr/bin/env bash
# usage: otel-collector-report.sh [collector-host]
set -euo pipefail
host=${1:-localhost}
m=$(curl -sf --max-time 5 "http://$host:8888/metrics") || { echo "cannot scrape $host:8888" >&2; exit 1; }
health=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "http://$host:13133" || true)
printf 'health_check: %s\n\n' "${health:-unreachable}"
printf '%-34s %-28s %12s\n' METRIC COMPONENT VALUE
printf '%s\n' "$m" | awk '
  /^otelcol_(receiver_refused|exporter_send_failed|exporter_enqueue_failed|processor_refused|exporter_queue_size|exporter_queue_capacity)/ {
    name = $1; sub(/\{.*/, "", name)
    comp = ""; if (match($0, /(receiver|exporter|processor)="[^"]*"/)) comp = substr($0, RSTART, RLENGTH)
    if ($2 + 0 > 0) printf "%-34s %-28s %12d\n", name, comp, $2
  }'
printf '\nrss: %s MiB\n' "$(printf '%s\n' "$m" | awk '/^otelcol_process_memory_rss/ {printf "%d", $2 / 1048576}')"
```

Generate a linked parent and child span with fresh IDs and post them over OTLP/HTTP using only the Python standard library, for checking a backend's trace view without installing an SDK.

```python
#!/usr/bin/env python3
"""usage: otel-test-trace.py [http://localhost:4318]"""
import json, secrets, sys, time, urllib.request

endpoint = (sys.argv[1] if len(sys.argv) > 1 else "http://localhost:4318").rstrip("/")
trace_id, parent_id, child_id = secrets.token_hex(16), secrets.token_hex(8), secrets.token_hex(8)
now = time.time_ns()

def span(span_id, name, start, end, parent=None, kind=2):
    s = {"traceId": trace_id, "spanId": span_id, "name": name, "kind": kind,
         "startTimeUnixNano": str(start), "endTimeUnixNano": str(end),
         "attributes": [{"key": "http.request.method", "value": {"stringValue": "GET"}}]}
    if parent:
        s["parentSpanId"] = parent
    return s

payload = {"resourceSpans": [{
    "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "test-trace"}}]},
    "scopeSpans": [{"scope": {"name": "manual"}, "spans": [
        span(parent_id, "GET /orders", now - 300_000_000, now),
        span(child_id, "SELECT orders", now - 250_000_000, now - 50_000_000, parent=parent_id, kind=3),
    ]}]}]}
req = urllib.request.Request(f"{endpoint}/v1/traces", data=json.dumps(payload).encode(),
                             headers={"Content-Type": "application/json"}, method="POST")
with urllib.request.urlopen(req, timeout=10) as resp:
    print(resp.status, resp.read().decode())
print(f"trace_id={trace_id}")
```

## Troubleshooting

| Symptom | Likely cause | Check |
| --- | --- | --- |
| No data from an SDK | Protocol and port mismatch, wrong endpoint path, SDK disabled | `env \| grep ^OTEL_`; SDK debug logging; send a test span from the same pod |
| Collector fails to start | Unknown component name or key, often a contrib component in a core build | `otelcol validate`; `otelcol components` lists what the binary contains |
| Deprecation warnings at start | Old component names on v0.144.0 or later | Rename to `otlp_grpc`, `otlp_http`, `k8s_attributes` and so on |
| `connection refused` from the SDK | Receiver bound to `localhost` inside a container | Set receiver `endpoint: 0.0.0.0:4317` |
| Service appears as `unknown_service` | `service.name` not set | Set `OTEL_SERVICE_NAME` |
| Traces split into several traces | Propagation lost at a hop, or mismatched propagators | Log incoming `traceparent` at each hop; align `OTEL_PROPAGATORS` |
| Traces with missing spans | Head sampling not parent-based, or tail sampling instances each saw part of the trace | Use `parentbased_*`; route by trace ID before `tail_sampling` |
| `otelcol_receiver_refused_*` rising | `memory_limiter` refusing data | Collector memory metrics; raise limits or scale out |
| `otelcol_exporter_queue_size` at capacity | Backend slow or down | `otelcol_exporter_send_failed_*`; backend health |
| `k8s_attributes` adds nothing | RBAC missing, or the agent sees a proxy IP instead of the pod IP | Collector logs for forbidden errors; send from pods directly to the node agent |
| Metrics duplicated or missing in Prometheus | Two Collectors write the same series, or resource attributes dropped | Compare `job` and `instance`; check `resource_to_telemetry_conversion` |
| Go service exports spans but nothing links across services | `otel.SetTextMapPropagator` never called; the default propagator is a no-op | Set the composite `TraceContext` and `Baggage` propagator; check outbound requests for `traceparent` |
| Last spans before exit never arrive | Process exited without `TracerProvider.Shutdown` (Go) or before the batch processor flushed (Python `atexit` skipped by `os._exit`) | Call shutdown with a timeout on `SIGTERM`; set `OTEL_BSP_SCHEDULE_DELAY` lower for short-lived jobs |
| Python: `Overriding of current TracerProvider is not allowed` | `set_tracer_provider` called twice, usually once by `opentelemetry-instrument` and once in code | Remove the manual call under the launcher, or drop the launcher |
| One span per health-check scrape | Probes instrumented like any request | `OTEL_PYTHON_EXCLUDED_URLS=healthz,metrics`; `otelhttp.WithFilter` in Go; or a `filter` processor on `url.path` |
| Same service shows as two in the backend | Different `service.name` from env and code, or one pod with the old `deployment.environment` and one with `.name` | `kubectl exec -- env \| grep OTEL_`; `otelcol` `debug` exporter output for the resource |
| Attribute names differ between services | Mixed old and stable HTTP semantic conventions | `OTEL_SEMCONV_STABILITY_OPT_IN=http` or `http/dup` on the older SDKs; upgrade instrumentation |
| Every metric has the same value across pods, or series flap | Resource attributes not converted to labels, so pods overwrite one another | `resource_to_telemetry_conversion` on the exporter or `resource_constant_labels` on `prometheus` |
| Delta counters missing after a Collector upgrade | Prometheus exporters drop delta temporality | `deltatocumulative` processor before the exporter, or `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative` in the SDK |
| Collector memory climbs until OOM with `tail_sampling` | `num_traces` times average trace size exceeds the limit, or `decision_wait` too long | `otelcol_processor_tail_sampling_*` metrics; lower `num_traces` or add replicas behind the load-balancing exporter |
| Config change has no effect | Collector does not reload on ConfigMap updates | `kubectl rollout restart` the Collector; confirm with `otelcol_process_uptime` |

## Further reading

- [Collector configuration](https://opentelemetry.io/docs/collector/configuration/) and [contrib component list](https://github.com/open-telemetry/opentelemetry-collector-contrib)
- [W3C Trace Context](https://www.w3.org/TR/trace-context/)
- [Semantic conventions](https://opentelemetry.io/docs/specs/semconv/)
- [Go SDK](https://opentelemetry.io/docs/languages/go/) and [Python zero-code configuration](https://opentelemetry.io/docs/zero-code/python/configuration/)
- [OTLP specification](https://opentelemetry.io/docs/specs/otlp/)
- [Jaeger getting started](https://www.jaegertracing.io/docs/latest/getting-started/)