Generic OTLP Integration¶

Click-Dog exports traces via OTLP/gRPC, making it compatible with any backend that accepts OpenTelemetry traces. This includes Grafana Tempo, Jaeger, Zipkin (via OTEL Collector), Elastic APM, and others.

Packaged backend experiences

Generic OTLP export works today. Backend-specific dashboards, saved views, and alert recipes for Tempo, Jaeger, Elastic, and other receivers are being prioritized from user demand. If one of these would help your rollout, tell us which backend and which views matter via Support.

Direct OTLP Backend¶

If your backend supports OTLP gRPC natively (e.g., Grafana Tempo, Jaeger with OTLP receiver):

Click-Dog blocks on this page are fragments

Merge each exporters.otel example into a complete config such as the minimal profile. The collector examples are separate OpenTelemetry Collector configurations.

exporters:
  otel:
    - collector_address: tempo.internal:4317
      service_name: click-dog-monitor

filters:
  # Compatibility default is raw. Prefer normalized_only or none when query
  # literals may contain secrets or PII.
  query_text_mode: normalized_only

The mode is applied before OTLP serialization and before multi-sink fan-out. normalized_only omits db.statement and retains an explicitly labeled bounded normalized preview only when ClickHouse provides one; it never falls back to raw SQL. All privacy-restricting modes also drop span attribute values carrying an HTTP query parameter so URL-encoded GET statements cannot bypass db.statement shaping. See Query Text Export Modes.

After configuring the backend, isolate OTLP delivery and then exercise the native ClickHouse tracing path:

click-dog test export -config /etc/click-dog/click-dog.yaml
click-dog test tracing -config /etc/click-dog/click-dog.yaml

Each sink is reported independently. PASS means the receiver accepted the batch, not that a downstream backend has indexed or rendered it.

Via OpenTelemetry Collector¶

For backends that don't support OTLP directly, use an OTEL Collector as a proxy.

Grafana Tempo¶

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  otlp:
    endpoint: tempo.internal:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]

Jaeger¶

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  otlp:
    endpoint: jaeger.internal:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]

Elastic APM¶

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  otlp/elastic:
    endpoint: apm-server.internal:8200
    headers:
      Authorization: "Bearer ${ELASTIC_APM_SECRET_TOKEN}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/elastic]

Multiple Backends¶

Send traces to multiple destinations simultaneously:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  otlp/tempo:
    endpoint: tempo.internal:4317
    tls:
      insecure: true

  otlp/jaeger:
    endpoint: jaeger.internal:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/tempo, otlp/jaeger]

TLS Configuration¶

Plain TLS¶

exporters:
  otel:
    - collector_address: otel-collector.internal:4317
      service_name: click-dog-monitor
      secure: true
      ca_cert: /path/to/ca.pem

Mutual TLS (mTLS)¶

exporters:
  otel:
    - collector_address: otel-collector.internal:4317
      service_name: click-dog-monitor
      secure: true
      ca_cert: /path/to/ca.pem
      client_cert: /path/to/client.pem
      client_key: /path/to/client-key.pem

Both client_cert and client_key must be provided together for mTLS, regardless of the secure setting. A lone half of the pair is an invalid configuration and causes -validate to fail.

When secure: true, -validate also checks that configured ca_cert, client_cert, and client_key files are readable. Certificate format and client certificate/key matching are checked when the TLS connection is built. Setting insecure_skip_verify: true requires secure: true; use it only for testing.

When secure: false, a configured CA certificate or complete client certificate/key pair is ignored and produces a validation warning. Remove the inactive TLS fields or enable TLS so the intended credentials take effect.

Batch size and retry implications¶

For live span batches, click-dog incrementally groups encoded OTLP spans into gRPC requests targeting at most 3,500,000 bytes (3.5 MB), leaving headroom below gRPC's common 4 MB default. If a collector still returns ResourceExhausted for a multi-span request, click-dog bisects that request and recursively sends the halves. A single span that receives ResourceExhausted cannot be split and fails the export call. Other gRPC errors are returned without an in-call retry.

Chunk delivery is all-or-nothing from the pipeline's accounting perspective. If early chunks (or the first half of a bisection) succeed and a later send fails, the exporter returns no accepted span keys. The next normal cycle then retries the original batch, including chunks the collector already accepted. Those spans retain their (trace_id, span_id), but downstream duplicate handling is backend-specific. Monitor export errors and size limits rather than assuming partial delivery is exactly-once.

monitor.max_query_length has two different effects depending on the path. On the query_log fetch — backfill and the analyze trace candidate search — it adds a length(query) predicate that excludes longer rows outright. Scheduled mode does not apply that predicate to the span-log fetch; there the key only bounds the enriched query_log.normalized_query attribute. Query-family rollups are a third case again: the analyze queries family rollups do not take the key at all, bounding preview text with their own limit instead. Exporter-specific max_query_length is what truncates outbound db.statement on both the live and backfill export paths. See Query Text Truncation.

Exported Data Format¶

Click-Dog exports standard OTLP trace data. Every span includes:

Resource Attributes¶

service.name: Configured via exporters.otel[].service_name

Instrumentation Scope¶

Scope name: clickhouse

Span Fields¶

trace_id: Original ClickHouse trace ID (16 bytes)
span_id: Original span ID (8 bytes)
parent_span_id: Parent span ID (if any)
name: Operation name from ClickHouse
kind: Span kind (INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER)
start_time: Microsecond precision from ClickHouse
end_time: Microsecond precision from ClickHouse
status: OK for live span-log spans; backfill query-log spans carry ERROR when the query failed (exception_code != 0)

Span Attributes¶

See Span Attributes Reference for the complete list.

Click-Dog Configuration¶

exporters:
  otel:
    - collector_address: your-endpoint:4317
      service_name: click-dog-monitor
      max_query_length: 100000

The endpoint, service name, and certificate-path string fields support environment variable expansion:

exporters:
  otel:
    - collector_address: ${OTEL_COLLECTOR_ADDRESS}
      service_name: ${OTEL_SERVICE_NAME}