Open-source · ClickHouse observability

Turn ClickHouse telemetry into query insight.

Click-Dog turns ClickHouse native spans and query logs into traces, normalized query families, known-good regression comparisons, findings, notifications, and dashboards for Datadog, Splunk, Honeycomb, Grafana Tempo, and any OTLP backend — read-only, with bounded source impact.

curl -fsSL https://github.com/coltconsulting/click-dog/releases/latest/download/install.sh -o install.sh && sudo bash install.sh

Signed and checksummed before install.

Get started View on GitHub

Public betaApache-2.0 licensed

Read-only by design

One static binary

Fork it freely

trace · slow-query-monitor live · clickhouse

SELECT events JOIN users GROUP BY day 1,284 ms

query · root1284ms

read events487ms

merge parts312ms

hash join users436ms

aggregate day231ms

sort + limit140ms

Exports to Datadog Splunk Honeycomb Grafana Tempo Jaeger Elastic Any OTLP

Why Click-Dog

Built to run in production.

Three things shape Click-Dog: it's manageable to run, it stays out of your source system's way, and it explains what your queries are doing.

Manageable in production

Run it as a single static binary, a container, or generated Kubernetes / Docker Compose manifests. Validate config before it talks to anything, dry-run exports, and monitor Click-Dog itself.

healthreadinessstatusmetrics

Low-impact source safety

The non-invasive property. Read-only connections, bounded polling, SQL-level filtering, batching, circuit-breaker protection, and adaptive backoff — so your source system stays in control.

read-onlycircuit breakerbackoffleader-gated HA

Query analysis, not trace shipping

Click-Dog exports spans, but the goal is to explain queries: slow-query traces, normalized query families, and bounded exported operational activity—not an audit log—across users, databases, tables, and operations.

slow queriesquery familiesexported user activity

Features

What Click-Dog ships out of the box.

Click-Dog does more than forward spans. It packages regression analysis, privacy controls, finding delivery, dashboards, export health, and rollout checks for a production telemetry path.

Native slow-query traces

Export ClickHouse's native trace topology, enriched with query-log context so slow traces carry the database behavior that caused them.

Normalized query families

Group thousands of statements into stable shapes that make p95, volume, and trend changes easier to reason about.

Baselines and regressions

Capture an explicit known-good window, then detect conservative latency regressions and failure spikes without silently moving the baseline.

Finding policy and notifications

Gate automation with severity-based exit codes and send privacy-reviewed new or critical findings to webhooks and Datadog Events.

Query-text privacy

Choose raw, redacted, normalized-only, or no query text at the final export boundary, with fail-closed normalized-only behavior.

Go-live smoke tests

Check source readiness, test every exporter with a local span, and prove ClickHouse native parent-child tracing before enabling the service.

Ready-made dashboards

Provision Datadog dashboards for application query analysis, exported user-activity review, and Click-Dog health—without starting from a blank canvas.

Export health and resilience

Track every sink independently while circuit-breaker protection and adaptive backoff keep failures from adding source pressure.

Deployment assets

Run as a static binary, systemd service, Docker Compose stack, Kubernetes deployment, or Ansible-managed sidecar fleet.

Architecture

One bounded pipeline, from source to sink.

Click-Dog sits beside your database — never in the request path — reads telemetry over a read-only connection, filters and batches at the source, then fans out to one or more backends.

Source

ClickHouse

system.opentelemetry_span_log

read-only · no DDL

Query log

system.query_log · backfill

Click-Dog

Collector

poll → filter → dedupe → batch

SQL filterLRU dedupebatchcircuit breakerbackoffleader election

Sinks · fan-out

Datadog

OTLP/gRPC

Splunk

HEC

Honeycomb

OTLP/gRPC

Any OTLP

Tempo · Jaeger · more

Query analysis

Follow a query from signal to finding.

Group and enrich queries

Roll normalized_query_hash values into stable shapes and add bounded query-log context.

Compare with known good

Capture a fixed baseline before a change, then compare a later non-overlapping window for latency regressions and failure spikes.

Gate and notify

Choose a warning or critical failure threshold and explicitly deliver eligible findings to webhooks or Datadog Events.

Drill into the evidence

Keep the bounded local report, then follow an exact hash through query and native-trace evidence.

query families · p95 last 24h 12 shapes

Query familyCallsp95Trend

SELECT events JOIN users …48.2k1.28s

SELECT count() FROM orders …211k340ms

INSERT INTO metrics …1.9M22ms

SELECT * FROM sessions WHERE …17.4k512ms

SELECT uniq(uid) FROM events …9.1k2.04s

Safety

Conservative by default — it runs next to your production database.

Click-Dog should make query telemetry easier to trust without making the source system harder to operate.

Source protection

Read-only ClickHouse sessionsAll runtime connections enforce readonly=2.

No schema changesNo DDL requirement and no write grants to your database.

Bounded pollingSQL-level filters, lookback windows, and max spans per cycle cap load.

Failure backpressureCircuit breaker and adaptive backoff reduce pressure during incidents.

Rollout control

Validate offlineCheck configuration before connecting to source or sink.

Test exporter routingclick-dog test export asks every configured sink to accept a local span.

Prove native tracingclick-dog test tracing verifies ClickHouse's sampled parent-child topology end to end.

Dry-run real readsExercise the fetch and filter path while discarding exports.

Probe readinessWatch /healthz, /readyz, /status, and metrics before and after enabling scheduled mode.

Use secure transportTLS and mTLS are available for collector and source links.

Deployment

Start small, then roll it out like infrastructure.

Single binary

Static Linux binary for a quick first node or a controlled manual install.

systemd

Installer-generated service, config validation, and local health checks.

Kubernetes

Generated manifests for a centralized cluster reader with readiness probes.

Ansible

Fleet rollout, canary playbook, bounded concurrency, and repeatable config.

Configuration example

Point it at ClickHouse. Pick your sinks. Run.

One YAML file and one command. Validate it, dry-run the export, then turn it on.

click-dog.yaml

clickhouse:
  host: localhost
  port: 9000
  username: default
  password: ${CLICKHOUSE_PASSWORD}

exporters:
  otel:
    - collector_address: localhost:4317
      service_name: click-dog-monitor

filters:
  query_text_mode: normalized_only

monitor:
  enabled: true
  min_trace_duration_ms: 1000
  check_interval_s: 30

log_level: info

terminal~/click-dog

# secret used by all three commands
$ export CLICKHOUSE_PASSWORD="secret"

# 1 · validate before it talks to anything
$ ./click-dog -validate -config click-dog.yaml
Config click-dog.yaml is valid.
  ClickHouse:  localhost:9000
  Exporters:   1 OTEL, 0 Splunk HEC
    OTEL[0]:       localhost:4317 (service=click-dog-monitor)
  Monitor:     min_trace=1000ms, interval=30s
  Self-metrics: OTLP push off

# 2 · dry-run the export pipeline
$ ./click-dog --dry-run -config click-dog.yaml
# earlier timestamped startup lines omitted
… [INFO] Dry-run mode: exports will be discarded
…
--- Dry Run Summary ---
# counts and top operations omitted; values depend on live data

# 3 · run it
$ ./click-dog -config click-dog.yaml
# earlier timestamped startup lines omitted
… [INFO] Starting scheduled mode: min_trace_duration=1000ms, interval=30s, lookback=40s (interval=30 + buffer=10)
# state-dependent startup check omitted

Put a watchdog on your queries.

Open source under Apache 2.0 — run, modify, and redistribute it under the license terms. Read-only by design, and live in minutes.

Get started — it's free Star on GitHub