Monitoring

Monitoring is enabled by default through kube-prometheus-stack. Prometheus stores metrics in-cluster unless you configure a remote write destination.

ClickStack is also enabled by default for operational logs, traces, and a HyperDX UI. It complements Prometheus rather than replacing it: Prometheus remains the metrics source, while ClickStack handles trace and log exploration for the deployment.

What's Scraped

The chart adds ServiceMonitors for:

• App (/api/metrics): app/admin API request counts, latency histograms, coarse rejections, and frontend error counts.
• HPS (/metrics): rule-engine request counts, latency histograms, rejections, Kafka worker wait time, bulk item volume, and memory cache stats.
• Supporting infrastructure where available: Kafka JMX and ClickHouse metrics. Traefik's Prometheus endpoint is enabled, but its ServiceMonitor is an explicit opt-in because the Traefik chart validates Prometheus Operator CRDs at render time.

Metric labels are intentionally bounded to avoid cardinality problems: route templates, methods, status classes, operations, and coarse reasons. They never include API keys, users, organizations, IP addresses, raw URLs, rule slugs, flow slugs, or error messages.

Useful queries:

histogram_quantile(0.95, sum(rate(rulebricks_hps_http_request_duration_seconds_bucket[5m])) by (le, route))
sum(rate(rulebricks_hps_rejections_total[5m])) by (route, reason)
histogram_quantile(0.95, sum(rate(rulebricks_hps_kafka_request_duration_seconds_bucket[5m])) by (le, operation))
sum(rate(rulebricks_hps_bulk_items_total[5m])) by (operation)
sum(rate(rulebricks_app_frontend_errors_total[5m])) by (source)

After install, verify scrape discovery:

kubectl get servicemonitor -n rulebricks
kubectl port-forward -n rulebricks svc/rulebricks-kube-prometheus-stack-prometheus 9090:9090

Built-in Observability

With global.clickstack.enabled: true, the chart deploys the ClickStack observability backend:

The default HyperDX ingress uses observability.<global.domain> when enabled. Keep access limited to your operational users, and use the ingress allow-list values if your cluster exposes it publicly.

If your organization already has a telemetry platform, disable ClickStack and configure the BYO paths instead:

global.tracing.* renders the chart-managed OpenTelemetry Collector only when ClickStack is disabled. The Rulebricks CLI asks these questions during rulebricks init and writes secret references where possible.

Remote Write

To ship metrics to AWS Managed Prometheus, Azure Monitor managed Prometheus, Grafana Cloud, or another remote-write-compatible backend:

kube-prometheus-stack:
  prometheus:
    prometheusSpec:
      remoteWrite:
        - url: 'https://prometheus-prod-XX.grafana.net/api/prom/push'
          basicAuth:
            username:
              name: prometheus-remote-write
              key: username
            password:
              name: prometheus-remote-write
              key: password

In-Cluster Retention

When keeping metrics in-cluster, give Prometheus persistent storage sized for your retention window:

kube-prometheus-stack:
  prometheus:
    prometheusSpec:
      retention: 30d
      storageSpec:
        volumeClaimTemplate:
          spec:
            storageClassName: gp3
            resources:
              requests:
                storage: 50Gi