Improve ClickHouse Keeper Grafana Dashboard

[discostur]

Fix ClickHouse Keeper Grafana Dashboard

fix #1844

Problem

The ClickHouse Keeper dashboard was not working correctly due to:

1. Incorrect Prometheus label names (pod_name vs pod, container_name vs container)
2. Template variables using incorrect label filters
3. Hardcoded metric names in template variable queries

Changes

Label Fixes

• Changed pod_name → pod in all queries and template variables
  • Prometheus/Mimir exports Kubernetes pod labels as pod, not pod_name
• Removed container_name="clickhouse-keeper" filters from all panel queries
  • Label should be container, not container_name
  • Filter not necessary as queries are already scoped to Keeper-specific metrics
• Updated legend formats from {{pod_name}} to {{pod}} across all panels

Template Variable Improvements

Before:

label_values(up{container_name="clickhouse-keeper"}, namespace) label_values(up{container_name="clickhouse-keeper", namespace=~"$namespace"}, pod_name) 

After:

label_values({__name__=~"ClickHouse.*Keeper.*",pod=~"chk-clickhouse-keeper.*"}, namespace) label_values({__name__=~"ClickHouse.*Keeper.*",pod=~"chk-clickhouse-keeper.*",namespace=~"$namespace"}, pod) 

Benefits:

• Uses generic metric selector instead of up metric with incorrect label
• Explicit pod name filter (chk-clickhouse-keeper.*) ensures only Keeper pods are shown
• Works with any ClickHouse Keeper metric, not dependent on specific metric availability

[Slach]

thanks for contribution, really appreciated to sync metrics names

look to https://github.com/Altinity/clickhouse-operator/blob/master/deploy/prometheus/prometheus-template.yaml#L52-L54

in our prometheus scraping config we use pod_name, not pod

please use following commands to check your dashboard changes

minikube delete minikube start kubectl create ns test helm install -n test clickhouse-operator ./deploy/helm/clickhouse-operator/ NO_WAIT=1 bash -xe ./deploy/prometheus/create-prometheus.sh NO_WAIT=1 bash -xe ./deploy/grafana/grafana-with-grafana-operator/install-grafana-operator.sh NO_WAIT=1 bash -xe ./deploy/grafana/grafana-with-grafana-operator/install-grafana-with-operator.sh kubectl apply -f ./docs/chk-examples/01-chi-simple-with-keeper.yaml 

[discostur]

@Slach ah i did miss the relabel config ... ok with that it works, correct. However i think this is kind of outdated and all other projects these days just use pod and container as labels instead of pod_name and container_name. So my suggestion would be to use the updated dashboard i provided and adjust the relabel config to be more streamlined to other metrics. In addition we would not "break" other setups because the relabel config would relabel all pod labels which have the annotation "prometheus_io_scrape=true".

[discostur]

I provided the corresponding changes and some minor fixes to the test setup itself you recommended ;)

I tried it with minikube and all metrics are working now:

[Slach]

too many changes which not related to dashboard improvements
thanks to adopt script to MacOS

But please don't break name conversion, look to initial prometheus scraping config and rollback to pod_name and container_name

[Slach]

rollback to pod_name please

[discostur]

why? most of the CNCF projects use pod instead of pod_name today. If you apply your prometheus scrape config other metrics in your cluster will be broken. With this change we would be more streamlined with other projects and won't break other labels.

If you stick to pod_name you have to deploy a separate prometheus instance which only scrapes altinity pod labels (and transforms them). With my approach one prometheus can scrape the whole labels from all pods in the cluster and it won't break.

[Slach]

why?

if you change pod_name > pod, container_name -> container
then you need also check it in tests/*

most of the CNCF projects use pod instead of pod_name today.

I agree.

Unfortunately, this break doesn't allow syncing too many things in our internal infrastructure, which is not open-sourced.

So if your improvements here just rename pod_name > pod, container_name > container, we need more valuable changes

[Slach]

i don't understand what exactly you trying to improve here? I don't see improvements, only breaks old name conversions

up{container_name='clickhouse-keeper'} will have the same result with {__name__=~\"ClickHouse.*Keeper.*\",pod=~\"chk-.*\"

both could be broken

chk-.* could be broken with chk.spec.templates.podTemplates.generateName
and clickhouse-keeper container name could be broken with chk.spec.templates.podTemplates.spec.containers

[Slach]

up is metrics which generated from prometheus itself after Kubernetes pod discovery.