Skip to content

query: query_exemplars does not account for external labels #4116

@tcolgate

Description

@tcolgate

@goku321

Thanos, Prometheus and Golang version used: v0.20.0 docker image

Object Storage Provider: gcs

What happened:

  • configured --exemplar=... on our local and global queriers
  • configure grafana for exemplars

What you expected to happen:

  • grafana "time series" panel should show exemplars.
    however...
    the time series panel includes the full query from the panel, including external labels used by the local prom
    instances.
    the query_range queries work as expected.
    query_exemplars did not.

If I send the query_exemplars a query including the external labels, I get an empty response.
If I remove the external labels and just use labels the local prom instance knows about, I get responses.

How to reproduce it (as minimally and precisely as possible):

  • configure a prometheus with external labels, with sidecar
  • configure thanos query to see the sidecar
  • query thanos-query with the external labels.
    (we have a second tier of queriers, but I do not think that is causing the issue.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions