- Notifications
You must be signed in to change notification settings - Fork 2.3k
Open
Description
Is your proposal related to a problem?
Compactor crashes with the following error:
panic: cannot reduce resolution to custom buckets schema
Describe the solution you'd like
Compactor support downsampling of native histograms custom buckets.
Describe alternatives you've considered
Disabling native histograms in prometheus.
Additional context
Prometheus: 3.9.1
Thanos : v0.41.0
Debug log:
ts=2026-02-26T18:13:18.445641724Z caller=streamed_block_writer.go:178 level=info msg="finalized downsampled block" mint=1770854400000 maxt=1772064000000 ulid=01KJDJBP3BKRF9RXC2Y7GAEQV1 resolution=300000 panic: cannot reduce resolution to custom buckets schema goroutine 3325 [running]: github.com/prometheus/prometheus/model/histogram.(*FloatHistogram).CopyToSchema(0x19c96d0eaff?, 0x0?) /go/pkg/mod/github.com/prometheus/prometheus@v0.308.0/model/histogram/float_histogram.go:154 +0x2e6 github.com/thanos-io/thanos/pkg/compact/downsample.(*histogramAggregator).add(0xc00146c390, {0x42d34f0?, 0xc00146c390?, 0xc0018dc3c0?}) /app/pkg/compact/downsample/downsample.go:363 +0x45 github.com/thanos-io/thanos/pkg/compact/downsample.downsampleBatch({0xc000256f08?, 0x6066d20?, 0x60068?}, 0x493e0, {0x42d34f0, 0xc00146c390}, 0xc000fb6e70) /app/pkg/compact/downsample/downsample.go:843 +0x7f github.com/thanos-io/thanos/pkg/compact/downsample.downsampleHistogramBatch({0xc000256f08?, 0x3386e00?, 0x164}, 0x493e0) /app/pkg/compact/downsample/downsample.go:455 +0x154 github.com/thanos-io/thanos/pkg/compact/downsample.downsampleRawLoop({0xc0032cc000?, 0xc000fb6ff8?, 0x1ec7033?}, 0x493e0, 0xc00316d200?, 0xc000fb6fd8, 0x3f09f30) /app/pkg/compact/downsample/downsample.go:692 +0x1aa github.com/thanos-io/thanos/pkg/compact/downsample.DownsampleRaw({0xc0032cc000?, 0x164, 0x11aaa}, 0x493e0) /app/pkg/compact/downsample/downsample.go:648 +0x165 github.com/thanos-io/thanos/pkg/compact/downsample.Downsample({0x42de890, 0xc0009d8000}, {0x42a55a0, 0xc000930140}, 0xc0012894a0, {0x42eb020, 0xc001457a20}, {0x7ffe17a25284, 0x9}, 0x493e0) /app/pkg/compact/downsample/downsample.go:190 +0xaff main.processDownsampling({0x42de890, 0xc0009d8000}, {0x42a55a0, 0xc000930140}, {0x4309568, 0xc00094b8f0}, 0xc0012894a0, {0x7ffe17a25284, 0x9}, 0x493e0, ...) /app/cmd/thanos/downsample.go:385 +0x845 main.downsampleBucket.func3() /app/cmd/thanos/downsample.go:265 +0x255 sync.(*WaitGroup).Go.func1() /usr/local/go/src/sync/waitgroup.go:239 +0x4a created by sync.(*WaitGroup).Go in goroutine 83 /usr/local/go/src/sync/waitgroup.go:237 +0x73 (Write your answer here.)
Reactions are currently unavailable