Skip to content

compact: implement native histogram custom buckets downsampling #8698

@bbondarenko

Description

@bbondarenko

Is your proposal related to a problem?

Compactor crashes with the following error:
panic: cannot reduce resolution to custom buckets schema

Describe the solution you'd like

Compactor support downsampling of native histograms custom buckets.

Describe alternatives you've considered

Disabling native histograms in prometheus.

Additional context

Prometheus: 3.9.1
Thanos : v0.41.0

Debug log:

ts=2026-02-26T18:13:18.445641724Z caller=streamed_block_writer.go:178 level=info msg="finalized downsampled block" mint=1770854400000 maxt=1772064000000 ulid=01KJDJBP3BKRF9RXC2Y7GAEQV1 resolution=300000 panic: cannot reduce resolution to custom buckets schema goroutine 3325 [running]: github.com/prometheus/prometheus/model/histogram.(*FloatHistogram).CopyToSchema(0x19c96d0eaff?, 0x0?)	/go/pkg/mod/github.com/prometheus/prometheus@v0.308.0/model/histogram/float_histogram.go:154 +0x2e6 github.com/thanos-io/thanos/pkg/compact/downsample.(*histogramAggregator).add(0xc00146c390, {0x42d34f0?, 0xc00146c390?, 0xc0018dc3c0?})	/app/pkg/compact/downsample/downsample.go:363 +0x45 github.com/thanos-io/thanos/pkg/compact/downsample.downsampleBatch({0xc000256f08?, 0x6066d20?, 0x60068?}, 0x493e0, {0x42d34f0, 0xc00146c390}, 0xc000fb6e70)	/app/pkg/compact/downsample/downsample.go:843 +0x7f github.com/thanos-io/thanos/pkg/compact/downsample.downsampleHistogramBatch({0xc000256f08?, 0x3386e00?, 0x164}, 0x493e0)	/app/pkg/compact/downsample/downsample.go:455 +0x154 github.com/thanos-io/thanos/pkg/compact/downsample.downsampleRawLoop({0xc0032cc000?, 0xc000fb6ff8?, 0x1ec7033?}, 0x493e0, 0xc00316d200?, 0xc000fb6fd8, 0x3f09f30)	/app/pkg/compact/downsample/downsample.go:692 +0x1aa github.com/thanos-io/thanos/pkg/compact/downsample.DownsampleRaw({0xc0032cc000?, 0x164, 0x11aaa}, 0x493e0)	/app/pkg/compact/downsample/downsample.go:648 +0x165 github.com/thanos-io/thanos/pkg/compact/downsample.Downsample({0x42de890, 0xc0009d8000}, {0x42a55a0, 0xc000930140}, 0xc0012894a0, {0x42eb020, 0xc001457a20}, {0x7ffe17a25284, 0x9}, 0x493e0)	/app/pkg/compact/downsample/downsample.go:190 +0xaff main.processDownsampling({0x42de890, 0xc0009d8000}, {0x42a55a0, 0xc000930140}, {0x4309568, 0xc00094b8f0}, 0xc0012894a0, {0x7ffe17a25284, 0x9}, 0x493e0, ...)	/app/cmd/thanos/downsample.go:385 +0x845 main.downsampleBucket.func3()	/app/cmd/thanos/downsample.go:265 +0x255 sync.(*WaitGroup).Go.func1()	/usr/local/go/src/sync/waitgroup.go:239 +0x4a created by sync.(*WaitGroup).Go in goroutine 83	/usr/local/go/src/sync/waitgroup.go:237 +0x73 

(Write your answer here.)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions