Thanos, Prometheus and Golang version used:

thanos: v0.40.1
prometheus: v3.9.1

Object Storage Provider:

AWS: S3

What happened:

I ran thanos store with the following configurations:

"store", "--http-address=0.0.0.0:10902", "--grpc-address=0.0.0.0:10901", "--data-dir=./data", "--cache-index-header", "--objstore.config-file=/etc/thanos/storager/bucket-tst.yml", "--index-cache.config-file=/etc/thanos/storager/cache-index-tst.yml", "--store.caching-bucket.config-file=/etc/thanos/storager/cache-bucket-tst.yml", "--consistency-delay=15s", "--min-time=-800d", "--log.level=warn", "--log.format=json" 

Note that --consistency-delay is set to only 15 seconds and --max-time is not configured. This is because I run the Thanos Receiver (ingestion) without persistent storage, and when all Receiver tasks are rotated (TSDBs are flushed to the S3 bucket), I want to be able to query recent metrics from the S3 bucket.

The Receiver (ingestion) tasks have been running for 2–3 days. Below is an example of the graph view of one of my metrics.

Then i proceed to redeploy the receiver (ingesting) service and all tasks are killed, result:

For about 3 minutes i lose the ability to query the last 2 hours of metrics, metrics that were once in the receiver (ingesting). Is this expected?

Eventually, I can see all my metrics again.

What you expected to happen:

I would expect to be able to query recent datapoints directly from the bucket immediately after they are uploaded.

How to reproduce it (as minimally and precisely as possible):

• Run a Thanos Receiver without persistent storage.
• Run Thanos Store with --consistency-delay=15s and without --max-time configured.
• Run Thanos Querier with both Thanos Receiver and Thanos Store endpoints configured.
• Kill all Thanos Receiver tasks.
• In the Querier graph view, observe that metric visibility is lost for a period of time.

Full logs to relevant components:

Anything else we need to know:
I also have a discussion open related to my current setup: #8650