Skip to content

Conversation

@arteam
Copy link
Contributor

@arteam arteam commented Oct 7, 2024

Prior to 7.8, whenever a cluster had only a single data node, the watermarks would not be respected. This was incompatible with how storage based autoscaling works and in order to change this in 7.8+ in a backwards compatible way, we had to introduce the cluster.routing.allocation.disk.watermark.enable_for_single_data_node node setting.

The setting was deprecated in #73733 (7.14), and was made to accept only true in #73737 (8.0).

@arteam arteam added >breaking WIP :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) labels Oct 7, 2024
@DaveCTurner
Copy link
Contributor

We haven't yet decided that we want to do this in v9, let's hold off here until that's settled.

…ata_node` setting Prior to 7.8, whenever a cluster had only a single data node, the watermarks would not be respected. This was incompatible with how storage based autoscaling on ESS/ECH works and in order to change this in 7.8+ in a backwards compatible way, we had to introduce the `cluster.routing.allocation.disk.watermark.enable_for_single_data_node node` setting. The setting was deprecated in elastic#73733 (7.14), and was made to accept only `true` in elastic#73737 (8.0).
@arteam arteam force-pushed the remove-watermark-enable-for-single-data-node branch from 50f6cae to ea54648 Compare October 7, 2024 10:19
@arteam
Copy link
Contributor Author

arteam commented Oct 7, 2024

We haven't yet decided that we want to do this in v9, let's hold off here until that's settled.

@DaveCTurner Yes, I'm aware that the breaking change hasn't been approved yet, that's why I created this PR as a draft and added a WIP label.

Copy link
Contributor

@henningandersen henningandersen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, though I think we should add a breaking change changelog entry to this PR. I thought that was half-automatic, but perhaps not from draft?

@arteam arteam removed the WIP label Nov 6, 2024
@arteam arteam marked this pull request as ready for review November 6, 2024 09:55
@elasticsearchmachine elasticsearchmachine added the needs:triage Requires assignment of a team area label label Nov 6, 2024
@elasticsearchmachine
Copy link
Collaborator

Hi @arteam, I've created a changelog YAML for you. Note that since this PR is labelled >breaking, you need to update the changelog YAML to fill out the extended information sections.

@arteam
Copy link
Contributor Author

arteam commented Nov 6, 2024

@elasticmachine update branch

@arteam arteam added Team:Distributed Coordination Meta label for Distributed Coordination team and removed needs:triage Requires assignment of a team area label labels Nov 6, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

@arteam
Copy link
Contributor Author

arteam commented Nov 6, 2024

Hi @henningandersen! I've made the PR ready to review and added the breaking changelog entry.

Copy link
Contributor

@henningandersen henningandersen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (with one breaking change documentation change).

arteam and others added 2 commits November 6, 2024 15:07
Co-authored-by: Henning Andersen <33268011+henningandersen@users.noreply.github.com>
Co-authored-by: Henning Andersen <33268011+henningandersen@users.noreply.github.com>
@arteam
Copy link
Contributor Author

arteam commented Nov 6, 2024

@elasticmachine update branch

@arteam arteam requested a review from a team as a code owner November 6, 2024 16:49
@arteam arteam removed the request for review from a team November 6, 2024 16:49
@arteam arteam force-pushed the remove-watermark-enable-for-single-data-node branch from afed3ad to 7dab95c Compare November 6, 2024 16:49
@arteam arteam added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Nov 6, 2024
@elasticsearchmachine elasticsearchmachine merged commit e09d406 into elastic:main Nov 6, 2024
16 checks passed
@arteam arteam deleted the remove-watermark-enable-for-single-data-node branch November 6, 2024 17:57
jozala pushed a commit that referenced this pull request Nov 13, 2024
…ata_node` setting (#114207) Prior to 7.8, whenever a cluster had only a single data node, the watermarks would not be respected. This was incompatible with how storage based autoscaling works and in order to change this in 7.8+ in a backwards compatible way, we had to introduce the `cluster.routing.allocation.disk.watermark.enable_for_single_data_node node` setting. The setting was deprecated in #73733 (7.14), and was made to accept only `true` in #73737 (8.0).
alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this pull request Nov 28, 2024
…ata_node` setting (elastic#114207) Prior to 7.8, whenever a cluster had only a single data node, the watermarks would not be respected. This was incompatible with how storage based autoscaling works and in order to change this in 7.8+ in a backwards compatible way, we had to introduce the `cluster.routing.allocation.disk.watermark.enable_for_single_data_node node` setting. The setting was deprecated in elastic#73733 (7.14), and was made to accept only `true` in elastic#73737 (8.0).
@leemthompo
Copy link
Contributor

@arteam is this PR relevant to the serverless changelog? [FYI this question is based on 9.0 breaking changes]

@arteam
Copy link
Contributor Author

arteam commented Apr 11, 2025

@leemthompo No, this change doesn't have any effect on ES Serverless

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) >breaking :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) Team:Distributed Coordination Meta label for Distributed Coordination team v9.0.0

6 participants