1

A general question. Assume a topic has 3 kafka partitions on different servers (brokers), each partition has 10 message with offset as its timestamp (0,1,...,9, greater number means stayed shorter time in partition, also means is newly came message). Let's say one partition happen to shut down since the server is done. What's the strategy for Kafka to re-balance the 10 message in the shut down partition into other partitions?

Visually, we have

broker 1 partition: |1-0|1-1|1-2|1-3|1-4|1-5|1-6|1-7|1-8|1-9|

broker 2 partition: |2-0|2-1|2-2|2-3|2-4|2-5|2-6|2-7|2-8|2-9|

broker 3 partition: |3-0|3-1|3-2|3-3|3-4|3-5|3-6|3-7|3-8|3-9|

Now if broker 3 is done, how will 3-0 to 3-9 be inserted into broker 1 and broker 2?

( My assumption is by default it will be spread half half randomly and inserted based on timestamp of broker 3, attached to tail of broker 1 and 2, and maybe there is somewhere one can configure behavior by code?)

Thanks in advance.

Improve this question
2
  • 1
    Kafka does not rebalance messages onto different partitions but rather partitions onto brokers. Commented May 8, 2019 at 18:16
  • Thanks for reminding. I will make my question more clear. Commented May 8, 2019 at 18:36

2 Answers 2

Reset to default
5

If a partition only exists on a single broker (replication factor 1) then when this broker is offline, the partition is not available. This is what you drew in your question.

To keep data available even when brokers go down you have to create topics with a replication factor greater than 1.

Then the data of the partition will be replicated onto several brokers and if one of them go offline, user traffic will be rediected to the available replicas.

I suggest you to go through the Replication section in the docs to understand how this works.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

2

The below diagram will help you understand how Kafka replicates partitions. If one broker is down, the consumer can read from the other broker because Kafka has a replication ability. (Of course, you need to set it like below) For example, if broker 1 dies, broker 2 will become a leader of topic1-part1, and a consumer can read from it.

enter image description here

Zookeeper will know if a broker( partition) is down, it will appoint another leader.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.