Spring Kafka Partitioning

Question

What is the difference in the behavior of the below two code snippets to publish a message?

Approach 1

Message<String> message = MessageBuilder.withPayload("testmsg") .setHeader(KafkaHeaders.MESSAGE_KEY, "key").setHeader(KafkaHeaders.TOPIC, "test").build(); ListenableFuture<SendResult<String, String>> future = kafkaTemplate.send(message);

Approach 2

ListenableFuture<SendResult<String, String>> future = kafkaTemplate.send("test", "testmsg");

Topic Config:

$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test Topic:test PartitionCount:3 ReplicationFactor:1 Configs: Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0 Topic: test Partition: 1 Leader: 0 Replicas: 0 Isr: 0 Topic: test Partition: 2 Leader: 0 Replicas: 0 Isr: 0

Observation:

If there are 3 consumers, one per partition; Approach 1 leads to all messages consumed by a single consumer from a single partition. With Approach 2; consumption is equally split between the 3 partitions/consumers.

Artem Bilan · Accepted Answer · 2017-08-07 21:59:40Z

But you have an answer in your code. The first one alongside with the topic provides messageKey.

The messageKey is really used to determine target partition if isn't specified explicitly:

/** * computes partition for given record. * if the record has partition returns the value otherwise * calls configured partitioner class to compute the partition. */ private int partition(ProducerRecord<K, V> record, byte[] serializedKey, byte[] serializedValue, Cluster cluster) { Integer partition = record.partition(); return partition != null ? partition : partitioner.partition( record.topic(), record.key(), serializedKey, record.value(), serializedValue, cluster); }

where DefaultPartitioner does this:

List<PartitionInfo> partitions = cluster.partitionsForTopic(topic); int numPartitions = partitions.size(); if (keyBytes == null) { int nextValue = nextValue(topic); ... } else { // hash the keyBytes to choose a partition return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions; }

So, all messages with the same key are sent to the same partition. Otherwise they are placed to the topic round-robin manner.

Collectives™ on Stack Overflow

Spring Kafka Partitioning

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related