Leader brokers without a matching listener error in kafka

Question

What is the meaning of this kafka error ?

[2018-08-22 11:40:49,429] WARN [Consumer clientId=consumer-1, groupId=console-consumer-62114] 1 partitions have leader brokers without a matching listener, including [topicname-0] (org.apache.kafka.clients.NetworkClient)

I'm getting it when running:

./kafka-console-consumer.sh --topic topicname --bootstrap-server localhost:9094

And I'm getting some errors inside a golang program when trying to read this topic:

2018/08/22 11:44:12 ReadOffsetWithRetryOnError conn error: < dial tcp :0: connect: connection refused > kafka0:9094 topic: 0

The code snippet:

conn, err := kafka.DialLeader(context.Background(), "tcp", ip, getTopic(topic), 0) if err != nil { log.Println("ReadOffsetWithRetryOnError conn error: <", err, "> ", ip, " topic:", topic) }

This is quite weird because, when reading on different topic it is working fine at the same time.

More error logs:

/kafka-topics.sh --describe --zookeeper localhost:2181 --topic topicname Topic:indexBlock PartitionCount:1
ReplicationFactor:1 Configs: Topic: topicname Partition: 0 Leader: -1 Replicas: 1002 Isr: 1002

I answered a similar question here: stackoverflow.com/a/53600352/1016425 check it out — lloiacono
– lloiacono, Commented Mar 20, 2019 at 12:23
Does this answer your question? Why could Kafka warn "partitions have leader brokers without a matching listener"? — Jacek Laskowski
– Jacek Laskowski, Commented Feb 23, 2020 at 13:32

halfer · Accepted Answer · 2020-04-16 13:13:18Z

[2018-08-22 11:40:49,429] WARN [Consumer clientId=consumer-1, groupId=console-consumer-62114] 1 partitions have leader brokers without a matching listener, including [topicname-0] (org.apache.kafka.clients.NetworkClient)

This error also happens if you try to run multiple consumers and the kafka topic contains only one partition. Generally one consumer should mapped with one partition. If you are using two consumers then you should have 2 partition in the kafka topic.

Small correction: if you try to run multiple consumers from the same consumer group.
I have 2 consumers and 3 partitions and error keep spamming the console.

jumping_monkey · Accepted Answer · 2020-12-28 17:17:26Z

In my case, i was getting this error when i was testing Kafka fail-over. I brought down 1 Kafka, and expected the message to be written to the other Kafka.

The issue was that topic replication-factor was set to 1, when i needed to set it to 2. (2 Kafka instances)

Bonus:
Check out the directories where the topics are created(in my case: kafka-logs-xx) for both Kafka, and you will understand why :-)

franck · Accepted Answer · 2018-08-23 05:18:53Z

I think my problem is I was running 2 instances, without setting anything special for replications. (prob no replication?) then I removed a broker. Some topics stopped working.

We were facing the same issue, started all the brokers and the issue was resolved.

Mukesh Chapagain · Accepted Answer · 2021-01-17 09:05:21Z

I had this issue while running Kafka in a docker container.

The following solution helped to resolve the issue.

As mentioned by @coldkreap in the comment of this answer: https://stackoverflow.com/a/58067985/327862

The kafka broker information is kept between restarts because the wurstmeister/kafka image creates a volume named 'kafka'. If you run docker volume ls you will see a kafka volume. Remove that volume and you will be able to recreate topics, etc.

If using docker-compose, you can run the following command to remove containers along with their associated volumes:

docker-compose down -v

OR

docker-compose rm -sv

Vezir · Accepted Answer · 2020-02-12 11:26:24Z

In my case, i was getting this error while updating my kafka cluster from v2.0 to v2.4. The reason was the configuration settings was wrong for log.dirs in server.properties file. Because I didnt noticed different disk name for different nodes and i missmatch the disknames in log.dirs settings.

stsmurf · Accepted Answer · 2020-07-27 17:59:02Z

jumping_monkey's bonus on checking the directories is helpful. For me, I was deploying with Bitnami Kafka. On first deploy I was not config in the helm values. I wanted to change retention down to minutes, and set that:

config: |- log.retention.minutes=10

This caused the log.dirs directory to switch from /bitnami/kafka/data to /tmp/logs.

Essentially where the data is being stored on the Kafka brokers caused the error to show up.

ntifi amine · Accepted Answer · 2020-04-14 20:36:22Z

I had this problem and i resolved it by creating a new broker and starting it. Afterwards you have to restart the different consumers, producers and so on.

Collectives™ on Stack Overflow

Leader brokers without a matching listener error in kafka

7 Answers 7

2 Comments

Comments

1 Comment

Comments

Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

2 Comments

Comments

1 Comment

Comments

Comments

Comments

Comments

Linked

Related