52

I'm trying to get Kafka to work on docker-compose for the first time. The application runs fine without docker. But on docker, I get the error as described below. Any reason why Kafka would throw this error?

The error:

email-service_1 | 2018-12-01 14:32:02.448 WARN 1 --- [ntainer#0-0-C-1] o.a.k.c.NetworkClient : [Consumer clientId=consumer-2, groupId=kafka] 1 partitions have leader brokers without a matching listener, including [email-token-0]

My docker-compose config:

version: '3.3' services: zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: image: wurstmeister/kafka command: [start-kafka.sh] environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_HOST_NAME: 192.168.23.134 KAFKA_CREATE_TOPICS: "email-token:1:1" volumes: - /var/run/docker.sock:/var/run/docker.sock ports: - "9092:9092" depends_on: - zookeeper email-service: build: ./email-service environment: SPRING_KAFKA_BOOTSTRAPSERVERS: kafka:9092 ports: - "8081:8081" depends_on: - kafka
Improve this question
2
  • 1
    I think your app is seeing the wrong advertised name for the broker. github.com/wurstmeister/kafka-docker/wiki/Connectivity Commented Dec 1, 2018 at 17:12
  • within the compose the KAFKA_ADVERTISED_HOST_NAME should match what you use to connect. To work inside the compose you can add in the kafka section: hostname: kafka and KAFKA_ADVERTISED_HOST_NAME: kafka. You can see this example with spring boot and kafka here Commented Dec 3, 2018 at 9:17

4 Answers 4

Reset to default
27

As stated in the comments to your question the problem seems to be with the advertised name for the Kafka broker. According to your docker-compose you should be using 192.168.23.134 but your email-service is using kafka:9092. You can try with this docker-compose. I replaced the wurstmeister services with the latest Zookeeper and Kafka provided by confluentinc and added your email-service. 

--- version: '2' services: zookeeper: image: confluentinc/cp-zookeeper:latest environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 kafka: image: confluentinc/cp-kafka:latest depends_on: - zookeeper ports: - 9092:9092 environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 email-service: build: ./email-service environment: SPRING_KAFKA_BOOTSTRAPSERVERS: kafka:29092 ports: - "8081:8081" depends_on: - kafka

advertised.listeners: Listeners to publish to ZooKeeper for clients to use, if different than the listeners config property. In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, the value for listeners will be used. Unlike listeners it is not valid to advertise the 0.0.0.0 meta-address.

Please note that KAFKA_ADVERTISED_HOST_NAME has been deprecated and it's recommended to use KAFKA_ADVERTISED_LISTENERS instead. For more information about KAFKA_ADVERTISED_LISTENERS check here.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

KAFKA_ADVERTISED_LISTENERS works great when connecting to the broker for the first time, do you have experience with connecting to the broker after it has restarted? From what I see, producers/consumers within the docker swarm can't reconnect to the broker as the dns is being cached somewhere as the container id and not as the advertised listener
above comment is solvable by setting a constant KAFKA_BROKER_ID
Any reason behind the use of port 29092?
16

This is Apache Kafka 2.4.0.

I'm sharing the low-level code-based findings to shed more light when this WARN message could be printed out and why. That's certainly a misconfiguration of a Kafka cluster. Read on and comment if there's something missing. Thanks!

The WARN message is printed out when the DefaultMetadataUpdater (of NetworkClient) is requested to handle a completed metadata response.

[count] partitions have leader brokers without a matching listener, including [partitions]

It is a warning that corresponds to Errors.LISTENER_NOT_FOUND that has the following default exception text:

There is no listener on the leader broker that matches the listener on which metadata request was processed.

That's on the client side.

Digging deeper you can find that this Errors.LISTENER_NOT_FOUND is used on a Kafka broker when MetadataCache is requested to find partition metadata. That's where you can find just before there's this DEBUG message:

Error while fetching metadata for [topicPartition]: listener [listenerName] not found on leader [leaderBrokerId]

Simply turn the DEBUG logging level for kafka.server.MetadataCache logger and you should see it in the controller broker's logs.

In this particular case, this MetadataCache is used by a broker (via KafkaApis) to handle TopicMetadata request where they say:

// In versions 5 and below, we returned LEADER_NOT_AVAILABLE if a matching listener was not found on the leader.

// From version 6 onwards, we return LISTENER_NOT_FOUND to enable diagnosis of configuration errors.

And at that moment, it's clear that the WARN message in question is for a connection on the listenerName.

In my case, when I was debugging the issue, it turned out that I used SSL://:9093 to connect to a Kafka broker while the partition leader was neither available nor configured to listen to the listeners configuration property.

I used kafka-topics to review the partition configuration and then reviewed the state of partitions in ZooKeeper.

get /brokers/topics/ssl/partitions/0/state {"controller_epoch":1,"leader":0,"version":1,"leader_epoch":0,"isr":[0]}

I had -1 for the leader, but the isr showed a broker that was simply misconfigured. That's why people reported they fixed the issue by restarting their clusters (to get all the brokers up and running) or fixing the broker ID to the one that worked previously.

Improve this answer

Comments

12

[context]

I am trying to run a docker compose with a kafka client using the registry of https://github.com/wurstmeister/kafka-docker I am trying to run a very simple kafka cluster with a single broker and 3 topics with each 1 partition and a replication factor of 1.

this great link explains connectivity for a kafka cluster with one broker, a kafka cluster with several brokers and also notions about the listeners, all using docker, please have a look : https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

[result]

The first time i run docker-compose up --force-recreate --build, everything runs just fine !
The topics are created automatically using KAFKA_CREATE_TOPICS and I can use kafka producer and consumer just fine.

list topics : bin/kafka-topics.sh --list --bootstrap-server localhost:9092
producer : bin/kafka-console-producer.sh --broker-list localhost:9092 --topic productadvisor_sales_dev
consumer : bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic productadvisor_sales_dev --from-beginning

After that, everytime I do docker-compose stop, and relaunch using docker-compose up --force-recreate --build and try to produce data I get the following error message ...

Error Message : [2019-09-23 19:41:33,037] WARN [Producer clientId=console-producer] 1 partitions have leader brokers without a matching listener, including [productadvisor_purchase_dev-0] (org.apache.kafka.clients.NetworkClient)

[Answer]

It appears you need to specify the value of KAFKA_BROKER_ID (=1 for instance) so that the zookeeper doesn't try to create a new broker which can't have a listener because it is binded to the old one.

[Code]

 services: zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: image: wurstmeister/kafka ports: - "9092:9092" environment: KAFKA_BROKER_ID: 1 KAFKA_HOST: localhost KAFKA_PORT: 9092 KAFKA_ADVERTISED_HOST_NAME: localhost KAFKA_ADVERTISED_PORT: 9092 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_CREATE_TOPICS: "productadvisor_sales_dev:1:1,productadvisor_stock_dev:1:1,productadvisor_purchase_dev:1:1" depends_on: - zookeeper command: [start-kafka.sh]

[Some documentation]

NB

if anyone has more information about the inner working of kafka, the zookeeper and the broker and why we need to specify it, why the information is kept even if I do a --force-recreate --build ... please do not hesitate. I am new to kafka and this is one of my first complete post on stackoverflow :)

Cheers !

Improve this answer

3 Comments

The kafka broker information is kept between restarts because the wurstmeister/kafka image creates a volume named 'kafka'. If you run docker volume ls you will see a kafka volume. Remove that volume and you will have to recreate topics, etc.
I have the same problem as you, even worse is that I can not specify the broker id of kafka, because I used docker swarm to deploy multiple kafka brokers based on global mode. Can you give me some advice now?
@coldkreap Thanks. Removing the container with its volume solved the issue for me: docker-compose down -v OR docker-compose rm -sv
1

In our case, one of broker in the cluster shutdown due to disk problem and we got this error in other replicas. When we solve the disk problem the problem was solved

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.