6

I'm running kafka locally via:

docker-compose.yml

 zookeeper: image: 'bitnami/zookeeper:latest' ports: - 2181:2181 environment: - ALLOW_ANONYMOUS_LOGIN=yes kafka: image: 'bitnami/kafka:latest' ports: - 9092:9092 environment: - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 - ALLOW_PLAINTEXT_LISTENER=yes - KAFKA_ADVERTISED_PORT=9092 - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092

My Spring Boot application is run with:

application.yml:

spring: application: name: testkafka kafka: bootstrap-servers: localhost:9092 server: port: 8080

When I run it and it tries to send to a topic on kafka, I get:

org.springframework.kafka.KafkaException: Reply timed out at org.springframework.kafka.requestreply.ReplyingKafkaTemplate.lambda$sendAndReceive$0(ReplyingKafkaTemplate.java:196) ~[spring-kafka-2.1.10.RELEASE.jar:2.1.10.RELEASE] at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.0.10.RELEASE.jar:5.0.10.RELEASE] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na] at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[na:na] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:na] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

If I run the spring boot from inside a docker container (using one docker compose file), then it does work:

Running Both in the Same Compose:

version: "3.0" services: service1: build: ./Service ports: - 8080:8080 - 5005:5005 links: - zookeeper - kafka zookeeper: image: 'bitnami/zookeeper:latest' ports: - 2181:2181 environment: - ALLOW_ANONYMOUS_LOGIN=yes kafka: image: 'bitnami/kafka:latest' ports: - 9092:9092 environment: - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 - ALLOW_PLAINTEXT_LISTENER=yes - KAFKA_ADVERTISED_PORT=9092 - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092

How can I get the kafka container to allow connections from outside itself/docker?

EDIT: Tried these changes:

kafka: image: 'bitnami/kafka:latest' ports: - 9092:9092 environment: - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 - ALLOW_PLAINTEXT_LISTENER=yes - KAFKA_ADVERTISED_PORT=9092 - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092

and:

spring: application: name: testkafka kafka: bootstrap-servers: kafka:9092 server: port: 8080

This still times out

Improve this question
1

1 Answer 1

Reset to default
10

If I run the spring boot from inside a docker container (using one docker compose file), then it does work

It shouldn't work, actually. Kafka isn't running as part of the application, so this section is not pointing at the Kafka container.

kafka: bootstrap-servers: localhost:9092

It would need to be kafka:9092 within the Docker network.

and ... bootstrap-servers: kafka:9092 ... still times out

That's because you still have KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092

Both inside and outside of Docker network, by specifying KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092, you're saying that your clients receive the Kafka bootstrap connection as localhost:9092, which would work outside of the Docker network only because you have made both a port forwarding and your container is running on localhost, however, inside the Docker network, as mentioned, localhost would mean that application container, not the broker.

The solution would be to create two port mappings via that property, as discussed in length by this blog post

Plus, Confluent provides a fully configured Docker Compose with appropriate mappings that'll work inside and out of Docker

 ports: - "9092:9092" - "29092:29092" environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181' ALLOW_PLAINTEXT_LISTENER: "yes" KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_LISTENERS: PLAINTEXT://:9092,PLAINTEXT_HOST://0.0.0.0:29092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092

For applications within Docker network, use kafka:9092, for applications outside, use localhost:29092

EDIT The bitnami config variables have changed. Refer this section of the README

https://github.com/bitnami/bitnami-docker-kafka/blob/master/README.md#accessing-kafka-with-internal-and-external-clients

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

in addition to Cricket answer you can then add kafka in your host file and connect to it from out side the docker-compose, given KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092, bootstrap-servers: kafka:9092 and ports - 9092:9092
@Paizo Editing the hosts file is a hack, and should be avoided. Correcting the listeners is the correct thing.
@DonRhummy See updated answer. The gist here is that you can't have the same spring config working inside and outside of Docker
@cricket_007 I get the error java.lang.IllegalArgumentException: requirement failed: advertised.listeners listener names must be equal to or a subset of the ones defined in listeners. Found PLAINTEXT,PLAINTEXT_HOST. The valid options based on the current configuration are PLAINTEXT
@DonRhummy Have you defined KAFKA_LISTENER_SECURITY_PROTOCOL_MAP, as mentioned in my answer?
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.