Messages are not delivered with kafka python

Question

I'm trying to set up a simple Kafka application with kafka-python. I've been trying to get some of the examples I found online to work but can't seem to make it. I have a kafka instance running in a docker container. I tested the shell tools and the instance is definitely working. I am able to send and receive messages. I suspect that the producer messages time out. Here are two versions of the code with basically the same behaviour:

import time from kafka import SimpleProducer, KafkaClient # connect to Kafka kafka = KafkaClient('localhost:9092') producer = SimpleProducer(kafka) # Assign a topic topic = 'test' producer.send_messages(topic, b'this is a message')

And the second version:

from kafka import KafkaProducer from kafka.errors import KafkaError producer = KafkaProducer(bootstrap_servers=['0.0.0.0:9092'], api_version=(0,10)) topic = "test" producer.send(topic, b'test message')

What logs are you seeing after the produce fails?

William Hammond
– William Hammond

2017-07-09 17:07:37 +00:00
Commented Jul 9, 2017 at 17:07 — William Hammond
– William Hammond, Commented Jul 9, 2017 at 17:07

Stav Shamir · Accepted Answer · 2019-12-17 09:48:36Z

Change the line: producer.send(topic, b'test message')

To: producer.send(topic, b'test message').get(timeout=30) (or any value the you see fit)

The problem is that the producer is killed before the message is sent, since this method is asynchronous. You could see it for yourself if you add:

import logging logging.basicConfig(level=logging.INFO)

And see that the timeout is 0.

William Hammond · Accepted Answer · 2017-07-09 19:03:40Z

It depends on how you ran docker but I believe your issue is with the host name you're trying to connect to. You need to point to the host set in the ADVERTISED_HOST environment variable. For example when I run kafka-docker as docker run --hostname kafka-1 -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST='kafka-1' --env ADVERTISED_PORT=9092 spotify/kafka I produce to kafka like such

from kafka import SimpleProducer, KafkaClient kafka = KafkaClient('kafka-1:9092') producer = SimpleProducer(kafka) topic = 'test' for i in range(100): producer.send_messages(topic, 'hullo-' + str(i))

additionally I needed to add 127.0.0.1 kafka-1 to my /etc/hosts file. After doing this I was able to consume the messages produced by the python client with bin/kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic test --from-beginning

Collectives™ on Stack Overflow

Messages are not delivered with kafka python

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related