0

I have 3 kafka brokers, with 3 partitions :

broker.id 1001: 10.18.0.73:9092 LEADER

broker.id 1002: 10.18.0.73:9093

broker.id 1005: 10.18.0.73:9094

Zookeeper set with 127.0.0.1:2181

Launch with:

1001 -> .\kafka-server-start.bat ..\..\config\server.properties 1002 -> .\kafka-server-start.bat ..\..\config\server1.properties 1005 -> .\kafka-server-start.bat ..\..\config\server2.properties 

This is server.properties

broker.id=-1 listeners=PLAINTEXT://10.18.0.73:9092 advertised.listeners=PLAINTEXT://10.18.0.73:9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/tmp/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=10.18.0.73:2181 zookeeper.connection.timeout.ms=6000 group.initial.rebalance.delay.ms=0 advertised.port=9092 advertised.host.name=10.18.0.73 port=9092 

This is server1.properties

broker.id=-1 listeners=PLAINTEXT://10.18.0.73:9093 advertised.listeners=PLAINTEXT://10.18.0.73:9093 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/tmp/kafka-logs4 num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=localhost:2181 zookeeper.connection.timeout.ms=6000 group.initial.rebalance.delay.ms=0 advertised.port=9093 advertised.host.name=10.18.0.73 port=9093 

This is server2.properties

broker.id=-1 listeners=PLAINTEXT://10.18.0.73:9094 advertised.listeners=PLAINTEXT://10.18.0.73:9094 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/tmp/kafka-logs2 num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=localhost:2181 zookeeper.connection.timeout.ms=6000 group.initial.rebalance.delay.ms=0 advertised.port=9094 advertised.host.name=10.18.0.73 port=9094 

in folder C:\kafka_2.12-2.4.0\config

Run All

enter image description here

Run Producer

.\kafka-console-producer.bat --broker-list 10.18.0.73:9092,10.18.0.73:9093,10.18.0.73:9094 --topic clinicaleventmanager 

Run Consumer

.\kafka-console-consumer.bat --bootstrap-server 10.18.0.73:9092,10.18.0.73:9093,10.18.0.73:9094 --topic clinicaleventmanager 

I launch a test message

enter image description here

Receive ok!

Now, i shutdown broker 1001 (the leader)

enter image description here

The new leader is 1002

enter image description here

In the consumer this message appeared for 1 second, I imagine for the time necessary for the election of the new leader

[2020-01-16 15:33:35,802] WARN [Consumer clientId=consumer-console-consumer-56669-1, groupId=console-consumer-56669] Connection to node 2147482646 (/10.18.0.73:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) 

if I try to send another message, this is not read by the consume enter image description here

The new leader 1002 does not appear to be sending messages. Why?

If i run 1001 broker.id, works all. Thanks

1 Answer 1

1

First, Kafka never "sends (pushes) messages", the consumer asks for them.

Second, it would seem you've changed nothing but the listeners, port, and log dir.

You don't explicitly create any topic, so you would end up with the defaults of one partition and one replica. For your topic and the internal consumer offsets topic

If any replica is offline from the broker you stopped, then no other process can read (or write) to that replica, regardless of which broker is the controller.

So, change the offsets (and transactions) replication factor to 3 and try again

Sign up to request clarification or add additional context in comments.

1 Comment

thanks, but I know that kafka does not automatically send messages to the consumer. I have modified the properties offsets.topic.replication.factor = 2 default.replication.factor = 2 and it worked. Thank!

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.