6

I am using Kafka 0.8.2-beta and have 2 Ubuntu 14 virtual machines:

  1. 172.30.141.127 is running Zookeeper
  2. 172.30.141.184 is running a Kafka broker

I'm starting the Zookeeper instance and all if fine. Then I try to start the broker and connect it to 172.30.141.127:2181. It seems to be able to connect and establish a session on a specific port, but then it losses connection due to some exception that doesn't seem to be logged.

The broker output:

[2015-01-19 11:03:55,029] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper) [2015-01-19 11:03:55,030] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [2015-01-19 11:03:55,031] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper) [2015-01-19 11:03:55,031] INFO Client environment:os.arch=i386 (org.apache.zookeeper.ZooKeeper) [2015-01-19 11:03:55,032] INFO Client environment:os.version=3.16.0-23-generic (org.apache.zookeeper.ZooKeeper) [2015-01-19 11:03:55,033] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper) [2015-01-19 11:03:55,033] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper) [2015-01-19 11:03:55,037] INFO Client environment:user.dir=/home/osboxes/Desktop/kafka_2.11-0.8.2-beta (org.apache.zookeeper.ZooKeeper) [2015-01-19 11:03:55,039] INFO Initiating client connection, connectString=172.30.141.127:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@1ecf473 (org.apache.zookeeper.ZooKeeper) [2015-01-19 11:03:55,129] INFO Opening socket connection to server 172.30.141.127/172.30.141.127:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2015-01-19 11:03:55,186] INFO Socket connection established to 172.30.141.127/172.30.141.127:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2015-01-19 11:03:55,203] WARN Session 0x0 for server 172.30.141.127/172.30.141.127:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) [2015-01-19 11:03:56,552] INFO Opening socket connection to server 172.30.141.127/172.30.141.127:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2015-01-19 11:03:56,555] INFO Socket connection established to 172.30.141.127/172.30.141.127:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2015-01-19 11:03:56,567] WARN Session 0x0 for server 172.30.141.127/172.30.141.127:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) [2015-01-19 11:03:57,131] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread) [2015-01-19 11:03:58,075] INFO Opening socket connection to server 172.30.141.127/172.30.141.127:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2015-01-19 11:03:58,077] INFO Socket connection established to 172.30.141.127/172.30.141.127:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2015-01-19 11:03:58,195] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper) [2015-01-19 11:03:58,196] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn) [2015-01-19 11:03:58,251] FATAL [Kafka Server 1], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 2000 at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880) at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98) at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84) at kafka.server.KafkaServer.initZk(KafkaServer.scala:157) at kafka.server.KafkaServer.startup(KafkaServer.scala:83) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:28) at kafka.Kafka$.main(Kafka.scala:46) at kafka.Kafka.main(Kafka.scala) [2015-01-19 11:03:58,279] INFO [Kafka Server 1], shutting down (kafka.server.KafkaServer) [2015-01-19 11:03:58,295] INFO [Kafka Server 1], shut down completed (kafka.server.KafkaServer) [2015-01-19 11:03:58,308] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable) org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 2000 at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880) at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98) at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84) at kafka.server.KafkaServer.initZk(KafkaServer.scala:157) at kafka.server.KafkaServer.startup(KafkaServer.scala:83) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:28) at kafka.Kafka$.main(Kafka.scala:46) at kafka.Kafka.main(Kafka.scala) [2015-01-19 11:03:58,335] INFO [Kafka Server 1], shutting down (kafka.server.KafkaServer)

Zookeper outputs:

[2015-01-19 11:03:55,245] INFO Accepted socket connection from /172.30.141.184:54089 (org.apache.zookeeper.server.NIOServerCnxnFactory) [2015-01-19 11:03:55,315] WARN Exception causing close of session 0x0 due to java.io.IOException: Connection reset by peer (org.apache.zookeeper.server.NIOServerCnxn) [2015-01-19 11:03:55,329] INFO Closed socket connection for client /172.30.141.184:54089 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn) [2015-01-19 11:03:56,613] INFO Accepted socket connection from /172.30.141.184:54090 (org.apache.zookeeper.server.NIOServerCnxnFactory) [2015-01-19 11:03:56,615] WARN Exception causing close of session 0x0 due to java.io.IOException: Connection reset by peer (org.apache.zookeeper.server.NIOServerCnxn) [2015-01-19 11:03:56,617] INFO Closed socket connection for client /172.30.141.184:54090 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn) [2015-01-19 11:03:58,133] INFO Accepted socket connection from /172.30.141.184:54091 (org.apache.zookeeper.server.NIOServerCnxnFactory) [2015-01-19 11:03:58,134] WARN Exception causing close of session 0x0 due to java.io.IOException: Connection reset by peer (org.apache.zookeeper.server.NIOServerCnxn) [2015-01-19 11:03:58,135] INFO Closed socket connection for client /172.30.141.184:54091 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)

Ping works between the 2 machines. Telnet to 2181 kind of works, in that it connects, but gets disconnected from time to time. This leads me to think that the problem is with the Zookeeper instance. Both processes are started as root.

Any ideas why this is happening? Thanks

Improve this question

3 Answers 3

Reset to default
3

You are probably hitting the max number of connections per host.

This happens if you have [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from /127.0.0.1 - max is 10 in your zookeeper logs.

Fix it by setting maxClientCnxns=[something more than 10; 0 for unlimited] in your conf/zoo.cfg.

Docs (search for maxClientCnxns)

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

No idea about the warning, but I had the same Error problem

org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 2000 at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880) ....

In my case when I was setting up producer and consumer, I provided wrong ip/port. When I changed it to correct one with:

bin/kafka-console-producer.sh --broker-list kafka_ip:kafka_port --topic test bin/kafka-console-consumer.sh --zookeeper zookeeper_id:zookeper_port --topic test --from-beginning

my problem was solved.

Improve this answer

Comments

0

As answered by @RickyA "You are probably hitting the max number of connections per host." Try clearing your logs folder which in my case was F:\tmp\kafka-logs.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.