I have 5 node Kafka cluster with Zookeeper running on 3 nodes. My consumer is not draining the messages from topic. My consumer app throws the below error
2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] TS=2018-05-23 02:00:56,thread=[kafka-coordinator-heartbeat-thread | dcm-onhands],level=ERROR,appName=${cf:application_name},space=${cf:space_name},class=AbstractCoordinator$HeartbeatThread:51,msg=Uncaught exception in thread 'kafka-coordinator-heartbeat-thread | dcm-onhands': 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:528) ~[kafka-clients-1.0.0.jar!/:?] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] java.lang.OutOfMemoryError: Direct buffer memory 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at java.nio.Bits.reserveMemory(Bits.java:694) ~[?:1.8.0_152] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[?:1.8.0_152] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_152] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241) ~[?:1.8.0_152] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at sun.nio.ch.IOUtil.read(IOUtil.java:195) ~[?:1.8.0_152] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_152] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:109) ~[kafka-clients-1.0.0.jar!/:?] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:145) ~[kafka-clients-1.0.0.jar!/:?] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93) ~[kafka-clients-1.0.0.jar!/:?] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:231) ~[kafka-clients-1.0.0.jar!/:?] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:192) ~[kafka-clients-1.0.0.jar!/:?] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:469) ~[kafka-clients-1.0.0.jar!/:?] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.Selector.poll(Selector.java:398) ~[kafka-clients-1.0.0.jar!/:?] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) ~[kafka-clients-1.0.0.jar!/:?] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:238) ~[kafka-clients-1.0.0.jar!/:?] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:275) ~[kafka-clients-1.0.0.jar!/:?] 2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:934) ~[kafka-clients-1.0.0.jar!/:?] My consumer configs are given below
maxpollrecordsconfig = 25000 sessiontimeoutms = 30000
Server Config KAFKA_HEAP_OPTS="-Xmx2G -Xms2G"
I am facing this issue with other consumers also
