I have one kafka producer and consumer.The kafka producer is publishing to one topic and the data is taken and some processing is done. The kafka consumer is reading from another topic about whether the processing of data from topic 1 was successful or not ie topic 2 has success or failure messages.Now Iam starting my consumer and then publishing the data to topic 1 .I want to make the producer and consumer synchronous ie once the producer publishes the data the consumer should read the success or failure message for that data and then the producer should proceed with the next set of data .
2 Answers
Apache Kafka and Publish/Subscribe messaging in general seeks to de-couple producers and consumers through the use of streaming async events. What you are describing is more like a batch job or a synchronous Remote Procedure Call (RPC) where the Producer and Consumer are explicitly coupled together. The standard Apache Kafka Producers/Consumer APIs do not support this Message Exchange Pattern but you can always write your own simple wrapper on top of the Kafka API's that uses Correlation IDs, Consumption ACKs, and Request/Response messages to make your own interface that behaves as you wish.
5 Comments
codehacker
Please can you give an code example on how to make it synchronous RPC
codehacker
Can I make the Kafka listener sleep or wait until the data arrives ?
Hans Jespersen
It sort of does that already. When you call poll() in your listener you give it a timeout in milliseconds for how long to wait for data. If you poll(60000) it will wait for up to 1 minute for data. If data arrives early, it will return the data early. If no data arrive during the minute it will wait the full 60 seconds and return null. Then you poll again. It's like HTTP long polling in a web app if you've ever done that before.
Novemberland
@HansJespersen I would like to ask if there a way to achieve synchronous communication in spark-streaming without Kafka? For example is there a way to use Http requests/responses?
Hans Jespersen
I can’t answer “without kafka”. I only know how to do this with Kafka Streams, not Spark Streaming
Short Answer : You can't do that, Kafka doesn't provide that support.
Long Answer: As Hans explained, Publish/Subscribe messaging model keeps Publish and subscribe completely unaware of each other and I believe that is where the power of this model lies. Producer can produce without worrying about if there is any consumer and consumer can consume without worrying about how many producers are there.
The closest you can do is, you can make your producer synchronous. Which means you can wait till your message is received and acknowledged by broker.
if you want to do that, flush after every send.
