Push Messages from AWS Lambda to Kafka

Question

I have a kafka machine running in AWS which consists of several topics. I have the following Lambda function which Produces a message and push that to one of the kafka topic.

import json from kafka import KafkaClient from kafka import SimpleProducer from kafka import KafkaProducer def lambda_handler(event, context): kafka = KafkaClient("XXXX.XXX.XX.XX:XXXX") print(kafka) producer = SimpleProducer(kafka, async = True) print(producer) task_op = { "'message": "Hai, Calling from AWS Lambda" } print(json.dumps(task_op)) producer.send_messages("topic_atx_ticket_update",json.dumps(task_op).encode('utf-8')) print(producer.send_messages) return ("Messages Sent to Kafka Topic")

But I see messages are not pushed as i expected.

Note: No Issues in Roles and Policies, Connectivity.

in lambda_handler.py file I am creating a kafka Client and using Kafka Producer API to produce messages. From my local laptop i was able to push the messages to the kafka topic — RajNikhil Marpu
– RajNikhil Marpu, Commented Jul 19, 2019 at 5:42
You say "no issues in… connectivity" -> can you detail how you have determined this? A common problem in this scenario is mis-configurtion of the Kafka listeners. — Robin Moffatt
– Robin Moffatt, Commented Jul 19, 2019 at 8:14
import json from kafka import KafkaClient from kafka import SimpleProducer from kafka import KafkaProducer def lambda_handler(event, context): kafka = KafkaClient("XXXX.XXX.XX.XX:XXXX") print(kafka) producer = SimpleProducer(kafka, async=True) print(producer) task_op= {"'message":"Hai, Calling from AWS Lambda"} print(json.dumps(task_op)) producer.send_messages("topic_atx_ticket_update", json.dumps(task_op).encode('utf-8')) print(producer.send_messages) return("Messages Sent to Kafka Topic") — RajNikhil Marpu
– RajNikhil Marpu, Commented Jul 22, 2019 at 13:22

RajNikhil Marpu · Accepted Answer · 2019-07-23 06:43:38Z

While Creating a Kafka Producer object,

producer = SimpleProducer(kafka, async=True)

"async" String should be False, like

producer = SimpleProducer(kafka, async=False)

Then,

you can send the Kafka Message to a topic from AWS Lambda.

Collectives™ on Stack Overflow

Push Messages from AWS Lambda to Kafka

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related