Skip to main content
Software Engineering

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

10
  • Each platform provides ways of doing this eg: "Kafka consumers within the same group can process messages in parallel" are you asking how those ways work? This doesn't seem like a hard problem to solve as the server will know what messages it has sent. Can you explain what part you are struggling with? Commented Nov 10, 2024 at 17:58
  • I'll provide an example to help clarify my query. Imagine you have a Kubernetes deployment with 5 replicas. These 5 pods connect to a single RabbitMQ stream and pull messages one by one, each at their own pace. If the offset is saved to an external database where the application both saves and reads from, how does pod X know which messages pods Z, Y, etc. have already processed? Or, if you can point me to a documentation explain something similar to my query it would be great. Commented Nov 10, 2024 at 18:44
  • 1
    This is core functionality of all messages queues - as such it will be well documented how do this in the docs of whichever MQ you select (different MQ's may have slightly different approaches). Commented Nov 10, 2024 at 20:08
  • yeah, I haven't used rabbitMQ streams, but the manual says they can be used in the same way as queues, and you can def do this in kafka, you just use the same consumer id. If your question is just about how to configure software X to do this then its prob off topic Commented Nov 10, 2024 at 20:12
  • RabbitMQ streams were deliberately designed so that each consumer gets a copy, not unique message. So why do you want to use a tool for thing it was deliberately designed to not support? And why don't you use a tool (RabbitMQ queues) that were designed to support this out of the box? Commented Nov 10, 2024 at 20:22