I noticed in the default configuration (maybe I have it configured wrong) if I have one app consisting of two consumers - and one producer produces 5,000 messages that both consumers receive all 5,000 messages, this seems like a big waste of resources, ideally each should receive 2,500.
It's been a while since I've worked with RabbitMQ so things may have changed, but this wasn't the default behavior. The current documentation seems to indicate that nothing has changed and, if messages are being published to one queue, RabbitMQ will use round-robin dispatching to send items from the work queue to each worker without duplication.
If multiple consumers are receiving the messages, perhaps the publish/subscribe configuration is being used.
My production environment has many apps which would be running multiple consumers of their own per app, is it desired in a streaming platform that all the consumers receive all the messages?
I can think of some cases where it would be desired that multiple (and perhaps all) consumers receive some or all of the messages. Consumers may have overlapping interests. This is a property of the system and not the streaming platform. I would generally want the streaming platform to support both message queuing and publish/subscribe.
Is it my responsibility to track each message received and determine if it had already been processed? Am I forced to reprocess payloads multiple times? or is there typically a way in other streaming platforms like kafka or amazon kinesis to fan messages out based on a consumer clients name?
This depends on the system, but I'd generally say that this should be a responsibility of the messaging or streaming platform. The publisher should communicate with a message broker to ensure reliable deliver. The broker should make messages available to consumers or deliver messages to consumers, depending on if it is supporting a pull or push model. There may be a need for the broker to handle receiving and deconflicting messages from multiple redundant publishers. The broker should also support the required delivery times (once, at least once). If the broker doesn't support any of these directly, then it may be necessary to implement them in the publisher or the consumers. Between message queuing and fanout exchanging and publish/subscribe functionality, many of these requirements can be met. Publish confirmation and support for clusters may handle others.