Questions tagged [producer-consumer]
The producer-consumer tag has no summary.
27 questions
0 votes
1 answer
63 views
Which techniques to follow to make a fast and consistent message consumer that writes to a RDBMS [closed]
I am building a consumer application that reads messages from a queue and runs basic sanitization and deduplication and persists them to an RDBMS. The functional requirements are messages must not be ...
- 131
1 vote
1 answer
584 views
What kind of queue should I use for processing large volume of data?
I have a segment of customers which may range between 1 and 5000. For each customers, I need to do some processing by making some query to the database using the data from the customer. The query is ...
- 121
4 votes
1 answer
2k views
One producer - multiple consumers queue, best way to guard against rare duplicates?
I have a classical one producer - multiple consumers queue, each item can be processed independently. On rare occasions (<1%) there can be multiple queue items related to the same object the ...
0 votes
0 answers
57 views
Architectural design for sending large amount of analytics data from production servers to s3 without impacting request performance
Lets say we have a server getting upto 1000 requests per second, serving them at p99 of 20ms (strong business case for not increasing this latency). The server gc parameters have been carefully tuned ...
- 109
3 votes
1 answer
4k views
What is the currently best C# implementation for a Producer/Consumer scenario?
I have a relatively simple task where I need some 10 consumers to consume work to be produced into a queue, continuously. This is my first time implementing this design pattern, so I have been ...
- 541
1 vote
1 answer
326 views
Implemeting a distributed n-Producers/1-Consumer service for critical-mission system
I am trying to implement a distributed version of multi-producer/1-consumer for a critical-mission system. I'm looking for good alternatives to the current approach based on RDBMS. The problem The ...
- 113
1 vote
1 answer
62 views
optimal way of dealing with subscriptions
I have a system in which some data is inserted into database. Whenever that data is inserted subscribers should be notified about that (it doesn't have to happen just after inserting, but there is a ...
- 111
1 vote
1 answer
869 views
Does the producer-consumer problem appear in both shared memory and distributed memory architectures?
I saw some books on operating system concepts mention the producer-consumer problem in the context of synchronizing concurrent accesses to shared resources. All seem to be in shared memory ...
- 5,555
5 votes
2 answers
1k views
What are the distinction and relation between batch processing and stream processing systems?
Design Data Intensive Applications says Batch processing systems (offline systems) Chapter 10 A batch processing system takes a large amount of input data, runs a job to pro‐ cess it, and ...
- 5,555
5 votes
2 answers
3k views
kafka consumer properties session.timeout.ms vs heartbeat.interval.ms
I am reading Kafka, the definitive guide and I came across the below point for consumers. heartbeat.interval.ms must be lower than session.timeout.ms, and is usually set to one-third of the timeout ...
- 59
1 vote
0 answers
30 views
Consumer triggered entity data agregation
I am currently using RabbitMQ to aggregate the output of various data sources. Functionally the process is the following: The user enters a product name. The request is sent to the different sources ...
- 119
0 votes
0 answers
104 views
How to send messages in the order they were queued, while ensuring that client B does not have to wait until client A has received his message?
I have a simplified producer/consumer pattern implemented below. The code outputs: "A" 1 second delay "B" 1 second delay "A" 1 second delay "B" ... What approach ...
- 49
1 vote
2 answers
1k views
Best design for a AMQP work queue
I have a system where a user uploads a file to import some users but it's necessary to do some validations and this take some time. I would like to create a work queue with RabbitMQ and the user will ...
- 11
2 votes
3 answers
4k views
How to build rest calls dynamically to interface with different APIs?
Lets say I am building an application where clients can book bus tickets. The only thing they do is select, origin, destination and the type of service (express or standard). The system will then book ...
- 327
1 vote
1 answer
389 views
Is my approach at building middleware using a queue a good approach?
I want to build a middleware system that my company can use and expand on to handle the integration points between our various bespoke and 3rd party systems as well as our hosted websites. I am ...
- 379