Limit the number of consumers for a queue who are running parallel

Question

I am using a queue, there are two threads. One is for Enqueue, the other is for Dequeue. Which are called as producer and consumer respectively. Produce can be unlimited. But I need to limit the consumers running at the same time. I read about “Task Parallel Library” and “Parallel.For”. But I’m not sure about the way that I should implement them here. Please advise me. Here are some of code segments for you to having a better understanding of the question

static void Main(string[] args) { // The Producer code comes here // ... // The Consumer code comes here Thread consumer = new Thread(new ThreadStart(PendingBookingConsumer)); consumer.Start(); } private static void PendingBookingConsumer() { try { while (true) { if (pendingBookingsQueue != null && pendingBookingsQueue.Count > 0) { PendingBooking oPendingBooking = pendingBookingsQueue.Dequeue(); //Run the Console App string command = @"C:\ServerAgentConsole.exe"; string args = oPendingBooking.Id + " " + oPendingBooking.ServiceAccountEmail.Trim() + " " + oPendingBooking.ServiceAccountPassword.Trim() + " " + oPendingBooking.ServiceAccountEmail.Trim() + " " + oPendingBooking.MailBoxOwnerEmail.Trim() + " " + oPendingBooking.Method.Trim(); Process process = new Process(); process.StartInfo.FileName = command; process.StartInfo.Arguments = args; process.EnableRaisingEvents = true; process.Exited += (sender, e) => { Process myProcess = (Process)sender; Console.WriteLine("Agent for booking ID :" + myProcess.StartInfo.Arguments[0] + " Done"); }; process.Start(); Thread.Sleep(2); } } } catch (Exception ex) { Console.WriteLine(ex.Message); } }

Community · Accepted Answer · 2017-05-23 12:30:08Z

Use one of the common techniques to process a BlockingCollection with a fixed degree of parallelism. Specify the DOP in the Parallel.ForEach options.

Then, make the processing function wait for the child process:

process.Start(); process.WaitForExit();

That way you have a fixed number of child processes outstanding at any time.

VMAtm · Accepted Answer · 2015-05-25 16:10:22Z

You can also consider the TPL Dataflow library, with easy implementation for Producer/Consumer pattern:

private static BufferBlock<int> m_buffer = new BufferBlock<int>>( new DataflowBlockOptions { BoundedCapacity = 10, MaxDegreeOfParallelism = 4 }); // Producer private static async void Producer() { while(true) { await m_buffer.SendAsync(Produce()); } } // Consumer private static async Task Consumer() { while(true) { Process(await m_buffer.ReceiveAsync()); } }

You can see the BoundedCapacity used for the throttling technique, limiting the queue size, and MaxDegreeOfParallelism is used for the limiting the parallel consuming tasks.

You can download introduction to TPL Dataflow from MSDN here.

PS: How to: Implement a Producer-Consumer Dataflow Pattern on MSDN

paparazzo · Accepted Answer · 2015-05-25 16:56:57Z

0

BlockingCollection supports an upper bound

BlockingCollection Constructor (Int32)

As Hans commented this is just size of collection.
Maybe you can do this with parallel in the consumer.

edited May 25, 2015 at 16:56

answered May 25, 2015 at 10:41

paparazzo

45.2k24 gold badges110 silver badges180 bronze badges

2 Comments

Hans Passant Over a year ago

That's for the bounded size of the collection, it has nothing to do with the number of consumer threads.

paparazzo Over a year ago

@HansPassant OK. Will delete when it gets an accepted answer.

Collectives™ on Stack Overflow

Limit the number of consumers for a queue who are running parallel

3 Answers 3

Comments

Comments

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

2 Comments

Linked

Related