How to manage multiple threads

Question

I have software that loads a list of up to 10,000 URLs which are used to scrape insurance prices for my website.

I have a single thread running at the moment per request which loads each URL from the list and fetches the data. What I want to do is run 20-30 requests per time. What's the best way to launch 20-30 threads at once whilst looping through the results from the textfile?

You could load the entire list at one go and hand off a chunk from it (say 50 URLs) to each new thread that you spawn, till you reach the max thread count (say 20 threads). Tweak the numbers as necessary. — Bhargav
– Bhargav, Commented Feb 28, 2012 at 14:34
I would probably scale back your ambitions, with that number of outbound request any website will rack up the bandwidth charges at an astronomical rate. — Lloyd
– Lloyd, Commented Feb 28, 2012 at 14:40

Matthias · Accepted Answer · 2012-02-28 14:31:28Z

3

Take a look at the Task Parallel Library and especially the Parallel.ForEach method.

answered Feb 28, 2012 at 14:31

Matthias

12.3k4 gold badges53 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Jonas Elfström · Accepted Answer · 2012-02-28 14:49:35Z

If you are on .NET 4 then you can take a look at TPL and something like the following.

const string path = @"c:\urls.txt"; string[] urls = File.ReadAllLines(path); var options = new ParallelOptions() { MaxDegreeOfParallelism = 20}; Parallel.ForEach(urls, options, url => { // Call your scraper here Debug.WriteLine(url); });

Collectives™ on Stack Overflow

How to manage multiple threads

2 Answers 2

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Linked

Related