I spent a whole day looking for the simplest possible multithreaded URL fetcher in Python, but most scripts I found are using queues or multiprocessing or complex libraries.
Finally I wrote one myself, which I am reporting as an answer. Please feel free to suggest any improvement.
I guess other people might have been looking for something similar.
Simplifying your original version as far as possible:
import threading import urllib2 import time start = time.time() urls = ["http://www.google.com", "http://www.apple.com", "http://www.microsoft.com", "http://www.amazon.com", "http://www.facebook.com"] def fetch_url(url): urlHandler = urllib2.urlopen(url) html = urlHandler.read() print "'%s\' fetched in %ss" % (url, (time.time() - start)) threads = [threading.Thread(target=fetch_url, args=(url,)) for url in urls] for thread in threads: thread.start() for thread in threads: thread.join() print "Elapsed Time: %s" % (time.time() - start) The only new tricks here are:
join already tells you that.Thread subclass, just a target function.
for loop?append, like this. Or, alternatively, write a wrapper which creates, starts, and returns a thread, like this. Either way, I think it's less simple (although the second one is a useful way to refactor complicated cases, it doesn't work when things are already simple).thread.start() returned the thread, you could put the creation and start together into a single expression. In C++ or JavaScript, you'd probably do that. The problem is that, while method chaining and other "fluent programming" techniques make things more concise, they can also breaks down the expression/statement boundary, and are often ambiguous. so Python goes in almost the exact opposite direction, and almost no methods or operators return the object they operate on. See en.wikipedia.org/wiki/Fluent_interface.multiprocessing has a thread pool that doesn't start other processes:
#!/usr/bin/env python from multiprocessing.pool import ThreadPool from time import time as timer from urllib2 import urlopen urls = ["http://www.google.com", "http://www.apple.com", "http://www.microsoft.com", "http://www.amazon.com", "http://www.facebook.com"] def fetch_url(url): try: response = urlopen(url) return url, response.read(), None except Exception as e: return url, None, e start = timer() results = ThreadPool(20).imap_unordered(fetch_url, urls) for url, html, error in results: if error is None: print("%r fetched in %ss" % (url, timer() - start)) else: print("error fetching %r: %s" % (url, error)) print("Elapsed Time: %s" % (timer() - start,)) The advantages compared to Thread-based solution:
ThreadPool allows to limit the maximum number of concurrent connections (20 in the code example)from urllib.request import urlopen on Python 3).
imap_unordered() returns the result as soon as it is ready. I assume the overhead is negligible compared to the time it takes to make the http request.
The main example in the concurrent.futures does everything you want, a lot more simply. Plus, it can handle huge numbers of URLs by only doing 5 at a time, and it handles errors much more nicely.
Of course this module is only built in with Python 3.2 or later… but if you're using 2.5-3.1, you can just install the backport, futures, off PyPI. All you need to change from the example code is to search-and-replace concurrent.futures with futures, and, for 2.x, urllib.request with urllib2.
Here's the sample backported to 2.x, modified to use your URL list and to add the times:
import concurrent.futures import urllib2 import time start = time.time() urls = ["http://www.google.com", "http://www.apple.com", "http://www.microsoft.com", "http://www.amazon.com", "http://www.facebook.com"] # Retrieve a single page and report the url and contents def load_url(url, timeout): conn = urllib2.urlopen(url, timeout=timeout) return conn.readall() # We can use a with statement to ensure threads are cleaned up promptly with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: # Start the load operations and mark each future with its URL future_to_url = {executor.submit(load_url, url, 60): url for url in urls} for future in concurrent.futures.as_completed(future_to_url): url = future_to_url[future] try: data = future.result() except Exception as exc: print '%r generated an exception: %s' % (url, exc) else: print '"%s" fetched in %ss' % (url,(time.time() - start)) print "Elapsed Time: %ss" % (time.time() - start) But you can make this even simpler. Really, all you need is:
def load_url(url): conn = urllib2.urlopen(url, timeout) data = conn.readall() print '"%s" fetched in %ss' % (url,(time.time() - start)) return data with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: pages = executor.map(load_url, urls) print "Elapsed Time: %ss" % (time.time() - start) I am now publishing a different solution, by having the worker threads not-deamon and joining them to the main thread (which means blocking the main thread until all worker threads have finished) instead of notifying the end of execution of each worker thread with a callback to a global function (as I did in the previous answer), as in some comments it was noted that such way is not thread-safe.
import threading import urllib2 import time start = time.time() urls = ["http://www.google.com", "http://www.apple.com", "http://www.microsoft.com", "http://www.amazon.com", "http://www.facebook.com"] class FetchUrl(threading.Thread): def __init__(self, url): threading.Thread.__init__(self) self.url = url def run(self): urlHandler = urllib2.urlopen(self.url) html = urlHandler.read() print "'%s\' fetched in %ss" % (self.url,(time.time() - start)) for url in urls: FetchUrl(url).start() #Join all existing threads to main thread. for thread in threading.enumerate(): if thread is not threading.currentThread(): thread.join() print "Elapsed Time: %s" % (time.time() - start) thread is threading.currentThread() isn't guaranteed to work (I think it always will for any CPython version so far, on any platform with real threads, if used in the main thread… but still, better not to assume). Safer to store all the Thread objects in a list (threads = [FetchUrl(url) for url in urls]), then start them, then join them with for thread in threads: thread.join().Thread subclass unless you have some kind of state to store or some API to interact with the threads from outside, just write a simple function, and do threading.Thread(target=my_thread_function, args=[url]).threading.enumerate() only includes the threads in the current process, so running multiple copies of the same script in separate instances of Python running as separate process isn't a problem. It's just that if you later decide to expand on this code (or use it in some other project) you may have daemon threads created in another part of the code, or what's now the main code may even be code running in some background thread.This script fetches the content from a set of URLs defined in an array. It spawns a thread for each URL to be fetch, so it is meant to be used for a limited set of URLs.
Instead of using a queue object, each thread is notifying its end with a callback to a global function, which keeps count of the number of threads running.
import threading import urllib2 import time start = time.time() urls = ["http://www.google.com", "http://www.apple.com", "http://www.microsoft.com", "http://www.amazon.com", "http://www.facebook.com"] left_to_fetch = len(urls) class FetchUrl(threading.Thread): def __init__(self, url): threading.Thread.__init__(self) self.setDaemon = True self.url = url def run(self): urlHandler = urllib2.urlopen(self.url) html = urlHandler.read() finished_fetch_url(self.url) def finished_fetch_url(url): "callback function called when a FetchUrl thread ends" print "\"%s\" fetched in %ss" % (url,(time.time() - start)) global left_to_fetch left_to_fetch-=1 if left_to_fetch==0: "all urls have been fetched" print "Elapsed Time: %ss" % (time.time() - start) for url in urls: "spawning a FetchUrl thread for each url to fetch" FetchUrl(url).start() urlsToFetch-=1. Inside the interpreter, that compiles into three separate steps to load urlsToFetch, subtract one, and store urlsToFetch. If the interpreter switches threads between the load and the store, you'll end up with thread 1 loading a 2, then thread 2 loading the same 2, then thread 2 storing a 1, then thread 1 storing a 1.threading.Lock around every access to the variable, or lots of other possibilities (use a counted semaphore instead of a plain integer, or use a barrier instead of counting explicitly, …), but you really don't need this global at all. Just join all the threads instead of daemonizing them, and it's done when you've joined them all.concurrent.futures whenever possible.