Python urllib2.urlopen() is slow, need a better way to read several urls

Edit: Please take a look at Wai's post for a better version of this code. Note that there is nothing wrong with this code and it will work properly, despite the comments below.

The speed of reading web pages is probably bounded by your Internet connection, not Python.

You could use threads to load them all at once.

import thread, time, urllib websites = {} def read_url(url): websites[url] = urllib.open(url).read() for url in urls_to_load: thread.start_new_thread(read_url, (url,)) while websites.keys() != urls_to_load: time.sleep(0.1) # Now websites will contain the contents of all the web pages in urls_to_load 

It is maby not perfect. But when I need the data from a site. I just do this:

import socket def geturldata(url): #NO HTTP URLS PLEASE!!!!! server = url.split("/")[0] args = url.replace(server,"") returndata = str() s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((server, 80)) #lets connect :p s.send("GET /%s HTTP/1.0\r\nHost: %s\r\n\r\n" % (args, server)) #simple http request while 1: data = s.recv(1024) #buffer if not data: break returndata = returndata + data s.close() return returndata.split("\n\r")[1] 

As a general rule, a given construct in any language is not slow until it is measured.

In Python, not only do timings often run counter to intuition but the tools for measuring execution time are exceptionally good.

Scrapy might be useful for you. If you don't need all of its functionality, you might just use twisted's twisted.web.client.getPage instead. Asynchronous IO in one thread is going to be way more performant and easy to debug than anything that uses multiple threads and blocking IO.

Not sure why nobody mentions multiprocessing (if anyone knows why this might be a bad idea, let me know):

import multiprocessing from urllib2 import urlopen URLS = [....] def get_content(url): return urlopen(url).read() pool = multiprocessing.Pool(processes=8) # play with ``processes`` for best results results = pool.map(get_content, URLS) # This line blocks, look at map_async # for non-blocking map() call pool.close() # the process pool no longer accepts new tasks pool.join() # join the processes: this blocks until all URLs are processed for result in results: # do something 

There are a few caveats with multiprocessing pools. First, unlike threads, these are completely new Python processes (interpreter). While it's not subject to global interpreter lock, it means you are limited in what you can pass across to the new process.

You cannot pass lambdas and functions that are defined dynamically. The function that is used in the map() call must be defined in your module in a way that allows the other process to import it.

The Pool.map(), which is the most straightforward way to process multiple tasks concurrently, doesn't provide a way to pass multiple arguments, so you may need to write wrapper functions or change function signatures, and/or pass multiple arguments as part of the iterable that is being mapped.

You cannot have child processes spawn new ones. Only the parent can spawn child processes. This means you have to carefully plan and benchmark (and sometimes write multiple versions of your code) in order to determine what the most effective use of processes would be.

Drawbacks notwithstanding, I find multiprocessing to be one of the most straightforward ways to do concurrent blocking calls. You can also combine multiprocessing and threads (afaik, but please correct me if I'm wrong), or combine multiprocessing with green threads.

1) Are you opening the same site many times, or many different site? If many different sites, I think urllib2 is good. If doing the same site over and over again, I have had some personal luck with urllib3 http://code.google.com/p/urllib3/

2) BeautifulSoup is easy to use, but is pretty slow. If you do have to use it, make sure to decompose your tags to get rid of memory leaks.. or it will likely lead to memory issues (did for me).

What do your memory and cpu look like? If you are maxing your CPU, make sure you are using real heavyweight threads, so you can run on more than 1 core.

How about using pycurl?

You can apt-get it by

$ sudo apt-get python-pycurl

First, you should try multithreading/multiprocessing packages. Currently, the three popular ones are multiprocessing;concurrent.futures and [threading][3]. Those packages could help you to open multi url at the same time, which could increase the speed.

More importantly, after using multithread processing, and if you try to open hundreds urls at the same time, you will find urllib.request.urlopen is very slow, and opening and read the context become the most time-consuming part. So if you want to make it even faster, you should try requests packages, requests.get(url).content() is faster than urllib.request.urlopen(url).read().

So, here I list two example to do fast multi url parsing, and the speed is faster than the other answers. The first example use classical threading package and generate hundreds thread at the same time. (One trivial shortcoming is it cannot keep the original order of the ticker.)

import time import threading import pandas as pd import requests from bs4 import BeautifulSoup ticker = pd.ExcelFile('short_tickerlist.xlsx') ticker_df = ticker.parse(str(ticker.sheet_names[0])) ticker_list = list(ticker_df['Ticker']) start = time.time() result = [] def fetch(ticker): url = ('http://finance.yahoo.com/quote/' + ticker) print('Visit ' + url) text = requests.get(url).content soup = BeautifulSoup(text,'lxml') result.append([ticker,soup]) print(url +' fetching...... ' + str(time.time()-start)) if __name__ == '__main__': process = [None] * len(ticker_list) for i in range(len(ticker_list)): process[i] = threading.Thread(target=fetch, args=[ticker_list[i]]) for i in range(len(ticker_list)): print('Start_' + str(i)) process[i].start() # for i in range(len(ticker_list)): # print('Join_' + str(i)) # process[i].join() print("Elapsed Time: %ss" % (time.time() - start)) 

The second example uses multiprocessing package, and it is little more straightforward. Since you just need to state the number of pool and map the function. The order will not change after fetching the context and the speed is similar to the first example but much faster than other method.

from multiprocessing import Pool import requests from bs4 import BeautifulSoup import pandas as pd import os import time os.chdir('file_path') start = time.time() def fetch_url(x): print('Getting Data') myurl = ("http://finance.yahoo.com/q/cp?s=%s" % x) html = requests.get(myurl).content soup = BeautifulSoup(html,'lxml') out = str(soup) listOut = [x, out] return listOut tickDF = pd.read_excel('short_tickerlist.xlsx') li = tickDF['Ticker'].tolist() if __name__ == '__main__': p = Pool(5) output = p.map(fetch_url, ji, chunksize=30) print("Time is %ss" %(time.time()-start))