I'm working with a process which is basically as follows:
- Take some list of urls.
- Get a
Responseobject from each. - Create a BeautifulSoup object from the
textof each Response. - Pull the text of a certain tag from that BeautifulSoup object.
From my understanding, this seems ideal for grequests:
GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily.
But yet, the two processes (one with requests, one with grequests) seem to be getting me different results, with some of the requests in grequests returning None rather than a response.
Using requests
import requests tickers = [ 'A', 'AAL', 'AAP', 'AAPL', 'ABBV', 'ABC', 'ABT', 'ACN', 'ADBE', 'ADI', 'ADM', 'ADP', 'ADS', 'ADSK', 'AEE', 'AEP', 'AES', 'AET', 'AFL', 'AGN', 'AIG', 'AIV', 'AIZ', 'AJG', 'AKAM', 'ALB', 'ALGN', 'ALK', 'ALL', 'ALLE', ] BASE = 'https://finance.google.com/finance?q={}' rs = (requests.get(u) for u in [BASE.format(t) for t in tickers]) rs = list(rs) rs # [<Response [200]>, # <Response [200]>, # <Response [200]>, # <Response [200]>, # <Response [200]>, # <Response [200]>, # ... # <Response [200]>] # All are okay (status_code == 200) Using grequests
# Restarted my interpreter and redefined `tickers` and `BASE` import grequests rs = (grequests.get(u) for u in [BASE.format(t) for t in tickers]) rs = grequests.map(rs) rs # [None, # <Response [200]>, # None, # None, # None, # None, # None, # None, # None, # None, # None, # None, # None, # None, # None, # None, # None, # None, # <Response [200]>, # <Response [200]>, # <Response [200]>, # <Response [200]>, # <Response [200]>, # <Response [200]>, # <Response [200]>, # <Response [200]>, # <Response [200]>, # <Response [200]>, # <Response [200]>, # <Response [200]>] Why the difference in results?
Update: I can print the exception type as follows. Related discussion here but I have no idea what's going on.
def exception_handler(request, exception): print(exception) rs = grequests.map(rs, exception_handler=exception_handler) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) # ("bad handshake: SysCallError(-1, 'Unexpected EOF')",) System/version info
- requests: 2.18.4
- grequests: 0.3.0
- Python: 3.6.3
- urllib3: 1.22
- pyopenssl: 17.2.0
- All via Anaconda
- System: same issue on both Mac OSX HS & Windows 10, build 10.0.16299
None. I'm guessing that Google is getting angry when you make too many unauthenticated requests all at once. Reading down slightly more in the README describes how to write an exception handler that would tell you what's going on.grequests.map(rs, size=2)