This is the first time I've tried to use a library with less-than-ideal levels of documentation and example code, so bear with me. I have a tiny bit of experience with the Requests library, but I need to send separate requests to a specific address every second:
- Without waiting for the first request to complete, handling the individual responses as they come in
- The responses' content need to be parsed separately
- While limiting the total number of connections
I can't figure out how to satisfy these conditions simultaneously. grequests.map() will give me the responses' content that I want, but only in a batch after they've all completed. grequests.send() seems to only return a response object that doesn't contain the html text of the web page. (I may be wrong about grequests.send(), but I haven't yet found an example that pulls content from that object)
Here's the code that I have so far:
import grequests from time import sleep def print_res(res, **kwargs): print res print kwargs headers = {'User-Agent':'Python'} req = grequests.get('http://stackoverflow.com', headers=headers, hooks=dict(response=print_res), verify=False) for i in range(3): job = grequests.send(req, grequests.Pool(10)) sleep(1) The response I get:
1 <Response [200]> {'verify': False, 'cert': None, 'proxies': {'http': 'http://127.0.0.1:8888', 'ht tps': 'https://127.0.0.1:8888'}, 'stream': False, 'timeout': None} 2 <Response [200]> {'verify': False, 'cert': None, 'proxies': {'http': 'http://127.0.0.1:8888', 'ht tps': 'https://127.0.0.1:8888'}, 'stream': False, 'timeout': None} 3 <Response [200]> {'verify': False, 'cert': None, 'proxies': {'http': 'http://127.0.0.1:8888', 'ht tps': 'https://127.0.0.1:8888'}, 'stream': False, 'timeout': None} I've tried accessing the html response with req.content, and job.content, but neither work.
