In what way is grequests asynchronous?

Question

I've been using the python requests library for some time, and recently had a need to make a request asynchronously, meaning I would like to send off the HTTP request, have my main thread continue to execute, and have a callback called when the request returns.

Naturally, I was lead to the grequests library (https://github.com/kennethreitz/grequests), but i'm confused about the behavior. For example:

import grequests def print_res(res): from pprint import pprint pprint (vars(res)) req = grequests.get('http://www.codehenge.net/blog', hooks=dict(response=print_res)) res = grequests.map([req]) for i in range(10): print i

The above code will produce the following output:

<...large HTTP response output...> 0 1 2 3 4 5 6 7 8 9

The grequests.map() call obviously blocks until the HTTP response is available. It seems likely I misunderstood the 'asynchronous' behavior here, and the grequests library is just for performing multiple HTTP requests concurrently and sending all responses to a single callback. Is this accurate?

Not sure, but could you not just use the built-in urllib module and run it in a background thread with the thread module? — Aya
– Aya, Commented Apr 15, 2013 at 13:06
I think i may have to do just that. I was just confused, and want to verify the expected behavior. — cacois
– cacois, Commented Apr 15, 2013 at 13:16
Sure. I just tend to stick with built-ins where possible to maximize portability. — Aya
– Aya, Commented Apr 15, 2013 at 13:22
For those coming to the question in recent years, after the whole Kenneth Reitz controversy the repo's control was transferred to github.com/spyoungtech/grequests — qwr
– qwr, Commented May 24, 2024 at 14:07

Martijn Pieters · Accepted Answer · 2019-01-28 14:47:10Z

.map() is meant to run retrieval of several URLs in parallel, and will indeed wait for these tasks to complete (gevent.joinall(jobs)) is called).

Use .send() instead to spawn jobs, using a Pool instance:

req = grequests.get('http://www.codehenge.net/blog', hooks=dict(response=print_res)) job = grequests.send(req, grequests.Pool(1)) for i in range(10): print i

Without the pool the .send() call will block still, but only for the gevent.spawn() call it executes.

@martijn pieters Can you give a sample with post. Not able to get it work with post calls
@Avinash: just use grequests.post() instead? As far as grequests is concerned, there is no difference between the various request methods.
What does .pool() actually do? I did not understand the documentation.

kylebebak · Accepted Answer · 2017-07-05 04:32:54Z

If you don't want to use grequests you can just implement requests with callbacks using requests + the threading module from the standard library. It's actually really easy, and if all you want to do is send requests with callbacks the API is nicer than the one provided by grequests.

from threading import Thread from requests import get, post, put, patch, delete, options, head request_methods = { 'get': get, 'post': post, 'put': put, 'patch': patch, 'delete': delete, 'options': options, 'head': head, } def async_request(method, *args, callback=None, timeout=15, **kwargs): """Makes request on a different thread, and optionally passes response to a `callback` function when request returns. """ method = request_methods[method.lower()] if callback: def callback_with_args(response, *args, **kwargs): callback(response) kwargs['hooks'] = {'response': callback_with_args} kwargs['timeout'] = timeout thread = Thread(target=method, args=args, kwargs=kwargs) thread.start()

You can verify that it works like an AJAX calls in JS: you send a request on another thread, do some stuff on the main thread, and when the request returns you invoke a callback. This callback just prints out the response content.

async_request('get', 'http://httpbin.org/anything', callback=lambda r: print(r.json())) for i in range(10): print(i)

Winand · Accepted Answer · 2017-02-15 06:32:18Z

Create a list of requests and then send them with .imap:

event_list = [grequests.get(url_viol_card, params={"viol": i}, session=session) for i in print_ev_list] for r in grequests.imap(event_list, size=5): print(r.request.url)

session is requests.Session() object (optional)
size=5 send 5 requests simultaneously: as soon as one of them is completed the next one is sent
In this example when any request (unordered) is completed its url is printed

Collectives™ on Stack Overflow

In what way is grequests asynchronous?

3 Answers 3

3 Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Linked

Related