Requests in Asyncio - Keyword Arguments in loop.run_in_executor

Question

I'm using asyncio with the requests module to make an asynchronous HTTP request.

I can make a GET request like this:

@asyncio.coroutine def do_checks(): loop = asyncio.get_event_loop() req = loop.run_in_executor(None, requests.get, 'https://api.github.com/user') resp = yield from req print(resp.status_code) loop = asyncio.get_event_loop() loop.run_until_complete(do_checks())

However, I need to do support Basic HTTP Auth (described here) in the request.

According to the documentation, url and auth are both named parameters for requests.get().

But, if I run this (note the addition of url='' and auth = ''):

@asyncio.coroutine def do_checks(): loop = asyncio.get_event_loop() req = loop.run_in_executor(None, requests.get, url='https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass')) resp = yield from req print(resp.status_code) loop = asyncio.get_event_loop() loop.run_until_complete(do_checks())

I get this error:

TypeError: run_in_executor() got an unexpected keyword argument 'url'

In the prototype for asyncio.run_in_executor(), additional arguments are supported:

BaseEventLoop.run_in_executor(executor, callback, *args)

requests.get() clearly supports named parameters (get, auth, etc.). What's wrong?

Thomas Orozco · Accepted Answer · 2014-05-30 14:21:01Z

Two ways to do that. Create a wrapper function, or just use a session to provide the auth.

Using a session:

@asyncio.coroutine def do_checks(): loop = asyncio.get_event_loop() session = requests.Session() session.auth = HTTPBasicAuth('user', 'pass') req = loop.run_in_executor(None, session.get, 'https://api.github.com/user') resp = yield from req print(resp.status_code)

Writing a wrapper function (note that I'm using def for clarity here, but that a lambda would obviously work too):

@asyncio.coroutine def do_checks(): def do_req(): return requests.get('https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass')) loop = asyncio.get_event_loop() req = loop.run_in_executor(None, do_req) resp = yield from req print(resp.status_code)

Thanks for the answer! The session method worked for me. If I use the wrapper function, I can't figure out how to use do_req() in a for loop to call multiple different URLs without redefining do_req() each time (since I can't pass in the URL as an argument).
@user3689902 Just redefine a new function every time. Doing this with a lambda is quite lightweight.
The asyncio docs recommend using functools.partial, "because asyncio can inspect functools.partial() object to display parameters in debug mode, whereas lambda functions have a poor representation." docs.python.org/3/library/…

dano · Accepted Answer · 2014-05-30 14:33:40Z

This is actually a design decision in asyncio. From the docstring of asyncio/base_events.py:

"""Base implementation of event loop. The event loop can be broken up into a multiplexer (the part responsible for notifying us of IO events) and the event loop proper, which wraps a multiplexer with functionality for scheduling callbacks, immediately or at a given time in the future. Whenever a public API takes a callback, subsequent positional arguments will be passed to the callback if/when it is called. This avoids the proliferation of trivial lambdas implementing closures. Keyword arguments for the callback are not supported; this is a conscious design decision, leaving the door open for keyword arguments to modify the meaning of the API call itself. """

Note the last sentence there.

The asyncio PEP notes this as well, and recommends a lambda to work around it:

This convention specifically does not support keyword arguments. Keyword arguments are used to pass optional extra information about the callback. This allows graceful evolution of the API without having to worry about whether a keyword might be significant to a callee somewhere. If you have a callback that must be called with a keyword argument, you can use a lambda. For example:

loop.call_soon(lambda: foo('abc', repeat=42))

pegasus · Accepted Answer · 2024-11-29 12:41:22Z

With Python 3.9+, you can simply use

req = asyncio.to_thread( requests.get, url='https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass') )

This is equivalent to (the following code does not work)

loop = asyncio.get_event_loop() req = loop.run_in_executor( None, requests.get, url='https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass') )

asyncio.to_thread accepts keyword arguments, while loop.run_in_executor does not.

Collectives™ on Stack Overflow

Requests in Asyncio - Keyword Arguments in loop.run_in_executor

3 Answers 3

3 Comments

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Related