python async post requests

Question

I was wondering if there was any way to make this script a lot faster - like instantly create 1000 accounts for example or at least in a matter of a few seconds. I’ve tried doing some async stuff myself but this is as far as I could get, I am just a beginner with asynchronous programming so any help is appreciated.

import asyncio import aiohttp async def make_numbers(numbers, _numbers): for i in range(numbers, _numbers): yield i async def make_account(): url = "https://example.com/sign_up.php" async with aiohttp.ClientSession() as session: async for x in make_numbers(35691, 5000000): async with session.post(url, data ={ "terms": 1, "captcha": 1, "email": "user%[email protected]" % str(x), "full_name": "user%s" % str(x), "password": "123456", "username": "auser%s" % str(x) }) as response: data = await response.text() print("-> Creating account number %d" % x) print (data) loop = asyncio.get_event_loop() try: loop.run_until_complete(make_account()) finally: loop.close()

user4815162342 · Accepted Answer · 2020-08-07 10:37:59Z

The code in the question executes all POST requests in a series, making the code no faster than if you used requests in a single thread. But unlike requests, asyncio makes it possible to parallelize them in the same thread:

async def make_account(): url = "https://example.com/sign_up.php" async with aiohttp.ClientSession() as session: post_tasks = [] # prepare the coroutines that post async for x in make_numbers(35691, 5000000): post_tasks.append(do_post(session, url, x)) # now execute them all at once await asyncio.gather(*post_tasks) async def do_post(session, url, x): async with session.post(url, data ={ "terms": 1, "captcha": 1, "email": "user%[email protected]" % str(x), "full_name": "user%s" % str(x), "password": "123456", "username": "auser%s" % str(x) }) as response: data = await response.text() print("-> Created account number %d" % x) print (data)

The above code will attempt to send all the POST requests at once. Despite the intention, it will be throttled by aiohttp.ClientSession's TCP connector which allows a maximum of 100 simultaneous connections by default. To increase or remove this limitation, you must set a custom connector on the session.

shaun shia · Accepted Answer · 2018-08-06 06:34:01Z

The async code you posted looks ok to me. You can speed it up by combining asyncio with multithreading/multiprocess.

But there are other limitations that prevent you from creating 1000 accounts in a second. For example, network speed, concurrent connections, rate limit, database IOPS on the server side.

Collectives™ on Stack Overflow

python async post requests

2 Answers 2

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Linked

Related