How to accelerate download speed of a file using Python?

Question

I am trying to download a file located on the web through write function with the wb mode. Files are being downloaded too slowly when it is compared to the speed of downloading through a web browser (as I have a high-speed internet connection). How can I accelerate the download speed? Is there a better way of handling file download?

Here is the way I use:

resp = session.get(download_url) with open(package_name + '.apk', 'wb+') as local_file: local_file.write(resp.content)

I have experimented that the download speeds of requests and urllib3 libraries are almost the same. Here is the experimental result to download a 15 MB file:

requests: 0:02:00.689587
urllib3: 0:02:05.833442

p.s. My Python version is 3.7.0, and my OS is Windows 10 version 1903.

p.s. I have investigated the reportedly similar question, but the answers/comments did not work.

Of course, for example, visit apkpure.com/true-fear-forsaken-souls-part-2/…, then click the Download button to generate and retrieve the download link. @Fozoro — talha06
– talha06, Commented Jun 15, 2019 at 18:34

fiacre · Accepted Answer · 2019-06-15 16:22:07Z

Seems odd, but it makes sense -- the browser caches the download, where writing directly to file does not.

Consider:

from tempfile import SpooledTemporaryFile temp = SpooledTemporaryFile() resp = session.get(download_url) temp.write(resp.content) temp.seek(0) with open(package_name + '.apk', 'wb') as local_file: local_file.write(resp.content)

That may be faster.

If you can create an asynchronous write to local file, you won't hold up your program.

 import asyncio async def write_to_local_file(name, spool_file): spool_file.seek(0) with open(package_name + '.apk', 'wb') as local_file: local_file.write(spool_file.read())

Then:

from tempfile import SpooledTemporaryFile temp = SpooledTemporaryFile() resp = session.get(download_url) temp.write(resp.content) asyncio.run(write_to_local_file("my_package_name", temp))

Getting ModuleNotFoundError: No module named '_contextvars' when I execute the script after importing SpooledTemporaryFile and asyncio.
@talha06 I am using python 3.7 on a Mac, what version of python what OS are you using? Can you try each import from the python shell?
My OS is Windows 10 (version 1903) and the version of Python is 3.7.0 as I have declared in the OP.
@talha06 : Yes, there seems to be a known bug with contextvars and Windows. I wonder if the code is reasonably fast without asyncio? The Spooled Temp File should help.
No, unfortunately, I have not recognized any concrete speedup.

Collectives™ on Stack Overflow

How to accelerate download speed of a file using Python?

1 Answer 1

7 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Linked

Related