PHP cUrl - Unable to fetch large files [duplicate]

Question

I'm developing a leecher website using PHP and cURL.

Here is the basic code:

$ch = curl_init(); $url = "http://somesite.com/somefile.part1.rar"; curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_HEADER, 0); $file_name = basename($url); $output = curl_exec($ch); curl_close($ch); file_put_contents($file_name, $output);

When the file size is small (like 15MB or so) this code works and the file is leeched to my server but when the file size is big (like 1GB or so) nothing is working.

I've tried setting 10000M file size limit for: post_max_size upload_max_filesize max_file_uploads but that didn't work.

I've tried to increase the memory limit up to 512M or even -1, but that didn't work ether.

So how can I fetch large files using cURL?

512MB ram, you must have at least 1GB ram to store the file contents in a php variable and then save, you should perhaps use CURLOPT_FILE and write directly to file instead. The script may also be timing out, after 60 seconds. — Lawrence Cherone
– Lawrence Cherone, Commented Dec 16, 2017 at 10:46
Guzzle it $client->request('GET', 'http://...', ['sink' => '/path/to/file']); — Lawrence Cherone
– Lawrence Cherone, Commented Dec 16, 2017 at 11:17

hanshenrik · Accepted Answer · 2017-12-16 11:19:02Z

what do you think this line does? curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); - it tells php to catch all stdout output of curl_exec, and save it all in memory at once, before doing anything else, that's both a very slow approach (because you don't start writing to disk before your download is 100% complete, and unless you're running on SSDs, disks are slow), and extremely memory hungry approach (because you store the entire file in memory at once), neither of those things are desirable. instead, do $fp=fopen(basename($url),'wb');curl_setopt($ch,CURLOPT_FILE,$fp); - now curl will write the content directly to the disk, thus being much faster (writing it to disk as it's being downloaded) AND just use a small amount of ram, no matter how big the download file is.

also note, if you're going to run large amount of slow downloads simultaneously, PHP-behind-a-webserver is simply a bad tool for the job, usually the amount of concurrent php processes you can run is very limited, and block your entire website from loading when all of them are busy, and php aborts if the client disconnect for some reason (see ignore_user_abort()), and many webservers will timeout if the script takes too long (see nginx proxy_read_timeout for example), and php often even kill itself for timeout reasons (see set_time_limit()) .. if that's the case, consider writing the downloader in another language (for example, Go's goroutines should be able to do a massive amount of concurrent slow downloads with little resource usage, unlike PHP)

Thank you. I did what you said and it worked! Could you please tell me from where and what books you learned these? I'm faaaaar beyond in working with files and downloading and uploading files and i'm working on a roject for a leecher site.
@ArashNaderi i don't really read books, and i don't remember (2006ish, php curl docs, i think?). btw, do you have a name for the project?
Thank you. I'm going to name it 'udu' for 'you download uplaod'
One other question: When I download a file with 1GB of size, the time to complete downloading would be 3 minutes and 20 seconds but with IDM, the same file is downloaded around 30 seconds. How can I achieve downloading files in shorter time? like parallel downloading.
@ArashNaderi downloading with many connections in parallel is another thing PHP is very bad at. PHP CAN do it, with the socket api and alternatively the curl_multi api, but the socket api is very difficult to use, and the curl_multi api use way more cpu than it should have been using (even with proper usage of curl_multi_select & co - not sure why) you'd be much better of using another tool than PHP for that job, maybe mget or wget2 or libtorrent - but personally i would probably use Go-curl

Collectives™ on Stack Overflow

PHP cUrl - Unable to fetch large files [duplicate]

1 Answer 1

11 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

11 Comments

Linked

Related