1

I'm downloading files in parallel through lftp onto a remote server, and I use the commands:

lftp ftp:*url* cd into directory mirror --parallel=10

I'm downloading a total of 365 files separated into 12 directories. When I use this command it takes a few hours, which isn't a deal breaker but I figured the more files I download in parallel the faster it will go. Obviously there has to be a point of diminishing returns, as I expect downloading all 365 files in parallel would overload the system. (Each file is little over a gigabyte). Plus, as I have increased the number of files to download in parallel, i increasingly get the message waiting to reconnect....trying in 30s etc. as I believe I am overtaxing the system.

Does anyone have thoughts on efficient ways to download files in parallel? Thanks in advance

Improve this question
1
  • Downloading N files at the same time, means that N processes are sending/recieving packets AT THE SAME TIME. Whichever part of the network between local and remote will be overloaded and slow down. Commented Dec 7, 2020 at 19:03

1 Answer 1

Reset to default
1

The more you run in parallel, the more load you put on the sender.

If the server has 1000 TCP connections and the bandwidth is limited, then you will get 0.1% of the server's bandwidth if you have 1 connection.

If you have 10 connections, you will get 1% of the server's bandwidth.

In general my recommendations would be:

  • If you are not in a hurry: use a single connection
  • If you control the server, do whatever you like
  • If you are in a hurry and you do not control the server, use caution: Some will have no problems accepting 1000 parallel requests. Others will struggle at 100. And if you are adding 10 extra parallel requests, that may strain the server.

In your situation it sounds as if you are in the first situation: You are not in a hurry, so you should use a single connection. The "Connection refused" is also a sign that the server is overloaded. If you want to use more than 1 connection, consider talking to whomever owns the server.

Improve this answer
0

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.