10

I regularly need to transfer millions of small files (small images, txt, json) with average of 5-50k per file between servers or to aws s3.

Is there a faster to merge them into a single file to optimize transfer speed other than zip/tar -cf?

Improve this question
2
  • 14
    What you have against tar/zip? Commented Aug 13, 2022 at 16:21
  • What exactly have you tried? Commented Aug 14, 2022 at 17:00

6 Answers 6

Reset to default
13

Something similar to tar cz * | ssh <host> "tar xfc -"? Seriously what is wrong with tar? This command does not create any intermediate files.

Improve this answer
9
  • 1
    ssh to access s3? Commented Aug 14, 2022 at 12:52
  • 3
    Or better tar --zstd c * | ssh <host> "tar xf -", since zstd mostly obsoletes gzip, being much faster at similar or better compression ratios. Commented Aug 15, 2022 at 3:46
  • @PeterCordes, how do you think to use ssh to access s3 and exec there tar command (w/o virtual machine)? Commented Sep 5, 2022 at 19:20
  • 1
    @RomeoNinov: don't ask me, ask the person who wrote this answer. For people who find this answer usable, my suggestion is an incremental improvement on it. For people using S3 like the original asker of the question, apparently this won't work, so they should see your answer. I'd expect zstd if better there, too, if S3 can extract a .tar.zst after you upload it. Commented Sep 5, 2022 at 19:28
  • 1
    @RomeoNinov: I haven't used S3 myself, thanks for the reminder that they charge per file as well as total size. And for the fact that millions of files would be prohibitively expensive to keep separately there. Yeah, maybe they are fine just keeping an archive there to grab from other servers, in which case .tar.zst is great. Commented Sep 5, 2022 at 20:38
9

Developing idea from other answer you can send information via pipe and even do not create locally a tar file for example. Command will be something like:

tar cf - * | aws s3 cp - s3://some-bucket/archive.tar

This command have advantage that will run tar and aws command in parallel. You can even add compression (which will execute again in parallel)

tar cf - * | gzip -c | aws s3 cp - s3://some-bucket/archive.tar.gz

For simplify the operation you can use top level directory for the files and not use *:

tar cf - top_level_directory | aws s3 cp - s3://some-bucket/archive.tar tar cf - top_level_directory | gzip -c | aws s3 cp - s3://some-bucket/archive.tar.gz

Inspired by other answer you can use cpio. Seems to be faster and produce smaller files:

ls |cpio -o |gzip -c | aws s3 cp - s3://some-bucket/archive.cpio.gz
Improve this answer
12
  • 2
    I don't think you need -c, tar -cf - * | gzip | ... should work. At least with the tools onmy Linux machine. Also, you don't really need gzip at all, you could do tar -czf - * unless there is some advantage in adding a separate call to gzip. Commented Aug 13, 2022 at 18:25
  • 3
    I don't think that's accurate. I just ran time tar czf file.tgz file and time tar cf - file | gzip -c > file.tgz using a 288M file on an SSD, and the first took 0m20.004s (real) while the second took 0m20.141s. So, at least in this particular test, using the pipe was marginally slower. Granted, the difference is trivial but could become relevant for larger files. Maybe? At the very least, the pipe does not seem to add any advantage to justify the increase in complexity of the command. Commented Aug 13, 2022 at 18:41
  • 3
    Running tar|gzip is the same as tar c, they both run in a piped parallel. To deny that would suggest it is either keeping the data in a buffer - resulting in memory exhaustion, or creating a temporary file, neither of which are true. Commented Aug 13, 2022 at 18:49
  • 2
    Heh, sounds like this might be worth a question! I suspect whether one uses an SSD or HDD for this will also be very relevant. Commented Aug 13, 2022 at 19:11
  • 4
    Good grief... run it then suspend it and look for what fd's are setup. They are both piped. Any difference in time is purely arbitrary. Commented Aug 13, 2022 at 19:18
7

Yes, there are a whole slew of options.

One is to never create an intermediate file, as suggested in the other answers. This cuts out local IO, though it makes resuming a partial upload impossible.

There are other options though to improve things further:

  • Use compression on the archive. GZip is the classic, but it’s somewhat slow. LZ4 is pretty widely available these days, wicked fast, and should still get you decent compression ratios with tar and what you’re describing. ZSTD is not as fast as LZ4, but will get you compression ratios similar to GZip in less time. Irrespective of the choice, this will cut down, possibly significantly, on the total data to transfer.
  • Consider using cpio instead of tar. tar is not exactly a space-efficient archive format. This usually doesn’t matter much, but if you’re dealing with millions of very small files, the overhead is actually pretty significant. cpio still has a nontrivial amount of overhead, but it’s less than tar does, so in theory using cpio here should result in a nontrivial reduction in the amount of data to transfer.
  • Consider creating multiple archives each containing a subset of the files (for example, maybe have no more than 100k files in each archive), and uploading those in parallel. Assuming the source system has a fast internet connection and relatively fast storage, this will (almost certainly) be faster than uploading one big archive because it can parallelize local IO better (and because AWS can process things in parallel on their end as well). ‘Optimal’ sizing here is generally small enough that you don’t need to use multipart uploads. This also helps with resuming partial uploads, even if you skip creating intermediate files locally, because you only have to upload the archives you haven’t already uploaded.
Improve this answer
6
  • zstd on lz4 on the tar or cpio stream might remove much of the difference, unless tar's extra metadata has low similarity within each entry or across files. Commented Aug 15, 2022 at 3:44
  • @PeterCordes True, but at the scale of millions of files, even compressed the overhead is likely to be measurable. Commented Aug 15, 2022 at 11:55
  • I tried archiving 100k empty files with simplistic names: mkdir {0001..1000} ; touch {0001..1000}/{001..100} ; tar cf foo.tar foo ; find foo/ | bsdcpio -o foo.cpio (the libarchive implementation) ; zstd foo.cpio foo.tar. The uncompressed tar archive was much bigger, 50M vs. 8.6M. But the compressed tar archive was half the size, 231K vs. 422K; tar compressed better. cpio -o --binary (aka -7 to use binary instead of ASCII text) doesn't work with that many files, nor does -6. And the other formats available, like --format newc, was larger than the default CPIO format. Commented Aug 16, 2022 at 9:02
  • Hmm, my test had identical sizes and permissions for all files, and timestamps within a few hundred milliseconds for all files (on tmpfs). So IDK how well it generalizes, but I think if tar is using binary to indicate record length vs. cpio using ASCII, that would compress better in general. The padding of 0 bytes in tar records compresses extremely well. (Especially if it's the same length every time, and probably exaggerating the effect by not having any file contents to make the compression dictionary work harder, not just use very short symbols for the similar metadata stuff. Commented Aug 16, 2022 at 9:07
  • @PeterCordes As you note yourself, it’s not exactly a fair comparison when all the archived files are empty. With actual file data, the difference is generally pretty small (at most a few percent), but tends to be in favor of cpio over tar. The cpio archives will also generally be processed faster, though the difference is again pretty small (at least, on modern systems, on older systems it could be pretty significant, which is part of why cpio stuck around). Commented Aug 16, 2022 at 14:04
2

It might be worth it to investigate the rsync command as what you want to do is transfer the files.

--- Added after comments.

This is an XY question: "It takes long time to transfer files so I want to compress them more?". I believe a better question would be: "It takes a long time to transfer the files, how can I make it happen faster?". Rsync is an answer to the second question.

Note that rsync creates and copies the files, it does not create an archive you later need to decompress (which would add to total time as well). As it copies, basically, "changed" or "new" files it can be set to run nearly continuously (say every hour) and will have no problems restarting after network failures. It pays to think a bit about directory structure though.

The exact options to use for rsync in different situations is dependant on the relative speeds of network compared to CPU speed. One important thing to note is that if you select compression it is done simultaneously with transfer. Compared to zipping or similar where you first compress, then transfer, then decompress. This can save a lot of "total time" even when maybe the compression is skipped. (Compression in my experience saves very little time in rsync when sending lots of small files).

Rsync is available on most if not all modern systems, including servers and aws s3 according to my understanding, but I may be wrong.

Improve this answer
7
  • 1
    Please edit your answer and give some details. What rsync options would you recommend? What command would you use? Compression? No compression? Which compression? Commented Aug 14, 2022 at 12:47
  • rsync with s3 aws? Commented Aug 14, 2022 at 12:50
  • TS asked for transfer of files between servers. I read the question as he wants to make the transfer as fast as possible. How to use rsync is well documented, but the results is dependant on the actual system setup. In brief, the -z option for compression may save time if the link is slow and CPU is fast, otherwise not. Commented Aug 14, 2022 at 14:00
  • 2
    rclone would probably work better here as it natively supports AWS (and parallelizes transfers, unlike rsync). Commented Aug 14, 2022 at 15:35
  • 1
    @AustinHemmelgarn, AFAIK outbound traffic from S3 if not free of charge. Also they have limit of free list, read, get properties operations. And by OPs word million of files may generate significant charge. Commented Aug 14, 2022 at 19:39
2

Heavily multithreaded use of rclone is what I use. I do a similar thing both between servers and S3.

Improve this answer
0

What is your storage?

This answer shows that on a single spindle, sorting by inode and running jobs in parallel will get the fastest results:

Speed up copying 1000000 small files

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.