7

I am running many tasks on a Linux cluster. Each task creates many output files. When all tasks are finished, I run something like tar cf foo.tar output_files/to create a tar archive. This is a very slow process since there are many thousands of files and directories.

Is there any way to do this in parallel as the output files are being created?

Is it possible to have multiple tar processes, spread across multiple machines, all adding their files to the same archive at once?

The cluster has a shared filesystem.

I am not interested in compression since it slows things down even more, and because all of the input files are themselves already compressed. Ideally the output would be a tar file, but I would consider other archive formats as well.

Improve this question
1
  • Usually the bottleneck is I/O, not the CPU. So why use several processes? Commented Jun 30, 2014 at 23:23

3 Answers 3

Reset to default
8

You can't have multiple processes adding to the same tar archive (or any other usual archive format, compressed or not). Each file is stored contiguously, and there is no way to insert data in a file, only to append or overwrite, so continuing to write to a file that isn't the last one would overwrite subsequent files.

If you know the file size in advance, you could reserve the size in the tar archive and have the program keep writing. That would require a lot of coding: it's a very unusual thing to do.

Unix has a feature designed to accommodate a group of files that are written to independently. It's called a directory.

There are very few cases where you'd gain anything from an uncompressed archive over a directory. Reading it might be slightly faster in some circumstances; this is an intrinsic consequence of the directory format (where each file entry is a pointer to its content) as opposed to the archive format (where each file entry is its content directly), which is precisely what makes it possible to build the directory piecewise. Transforming a directory tree to an archive is post-processing that needs to be done sequentially.

Improve this answer
2
  • One of the reasons to avoid a directory here is because there are so many files. If we do too many runs without archiving the results, the shared filesystem can run out of inodes. Commented Jun 30, 2014 at 23:33
  • 4
    @DavidKelly999 Then your filesystem is not adapted to your workload. Create a filesystem with more inodes, or use a different filesystem type, or use a database. An archive is not at all adapted to this problem, you're trying to fit a square peg in a round hole. Commented Jun 30, 2014 at 23:38
4

You can start the creation of the final tar file before all output files are created: Maybe that achieves the speed up you want.

You can call tar this way:

tar -cf foo.tar -T file-list

file-list would be a FIFO. You need a script which detects

  1. new files in the source directory (inotifywatch)

  2. when each of these new files is finished (fuser)

If a file is finished then its path is written to the FIFO. Maybe it is useful not to create an archive with completely mixed paths. You can start with the directory which gets the first input file and add new directories only after their last file has been finished (create a flag file after the respective process has been finished). The first approach has the advantage that probably the file is completely in the cache yet.

Improve this answer
0

GNU tar has --append:

tar -f foo.tar --append newfiles

Unfortunately it reads the full tar file.

Improve this answer

You must log in to answer this question.