0

I have set of large number of files to be archived in a linux server. I need to compress those files in to set of 1GB compressed files.

I am using gzip to compress the files but I don't know the final size of the compressed folders until I really compressed the files. I am using below command to get the final size of the compressed file but it's not really efficient as I need to keep adding new files and compress until I get the desired final size.

tar -czf - /path/to/compressed-directory | wc -c

Is there an efficient way to do this? Or should I consider compressing individual files and then create the final archive using individual compressed files?

Improve this question

1 Answer 1

Reset to default
3

There is no way other than trial and error to determine if adding the next file to the archive will push it over your 1GB limit. Also it's possible to have a file greater than 1GB compressed, in which case you will not be able to do what you want.

I would recommend that you use split to break up a tar.gz archive of all of your files into 1GB pieces. Each piece individually will not be extractable. Instead you will need to recombine them on the other end back into the one large .tar.gz for extraction.

E.g.:

tar -czf - /path/to/whatever | split -b 1024m - prefix

where prefix is the leading characters of the file names of the resulting split files. To recombine and extract on the other end:

cat prefix* | tar -xzf -
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.