0

I have large directory that contains several millions+ sub-dir, each sub-dir contain 3 or less image files. What is the most efficient way for me to zip them so I can move it to a new server?

The sub-dir are 5 letter words compose of random letters + numbers.

For ex:
/foo/re63d/image.jpg, /foo/re63d/image2.jpg
/foo/4sf3g/image.jpg, /foo/4sf3g/image2.jpg
/foo/vj33s/image.jpg, /foo/vj33s/image2.jpg
and so on several millions times

I was thinking of using shell script to zip/tar them alphabetically. For ex every folders starting with

00* zip them into 0_0.zip
01* zip them into 0_1.zip
a0* into a_0.zip
a1* into a_1.zip
aa* into a_a.zip

This way it's easier for me to manage them. I want to be able to choose what sub section I want to zip at a time like aa* to af* as I want to be able control how long each process run. How would I go about doing this?

Improve this question
8
  • 4
    Not a real answer, but if what you want is to reliably move a lot of files to another server, I'd say rsync would be a better tool. Commented Dec 21, 2018 at 23:34
  • I like the ability to see what I transfer. How much should I be worry about the integrity of the transfers if there's a broken connection? Commented Dec 21, 2018 at 23:43
  • 1
    @nxnev I would say it constitutes an answer because it's better for OPs use-case (It's not what he/she wants, but what he/she needs.) Perz: rsync is rock solid and will continue where it left off when there is a broken connection and has a compression algorithm built-in Commented Dec 21, 2018 at 23:48
  • Thank you. I will have a look at rsync then. Commented Dec 21, 2018 at 23:52
  • What do you mean by efficient: Smallest size, least cpu time, least effort my you? Commented Dec 22, 2018 at 12:45

0

Reset to default

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.