0

What I am trying to accomplish is I have a directory with lets say 176,000 files. I want the script to split the 176,000 files into folders with lets say 20,000 files each and the last one containing the odd ball #. I would like to copy the files sequentially and keeping them in order alpha numerical. This is what I currently have but it seems to be missing something, the first 2 folders it creates are out of order, the other 3 folders it creates are sequential.

Bottom line, folder contains 176,000 files, script would move the first 20,000 into a folder called 'split' then takes the next 20,000 and so on, creating split2, split3, etc..

Here is what I have:

#!/bin/bash dir_size=20000 dir_name="split" n=$((`find . -maxdepth 1 -type f | wc -l`/$dir_size+1)) for i in `seq 1 $n`; do mkdir -p "$dir_name$i"; find . -maxdepth 1 -type f | sort -n -z | head -n $dir_size | xargs -i mv "{}" "$dir_name$i" done
Improve this question
0

1 Answer 1

Reset to default
0
  • Make a list of your files (note that you cannot use ls * or ls*.dpx because this would create a command line with 176000 parameters to ls) and use split to split this list in chunks (files named files.01, files.02...)(the lone - is telling split to work on standard input).
ls | split -d --lines 20000 - files.
  • For more stringent filtering, you can also use find as you did, piping output to split after a sort:
find . type f | sort | split -d --lines 20000 - files.
  • Iterate the produced files.*, create the directory name using the numeric part, and then move the files:
for f in files.* # iterate the files.* do dirname="/path/to/dir${f#*.}" # generate directory name from numeric suffix mkdir -p "$dirname" # create the directory xargs mv -t "$dirname" < $f # move the files done

Warning: untested, try on a small sample first.

If you have spaces in the file names use find -print0, sort -z, split --separator '\0' and xargs -0.

Improve this answer
6
  • So what would the full syntax be? I'm fairly new with bash I just dont want to break the whole script :( I know I'm close Commented Feb 25, 2020 at 18:03
  • See augmented answer Commented Feb 25, 2020 at 19:21
  • thanks man! in this line, how can I specify it to make the folders in the current working directory vs a specified path? dirname="/path/to/dir${f#*.}" # generate directory name from numeric suffix Commented Feb 26, 2020 at 17:56
  • Basically I want it to create the 'split' folders in the working directory Commented Feb 26, 2020 at 18:14
  • ` dirname="dirprefix${f#*.}"` where "dirprefix" is anything you like. Commented Feb 26, 2020 at 18:55

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.