2

I have created a text file with a list of file names like below 

022694-39.tar 022694-39.tar.2017-05-30_13:56:33.OLD 022694-39.tar.2017-07-04_09:22:04.OLD 022739-06.tar 022867-28.tar 022867-28.tar.2018-07-18_11:59:19.OLD 022932-33.tar

I am trying to read the file line by line then strip anything after .tar with awk and use this to create a folder unless it exists.

Then the plan is to copy the original file to the new folder with the original full name stored in $LINE.

$QNAP= "Path to storage" $LOG_DIR/$NOVA_TAR_LIST= "Path to text file containing file names" 
while read -r LINE; do CURNT_JOB_STRIPED="$LINE | `awk -F ".tar" '{print $1}'`" if [ ! -d "$QNAP/$CURNT_JOB_STRIPED" ] then echo "Folder $QNAP/$CURNT_JOB_STRIPED doesn't exist." #mkdir "$QNAP/$CURNT_JOB_STRIPED" fi done <"$LOG_DIR/$NOVA_TAR_LIST"

Unfortunately this seems to be trying to join all the file names together when trying to create the directories rather than doing them one by one and I get a File name too long

output:

......951267-21\n951267-21\n961075-07\n961148-13\n961520-20\n971333-21\n981325-22\n981325-22\n981743-40\n999111-99\n999999-04g\n999999-44': File name too long

Apologies if this is trivial, bit of a rookie...

Improve this question

2 Answers 2

Reset to default
1

Try modifying your script as follows:

CURNT_JOB_STRIPED=$(echo "${LINE}" | awk -F ".tar" '{print $1}')

You have to use $(...) for command substitution. Also, you should print the variable LINE in order to prevent the shell from interpreting its value as a command but passing it to the next command of the pipe (as an input) instead. Finally, you should remove the backticks from the awk expression (this is the deprecated syntax for command substitution) since what you want is the result from the piping commands.

For further information, take a look over http://tldp.org/LDP/abs/html/commandsub.html

Alternatively, and far less readable (neither with a higher performance, thus just as a "curiosity"), you can just use instead of the whole while loop:

xargs -I{} bash -c 'mkdir -p "${2}/${1%.tar*}"' - '{}' "${QNAP}" < "${LOG_DIR}/${NOVA_TAR_LIST}"
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Hi danlor thanks for the solution and additional info.... i think i will skip the xargs one though, as you said far less readable.
1

The problem is with the CURNT_JOB_STRIPED="$LINE | `awk -F ".tar" '{print $1}'`" line.

The `command` is legacy a syntax, $(command) should be used instead.

$LINE variable should be printed so awk can receive its value trough a pipe.

If you run the whole thing in a sub shell ( $(command) ) you can assign the output into a variable: var=$(date)

Is is safer to put variables into ${} so if there is surrounding text you will not get unexpected results.

This should work: CURNT_JOB_STRIPED=$(echo "${LINE}" | awk -F '.tar' '{print $1}')

With variable substitution this can be achieved with more efficient code, and it also clean to read I believe.

Variable substitution is not change the ${LINE} variable so it can be used later as the variable that have the full filename unchanged while ${LINE%.tar*} cut the last .tar text from the variable value and with * anything after that.

while read -r LINE; do if [ ! -d "${QNAP}/${LINE%.tar*}" ] then echo "Folder ${QNAP}/${LINE%.tar*} doesn't exist." #mkdir "${QNAP}/${LINE%.tar*}" fi done <"${LOG_DIR}/${NOVA_TAR_LIST}"

This way you not store the directory name as variable and ${LINE} only store the filename. If You need it into a variable you can do that easily: var="${LINE%.tar*}"

Variable Substitution:

There is more i only picked this 4 for now as they similar and relevant here.

${var#pattern} - Use value of var after removing text that match pattern from the left

${var##pattern} - Same as above but remove the longest matching piece instead the shortest

${var%pattern} - Use value of var after removing text that match pattern from the right

${var%%pattern} - Same as above but remove the longest matching piece instead the shortest

Improve this answer

5 Comments

Note that all-caps names are used for variables meaningful to the shell and POSIX-specified utilities; application-defined names should have at least one lowercase character. See fourth paragraph of pubs.opengroup.org/onlinepubs/9699919799/basedefs/… for the relevant specification.
(I don't, personally, tend to agree that "${foo}" is any safer than "$foo" except in clearly-identifiable circumstances -- $foo_suffix is a very clear and explicit case -- but... folks' style views differ; to-may-toe, to-mah-toe. OTOH, I recently spotted a security bug caused by the typo $foo} causing a test to operate against the wrong filename, so arguably, increasing the amount of syntax used increases the number of potential mistakes).
...the above nitpicks notwithstanding, this answer has very good advice overall; welcome to the site!
...oh, you might consider <<<"$LINE" instead of echo "$LINE" |, as well; counterintuitive as it may seem, setting up a tempfile for a heredoc is often faster than forking a subshell to handle the write end of a pipeline. See the APPLICATION USAGE and RATIONALE sections of the POSIX spec for echo.
@CharlesDuffy Yes I missed to mention var names with capital letters are something that we should avoid as well. I thought about mention <<< operator but I had to work older systems where that not exist, so I decided to avoid to do so, to keep code more portable.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.