5

I'm trying to build a wrapper to execute a tool multiple times, then concatenate some of the results. I'd like to pass two sets of files to my wrapper script, then run the tool for each pair of files. I'd like it to behave like this:

multitool.sh -a a*.txt -b b*.txt (expanding the wildcards to match all files available)

Then inside multitool.sh, I run the tool on a1.txt b1.txt, a2.txt b1.txt, a1.txt b2.txt, a2.txt b2.txt, etc, with variable numbers of a and b files.

I followed this tutorial explaining the basics of option handling, and I'm able to use getops to handle a -h, but nothing else.

Here is where I'm at now:

#!/bin/bash while getopts ":hpg:" option; do case $option in h) # display help echo "-h to print this help and exit, -p is peak files, -g is bedgraph files." exit;; p) # get list of peak files to process l_peaks=$OPTARG;; g) # get list of bedgraph files to process l_bgraphs=$OPTARG;; \?) # Invalid option echo "Error! Invalid option!" echo "-h to print this help and exit, -p is peak files, -g is bedgraph files." exit;; esac done echo "$l_peaks" echo "$l_bgraphs"

I'm working with a team who aren't particularly computer literate, so if I can keep the wrapper to a simple one line execution, that would be best.

How can I pass these lists of files as one option each?

Improve this question
5
  • Please edit your question and add more details. When you run multitool.sh -a a*.txt -b b*.txt, the shell will normally expand a*.txt to the list of the matching files. So for your script it looks like you ran multitool.sh -a a1.txt a2.txt -b b1.txt b2.txt Is this what you want? Or do you want to get the literal a*.txt in your script? Then you would need to quote the patterns in the command line. Commented Feb 24 at 13:26
  • @Bodo I want it to expand following the pattern Commented Feb 24 at 13:36
  • 1
    Do you really need to build your own? GNU parallel can do this, so can a combination of paste and xargs. I would urge you to ask a new question with the details of what you need to do because I really doubt rolling your own script will be the best solution here. Commented Feb 24 at 18:42
  • @terdon there's more to do in the script than just run a tool with paired inputs, and again I'm writing it from people who don't have much technical knowledge with the command line, so I'd rather write a script with some comments and help rather than a one-liner that someone could mess up during a copy+paste. My boss has a "pipeline" that's a bunch of bash commands written in a word document. Commented Feb 25 at 10:14
  • 1
    I get that, but that's precisely why I would think using standard tools and then writing a simple bash wrapper to call them would be easier. Absolutely nothing wrong with the answer you have though, so if that's enough for your needs, go for it. Commented Feb 25 at 14:12

4 Answers 4

Reset to default
10

No, don't squash multiple filenames into a single string. For one, because the shells don't support that(*), but also because in general, file names can be any strings. There's no character or character sequence you could use as a separator that wouldn't be valid in a filename. Except for the NUL byte, but you can't pass that as a command line argument anyway.

(* With the possible exception of zsh.)

So, just fill two arrays with the filenames you get as arguments. From there, it's also simple to loop over the lists.

Though, getopts will stop when it sees a non-option argument, so we'll have to look manually for some separator string. (Which again, necessarily is something that could be a filename, but at least it'll need to match a complete filename for things to get messed up. Let's pick ::, and if you ever have a filename like that, be sure to pass it as ./::.)

#!/bin/bash while getopts "xy" option; do # silly placeholder echo "option $option" done shift $(( OPTIND - 1 )) list=a lista=() listb=() while [ "$1" ]; do case $1 in ::) list=b;; *) if [ "$list" = a ]; then lista+=("$1") else listb+=("$1") fi;; esac shift done printf "files in list a:\n" printf " <%s>\n" "${lista[@]}" printf "\n" printf "files in list b:\n" printf " <%s>\n" "${listb[@]}" printf "\n" printf "combinations:\n" for b in "${listb[@]}"; do for a in "${lista[@]}"; do printf "<%s> <%s>\n" "$a" "$b" done done

running that gives something like this:

$ bash lists.sh -x a* :: b* option x files in list a: <a1.txt> <a2.txt> <a3.txt> files in list b: <b1.txt> <b2.txt> combinations: <a1.txt> <b1.txt> <a2.txt> <b1.txt> <a3.txt> <b1.txt> <a1.txt> <b2.txt> <a2.txt> <b2.txt> <a3.txt> <b2.txt>

Adjust the getopts loop as necessary, but remember to keep the shift command to clear the options getopts already processed.

That is, I assumed above you wanted to combine first A with all Bs, and second A with all Bs, etc.

If, instead, you want first A with first B, second A with second B, etc., then that last loop would need to be something like:

i=0 # stop when either list ends while [ "${lista[i]}" ] && [ "${listb[i]}" ]; do printf "<%s> <%s>\n" "${lista[i]}" "${listb[i]}" i=$((i + 1)) done

Though, as @terdon noted in a comment, GNU parallel could do this directly.

Like so:

$ parallel echo {1} {2} ::: a*.txt ::: b*.txt a1.txt b1.txt a1.txt b2.txt a2.txt b1.txt a2.txt b2.txt a3.txt b1.txt a3.txt b2.txt # or $ parallel echo {1} {2} ::: a*.txt :::+ b*.txt a1.txt b1.txt a2.txt b2.txt

It passes the given command through a shell, se special characters would need some extra quoting. And of course you need to be careful with the separators, with that extra + being rather meaningful.

Improve this answer
2
  • @Whitehot, hmm. I have no idea about the subject matter your post hinted at, but I took your phrasing to mean you wanted every A paired against every B (so total n*m pairs), while one might also want first with first, second with second etc (total min(n,m) pairs) is. Anyway, edited to add the other one too. Commented Feb 24 at 21:22
  • 1
    In my post I intended to pair every A with every B (like parallel, which I intended to use in the script until I realised I could batch A files together with the tool I am using), which is why I explicitly mentioned the pairs a2.txt b1.txt and a1.txt b2.txt. So you were safe in your assumption :) Commented Feb 25 at 10:11
4

If you're happy to copy the approach used by find and quote the two wildcard glob patterns you can use your script to expand the values appropriately. In this configuration the quoted patterns are then handled by the shell (and therefore by getopts) as only one argument each.

Here's an example, where I've taken your existing code pretty much unchanged and shrunk it for (my) convenience. Your code is laid out better, but the interesting parts are the processing of the -a and -b arguments:

#!/bin/bash # # Multiple arguments: multitool -hp -a 'a*.txt' -b 'b*.txt' # ######################################################################## # while getopts ':hpg:a:b:' option do case "$option" in a) a_list=($OPTARG) ;; # Expand the (quoted) argument to the a_list array b) b_list=($OPTARG) ;; # Expand the (quoted) argument to the b_list array h) echo "HELP!" >&2; exit 1 ;; p) l_peaks="$OPTARG" ;; g) l_bgraphs="$OPTARG" ;; \?) echo "ERROR!" >&2; exit 1 ;; esac done shift $(( OPTIND - 1 )) # Checking values # echo "l_peaks=$l_peaks, l_bgraphs=$l_bgraphs" echo "a_list=(${a_list[@]})" echo "b_list=(${b_list[@]})" echo for ((i=0; i<="${#a_list[@]}"; i++)) do printf '%d\t%s\t%s\n' "$i" "${a_list[$i]}" "${b_list[$i]}" done

Example

ls aapple abanana acherry bone bthree btwo ./multitool -p -a 'a*' -b 'b*' l_peaks=, l_bgraphs= a_list=(aapple abanana acherry) b_list=(bone bthree btwo) 0 aapple bone 1 abanana bthree 2 acherry btwo
Improve this answer
3

One of the common approaches commands handle options that take more than one arguments, is to call the options multiple times as in:

cmd -vvi file1 -i file2 -ifile3 -o out1 -ifile4

(above showing examples where the argument is provided as a separate argument or in the same argument)

To apply that to globs, if using zsh, you could do:

cmd -vv file*(nP[-i]) -o out1

Where the P[prefix] glob qualifier inserts an extra -i argument before each generated file so it ends up running cmd -vv -i file1 -i file2 ... -o out.

(also note the n qualifier to make sure files are sorted numerically so file10 comes after file9 and not between file1 and file2 as it would in the default lexical order).

Or:

cmd -vv file*(ne:REPLY[1,0]=-i:) -o out1

Or:

files=(file*(n)); cmd -vv -i$^files -o out1

Or, with the histsubstpattern option enabled:

cmd -vv file*(n:s/#/-i) -o out1

To have -i prepended (in the same argument) to each file name, so it ends up running cmd -vv -ifile1 -ifile2 ... -o out1.

In bash (or ksh93 where bash copied that syntax from), you can do:

files=(file*); cmd -vv "${files[@]/#/-i}" -o out

Though beware that bash has no equivalent for zsh's n qualifier so file10 will be sorted between file1 and file2, and unless you turn the failglob option on, that will call cmd with a literal -ifile* as argument if there's no matching file.

In the cmd script itself, you'd do something like:

inputs=() while getopts i: option; do case $option in (i) inputs+=( "$OPTARG" );; # rest of option parsing esac done echo "I got ${#inputs[@]} input files"
Improve this answer
0

We know that the list of files will be an even number, given that they should be paired up with each other. Using this, we can use a loop over all given filenames, and while we're iterating over them, we count them. Once we have reached the mid-point, we can start pairing up the first name in the list with the current name in the loop, then shift (which removes the first list item) and repeat.

This works because a for loop always iterates over a static list. We thus "save" the list of names by looping over it, and can modify the list of positional parameters inside the loop without disturbing the loop itself.

#!/bin/sh n=$(( $# / 2 )) for name do if [ "$n" -le 0 ]; then # The pair is "$1" (from the 1st set of files) # and "$name" (from the 2nd set of files). printf 'a-file = "%s"\tb-file = "%s"\n' "$1" "$name" shift fi n=$(( n - 1 )) done

Note that none of this requires the bash shell; we can run it with a simpler sh shell.

Testing:

$ ls a1.txt a2.txt a4.txt a6.txt a8.txt b1.txt b2.txt b4.txt b6.txt b8.txt script a10.txt a3.txt a5.txt a7.txt a9.txt b10.txt b3.txt b5.txt b7.txt b9.txt 
$ sh script a* b* a-file = "a1.txt" b-file = "b1.txt" a-file = "a10.txt" b-file = "b10.txt" a-file = "a2.txt" b-file = "b2.txt" a-file = "a3.txt" b-file = "b3.txt" a-file = "a4.txt" b-file = "b4.txt" a-file = "a5.txt" b-file = "b5.txt" a-file = "a6.txt" b-file = "b6.txt" a-file = "a7.txt" b-file = "b7.txt" a-file = "a8.txt" b-file = "b8.txt" a-file = "a9.txt" b-file = "b9.txt"

If you need to do option parsing, do that before the code that pairs up the filenames:

#!/bin/sh myself=$0 usage () { cat <<-USAGE_MESSAGE USAGE: $myself -h $myself -- {File set A} {File set B} -h Display usage info File set A Filenames for file set A File set B Filenames for file set B USAGE_MESSAGE } while getopts h opt; do case $opt in h) usage exit ;; *) printf '%s: Error in option parsing\n' "$myself" >&2 exit 1 esac done shift "$(( OPTIND - 1 ))" if [ "$#" -eq 0 ]; then usage >&2 exit 1 fi n=$(( $# / 2 )) for name do if [ "$n" -le 0 ]; then printf 'a-file = "%s"\tb-file = "%s"\n' "$1" "$name" shift fi n=$(( n - 1 )) done
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.