How may I generalize an awk command into a script? (extracting/rearranging columns from file)

You can use bash's getopts (you have to scroll down a little bit) to do some command line parsing:

#!/bin/bash delimiter=: first=1 second=2 while getopts d:f:s: FLAG; do case $FLAG in d) delimiter=$OPTARG;; f) first=$OPTARG;; s) second=$OPTARG;; *) echo error >&2; exit 2;; esac done shift $((OPTIND-1)) awk -F"$delimiter" -v "OFS=$delimiter" -v first="$first" -v second="$second" '{ print $first OFS $second }' "$@" 

The following shell script takes an optional -d option to set the delimiter (tab is default), as well as a non-optional -c option with a column specification.

The column specification is similar to that of cut but also allows for rearranging and duplicating the output columns, as well as specifying ranges backwards. Open ranges are also supported.

The file to parse is given on the command line as the last operand, or passed on standard input.

#!/bin/sh delim='\t' # tab is default delimiter # parse command line option while getopts 'd:c:' opt; do case $opt in d) delim=$OPTARG ;; c) cols=$OPTARG ;; *) echo 'Error in command line parsing' >&2 exit 1 esac done shift "$(( OPTIND - 1 ))" if [ -z "$cols" ]; then echo 'Missing column specification (the -c option)' >&2 exit 1 fi # ${1:--} will expand to the filename or to "-" if $1 is empty or unset cat "${1:--}" | awk -F "$delim" -v cols="$cols" ' BEGIN { # output delim will be same as input delim OFS = FS # get array of column specs ncolspec = split(cols, colspec, ",") } { # get fields of current line # (need this as we are rewriting $0 below) split($0, fields, FS) nf = NF # save NF in case we have an open-ended range $0 = ""; # empty $0 # go through given column specification and # create a record from it for (i = 1; i <= ncolspec; ++i) if (split(colspec[i], r, "-") == 1) # single column spec $(NF+1) = fields[colspec[i]] else { # column range spec if (r[1] == "") r[1] = 1 # open start range if (r[2] == "") r[2] = nf # open end range if (r[1] < r[2]) # forward range for (j = r[1]; j <= r[2]; ++j) $(NF + 1) = fields[j] else # backward range for (j = r[1]; j >= r[2]; --j) $(NF + 1) = fields[j] } print }' 

There's a slight inefficiency in this as the code needs to re-parse the column specification for each new line. If support for open-ended ranges is not needed, or if all lines are assumed to have exactly the same number of columns, only a single pass over the specification can be done in the BEGIN block (or in a separat NR==1 block) to create an array of fields that should be outputted.

Missing: Sanity check for column specification. A malformed specification string may well cause weirdness.

Testing:

$ cat file 1:2:3 a:b:c @:(:) 

$ sh script.sh -d : -c 1,3 <file 1:3 a:c @:) 

$ sh script.sh -d : -c 3,1 <file 3:1 c:a ):@ 

$ sh script.sh -d : -c 3-1,1,1-3 <file 3:2:1:1:1:2:3 c:b:a:a:a:b:c ):(:@:@:@:(:) 

$ sh script.sh -d : -c 1-,3 <file 1:2:3:3 a:b:c:c @:(:):) 

Thanks for replies. Here is my script. I created it by trial and error which doesn't often lead to a working solution, and don't have a systematic way of coming up with a script which I always aim at. Please provide some code review if you can. Thanks.

The script works in the following examples (not sure if works in general):

$ projection -d ":" /etc/passwd 4 3 6 7 $ projection -d "/" /etc/passwd 4 3 6 7 

Script projection is:

#! /bin/bash # default arg value delim="," # CSV by default # Parse flagged arguments: while getopts "td:" flag do case $flag in d) delim=$OPTARG;; t) delim="\t";; ?) exit;; esac done # Delete the flagged arguments: shift $(($OPTIND -1)) inputfile="$1" shift 1 fs=("$@") # prepend "$" to each field number fields=() for f in "${fs[@]}"; do fields+=(\$"$f") done awk -F"$delim" "{ print $(join_by.sh " \"$delim\" " "${fields[@]}") }" "$inputfile" 

where join_by.sh is

#! /bin/bash # https://stackoverflow.com/questions/1527049/join-elements-of-an-array # https://stackoverflow.com/a/2317171/ # get the separator: d="$1"; shift; # interpolate other parameters by teh separator # by treating the first parameter specially echo -n "$1"; shift; printf "%s" "${@/#/$d}";