7

Suppose we have these two files:

$ cat ABC.txt ABC DEF $ cat PQR.txt PQR XTZ

And we want to form a new file with the 1st column of each file. This can be achieved by:

$ paste -d ' ' <(cut -d ' ' -f 1 ABC.txt) <(cut -d ' ' -f 1 PQR.txt ) ABC PQR

But I want to use this with tons of files in the input, not only ABC.txt and PQR.TXT, but a lot of them. How we can generalize this situation to pass each file in the collection to cut and then pass all the outputs to paste (I know that this may be done better with awk but I want to know how to solve this using this approach).

Edit 1

I have discovered a dirty, dirty way of doing this:

$ str=''; for i in *.txt; \ do str="${str} <(cut -d ' ' -f 1 ${i})"; \ done ; \ str="paste -d ' ' $str"; \ eval $str

But please, free my soul with an answer that does not involve going to Computer Science Hell.

Edit 2

Each file can have n rows, if this matters.

Improve this question
2
  • do you have only one row for each file? Commented Apr 21, 2016 at 18:28
  • No, each file have n rows. Commented Apr 21, 2016 at 18:28

3 Answers 3

Reset to default
5

Process substitution <(somecommand) doesn't pipe to stdin, it actually opens a pipe on a separate file descriptor, e.g. 63, and passes in /dev/fd/63. When this "file" is opened, the kernel* duplicates the fd instead of opening a real file.

We can do something similar by opening a bunch of file descriptors and then passing them to the command:

# Start subshell so all files are automatically closed ( fds=() n=0 # Open a new fd for each process subtitution for file in ./*.txt do exec {fds[n++]}< <(cut -d ' ' -f 1 "$file") done # fds now contain a list of fds like 12 14 # prepend "/dev/fd/" to all of them parameters=( "${fds[@]/#//dev/fd/}" ) paste -d ' ' "${parameters[@]}" )

{var}< file is bash's syntax for dynamic file descriptor assignment. like var=4; exec 4< file; but without having to hardcode the 4 and instead let bash pick a free file descriptor. exec opens it in the current shell.

* Linux, FreeBSD, OpenBSD and XNU/OSX anyways. This is not POSIX, but neither is <(..)

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Nicely done; it's worth mentioning that the {var} method of defining file descriptors requires Bash 4.1+.
Thanks for the great answer! By the way i suppose you mean var=4; exec 4< file;, isn't it?
1

Given space delimited input files, and provided ':' is a safe delimiter, (i.e. if there are no colons in the input), this paste to sed one-liner works:

paste -d':' *.txt | sed 's/ [^:]*$//;s/ [^:]*:*/ /g;s/://g'

(POSIX, with no eval, exec, bashisms, subshells, or loops.)

Improve this answer

6 Comments

@that-other-guy's and my answers are ~50x as fast as this (tested with 3 10,000,000-line .txt files).
@webb, that's cool, but didn't the OP say he was testing a lot of little files, rather than a few big files? A benchmark for 10,000,000 3-line text files might be more relevant.
interesting point. your answer is 50x faster for 2000 single-line files, e.g., 40,000 files/second vs 800 files/second for @that-other-guy's and my answers! additionally, all three answers fail completely for e.g. 3000 (or more) files.
@webb, that's a surprise, 50x on opposite sides... a good illustration of algorithms for file length vs. file size. If time permits, do provide more detail in what way all three answers "fail completely" at some point. Did they slow down, return wrong answers, or what?
they fail because of too many open files (yours) or file descriptors (mine & @that-other-guy's).
|
1

After a closer look, I see that @that-other-guy's answer is awesome, but here also is another dirty dirty way that's roughly the same under the hood.

eval "paste -d' ' "$(find *.txt -printf " <(cut -d' ' -f1 '%f')")
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.