2

I'm using GNU parallel to execute computation heavy transformations on remote hosts that are more powerful than the one requesting the work.

I am using a bash script located in /usr/local/lib/myscript, and I'd like to use it on the remote hosts. However, I don't want to require the hosts to have this script, so I'm sending it along the work with --basefile.

If I supply the filename only, all goes well. But if I supply the full path to a basefile, it doesn't work.

To illustrate:

$ cat /tmp/common.sh #!/usr/bin/env bash echo "Hello world! from $(hostname)" $ cd /tmp $ parallel --nonall -S 2/user@remote-host --basefile common.sh --cleanup bash common.sh Hello world! from remote-host $ parallel --nonall -S 2/user@remote-host --basefile /tmp/common.sh --cleanup "bash {}" could not make way for new symlink: tmp rsync error: some files could not be transferred (code 23) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/main.c(992) [sender=2.6.9] cannot delete non-empty directory: tmp

Is there a way to use --basefile with files outside of the current path? Or should I use a second --trc and take advantage of the {1} and {2} substitution patterns to emulate --basefile?

Improve this question
1
  • Use /./ in the path. Commented Nov 28, 2016 at 8:36

1 Answer 1

Reset to default
2

Transferring of files (i.e. --basefile, --transfer-file, --transfer, --return) in GNU Parallel uses rsync's /./ magic. So when you ask to transfer /tmp/common.sh it will be put in /tmp/common.sh on the remote side. But if you ask to transfer /tmp/./common.sh it will transfer /tmp/common.sh to $(pwd)/..

In other words:

(local file) => (remote file) dir/file => ./dir/file /tmp/sub/dir/file => /tmp/sub/dir/file /tmp/sub/./dir/file => ./dir/file

The error you see is probably due to /tmp being a symlink on the remote system.

If you use --workdir ... you will have new workdir for each job. As --basefile only copies the file before the first job is run, this will not work.

Instead you can transfer common.sh for each job:

parallel --wd ... --tf {1} -S server 'pwd ; ls; echo {2}' ::: /tmp/./common.sh ::: foo bar
Improve this answer
3
  • I don't think I understand how this works. If I observe the remote system where the command is executed, I can see the basefile transferred for the first job in ~/.parallel/tmp/host-1234/. But for the second and third jobs, only the files after ::: get transferred, the basefile isn't. And so the first job completes, cleans up, and the two other jobs fail because they don't have the basefile they need. I managed to sort out the /./ magic, but only the first job gets completed and retrieved. Commented Nov 28, 2016 at 22:26
  • Then please update the question: You forgot to mention you use ':::' and '--workdir' and not '--nonall'. It makes it harder to find a working solution for you if you leave that out. Commented Nov 29, 2016 at 7:59
  • My bad, I tried to simplify the command so readers don't get caught up in the details, I didn't understand workdir ... would have an impact. Commented Nov 29, 2016 at 15:29

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.