2

I realize this might be hard to answer without you knowing how my cluster is setup, but I' am trying to submit jobs (via SGE) to a cluster, but the environment is not setup correctly and the jobs fails. Moreover, there are two different master nodes I can login into to submit jobs to the same cluster, and my scripts work on one while not on the other.

The is the machine info for the master node that my script does work on:

cat /proc/version Linux version 2.6.32-279.el6.x86_64 ([email protected]) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Wed Jun 13 18:24:36 EDT 2012

The machine it does not work on:

cat /proc/version Linux version 3.10.0-514.6.2.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Thu Feb 23 03:04:39 UTC 2017

Here is a test script I' am using:

#!/bin/bash -I #$ -wd ~ #$ -N test #$ -o ~/test.log #$ -j y #$ -terse #$ -V #$ -notify #$ -l h_vmem=2G -pe smp 1 -l athena=true ls hostname nproc

Here is the output after running "qsub test.sh":

/bin/bash: module: line 1: syntax error: unexpected end of file /bin/bash: error importing function definition for `BASH_FUNC_module' /opt/sge/default/spool/execd/node156/job_scripts/1063646: line 11: ls: command not found /opt/sge/default/spool/execd/node156/job_scripts/1063646: line 12: hostname: command not found

To add to the confusion, when I ssh directly into those job nodes (node156 in the above example) I can run the ls and hostname commands just fine!

I've been in contact with the cluster admins, and they are unable to replicate my issue (even if they login as me). We first tested that if setting ~/.bashrc and ~/.bash_profile to default settings would fix it, but it did not. Here are those files:

cat ~/.bashrc # .bashrc # Source global definitions if [ -f /etc/bashrc ]; then . /etc/bashrc fi

.bash_profile:

cat ~/.bash_profile # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs

Any suggestions?

Improve this question

1 Answer 1

Reset to default
0

I don't have a full solution, because I don't know anything about SGE. But I can explain part of the problem.

The machine where your script works is running an old version of the operating system. This is evident not only by the kernel version number, but also by the fact that it hasn't received security updates in a while. Specifically, I think it's running a version of bash that's vulnerable to the Shellshock bug.

Bash (ab)uses the environment to pass functions. Normally the environment is only used to pass data, in the form of a series of items of the form NAME=VALUE. Older versions of bash add items of the form NAME=() {CODE}, which in some circumstances allowed injecting code by defining a variable that a script would never use — the shellshock bug. The fix for the bug changed the way functions are encoded to BASH_FUNC_NAME%%=() {CODE}.

Evidently some part of your setup dumps out the environment and parses it. This may either be a part of SGE or something specific to your setup. A plausible reason to do this is to save the environment in which a job was submitted, to execute the job in the same environment.

Something somewhere is defining a function called module in bash, and exporting it. The code would look something like

module () { … } export -f module

The fix is either to upgrade the environment parser to something that can cope with the new bash encoding, or to stop exporting functions.

Improve this answer
1
  • This is the answer, thanks! "declare -F" shows that there is a function named "module" defined in the environment on the machine the code does not work on. It is defined in the "old" way as shown at the bottom of your post. Moreover, the -V option in my SGE job script copies tells the tool to copy over all environment variables. Removing -V fixes the issue. Commented May 30, 2017 at 15:48

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.