0

I'm trying to understand the need for using nohup with background commands in ssh. My shell is csh on CentOS.

  1. The background command below continues running even after ssh exits. I was expecting this to only happen if nohup was prefixing the command.

    What's the scenario where nohup would be needed?

     ssh host 'sleep 80 >& /dev/null &' 
  2. I also tried an interactive shell, and the PID of the background job still exists on host after the ssh exits.

     ssh host sleep 80 >& /dev/null & exit 
  3. I also tried terminating the interactive session with kill -HUP PID rather than exit, and the PID of the background job still exists on host after the ssh exits.

Anything I'm doing wrong?

1 Answer 1

1

No, a background process group (job) is NOT killed by default when the session leader (the shell) exits, or when its controlling terminal is torn down.

There are only some special cases when that happens:

(1) The background job is stopped, in which case it will be sent a SIGHUP/SIGCONT pair of signals by the kernel. If the SIGHUP signal is not caught or ignored by a process, the process will terminate.

The definition of a stopped job is: any job containing a stopped process. A process sleeping on a blocking system call like nanosleep(2) or read(2) is NOT considered stopped.

(2) A process tries to read or write to the terminal which no longer exists, and exits (of its own volition) because of the errors it gets when trying to do so.

(3) The job is actually a foreground job. The kernel sends a SIGHUP signal to the foreground process group when the session leader / controlling process (ie the shell) terminates. The controlling process is itself signaled with SIGHUP when its controlling terminal is torn down, which usually causes it to terminate.

Even commands started with & are actually part of the foreground process group when they're started from a shell with no job control (which in most shells --but not in csh-- is the default when running scripts and subshells).

(4) You're using a shell like bash or zsh, which goes out of its way to send a SIGHUP signal to all its jobs when it's itself signaled with SIGHUP (per point 3. above, the shell being the controlling process), or simply when it exits (the latter only the default in zsh, but not the default and subject to the shopt huponexit option in bash).

The csh shell (either the real csh or tcsh) does not have that behaviour from bash or zsh. In tcsh (but NOT in the real csh) you can start a command with the hup builtin in order to have it hup'ed when the shell exits:

tcsh% hup sleep 3600 & tcsh% exit $ pgrep sleep [nothing] 

(5) Your init system goes out of its way to cleanup any terminated user sessions. In its default configuration, systemd will signal all the processes from a scope with a SIGTERM followed by a SIGKILL after a delay, so nohup will NOT help you with that anyway. Also, systemd's idea of a scope doesn't match a Unix process session, so running a command with setsid(1) will not let it escape it, either.

You can probably change systemd's behaviour by tweaking away from their defaults the KillUserProcesses=yes, KillMode=control-group, KillSignal=SIGTERM and SendSIGKILL=yes options.

13
  • Thanks. So, in real csh: what's the scenario where nohup would be needed? Is it not needed for real csh? Commented Apr 7, 2020 at 17:57
  • (1) and (3). And (5) with a specially configured systemd. Commented Apr 7, 2020 at 17:59
  • FWIW, notice that nohup is a builtin in csh, and does NOT do some of the actions that a standalone nohup program may do (like redirect stdout and stderr, and maybe even stdin). It's a perfectly fine idea to read the csh(1) manpage, too. And in general, nohup is not some magic tool -- it just starts a program with SIGHUP ignored. The standalone nohup is also subject to nasty races. Commented Apr 7, 2020 at 18:38
  • If you find anything dubious in this answer, please put up directly. Don't let it "float". They were already similar questions like this or this. Commented Apr 7, 2020 at 18:51
  • Especially this comment applies here too. Commented Apr 7, 2020 at 18:58

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.