nohup and keeping work alive

The hangup

You start a three-hour job over SSH with &, close the laptop, and the next morning the job is dead. The culprit is SIGHUP, signal 1, named after the modem *hang-up* of the days when terminals were connected by phone line. When a terminal goes away, the kernel sends HUP to the session's leader, your shell, and the shell passes it on to its jobs. HUP's default action is to terminate, so every job started from that terminal dies with it, background or not. The & only freed your keyboard; it never detached the job from the terminal.

nohup: ignore the hangup

nohup CMD sets HUP to *ignored* and then runs CMD in its own place. Ignored signals stay ignored across exec, so CMD and everything it starts shrug the hangup off. nohup also worries about output: if standard output is a terminal, which will soon disappear, it sends the output to a file called nohup.out in the current directory (or $HOME/nohup.out if that is not writable), and standard error follows it. nohup does not put the command in the background; you still add &. The professional form names its own log so nothing lands in a surprise nohup.out:

~% nohup sh -c 'echo started; sleep 600' &
~% cat nohup.out
started
~% nohup sleep 1500 > /root/lab/job.log 2>&1 &
[1] 946
~% grep SigIgn /proc/946/status
SigIgn:	0000000000000001

Proving it with /proc and killall -HUP

Every process's /proc/PID/status has a SigIgn line: a hexadecimal bit mask of the signals it ignores, where bit 1 (value 1) is HUP, value 2 is INT and value 4 is QUIT. If the last hex digit is odd, HUP is ignored and the process will survive a hangup. Depending on how it was started you may see …0006 for a plain background job (the shell made it ignore INT and QUIT) and …0007 for the same job under nohup. Or simply test it: start one plain and one nohup'd sleep, then killall -HUP sleep. Only the protected one is still in ps afterwards.

setsid: leave the terminal behind

~% setsid sleep 1502 &
~% ps -o pid,pgid,sid,tty,args | grep '[s]leep 1502'
  954   954   954 ?      sleep 1502

setsid CMD goes further: it runs CMD in a brand-new session with no controlling terminal, so there is no terminal whose hangup could reach it, and Ctrl-C in your terminal never touches it either. ps shows its session ID equal to its own PID and its TTY as ?. Inside a script, trap '' HUP makes the script itself ignore the hangup, just like nohup. bash users may know disown; ash does not have it. For anything that must run for days, the right home is a service managed by init, or a terminal multiplexer such as tmux or screen on a full system.