job control
Suspend, resume and detach what you started
A job is one thing your shell started, and it is not the same unit as a process: a pipeline of
four commands is four processes and one job. jobs, fg, bg and disown are shell builtins
that work on that list, so they only ever know about commands this shell started. Another
terminal's background job is invisible to them, and only ps sees across the
whole machine.
The numbers in brackets are not PIDs
[1] is a job number, counted per shell and reused as jobs finish. Anywhere a command takes a
job, %1 names it, %+ is the current job and %- the previous one, and %str matches a job
whose command begins with str. The + and - in the jobs output mark those last two.
kill accepts a job spec as well as a PID, so kill %1 and kill 4749 can be
the same instruction. $! holds the PID of the job you started most recently. Save it
into a variable if you want to signal that job later.
None of this works in a script
Job control is a feature of an interactive shell, and bash switches it off everywhere else. A
script that calls fg gets no job control and stops; %1 is not understood; Ctrl-Z has no
meaning where there is no terminal to send it. set -m turns it back on, and is occasionally the
right answer, but a script that wants to manage several children usually wants $! and wait
instead.
The examples below use set -m wherever a job has to be suspended, because otherwise the shell
never notices it happened.
Ctrl-Z, and what comes after it
Ctrl-Z suspends the foreground job by sending it TSTP. The process stops where it is, keeping
its memory and its open files, and does nothing until something resumes it. fg resumes it in the
foreground, bg resumes it in the background, and both send CONT to do so.
The usual reason to want it is that you started something long without an & and would like your
prompt back. Ctrl-Z, then bg, then disown if you also intend to close the terminal.
Keeping a program running after you log out has
the case where that is not good enough.
The jobs table
Every example here starts its own jobs, so there is no sample data to declare. They all send the job's output to /dev/null, as you would with a real background job: anything it prints otherwise lands in the middle of whatever you are typing. jobs echoes the command back exactly as you gave it, redirect included.
List the shell's jobs
sleep 300 >/dev/null 2>&1 & jobs& starts the command in the background and the shell adds it to a table. jobs prints that table: the job number in brackets, the state, and the command as you typed it.
Show output
[1]+ Running sleep 300 > /dev/null 2>&1 &
Read the plus and minus markers
sleep 300 >/dev/null 2>&1 & sleep 200 >/dev/null 2>&1 & sleep 100 >/dev/null 2>&1 & jobs+ marks the current job and - the one before it. Anything that takes a job spec and is given none acts on the + job, so those two characters decide what a bare fg or bg will do.
Show output
[1] Running sleep 300 > /dev/null 2>&1 &
[2]- Running sleep 200 > /dev/null 2>&1 &
[3]+ Running sleep 100 > /dev/null 2>&1 &
Start a pipeline in the background
sleep 300 2>&1 | cat >/dev/null 2>&1 & jobsThe two programs share a single entry. A whole pipeline counts as one job, so suspending or killing it takes every stage with it instead of leaving cat reading from a pipe nothing writes to.
Show output
[1]+ Running sleep 300 2>&1 | cat > /dev/null 2>&1 &
See the processes inside that one job
sleep 300 2>&1 | cat >/dev/null 2>&1 & jobs -l-l breaks the job open: one PID per stage, indented under the job number. Job and process are different units, and this is the listing where that stops being an abstract point.
Show output
Your output will differ: the PIDs will differ, and their width sets the indentation of the second line
[1]+ 196 Running sleep 300 2>&1
197 | cat > /dev/null 2>&1 &
Show the PID beside each job
sleep 300 >/dev/null 2>&1 & sleep 200 >/dev/null 2>&1 & jobs -l-l adds the process id. That is the column you need when handing a job to something outside the shell.
Show output
Your output will differ: the PIDs will differ
[1]- 4749 Running sleep 300 > /dev/null 2>&1 &
[2]+ 4750 Running sleep 200 > /dev/null 2>&1 &
Print only the PIDs
sleep 300 >/dev/null 2>&1 & sleep 200 >/dev/null 2>&1 & jobs -p-p drops everything but the PIDs, a line each, ready to pipe into kill or xargs.
Show output
Your output will differ: the PIDs will differ
4759
4760
Ask about one job
sleep 300 >/dev/null 2>&1 & sleep 200 >/dev/null 2>&1 & jobs %2A job spec narrows the listing to one entry.
Show output
[2]+ Running sleep 200 > /dev/null 2>&1 &
Separate the running from the stopped
set -m; sleep 300 >/dev/null 2>&1 & sleep 200 >/dev/null 2>&1 & kill -STOP %1; sleep 0.3; jobs -r; echo ---; jobs -s-r lists only running jobs and -s only stopped ones. Note that stopping job 1 also made it the current job: the + moved.
Show output
[2]- Running sleep 200 > /dev/null 2>&1 &
---
[1]+ Stopped sleep 300 > /dev/null 2>&1
See a job that has finished
sleep 0.2 & sleep 0.5; jobsA completed job stays in the table until it is reported once, then disappears. That is why an interactive shell prints [1]+ Done at some arbitrary later moment rather than the instant the job ended: it tells you the next time it draws a prompt.
Show output
[1]+ Done sleep 0.2
See a job that was signalled
sleep 300 >/dev/null 2>&1 & kill %1; sleep 0.3; jobsTerminated rather than Done, because the job did not choose to exit. A job killed with -9 reports Killed here instead.
Show output
[1]+ Terminated sleep 300 > /dev/null 2>&1
Naming a job
Signal a job by number
sleep 300 >/dev/null 2>&1 & kill %1; sleep 0.3; jobskill takes %1 wherever it takes a PID. The shell resolves the spec and signals the job's process group, so a suspended pipeline goes as one thing rather than a stage at a time.
Show output
[1]+ Terminated sleep 300 > /dev/null 2>&1
Name the current and previous jobs
sleep 300 >/dev/null 2>&1 & sleep 200 >/dev/null 2>&1 & jobs %+; jobs %-%+ and %- are the two the markers point at. %% is another spelling of %+.
Show output
[2]+ Running sleep 200 > /dev/null 2>&1 &
[1]- Running sleep 300 > /dev/null 2>&1 &
Name a job by how its command starts
sleep 300 >/dev/null 2>&1 & jobs %sle%str matches the job whose command line begins with str. It is an error rather than a guess if two jobs match, so this is safe to use interactively and unwise in a script.
Show output
[1]+ Running sleep 300 > /dev/null 2>&1 &
Name a job by any part of its command
sleep 300 >/dev/null 2>&1 & kill %?300; sleep 0.3; jobs%?str matches on a substring rather than a prefix. Use it when the interesting part of the command is an argument.
Show output
[1]+ Terminated sleep 300 > /dev/null 2>&1
Capture the PID of the job you just started
sleep 300 >/dev/null 2>&1 & p=$!; kill "$p"; sleep 0.3; jobs$! is the PID of the most recent background command. Save it into a variable straight away, because the next & overwrites it, and a script that wants to signal one particular child later has nothing else to go on.
Show output
[1]+ Terminated sleep 300 > /dev/null 2>&1
Wait for a specific job
sleep 1 & p=$!; wait $p; echo "exit was $?"wait blocks until the job finishes and then reports its exit status. A script that fans work out collects it back this way.
Show output
exit was 0
Wait for everything
sleep 0.5 & sleep 1 & wait; echo "all done"With no argument, wait returns when every background job has finished. This is the last line of most scripts that fan work out.
Show output
all done
Wait for whichever finishes first
sleep 0.5 & sleep 2 & wait -n; echo "first finished"wait -n returns as soon as any one job ends. A script keeping a fixed number of workers busy needs that, since waiting for the whole batch idles every worker until the slowest finishes.
Show output
first finished
Suspending and resuming
Ctrl-Z sends TSTP to the foreground job. No example can press it, so these use kill -STOP, the same thing minus the ability to be caught. All of them set set -m first, for the reason the next section covers.
Suspend a running job
set -m; sleep 300 >/dev/null 2>&1 & kill -STOP %1; sleep 0.3; jobsThe job stops where it is. It keeps its memory, its open files and any lock it was holding, so a suspended job is not a cheap thing to leave lying about.
Show output
[1]+ Stopped sleep 300 > /dev/null 2>&1
Resume it in the background
set -m; sleep 300 >/dev/null 2>&1 & kill -STOP %1; sleep 0.3; bg %1; sleep 0.3; jobsbg sends CONT and leaves the job in the background. It echoes the job it acted on, and that is the first line here.
Show output
[1]+ sleep 300 > /dev/null 2>&1 &
[1]+ Running sleep 300 > /dev/null 2>&1 &
Resume it in the foreground
set -m; sleep 1 & fgfg prints the command it is resuming and then waits for it, so the shell has no prompt until the job ends. A one-second job is used here for the obvious reason.
Show output
sleep 1
Bring back the current job with no argument
set -m; sleep 1 & sleep 300 >/dev/null 2>&1 & fg %1A bare fg takes the + job: the most recently started or suspended one, rather than the one you were last thinking about. Name the job when there is more than one.
Show output
sleep 1
Check that a stopped job is really stopped
set -m; sleep 300 >/dev/null 2>&1 & kill -STOP %1; sleep 0.3; ps -o state= -p $!T in the ps state column, the same letter the kill page uses. The shell's Stopped and the kernel's T are two views of one fact.
Show output
T
Why none of it works in a script
Job control belongs to an interactive shell. Everywhere else bash switches it off, and the failure is quiet in one direction and loud in the other.
Call fg from a script
printf "fg\n" > try.sh; bash try.shThe loud direction. fg, bg and %1 all need job control, and a script has none, so the builtin refuses rather than guessing.
Show output
try.sh: line 1: fg: no job control
Stop a job without job control
sleep 300 >/dev/null 2>&1 & kill -STOP %1; sleep 0.5; jobs; ps -o state= -p $!The quiet direction, and the more dangerous of the two. The process really is stopped, which the T on the second line shows, but the shell still calls it Running: without job control it is not told when a child stops, only when one exits.
Show output
[1]+ Running sleep 300 > /dev/null 2>&1 &
T
Turn job control on
set -m; sleep 300 >/dev/null 2>&1 & kill -STOP %1; sleep 0.5; jobs; ps -o state= -p $!The same script with set -m. Nothing about the process changed; the shell is now watching for the notification it was ignoring before.
Show output
[1]+ Stopped sleep 300 > /dev/null 2>&1
T
Outliving the shell
A background job is still the shell's child, and it is still in the shell's jobs table. Both of those have to be dealt with for it to survive the terminal going away, and the three usual commands each deal with a different one.
Read the mask of signals a background job ignores
sleep 300 >/dev/null 2>&1 &
sleep 0.3
mask=$(awk '/^SigIgn/{print $2}' /proc/$!/status)
printf '%08x\n' $(( 0x$mask & 0x7fffffff ))SigIgn in /proc/<pid>/status is the set of signals a process ignores, as hex, one bit per signal with bit 0 standing for signal 1. 6 is binary 110, so bits 1 and 2, so signals 2 and 3: INT and QUIT. Bash starts every background job that way, which is why Ctrl-C has never reached one. The & 0x7fffffff keeps signals 1 to 31, the standard ones. Above those sit the real-time signals, which the C library claims a couple of for its own use, and which of them show up here is not the same on every machine.
Show output
00000006
See the one bit nohup sets
nohup sleep 300 >/dev/null 2>&1 &
sleep 0.3
mask=$(awk '/^SigIgn/{print $2}' /proc/$!/status)
printf '%08x\n' $(( 0x$mask & 0x7fffffff ))7 rather than 6: bit 0 is now set, so HUP joins the two bash had already blocked. nohup sets that and then executes the program, so the setting is in place before the program runs a single instruction and there is nothing for it to override.
Show output
00000007
Read those bits by name instead of counting them
sleep 300 >/dev/null 2>&1 &
sleep 0.3
mask=$(awk '/^SigIgn/{print $2}' /proc/$!/status)
for sig in HUP INT QUIT; do
echo "$sig $(( (0x$mask >> ($(kill -l $sig) - 1)) & 1 ))"
donekill -l turns a signal name into its number, and the number into the bit to shift down to. The same 6 as two examples ago, spelled out: HUP is the gap, and it is the gap the three commands below each close differently.
Show output
HUP 0
INT 1
QUIT 1
Confirm nohup closed the gap
nohup sleep 300 >/dev/null 2>&1 &
sleep 0.3
mask=$(awk '/^SigIgn/{print $2}' /proc/$!/status)
for sig in HUP INT QUIT; do
echo "$sig $(( (0x$mask >> ($(kill -l $sig) - 1)) & 1 ))"
doneAll three ignored. This is the whole of what nohup does about signals, and it is worth knowing how little that is: the job still dies if anything sends it TERM, and it is still in the shell's jobs table until something takes it out.
Show output
HUP 1
INT 1
QUIT 1
Start something with nohup
nohup ./long-import.sh > import.log 2>&1 &The form to actually type. Without a redirect nohup sends output to nohup.out in the current directory, but only when it started from a terminal, so a redirect you chose yourself is one less thing to be surprised by. There is no output block here because that terminal-only behaviour cannot be reproduced in a pipeline.
Take a job out of the shell's table
sleep 300 >/dev/null 2>&1 & disown; jobs; echo "still running: $(pgrep -c -x sleep)"jobs prints nothing and the process is still there. disown removes the entry, so the shell no longer signals it on exit and no longer reports on it.
Show output
still running: 1
Disown one job by name
sleep 300 >/dev/null 2>&1 & sleep 200 >/dev/null 2>&1 & disown %1; jobsWith a job spec, only that entry goes. Job 2 is still the shell's business.
Show output
[2]+ Running sleep 200 > /dev/null 2>&1 &
Disown everything at once
sleep 300 >/dev/null 2>&1 & sleep 200 >/dev/null 2>&1 & disown -a; jobs; echo "still running: $(pgrep -c -x sleep)"-a clears the table. Both processes are running and neither is the shell's concern any more.
Show output
still running: 2
Keep the job listed but shield it from HUP
sleep 300 >/dev/null 2>&1 & disown -h %1; jobs-h is the middle option: the job stays in the table, so fg and jobs still work on it, but the shell will not send it HUP when it exits. Reach for this when you want the job back later and are only worried about the disconnection.
Show output
[1]+ Running sleep 300 > /dev/null 2>&1 &
Start something in a session of its own
setsid sleep 300 >/dev/null 2>&1 & sleep 0.4; pgrep -c -x sleepsetsid does not touch the jobs table or the signal mask. It puts the process in a new session with no controlling terminal at all, so there is nothing that could hang up on it. This is the mechanism a daemon uses.
Show output
1
Check whether a job survived the shell
bash -c 'sleep 300 >/dev/null 2>&1 & disown' ; sleep 0.4; pgrep -c -x sleepThe inner shell started a job, disowned it and exited. The sleep is still running, now reparented to PID 1. Processes and signals covers what reparenting means for anything you left behind.
Show output
1