Lesson 4 of 8 in Linux and the shell

Processes and signals: see, stop and manage programs

Core11 min

A process is a running copy of a program, with its own process ID and a parent that started it. A signal is a short message the kernel delivers to a process, such as a request to stop, which the process can handle in its own way.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Users, groups, permissions and sudo

The problem this solves

On a server, problems often look like a program that is still running when it should not be, or one that is not responding. You need to see what is running, who started it and how to stop it without damaging data.

Stopping a program is not one action. Some programs need a moment to finish writing a file or close a connection. Others hang and have to be removed by force. Linux gives you different signals for each case.

Knowing the difference between asking and forcing is what keeps a quick fix from corrupting a database or leaving a lock file behind.

How it works

Every process has a parent that started it
Every process has a parent that started itA login through sshd starts your shell, and every command you run is a child of that shell, with systemd as the ancestor of the whole tree.startsstartsstartssystemdPID 1, starts the restsshdhandles your loginbashyour shellsleep 300a command you ran
  • Control
  • Runs code

A login through sshd starts your shell, and every command you run is a child of that shell, with systemd as the ancestor of the whole tree.

  1. systemd to sshd: starts
  2. sshd to bash: starts
  3. bash to sleep 300: starts
What happens when you run kill
What happens when you run killYou run kill with a process ID, the kernel delivers the signal to that process, and a process that handles SIGTERM cleans up and exits.1 run kill2 asks tosend3 SIGTERM arrives4 handlerrunsYoukill <pid>kill commandjust sends a requestKerneldelivers the signalTarget processreceives SIGTERMCleans up, exitsits own handler
  • People and clients
  • Control
  • Runs code

You run kill with a process ID, the kernel delivers the signal to that process, and a process that handles SIGTERM cleans up and exits.

  1. You to kill command: 1 run kill
  2. kill command to Kernel: 2 asks to send
  3. Kernel to Target process: 3 SIGTERM arrives
  4. Target process to Cleans up, exits: 4 handler runs

Every process has a process ID, or PID, and a parent PID. The first process, PID 1, is systemd on most distributions, and it starts everything else. When you log in over SSH, sshd starts your shell, and each command you type becomes a child of that shell.

A signal is a numbered message. kill is badly named, because it only sends a signal. By default it sends SIGTERM, signal 15, which means please terminate. The process can catch it, finish its work and exit cleanly.

SIGKILL, signal 9, cannot be caught or ignored. The kernel removes the process immediately, so it gets no chance to clean up. Treat it as a last resort. Ctrl+C sends SIGINT, and closing a terminal or dropping an SSH connection sends SIGHUP to the jobs started from it.

When a process ends, it leaves an exit code. A process killed by a signal reports 128 plus the signal number, so SIGTERM shows up as 143 and SIGKILL as 137.

Try it yourself

Run these on Linux or WSL in a normal shell. Everything is a harmless sleep or a small script, and each step stops what it starts.

  1. 1
    Find your shell in the process list
    echo $$
    ps -o pid,ppid,stat,comm -p $$

    $$ is the PID of your shell. The ppid column is its parent, which is usually sshd or your terminal program.

  2. 2
    Start something in the background and look at it
    sleep 300 &
    ps -o pid,ppid,stat,comm -p $!

    & runs the command in the background and $! holds its PID. The ppid will match your shell's PID from the first step.

  3. 3
    Stop it politely and read the exit code
    sleep 300 & pid=$!; kill $pid; wait $pid; echo $?

    You should see

    143

    kill sent SIGTERM. wait collects the exit code, and 143 is 128 plus 15.

  4. 4
    Write a program that handles SIGTERM
    mkdir -p ~/learn-linux
    cat > ~/learn-linux/tidy.sh <<'EOF'
    #!/bin/bash
    trap 'echo "got TERM, cleaning up"; exit 0' TERM
    echo "running as PID $$"
    while true; do sleep 1; done
    EOF
    chmod u+x ~/learn-linux/tidy.sh
    ~/learn-linux/tidy.sh & pid=$!; sleep 2; kill $pid; wait $pid; echo "exit: $?"

    You should see

    running as PID <pid>
    got TERM, cleaning up
    exit: 0

    trap registers the handler. Your shell may also print a line like [1] <pid> first. This is what well-behaved services do on SIGTERM.

  5. 5
    See what is using the CPU and memory
    ps aux --sort=-%mem | head -n 5
    top -b -n 1 | head -n 12

    ps aux lists all processes. top -b -n 1 prints one snapshot and exits, so it does not take over your terminal.

  6. 6
    Clean up
    rm -r ~/learn-linux

    Removes only the practice folder.

What breaks, and where to look

  • kill -9 does nothing and the process stays in the list, with state D in ps.

    WHY IT HAPPENS
    State D means uninterruptible sleep, usually waiting on a disk or a network share that is not answering. The process cannot receive even SIGKILL until that I/O returns.
    CHECK FIRST
    Run ps -o pid,stat,wchan:20,cmd -p <pid> to see what it is waiting on, then check the disk or the mount, for example with df -h and dmesg | tail.
  • bash: kill: (4321) - Operation not permitted

    WHY IT HAPPENS
    You can only signal processes that belong to your own user. The process runs as another user, often a service account or root.
    CHECK FIRST
    Run ps -o user,pid,cmd -p 4321 to see the owner. Use sudo kill 4321 if you are sure, or better, stop it through its service with systemctl.
  • A long job you started over SSH was gone after your connection dropped.

    WHY IT HAPPENS
    When the terminal closes, the shell sends SIGHUP to its jobs and most programs exit on it.
    CHECK FIRST
    Run the work under tmux or as a systemd service. For a quick fix, start it with nohup <command> &. Look for it afterwards with ps aux | grep <name>.

When not to use this

  • Do not reach for kill -9 first. It skips cleanup, so it can leave half-written files, stale lock files and open transactions. Send SIGTERM and wait.
  • Do not hand-manage a long-running service with & and kill. Run it under systemd so it is started, restarted and stopped the same way every time.

Check yourself

What is the difference between SIGTERM and SIGKILL?

SIGTERM asks the process to stop and it can catch the signal and clean up. SIGKILL cannot be caught, so the kernel removes the process at once.

What does an exit code of 143 tell you?

The process was ended by SIGTERM, because the code is 128 plus the signal number 15.

Why can a command run with & die when your SSH session closes?

The shell sends SIGHUP to its jobs when the session ends, and most programs exit on it. Use nohup, tmux or a service.

Lesson 11 of 92 in the DevOps path

Search

Search cheat sheets and projects