Lesson 4 of 8 in Linux and the shell
Processes and signals: see, stop and manage programs
A process is a running copy of a program, with its own process ID and a parent that started it. A signal is a short message the kernel delivers to a process, such as a request to stop, which the process can handle in its own way.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Users, groups, permissions and sudo
The problem this solves
On a server, problems often look like a program that is still running when it should not be, or one that is not responding. You need to see what is running, who started it and how to stop it without damaging data.
Stopping a program is not one action. Some programs need a moment to finish writing a file or close a connection. Others hang and have to be removed by force. Linux gives you different signals for each case.
Knowing the difference between asking and forcing is what keeps a quick fix from corrupting a database or leaving a lock file behind.
How it works
- Control
- Runs code
A login through sshd starts your shell, and every command you run is a child of that shell, with systemd as the ancestor of the whole tree.
- systemd to sshd: starts
- sshd to bash: starts
- bash to sleep 300: starts
- People and clients
- Control
- Runs code
You run kill with a process ID, the kernel delivers the signal to that process, and a process that handles SIGTERM cleans up and exits.
- You to kill command: 1 run kill
- kill command to Kernel: 2 asks to send
- Kernel to Target process: 3 SIGTERM arrives
- Target process to Cleans up, exits: 4 handler runs
Every process has a process ID, or PID, and a parent PID. The first process, PID 1, is systemd on most distributions, and it starts everything else. When you log in over SSH, sshd starts your shell, and each command you type becomes a child of that shell.
A signal is a numbered message. kill is badly named, because it only sends a signal. By default it sends SIGTERM, signal 15, which means please terminate. The process can catch it, finish its work and exit cleanly.
SIGKILL, signal 9, cannot be caught or ignored. The kernel removes the process immediately, so it gets no chance to clean up. Treat it as a last resort. Ctrl+C sends SIGINT, and closing a terminal or dropping an SSH connection sends SIGHUP to the jobs started from it.
When a process ends, it leaves an exit code. A process killed by a signal reports 128 plus the signal number, so SIGTERM shows up as 143 and SIGKILL as 137.
Try it yourself
Run these on Linux or WSL in a normal shell. Everything is a harmless sleep or a small script, and each step stops what it starts.
- 1Find your shell in the process list
echo $$ ps -o pid,ppid,stat,comm -p $$
$$is the PID of your shell. Theppidcolumn is its parent, which is usually sshd or your terminal program. - 2Start something in the background and look at it
sleep 300 & ps -o pid,ppid,stat,comm -p $!
&runs the command in the background and$!holds its PID. Theppidwill match your shell's PID from the first step. - 3Stop it politely and read the exit code
sleep 300 & pid=$!; kill $pid; wait $pid; echo $?
You should see
143
killsent SIGTERM.waitcollects the exit code, and 143 is 128 plus 15. - 4Write a program that handles SIGTERM
mkdir -p ~/learn-linux cat > ~/learn-linux/tidy.sh <<'EOF' #!/bin/bash trap 'echo "got TERM, cleaning up"; exit 0' TERM echo "running as PID $$" while true; do sleep 1; done EOF chmod u+x ~/learn-linux/tidy.sh ~/learn-linux/tidy.sh & pid=$!; sleep 2; kill $pid; wait $pid; echo "exit: $?"
You should see
running as PID <pid> got TERM, cleaning up exit: 0
trapregisters the handler. Your shell may also print a line like[1] <pid>first. This is what well-behaved services do on SIGTERM. - 5See what is using the CPU and memory
ps aux --sort=-%mem | head -n 5 top -b -n 1 | head -n 12
ps auxlists all processes.top -b -n 1prints one snapshot and exits, so it does not take over your terminal. - 6Clean up
rm -r ~/learn-linux
Removes only the practice folder.
What breaks, and where to look
kill -9 does nothing and the process stays in the list, with state D in
ps.- WHY IT HAPPENS
- State D means uninterruptible sleep, usually waiting on a disk or a network share that is not answering. The process cannot receive even SIGKILL until that I/O returns.
- CHECK FIRST
- Run
ps -o pid,stat,wchan:20,cmd -p <pid>to see what it is waiting on, then check the disk or the mount, for example withdf -handdmesg | tail.
bash: kill: (4321) - Operation not permitted
- WHY IT HAPPENS
- You can only signal processes that belong to your own user. The process runs as another user, often a service account or root.
- CHECK FIRST
- Run
ps -o user,pid,cmd -p 4321to see the owner. Usesudo kill 4321if you are sure, or better, stop it through its service withsystemctl.
A long job you started over SSH was gone after your connection dropped.
- WHY IT HAPPENS
- When the terminal closes, the shell sends SIGHUP to its jobs and most programs exit on it.
- CHECK FIRST
- Run the work under
tmuxor as a systemd service. For a quick fix, start it withnohup <command> &. Look for it afterwards withps aux | grep <name>.
When not to use this
- Do not reach for
kill -9first. It skips cleanup, so it can leave half-written files, stale lock files and open transactions. Send SIGTERM and wait. - Do not hand-manage a long-running service with
&andkill. Run it under systemd so it is started, restarted and stopped the same way every time.
Check yourself
What is the difference between SIGTERM and SIGKILL?
SIGTERM asks the process to stop and it can catch the signal and clean up. SIGKILL cannot be caught, so the kernel removes the process at once.
What does an exit code of 143 tell you?
The process was ended by SIGTERM, because the code is 128 plus the signal number 15.
Why can a command run with & die when your SSH session closes?
The shell sends SIGHUP to its jobs when the session ends, and most programs exit on it. Use nohup, tmux or a service.
Keep going
Lesson 11 of 92 in the DevOps path