Lesson 6 of 8 in Linux and the shell
systemd: run programs as services
systemd is the program that starts first on most Linux distributions and manages everything else as units. A service unit describes one background program, how to start it and when it should run, and systemctl is the tool you use to control it.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Processes and signals: see, stop and manage programs, Package management with apt and dnf
The problem this solves
A web server or database must start when the machine boots, restart if it crashes, and run without a terminal attached. Starting it by hand with & fails all three, as it dies when you log out and does not return after a reboot.
systemd solves this with a small text file per service. The file says what to run, which user to run it as, what must start first and what to do on failure. The same commands then work for every service on the machine.
It also collects the output of each service, so when something fails you have one place to read why.
How it works
- Data
- Control
- Runs code
- People and clients
You ask systemd through systemctl to start a service, systemd reads the unit file, starts the program, and the program's output goes to the journal.
- systemctl to systemd: 1 start myapp
- systemd to Unit file: 2 reads the file
- systemd to Your program: 3 runs ExecStart
- Your program to Journal: 4 output is kept
- Other
- Control
- Data
- Runs code
Enabling a service puts a link in the target's wants folder, and at boot systemd reaches the target, follows the link and starts the service.
- Boot to multi-user.target: 1 boot reaches it
- multi-user.target to .wants/ link: 2 looks in .wants/
- .wants/ link to myapp.service: 3 starts what it links
systemd is PID 1. Everything it manages is a unit, and a service is the unit type for a long-running program. Units live in /etc/systemd/system for ones you write and /usr/lib/systemd/system for ones that packages install.
A service file has three parts. [Unit] describes it and its ordering. [Service] holds ExecStart, the command to run, plus settings like User= and Restart=. [Install] says which target should pull it in when you enable it.
systemctl start and stop act now. systemctl enable and disable change what happens at boot by creating or removing a link. They are separate on purpose, and enable --now does both.
After you edit a unit file, run systemctl daemon-reload so systemd rereads it. The service's output goes to the journal, which you read with journalctl, covered in the next concept.
Try it yourself
This creates a small service for your own user, so no root is needed and nothing system-wide changes. It needs a distribution running systemd, which includes recent Ubuntu, Debian and Fedora and WSL with systemd enabled.
- 1Look at an existing service
systemctl status systemd-journald --no-pager systemctl list-units --type=service --state=running --no-pager | head -n 10
active (running)is the healthy state. Reading status does not need sudo. - 2Write a unit file for your own user
mkdir -p ~/.config/systemd/user cat > ~/.config/systemd/user/hello.service <<'EOF' [Unit] Description=Hello demo [Service] ExecStart=/bin/sh -c 'while true; do echo hello from hello.service; sleep 5; done' Restart=on-failure [Install] WantedBy=default.target EOF systemctl --user daemon-reload
--usermanages services for your own account.daemon-reloadmakes systemd read the new file. - 3Start it and check on it
systemctl --user start hello.service systemctl --user status hello.service --no-pager
Look for
Active: active (running)and the program's PID. The last lines of the status show its recent output. - 4Read what it printed
journalctl --user -u hello.service -n 5 --no-pager
-uselects the unit and-n 5shows the last five lines. - 5Control it, then clean up
systemctl --user restart hello.service systemctl --user stop hello.service rm ~/.config/systemd/user/hello.service systemctl --user daemon-reload
restartis stop then start. The last two lines delete the demo unit and tell systemd it is gone.
What breaks, and where to look
Main process exited, code=exited, status=203/EXEC
- WHY IT HAPPENS
- systemd could not execute the
ExecStartcommand. The path is wrong, the file is not executable, or the interpreter named in its first line does not exist. Relative paths do not work inExecStart. - CHECK FIRST
- Run
systemctl status <name>andjournalctl -u <name> -n 30. Then test the exact path fromExecStartin a shell withls -l <path>, and use an absolute path.
Warning: The unit file ... changed on disk. Run 'systemctl daemon-reload' to reload units.
- WHY IT HAPPENS
- systemd keeps a loaded copy of each unit in memory. You edited the file, but the running copy is still the old one.
- CHECK FIRST
- Run
sudo systemctl daemon-reload, then restart the service.systemctl cat <name>shows the file systemd has on disk.
Failed with result 'start-limit-hit', or the service keeps restarting then stops trying.
- WHY IT HAPPENS
- The program crashes right after starting. With
Restart=set, systemd retries, then gives up when it restarts too many times in a short window. - CHECK FIRST
- Read why it crashes with
journalctl -u <name> -n 50. After fixing, clear the failed state withsudo systemctl reset-failed <name>and start it again.
When not to use this
- Inside most containers there is no systemd. The container's main process is the service, and restarts are handled by Docker or Kubernetes instead.
- Distributions without systemd, such as Alpine, use other init systems with different commands. Check
ps -p 1 -o comm=to see what is PID 1 before assuming.
Check yourself
What is the difference between systemctl start and systemctl enable?
start runs the service now. enable makes it start at boot. Use enable --now for both.
You edited a unit file and the service behaves as before. What did you forget?
Run systemctl daemon-reload so systemd rereads the file, then restart the service.
Where do you read why a service failed?
In systemctl status <name> for a short view, and journalctl -u <name> for the full output.
Keep going
Lesson 13 of 92 in the DevOps path