Lesson 6 of 8 in Linux and the shell

systemd: run programs as services

Core12 min

systemd is the program that starts first on most Linux distributions and manages everything else as units. A service unit describes one background program, how to start it and when it should run, and systemctl is the tool you use to control it.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Processes and signals: see, stop and manage programs, Package management with apt and dnf

The problem this solves

A web server or database must start when the machine boots, restart if it crashes, and run without a terminal attached. Starting it by hand with & fails all three, as it dies when you log out and does not return after a reboot.

systemd solves this with a small text file per service. The file says what to run, which user to run it as, what must start first and what to do on failure. The same commands then work for every service on the machine.

It also collects the output of each service, so when something fails you have one place to read why.

How it works

systemd reads a unit file and runs what it describes
systemd reads a unit file and runs what it describesYou ask systemd through systemctl to start a service, systemd reads the unit file, starts the program, and the program's output goes to the journal.1 start myapp2 reads thefile3 runsExecStart4 output is keptUnit filemyapp.servicesystemdPID 1, the managerYour programthe ExecStart commandsystemctlwhat you typeJournalcollects its output
  • Data
  • Control
  • Runs code
  • People and clients

You ask systemd through systemctl to start a service, systemd reads the unit file, starts the program, and the program's output goes to the journal.

  1. systemctl to systemd: 1 start myapp
  2. systemd to Unit file: 2 reads the file
  3. systemd to Your program: 3 runs ExecStart
  4. Your program to Journal: 4 output is kept
Enable means start at boot, start means start now
Enable means start at boot, start means start nowEnabling a service puts a link in the target's wants folder, and at boot systemd reaches the target, follows the link and starts the service.1 bootreaches it2 looks in .wants/3 starts what it linksBootmachine powers onmulti-user.targetthe normal running state.wants/ linkcreated by enablemyapp.servicestarts automatically
  • Other
  • Control
  • Data
  • Runs code

Enabling a service puts a link in the target's wants folder, and at boot systemd reaches the target, follows the link and starts the service.

  1. Boot to multi-user.target: 1 boot reaches it
  2. multi-user.target to .wants/ link: 2 looks in .wants/
  3. .wants/ link to myapp.service: 3 starts what it links

systemd is PID 1. Everything it manages is a unit, and a service is the unit type for a long-running program. Units live in /etc/systemd/system for ones you write and /usr/lib/systemd/system for ones that packages install.

A service file has three parts. [Unit] describes it and its ordering. [Service] holds ExecStart, the command to run, plus settings like User= and Restart=. [Install] says which target should pull it in when you enable it.

systemctl start and stop act now. systemctl enable and disable change what happens at boot by creating or removing a link. They are separate on purpose, and enable --now does both.

After you edit a unit file, run systemctl daemon-reload so systemd rereads it. The service's output goes to the journal, which you read with journalctl, covered in the next concept.

Try it yourself

This creates a small service for your own user, so no root is needed and nothing system-wide changes. It needs a distribution running systemd, which includes recent Ubuntu, Debian and Fedora and WSL with systemd enabled.

  1. 1
    Look at an existing service
    systemctl status systemd-journald --no-pager
    systemctl list-units --type=service --state=running --no-pager | head -n 10

    active (running) is the healthy state. Reading status does not need sudo.

  2. 2
    Write a unit file for your own user
    mkdir -p ~/.config/systemd/user
    cat > ~/.config/systemd/user/hello.service <<'EOF'
    [Unit]
    Description=Hello demo
    
    [Service]
    ExecStart=/bin/sh -c 'while true; do echo hello from hello.service; sleep 5; done'
    Restart=on-failure
    
    [Install]
    WantedBy=default.target
    EOF
    systemctl --user daemon-reload

    --user manages services for your own account. daemon-reload makes systemd read the new file.

  3. 3
    Start it and check on it
    systemctl --user start hello.service
    systemctl --user status hello.service --no-pager

    Look for Active: active (running) and the program's PID. The last lines of the status show its recent output.

  4. 4
    Read what it printed
    journalctl --user -u hello.service -n 5 --no-pager

    -u selects the unit and -n 5 shows the last five lines.

  5. 5
    Control it, then clean up
    systemctl --user restart hello.service
    systemctl --user stop hello.service
    rm ~/.config/systemd/user/hello.service
    systemctl --user daemon-reload

    restart is stop then start. The last two lines delete the demo unit and tell systemd it is gone.

What breaks, and where to look

  • Main process exited, code=exited, status=203/EXEC

    WHY IT HAPPENS
    systemd could not execute the ExecStart command. The path is wrong, the file is not executable, or the interpreter named in its first line does not exist. Relative paths do not work in ExecStart.
    CHECK FIRST
    Run systemctl status <name> and journalctl -u <name> -n 30. Then test the exact path from ExecStart in a shell with ls -l <path>, and use an absolute path.
  • Warning: The unit file ... changed on disk. Run 'systemctl daemon-reload' to reload units.

    WHY IT HAPPENS
    systemd keeps a loaded copy of each unit in memory. You edited the file, but the running copy is still the old one.
    CHECK FIRST
    Run sudo systemctl daemon-reload, then restart the service. systemctl cat <name> shows the file systemd has on disk.
  • Failed with result 'start-limit-hit', or the service keeps restarting then stops trying.

    WHY IT HAPPENS
    The program crashes right after starting. With Restart= set, systemd retries, then gives up when it restarts too many times in a short window.
    CHECK FIRST
    Read why it crashes with journalctl -u <name> -n 50. After fixing, clear the failed state with sudo systemctl reset-failed <name> and start it again.

When not to use this

  • Inside most containers there is no systemd. The container's main process is the service, and restarts are handled by Docker or Kubernetes instead.
  • Distributions without systemd, such as Alpine, use other init systems with different commands. Check ps -p 1 -o comm= to see what is PID 1 before assuming.

Check yourself

What is the difference between systemctl start and systemctl enable?

start runs the service now. enable makes it start at boot. Use enable --now for both.

You edited a unit file and the service behaves as before. What did you forget?

Run systemctl daemon-reload so systemd rereads the file, then restart the service.

Where do you read why a service failed?

In systemctl status <name> for a short view, and journalctl -u <name> for the full output.

Lesson 13 of 92 in the DevOps path

Search

Search cheat sheets and projects