Lesson 7 of 8 in Linux and the shell

Logs and journalctl: find out what happened

Core10 min

A log is a timestamped record of what a program or the system did. On systemd machines most logs go to the journal, which you search with journalctl, and some programs also write plain text files under /var/log.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: systemd: run programs as services

The problem this solves

When a service fails, the only evidence is what it wrote before it stopped. If you cannot find or filter that output, you end up guessing and restarting things until it works.

Logs from many programs also arrive in different shapes. Some are text files, some go to the journal, and a busy server can produce thousands of lines a minute. You need a way to ask for just one service, one hour or only the errors.

journald stores everything in one indexed place, so a single command answers questions such as what this service printed since it last started.

How it works

Where a program's log lines end up
Where a program's log lines end upA program prints to stdout, journald collects it into the journal, and journalctl searches the journal, while some programs also write their own log file.1 prints tostdout2 stores eachlineor writes a file3 search by unit or timeYour programprints messages/var/log filesif the app writes its ownjournaldthe collectorThe journalindexed, with timestampsjournalctlwhat you type
  • Runs code
  • Data
  • Control
  • People and clients

A program prints to stdout, journald collects it into the journal, and journalctl searches the journal, while some programs also write their own log file.

  1. Your program to journald: 1 prints to stdout
  2. journald to The journal: 2 stores each line
  3. Your program to /var/log files: or writes a file
  4. journalctl to The journal: 3 search by unit or time

For a service run by systemd, anything the program prints to stdout or stderr is picked up by journald and stored with a timestamp, the unit name and a priority level from emergency down to debug. You do not configure anything in the program itself.

journalctl reads that store. Without arguments it prints everything, oldest first. You narrow it with filters: -u for a unit, -b for the current boot, --since and --until for time, -p for priority and -f to follow new lines as they arrive.

Some software keeps its own text files under /var/log, for example web servers with access and error logs. Those are plain files you read with tail, less and grep. Which names exist depends on the distribution.

To test a pipeline end to end you can write your own message with logger, then find it with journalctl.

Try it yourself

These commands only read, apart from one test message you write yourself. Use a systemd Linux machine or VM. Some need sudo to see system-wide logs.

  1. 1
    See how much the journal holds
    journalctl --disk-usage
    journalctl --list-boots | tail -n 3

    --list-boots shows how many boots are stored. If it lists only the current one, the journal is probably not persistent.

  2. 2
    Read recent warnings and errors
    sudo journalctl -b -p warning -n 20 --no-pager

    -b limits to this boot, -p warning shows warnings and anything more serious, and -n 20 shows the last 20 lines.

  3. 3
    Filter by service and time
    sudo journalctl -u ssh --since "1 hour ago" --no-pager | tail -n 20

    The SSH service is called ssh on Debian and Ubuntu and sshd on Fedora and RHEL. If the unit does not exist you will see no entries.

  4. 4
    Write your own message and find it
    logger -t learn-demo "hello from logger"
    journalctl -t learn-demo -n 3 --no-pager

    -t sets a tag, and journalctl -t filters by it. This proves logs can reach the journal.

  5. 5
    Follow logs live
    journalctl -f

    New lines appear as they are written. In a second terminal run logger -t learn-demo "second message" to see one arrive. Stop with Ctrl+C.

  6. 6
    Look for plain log files
    ls /var/log | head -n 20
    sudo tail -n 5 /var/log/syslog

    Debian and Ubuntu usually have /var/log/syslog. Fedora and RHEL rely on the journal, so the file may not exist.

What breaks, and where to look

  • Hint: You are currently not seeing messages from other users and the system.

    WHY IT HAPPENS
    Your account is not allowed to read the system journal. Only root and members of groups such as adm or systemd-journal can.
    CHECK FIRST
    Run the command with sudo, or check your groups with id. An administrator can add you with sudo usermod -aG systemd-journal <you>, and you must log in again.
  • The logs from before the last reboot are gone.

    WHY IT HAPPENS
    On many systems the journal is kept only in memory unless the folder /var/log/journal exists. When it is missing, the journal starts empty at every boot.
    CHECK FIRST
    Run ls -d /var/log/journal and journalctl --list-boots. To keep logs, run sudo mkdir -p /var/log/journal and sudo systemctl restart systemd-journald.
  • The disk is full and /var/log is the biggest folder.

    WHY IT HAPPENS
    A noisy program, or a log file that is never rotated, has grown without limit. The journal also has a size cap, but plain files do not.
    CHECK FIRST
    Run sudo du -h --max-depth=1 /var/log | sort -h | tail to find the largest entry. For the journal use journalctl --disk-usage, then sudo journalctl --vacuum-size=200M.

When not to use this

  • Reading logs on each machine does not scale to dozens of servers or short-lived containers. Ship logs to a central system such as Loki or an ELK stack so they survive the machine.
  • A log says what the program chose to report. For a hang or a slow request that logs nothing, use metrics or tracing and tools such as strace.

Check yourself

How do you show only one service's logs from the last hour?

journalctl -u <service> --since "1 hour ago".

Why might the journal be empty after a reboot?

The journal was only stored in memory because /var/log/journal does not exist, so it is lost at every boot.

What does journalctl -f do?

It follows the journal live and prints new lines as they arrive, like tail -f.

Lesson 14 of 92 in the DevOps path

Search

Search cheat sheets and projects