Lesson 7 of 8 in Linux and the shell
Logs and journalctl: find out what happened
A log is a timestamped record of what a program or the system did. On systemd machines most logs go to the journal, which you search with journalctl, and some programs also write plain text files under /var/log.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: systemd: run programs as services
The problem this solves
When a service fails, the only evidence is what it wrote before it stopped. If you cannot find or filter that output, you end up guessing and restarting things until it works.
Logs from many programs also arrive in different shapes. Some are text files, some go to the journal, and a busy server can produce thousands of lines a minute. You need a way to ask for just one service, one hour or only the errors.
journald stores everything in one indexed place, so a single command answers questions such as what this service printed since it last started.
How it works
- Runs code
- Data
- Control
- People and clients
A program prints to stdout, journald collects it into the journal, and journalctl searches the journal, while some programs also write their own log file.
- Your program to journald: 1 prints to stdout
- journald to The journal: 2 stores each line
- Your program to /var/log files: or writes a file
- journalctl to The journal: 3 search by unit or time
For a service run by systemd, anything the program prints to stdout or stderr is picked up by journald and stored with a timestamp, the unit name and a priority level from emergency down to debug. You do not configure anything in the program itself.
journalctl reads that store. Without arguments it prints everything, oldest first. You narrow it with filters: -u for a unit, -b for the current boot, --since and --until for time, -p for priority and -f to follow new lines as they arrive.
Some software keeps its own text files under /var/log, for example web servers with access and error logs. Those are plain files you read with tail, less and grep. Which names exist depends on the distribution.
To test a pipeline end to end you can write your own message with logger, then find it with journalctl.
Try it yourself
These commands only read, apart from one test message you write yourself. Use a systemd Linux machine or VM. Some need sudo to see system-wide logs.
- 1See how much the journal holds
journalctl --disk-usage journalctl --list-boots | tail -n 3
--list-bootsshows how many boots are stored. If it lists only the current one, the journal is probably not persistent. - 2Read recent warnings and errors
sudo journalctl -b -p warning -n 20 --no-pager
-blimits to this boot,-p warningshows warnings and anything more serious, and-n 20shows the last 20 lines. - 3Filter by service and time
sudo journalctl -u ssh --since "1 hour ago" --no-pager | tail -n 20
The SSH service is called
sshon Debian and Ubuntu andsshdon Fedora and RHEL. If the unit does not exist you will see no entries. - 4Write your own message and find it
logger -t learn-demo "hello from logger" journalctl -t learn-demo -n 3 --no-pager
-tsets a tag, andjournalctl -tfilters by it. This proves logs can reach the journal. - 5Follow logs live
journalctl -f
New lines appear as they are written. In a second terminal run
logger -t learn-demo "second message"to see one arrive. Stop with Ctrl+C. - 6Look for plain log files
ls /var/log | head -n 20 sudo tail -n 5 /var/log/syslog
Debian and Ubuntu usually have
/var/log/syslog. Fedora and RHEL rely on the journal, so the file may not exist.
What breaks, and where to look
Hint: You are currently not seeing messages from other users and the system.
- WHY IT HAPPENS
- Your account is not allowed to read the system journal. Only root and members of groups such as
admorsystemd-journalcan. - CHECK FIRST
- Run the command with
sudo, or check your groups withid. An administrator can add you withsudo usermod -aG systemd-journal <you>, and you must log in again.
The logs from before the last reboot are gone.
- WHY IT HAPPENS
- On many systems the journal is kept only in memory unless the folder
/var/log/journalexists. When it is missing, the journal starts empty at every boot. - CHECK FIRST
- Run
ls -d /var/log/journalandjournalctl --list-boots. To keep logs, runsudo mkdir -p /var/log/journalandsudo systemctl restart systemd-journald.
The disk is full and
/var/logis the biggest folder.- WHY IT HAPPENS
- A noisy program, or a log file that is never rotated, has grown without limit. The journal also has a size cap, but plain files do not.
- CHECK FIRST
- Run
sudo du -h --max-depth=1 /var/log | sort -h | tailto find the largest entry. For the journal usejournalctl --disk-usage, thensudo journalctl --vacuum-size=200M.
When not to use this
- Reading logs on each machine does not scale to dozens of servers or short-lived containers. Ship logs to a central system such as Loki or an ELK stack so they survive the machine.
- A log says what the program chose to report. For a hang or a slow request that logs nothing, use metrics or tracing and tools such as
strace.
Check yourself
How do you show only one service's logs from the last hour?
journalctl -u <service> --since "1 hour ago".
Why might the journal be empty after a reboot?
The journal was only stored in memory because /var/log/journal does not exist, so it is lost at every boot.
What does journalctl -f do?
It follows the journal live and prints new lines as they arrive, like tail -f.
Keep going
Lesson 14 of 92 in the DevOps path