Lesson 1 of 8 in Linux and the shell
Shell basics: commands, pipes and redirection
The shell is the program that reads what you type, starts other programs for you and connects their input and output. A pipe feeds one program's output into the next, and a redirect sends output to a file instead of the screen.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: DevOps and delivery fundamentals
The problem this solves
A server usually has no desktop. Everything you do happens by typing commands, and most of the real work is combining small tools: find the lines, count them, save the result.
Without a mental model of how the shell wires programs together, commands look like magic strings. You copy them, they work once, and you cannot adapt them when the problem changes.
The model is small. Every program has one input and two outputs, and the shell lets you rewire them.
How it works
- Data
- Runs code
- People and clients
A file is read by grep, grep passes its matching lines through a pipe to wc, and wc either prints a count on your terminal or writes it to a file.
- access.log to grep 404: 1 lines go in
- grep 404 to wc -l: 2 pipe |
- wc -l to Terminal: 3 result printed
- wc -l to count.txt: or redirect >
- Other
- Runs code
- People and clients
A program reads from stdin and writes normal results to stdout and problems to stderr, and both outputs land on your terminal unless you redirect one of them.
- stdin (0) to Program: reads
- Program to stdout (1): writes results
- Program to stderr (2): writes errors
- stdout (1) to Terminal: shown
- stderr (2) to Terminal: also shown
When you type a command, the shell finds the program, starts it and waits. The program reads from standard input, called stdin, and writes to standard output, called stdout. Errors go to a separate stream, standard error or stderr, so they do not get mixed into the results.
A pipe, written |, connects the stdout of the program on its left to the stdin of the program on its right. Each tool stays small and the pipeline does the combining.
A redirect points a stream somewhere else. > writes stdout to a file and replaces what was there, >> appends, and 2> redirects only stderr.
When a program finishes it also hands back an exit code. Zero means success and anything else means failure. The shell keeps the last one in $?, and && and || run the next command only when the previous one succeeded or failed.
Try it yourself
Run these on any Linux machine or WSL. Everything happens inside a throwaway folder called learn-linux in your home directory, and the last step removes it.
- 1Make a practice folder and a small log
mkdir -p ~/learn-linux && cd ~/learn-linux printf 'GET /a 200\nGET /b 404\nGET /c 404\nPOST /d 500\n' > access.log cat access.log
You should see
GET /a 200 GET /b 404 GET /c 404 POST /d 500
printfwrites the four lines and>sends them into the file instead of the screen. - 2Chain two programs with a pipe
grep 404 access.log | wc -l
You should see
2
grepkeeps the lines containing 404 andwc -lcounts what arrives on its stdin. - 3Redirect to a file, then append
grep 404 access.log > errors.txt echo 'POST /e 404' >> errors.txt cat errors.txt
You should see
GET /b 404 GET /c 404 POST /e 404
Run the first line twice and the file is replaced each time. The
>>line adds to the end instead. - 4Separate errors from results
ls /nope 2> err.txt echo $? cat err.txt
You should see
2 ls: cannot access '/nope': No such file or directory
Nothing printed for the
lsitself because its error went intoerr.txt. The exit code 2 is the GNUlscode for a serious problem. The message text is from GNU coreutils, so other systems may word it differently. - 5Act on success or failure
grep -q 500 access.log && echo 'has 500s' || echo 'clean' grep -q 999 access.log && echo 'has 999s' || echo 'no 999s'
You should see
has 500s no 999s
-qmakes grep print nothing and only set the exit code.&&runs when the exit code is 0 and||runs when it is not. - 6Clean up
cd ~ && rm -r ~/learn-linux
This removes only the practice folder.
What breaks, and where to look
bash: foo: command not found
- WHY IT HAPPENS
- The shell looks for a program only in the folders listed in the
PATHvariable. Either the program is not installed, you mistyped it, or it lives in a folder that is not onPATH. - CHECK FIRST
- Run
command -v footo see whether the shell can find it, andecho $PATHto see where it looks. For a script in the current folder, run it as./foo.
You ran
sort data.txt > data.txtand the file is now empty.- WHY IT HAPPENS
- The shell opens and empties the output file before it starts the program. By the time sort reads
data.txtthere is nothing left in it. - CHECK FIRST
- Restore the file from a backup or version control. Next time write to a different file and then rename it, or use the tool's own option such as
sort -o data.txt data.txt.
rm: cannot remove 'my': No such file or directory (for a file called
my file.txt)- WHY IT HAPPENS
- The shell splits arguments on spaces. Without quotes,
myandfile.txtbecome two separate names. - CHECK FIRST
- Quote names that contain spaces, as in
ls "my file.txt". Runlsfirst with the same arguments to see what the shell will pass.
When not to use this
- Pipes move plain text. When the data is JSON or tables with awkward quoting, a tool built for that format such as
jqis safer than chaininggrepandcut. - Once a shell script needs loops, functions and error handling across many files, a scripting language like Python is usually easier to test and read.
Check yourself
What is the difference between > and >>?
> replaces the file with the new output, and >> adds the output to the end of the file.
Why does ls /nope > out.txt still print an error on screen?
The error goes to stderr, and > redirects only stdout. Use 2> to redirect stderr as well.
What does cmd1 && cmd2 do?
It runs cmd2 only if cmd1 finished with exit code 0.
Keep going
Lesson 8 of 92 in the DevOps path