Lesson 1 of 8 in Linux and the shell

Shell basics: commands, pipes and redirection

Core10 min

The shell is the program that reads what you type, starts other programs for you and connects their input and output. A pipe feeds one program's output into the next, and a redirect sends output to a file instead of the screen.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: DevOps and delivery fundamentals

The problem this solves

A server usually has no desktop. Everything you do happens by typing commands, and most of the real work is combining small tools: find the lines, count them, save the result.

Without a mental model of how the shell wires programs together, commands look like magic strings. You copy them, they work once, and you cannot adapt them when the problem changes.

The model is small. Every program has one input and two outputs, and the shell lets you rewire them.

How it works

A pipeline passes text from one program to the next
A pipeline passes text from one program to the nextA file is read by grep, grep passes its matching lines through a pipe to wc, and wc either prints a count on your terminal or writes it to a file.1 lines goin2 pipe |3 resultprintedor redirect >access.loga text filegrep 404keeps matching lineswc -lcounts linesTerminalwhat you seecount.txta file you chose
  • Data
  • Runs code
  • People and clients

A file is read by grep, grep passes its matching lines through a pipe to wc, and wc either prints a count on your terminal or writes it to a file.

  1. access.log to grep 404: 1 lines go in
  2. grep 404 to wc -l: 2 pipe |
  3. wc -l to Terminal: 3 result printed
  4. wc -l to count.txt: or redirect >
Every program has one input and two outputs
Every program has one input and two outputsA program reads from stdin and writes normal results to stdout and problems to stderr, and both outputs land on your terminal unless you redirect one of them.readswritesresultswrites errorsshownalso shownstdin (0)keyboard or a pipeProgramany commandstdout (1)normal outputstderr (2)error messagesTerminalshows both by default
  • Other
  • Runs code
  • People and clients

A program reads from stdin and writes normal results to stdout and problems to stderr, and both outputs land on your terminal unless you redirect one of them.

  1. stdin (0) to Program: reads
  2. Program to stdout (1): writes results
  3. Program to stderr (2): writes errors
  4. stdout (1) to Terminal: shown
  5. stderr (2) to Terminal: also shown

When you type a command, the shell finds the program, starts it and waits. The program reads from standard input, called stdin, and writes to standard output, called stdout. Errors go to a separate stream, standard error or stderr, so they do not get mixed into the results.

A pipe, written |, connects the stdout of the program on its left to the stdin of the program on its right. Each tool stays small and the pipeline does the combining.

A redirect points a stream somewhere else. > writes stdout to a file and replaces what was there, >> appends, and 2> redirects only stderr.

When a program finishes it also hands back an exit code. Zero means success and anything else means failure. The shell keeps the last one in $?, and && and || run the next command only when the previous one succeeded or failed.

Try it yourself

Run these on any Linux machine or WSL. Everything happens inside a throwaway folder called learn-linux in your home directory, and the last step removes it.

  1. 1
    Make a practice folder and a small log
    mkdir -p ~/learn-linux && cd ~/learn-linux
    printf 'GET /a 200\nGET /b 404\nGET /c 404\nPOST /d 500\n' > access.log
    cat access.log

    You should see

    GET /a 200
    GET /b 404
    GET /c 404
    POST /d 500

    printf writes the four lines and > sends them into the file instead of the screen.

  2. 2
    Chain two programs with a pipe
    grep 404 access.log | wc -l

    You should see

    2

    grep keeps the lines containing 404 and wc -l counts what arrives on its stdin.

  3. 3
    Redirect to a file, then append
    grep 404 access.log > errors.txt
    echo 'POST /e 404' >> errors.txt
    cat errors.txt

    You should see

    GET /b 404
    GET /c 404
    POST /e 404

    Run the first line twice and the file is replaced each time. The >> line adds to the end instead.

  4. 4
    Separate errors from results
    ls /nope 2> err.txt
    echo $?
    cat err.txt

    You should see

    2
    ls: cannot access '/nope': No such file or directory

    Nothing printed for the ls itself because its error went into err.txt. The exit code 2 is the GNU ls code for a serious problem. The message text is from GNU coreutils, so other systems may word it differently.

  5. 5
    Act on success or failure
    grep -q 500 access.log && echo 'has 500s' || echo 'clean'
    grep -q 999 access.log && echo 'has 999s' || echo 'no 999s'

    You should see

    has 500s
    no 999s

    -q makes grep print nothing and only set the exit code. && runs when the exit code is 0 and || runs when it is not.

  6. 6
    Clean up
    cd ~ && rm -r ~/learn-linux

    This removes only the practice folder.

What breaks, and where to look

  • bash: foo: command not found

    WHY IT HAPPENS
    The shell looks for a program only in the folders listed in the PATH variable. Either the program is not installed, you mistyped it, or it lives in a folder that is not on PATH.
    CHECK FIRST
    Run command -v foo to see whether the shell can find it, and echo $PATH to see where it looks. For a script in the current folder, run it as ./foo.
  • You ran sort data.txt > data.txt and the file is now empty.

    WHY IT HAPPENS
    The shell opens and empties the output file before it starts the program. By the time sort reads data.txt there is nothing left in it.
    CHECK FIRST
    Restore the file from a backup or version control. Next time write to a different file and then rename it, or use the tool's own option such as sort -o data.txt data.txt.
  • rm: cannot remove 'my': No such file or directory (for a file called my file.txt)

    WHY IT HAPPENS
    The shell splits arguments on spaces. Without quotes, my and file.txt become two separate names.
    CHECK FIRST
    Quote names that contain spaces, as in ls "my file.txt". Run ls first with the same arguments to see what the shell will pass.

When not to use this

  • Pipes move plain text. When the data is JSON or tables with awkward quoting, a tool built for that format such as jq is safer than chaining grep and cut.
  • Once a shell script needs loops, functions and error handling across many files, a scripting language like Python is usually easier to test and read.

Check yourself

What is the difference between > and >>?

> replaces the file with the new output, and >> adds the output to the end of the file.

Why does ls /nope > out.txt still print an error on screen?

The error goes to stderr, and > redirects only stdout. Use 2> to redirect stderr as well.

What does cmd1 && cmd2 do?

It runs cmd2 only if cmd1 finished with exit code 0.

Lesson 8 of 92 in the DevOps path

Search

Search cheat sheets and projects