Lesson 8 of 8 in Linux and the shell

SSH: log in to servers with keys, safely

Core13 min

SSH is the protocol for logging in to a remote machine over an encrypted connection. With key-based login you prove your identity with a private key that never leaves your computer, instead of sending a password.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Users, groups, permissions and sudo, systemd: run programs as services

The problem this solves

Servers are somewhere else, so you need a way to run commands on them over a network you do not control. Anything sent in plain text on that network can be read or changed.

SSH solves two problems at once. It encrypts everything, and it lets each side check who the other is. You check the server's identity so you do not hand your login to an impostor, and the server checks yours.

A server on the internet is probed by automated login attempts within minutes of appearing. Passwords can be guessed, while a key pair cannot, so key login with passwords switched off is the standard baseline.

How it works

Who checks what when you connect with SSH
Who checks what when you connect with SSHYour computer connects to port 22 on the server, checks the server's host key against known_hosts, and the server checks your public key against authorized_keys.1 connect2 reach sshd3 is this server known?4 is your key allowed?Your computerprivate key stays hereNetworkport 22Server (sshd)has a host keyknown_hostsserver keys you trustauthorized_keyspublic keys allowed in
  • People and clients
  • Network
  • Runs code
  • Security

Your computer connects to port 22 on the server, checks the server's host key against known_hosts, and the server checks your public key against authorized_keys.

  1. Your computer to Network: 1 connect
  2. Network to Server (sshd): 2 reach sshd
  3. Your computer to known_hosts: 3 is this server known?
  4. Server (sshd) to authorized_keys: 4 is your key allowed?

There are two key checks, in opposite directions. First the server shows its host key. Your client looks it up in ~/.ssh/known_hosts. The first time you see a server it asks you to confirm and saves the fingerprint, and on later visits it warns loudly if the key has changed.

Then you prove who you are. A key pair is a private key and a matching public key. You put the public key in ~/.ssh/authorized_keys on the server, and the private key stays on your machine. The client signs a challenge with the private key, and the server verifies it with the public key. The private key is never sent.

Protect the private key with a passphrase and with file permissions. SSH ignores a private key that other users can read, and the server refuses keys when ~/.ssh or authorized_keys can be written by others.

Once keys work, set safe defaults in /etc/ssh/sshd_config: PasswordAuthentication no and PermitRootLogin no. Log in as a normal user and use sudo for admin work.

Try it yourself

These steps create a practice key and test your client configuration without connecting anywhere. If you have a VM or server you can recover, the hardening step shows what to change there.

  1. 1
    Create a practice key pair
    mkdir -p ~/learn-ssh && chmod 700 ~/learn-ssh
    ssh-keygen -t ed25519 -f ~/learn-ssh/demo_key -N "" -C "learn-demo"
    ls -l ~/learn-ssh

    -N "" means no passphrase, which is fine for a throwaway key. For a real key, leave it out and ssh-keygen will prompt you to set one. You get demo_key (private) and demo_key.pub (public).

  2. 2
    Inspect the public key and its fingerprint
    cat ~/learn-ssh/demo_key.pub
    ssh-keygen -lf ~/learn-ssh/demo_key.pub

    The public key is one line and is safe to share. The fingerprint is a short summary you can compare with what a server shows.

  3. 3
    Name a server in a config file and test it offline
    cat > ~/learn-ssh/config <<'EOF'
    Host web
      HostName 203.0.113.10
      User deploy
      IdentityFile ~/learn-ssh/demo_key
    EOF
    ssh -F ~/learn-ssh/config -G web | grep -E '^(hostname|user|port|identityfile) '

    203.0.113.10 is a reserved documentation address, so nothing real is contacted. ssh -G only prints the settings SSH would use. Normally this block goes in ~/.ssh/config, after which ssh web is enough.

  4. 4
    Install your public key on a server you control
    ssh-copy-id -i ~/learn-ssh/demo_key.pub <user>@<server>
    ssh -i ~/learn-ssh/demo_key <user>@<server>

    Only for a VM or server of your own. ssh-copy-id appends the public key to ~/.ssh/authorized_keys on the server. For real use, make the key with a passphrase first.

  5. 5
    Harden the server, then test before you disconnect
    # on the server, with a second session still open:
    sudo sshd -T | grep -iE 'passwordauthentication|permitrootlogin'
    # edit /etc/ssh/sshd_config: PasswordAuthentication no, PermitRootLogin no
    sudo sshd -t
    sudo systemctl reload ssh

    The service is ssh on Debian and Ubuntu and sshd on Fedora and RHEL. sshd -t checks the config for errors, and sshd -T shows the settings actually in effect. Keep your current session open and test a new login in another window before you close it.

  6. 6
    Clean up
    rm -r ~/learn-ssh

    Removes the practice key. If you installed it on a server in step 4, delete that line from the server's ~/.ssh/authorized_keys too.

What breaks, and where to look

  • WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! ... Offending key in ~/.ssh/known_hosts:3

    WHY IT HAPPENS
    The key the server shows is not the one you saved. The usual cause is that the server was rebuilt or the IP address now belongs to another machine. It can also be someone intercepting the connection.
    CHECK FIRST
    First confirm with whoever runs the server that the change is expected. Then remove the old entry with ssh-keygen -R <host> and connect again to accept the new fingerprint.
  • <user>@<host>: Permission denied (publickey).

    WHY IT HAPPENS
    The server did not accept any key you offered. Common causes are the public key missing from authorized_keys, the wrong user name, or ~/.ssh and authorized_keys being writable by others, which sshd refuses by default.
    CHECK FIRST
    Run ssh -v <user>@<host> to see which keys were offered. On the server, check ls -ld ~ ~/.ssh ~/.ssh/authorized_keys and sudo journalctl -u ssh -n 20. Directories should be 700 and the file 600.
  • WARNING: UNPROTECTED PRIVATE KEY FILE! Permissions 0644 for 'key' are too open. This private key will be ignored.

    WHY IT HAPPENS
    SSH will not use a private key that other users can read, because they could copy it.
    CHECK FIRST
    Run ls -l <keyfile> and fix it with chmod 600 <keyfile>.

When not to use this

  • A key on a laptop can be copied or lost. For many servers and many people, short-lived certificates from an SSH certificate authority or a cloud identity service are easier to revoke than a pile of authorized_keys files.
  • Do not leave SSH reachable from the whole internet if you can avoid it. Restrict port 22 with a firewall or security group to your own IP or a VPN, and keep password login off.
  • Settings in /etc/ssh/sshd_config.d/*.conf can override your edit, since sshd uses the first value it reads for each setting. If sshd -T still shows passwordauthentication yes, look in that folder.

Check yourself

Which key goes on the server and which stays on your computer?

The public key goes into ~/.ssh/authorized_keys on the server. The private key stays on your computer and is never sent.

What does the known_hosts warning about a changed host key protect you from?

It stops you from logging in to an impostor that is pretending to be the server. Confirm the change is legitimate before you remove the old entry.

Why test a new login in a second terminal before closing the first?

If a config change locks you out, the open session is your way back in to fix it.

Lesson 15 of 92 in the DevOps path

Search

Search cheat sheets and projects