Lesson 6 of 8 in Networking for DevOps

Firewalls and security groups

Core8 min

A firewall is a set of rules that decides which network traffic may pass, based on protocol, port and the source or destination address. A cloud security group is a firewall attached to a server's network interface that lists only what is allowed.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Ports, TCP and UDP, NAT and routing: how traffic leaves

The problem this solves

A service can be running, listening and routed correctly and still be unreachable, because a rule drops the traffic before it arrives. The failure looks like the server is down.

The opposite mistake is worse. A rule opened to the whole internet for convenience stays open and gets found by scanners.

How it works

A firewall allows matching traffic and drops the rest
A firewall allows matching traffic and drops the restA packet reaches the firewall, which checks its rules, passes it to the server if a rule allows it, and otherwise drops it silently.1 packetarrives2 check rules3 allowed:pass3 no match: dropClientsource addressFirewallor security groupServerlistening on 443Rulesprotocol, port, sourceDroppedno reply at all
  • People and clients
  • Security
  • Runs code
  • Control
  • Other

A packet reaches the firewall, which checks its rules, passes it to the server if a rule allows it, and otherwise drops it silently.

  1. Client to Firewall: 1 packet arrives
  2. Firewall to Rules: 2 check rules
  3. Firewall to Server: 3 allowed: pass
  4. Firewall to Dropped: 3 no match: drop

Most firewalls follow default deny for inbound traffic. Nothing gets in unless a rule allows it. A rule names a protocol, a port or port range, and a source, written as a CIDR block such as 203.0.113.0/24 or a reference to another group.

Most modern firewalls are stateful. When they allow an outgoing request, they automatically allow the reply that belongs to that connection, so you do not write a rule for return traffic. AWS security groups behave this way and contain only allow rules. In Linux iptables and nftables, rules are evaluated in order and the first match decides.

Filters exist at several levels, and a packet must satisfy all of them: a firewall on the host (ufw, nftables), the cloud security group on the interface, optional subnet-level network ACLs, and Kubernetes network policies. When something is blocked, you need to find which layer did it.

The behaviour tells you which. A closed port with nothing listening usually gets an immediate refusal. A firewall that drops packets sends nothing back, so the client waits until it times out.

Try it yourself

You will tell a refusal from a timeout and read your host firewall without changing it. Start with no listener on port 9, which is almost never used.

  1. 1
    See a refusal (nothing listening)
    curl -sS --connect-timeout 3 http://127.0.0.1:9/; echo "exit=$?"

    It fails at once with Connection refused and exit code 7. The machine answered and said no program is on that port.

  2. 2
    See a timeout (no answer at all)
    curl -sS --connect-timeout 3 http://192.0.2.1/; echo "exit=$?"

    192.0.2.1 is a documentation address that is never in use. Expect a wait of about 3 seconds and Connection timed out with exit code 28, because nothing replies. With no network you may see Network is unreachable instead.

  3. 3
    Read your host firewall rules (read-only)
    sudo ufw status verbose
    sudo nft list ruleset

    Use whichever tool exists. ufw may be missing, and nft prints an empty result on systems with no rules. These commands only list rules. Do not change firewall rules on a remote machine without a way back in.

  4. 4
    Read a rule as a sentence

    A security group rule such as inbound, TCP, port 443, source 0.0.0.0/0 reads as: allow anyone on the internet to connect to port 443 on this server. A rule with TCP port 22 and source 203.0.113.0/24 allows SSH from that one range only. Write the sentence aloud and check that you meant it.

What breaks, and where to look

  • curl: (28) Connection timed out or SSH hangs, but the service is running on the server.

    WHY IT HAPPENS
    A firewall or security group is dropping the packets, so no reply ever comes back. A timeout means no answer, and an unreachable route can cause it too.
    CHECK FIRST
    On the server run ss -tlnp | grep <port> to confirm it listens. From a client run nc -vz -w 3 <host> <port>. If it listens and still times out, check the security group inbound rule, then the host firewall.
  • Connection refused right away.

    WHY IT HAPPENS
    In most cases a firewall is not involved. The host answered and nothing is listening on that port, or the app is bound to 127.0.0.1. A firewall rule set to reject, rather than drop, can also produce it.
    CHECK FIRST
    Run ss -tlnp on the server and look at the port and the local address column.
  • Unexpected login attempts fill /var/log/auth.log and a scanner finds your database port.

    WHY IT HAPPENS
    A rule allows 0.0.0.0/0 on a port that only your team needs, such as SSH or a database.
    CHECK FIRST
    List the inbound rules and look at every source of 0.0.0.0/0. Replace it with your office range, a VPN range or another security group.

When not to use this

  • A firewall limits who can connect and does not fix a vulnerable service. Keep software patched and require authentication, even for services behind a firewall.
  • Do not expect a rule change on one layer to fix a block at another. Security groups, host firewalls and network policies are independent and all have to allow the traffic.

Check yourself

What does a timeout tell you compared with a refusal?

A refusal means the host answered and nothing listens on that port. A timeout means no answer came back, which points to a drop by a firewall or a bad route.

What does stateful mean for a firewall?

Replies to connections it already allowed are let through automatically, so you do not write rules for return traffic.

Why is a source of 0.0.0.0/0 on SSH risky?

It lets the whole internet attempt to connect to the port, so scanners and attackers will find it.

Lesson 21 of 92 in the DevOps path

Search

Search cheat sheets and projects