Lesson 6 of 8 in Networking for DevOps
Firewalls and security groups
A firewall is a set of rules that decides which network traffic may pass, based on protocol, port and the source or destination address. A cloud security group is a firewall attached to a server's network interface that lists only what is allowed.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Ports, TCP and UDP, NAT and routing: how traffic leaves
The problem this solves
A service can be running, listening and routed correctly and still be unreachable, because a rule drops the traffic before it arrives. The failure looks like the server is down.
The opposite mistake is worse. A rule opened to the whole internet for convenience stays open and gets found by scanners.
How it works
- People and clients
- Security
- Runs code
- Control
- Other
A packet reaches the firewall, which checks its rules, passes it to the server if a rule allows it, and otherwise drops it silently.
- Client to Firewall: 1 packet arrives
- Firewall to Rules: 2 check rules
- Firewall to Server: 3 allowed: pass
- Firewall to Dropped: 3 no match: drop
Most firewalls follow default deny for inbound traffic. Nothing gets in unless a rule allows it. A rule names a protocol, a port or port range, and a source, written as a CIDR block such as 203.0.113.0/24 or a reference to another group.
Most modern firewalls are stateful. When they allow an outgoing request, they automatically allow the reply that belongs to that connection, so you do not write a rule for return traffic. AWS security groups behave this way and contain only allow rules. In Linux iptables and nftables, rules are evaluated in order and the first match decides.
Filters exist at several levels, and a packet must satisfy all of them: a firewall on the host (ufw, nftables), the cloud security group on the interface, optional subnet-level network ACLs, and Kubernetes network policies. When something is blocked, you need to find which layer did it.
The behaviour tells you which. A closed port with nothing listening usually gets an immediate refusal. A firewall that drops packets sends nothing back, so the client waits until it times out.
Try it yourself
You will tell a refusal from a timeout and read your host firewall without changing it. Start with no listener on port 9, which is almost never used.
- 1See a refusal (nothing listening)
curl -sS --connect-timeout 3 http://127.0.0.1:9/; echo "exit=$?"
It fails at once with
Connection refusedand exit code 7. The machine answered and said no program is on that port. - 2See a timeout (no answer at all)
curl -sS --connect-timeout 3 http://192.0.2.1/; echo "exit=$?"
192.0.2.1is a documentation address that is never in use. Expect a wait of about 3 seconds andConnection timed outwith exit code 28, because nothing replies. With no network you may seeNetwork is unreachableinstead. - 3Read your host firewall rules (read-only)
sudo ufw status verbose sudo nft list ruleset
Use whichever tool exists.
ufwmay be missing, andnftprints an empty result on systems with no rules. These commands only list rules. Do not change firewall rules on a remote machine without a way back in. - 4Read a rule as a sentence
A security group rule such as inbound, TCP, port 443, source 0.0.0.0/0 reads as: allow anyone on the internet to connect to port 443 on this server. A rule with TCP port 22 and source 203.0.113.0/24 allows SSH from that one range only. Write the sentence aloud and check that you meant it.
What breaks, and where to look
curl: (28) Connection timed outor SSH hangs, but the service is running on the server.- WHY IT HAPPENS
- A firewall or security group is dropping the packets, so no reply ever comes back. A timeout means no answer, and an unreachable route can cause it too.
- CHECK FIRST
- On the server run
ss -tlnp | grep <port>to confirm it listens. From a client runnc -vz -w 3 <host> <port>. If it listens and still times out, check the security group inbound rule, then the host firewall.
Connection refusedright away.- WHY IT HAPPENS
- In most cases a firewall is not involved. The host answered and nothing is listening on that port, or the app is bound to
127.0.0.1. A firewall rule set to reject, rather than drop, can also produce it. - CHECK FIRST
- Run
ss -tlnpon the server and look at the port and the local address column.
Unexpected login attempts fill
/var/log/auth.logand a scanner finds your database port.- WHY IT HAPPENS
- A rule allows
0.0.0.0/0on a port that only your team needs, such as SSH or a database. - CHECK FIRST
- List the inbound rules and look at every source of
0.0.0.0/0. Replace it with your office range, a VPN range or another security group.
When not to use this
- A firewall limits who can connect and does not fix a vulnerable service. Keep software patched and require authentication, even for services behind a firewall.
- Do not expect a rule change on one layer to fix a block at another. Security groups, host firewalls and network policies are independent and all have to allow the traffic.
Check yourself
What does a timeout tell you compared with a refusal?
A refusal means the host answered and nothing listens on that port. A timeout means no answer came back, which points to a drop by a firewall or a bad route.
What does stateful mean for a firewall?
Replies to connections it already allowed are let through automatically, so you do not write rules for return traffic.
Why is a source of 0.0.0.0/0 on SSH risky?
It lets the whole internet attempt to connect to the port, so scanners and attackers will find it.
Lesson 21 of 92 in the DevOps path