Lesson 8 of 8 in Cloud fundamentals with AWS
Find and delete surprise AWS costs
Cleanup is the habit of listing what you created and deleting what you no longer need, because the cloud keeps billing for anything that exists. Surprise bills almost always come from a few forgotten resources.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Accounts and billing safety before you build, Virtual machines on EC2
The problem this solves
Most surprise bills have nothing to do with heavy usage. They come from a small thing left running for weeks: a VM, an unattached disk, a NAT gateway or an unused public IP address. The charge per hour is small, which is why nobody notices until the bill arrives.
The cloud console does not warn you when a lab is finished. Deleting is your job, and it is easy to look in the wrong place. Resources live in one Region, and a single forgotten Region can hold the problem.
A sweep is a short list of read-only commands you run after every lab and once a week. It costs nothing and turns the bill from a surprise into something you already knew.
How it works
- Control
- Runs code
- Network
- Data
Forgotten virtual machines, NAT gateways, unattached volumes and unused public IP addresses all feed into your bill because each is charged by time or size.
- Forgotten VM to Your bill: compute hours
- NAT gateway to Your bill: hours and traffic
- Unattached volume to Your bill: storage
- Unused public IP to Your bill: address hours
- Control
- People and clients
- Runs code
You list everything, check each item against your own notes, delete only what is yours, and then list again to prove it is gone.
- List to Is it mine?: 1 compare to your notes
- Is it mine? to Delete: 2 only your own
- Delete to List again: 3 re-run the list
Four causes explain most surprises. A stopped or running instance bills for compute or for its disk. A volume left behind after its instance was deleted still bills for its size. A NAT gateway bills for every hour it exists and for the data it moves. A public IPv4 address bills hourly, including an Elastic IP that is not attached to anything.
The sweep lists each of those. A volume in the available state is not attached to anything, which makes it a prime suspect. An Elastic IP with no association is unused. A NAT gateway that is available is billing right now. Run the list again in every Region you have ever used, because the console only shows one at a time.
Delete from the inside out: instances first, then volumes, then addresses and gateways, and last the network. Azure has the same traps, with budgets in Cost Management and resource groups that you can delete in one go. Google Cloud has budgets and alerts under Billing and lets you delete a whole project to remove everything in it.
Try it yourself
Every command here is a read-only list, free to run in CloudShell. The final step shows delete commands as templates, with placeholders that you replace only for things you created.
- 1Read the bill by service
In the console open Billing and Cost Management, then Bills. Expand the services with a charge and note the Region shown for each. Open the Free Tier page to see how much of each free allowance you have used. Use the console for this rather than the Cost Explorer API, because API calls to Cost Explorer are charged per request.
- 2Sweep the current Region
aws ec2 describe-instances --filters Name=instance-state-name,Values=pending,running,stopping,stopped --query "Reservations[].Instances[].[InstanceId,InstanceType,State.Name]" --output table aws ec2 describe-volumes --query "Volumes[].[VolumeId,Size,State]" --output table aws ec2 describe-addresses --query "Addresses[].[PublicIp,AssociationId]" --output table aws ec2 describe-nat-gateways --filter Name=state,Values=pending,available --query "NatGateways[].[NatGatewayId,State]" --output table aws ec2 describe-snapshots --owner-ids self --query "Snapshots[].[SnapshotId,VolumeSize]" --output table aws elbv2 describe-load-balancers --query "LoadBalancers[].[LoadBalancerName,Type]" --output table aws rds describe-db-instances --query "DBInstances[].[DBInstanceIdentifier,DBInstanceStatus]" --output table
Empty output means nothing of that kind exists. A volume in the
availablestate is attached to nothing. An address row with no association is unused. - 3Sweep every Region for the big three
for r in $(aws ec2 describe-regions --query "Regions[].RegionName" --output text); do echo "== $r" aws ec2 describe-instances --region "$r" --filters Name=instance-state-name,Values=pending,running,stopping,stopped --query "Reservations[].Instances[].InstanceId" --output text aws ec2 describe-volumes --region "$r" --query "Volumes[].VolumeId" --output text aws ec2 describe-nat-gateways --region "$r" --filter Name=state,Values=pending,available --query "NatGateways[].NatGatewayId" --output text done
Each Region should print its name and then nothing. Any ID that appears under a Region is something to look at. This checks the Regions enabled for your account. It does not cover every service, so keep reading the Bills page too.
- 4Delete what is yours, inside out
aws ec2 terminate-instances --instance-ids <instance-id> aws ec2 delete-volume --volume-id <volume-id> aws ec2 delete-nat-gateway --nat-gateway-id <nat-gateway-id> aws ec2 release-address --allocation-id <allocation-id>
Run a line only for something you recognise as your own. Replace the placeholder first, and add
--region <name>if it is in another Region. A NAT gateway takes a short while to delete, and its Elastic IP is not released with it, so release that address too. After deleting, run the sweep again. - 5Check again tomorrow
Billing data lags, so a mistake may not show up for hours. Look at the Bills page again the next day and confirm that nothing new has appeared. Put a weekly reminder in your calendar to run the sweep.
What breaks, and where to look
The bill shows charges for a service you never touched, but the console shows nothing.
- WHY IT HAPPENS
- The resource is in a Region you are not looking at, often one a tutorial or template used by default.
- CHECK FIRST
- On the Bills page, expand the service and read its Region. Then run the sweep with
--region <that-region>and look in the console with that Region selected.
You terminated the instance, but a disk or address is still on the bill.
- WHY IT HAPPENS
- A volume attached as a second disk, or one created on its own, is not deleted with the instance unless it was set to be. An Elastic IP stays allocated until you release it.
- CHECK FIRST
- Run
aws ec2 describe-volumes --filters Name=status,Values=availablefor orphaned disks andaws ec2 describe-addressesfor unattached addresses.
An instance shows as stopped, but you still see charges.
- WHY IT HAPPENS
- Stopping ends the compute charge but not the cost of the attached disk, and a public Elastic IP kept on the instance keeps billing.
- CHECK FIRST
- Run
aws ec2 describe-instances --filters Name=instance-state-name,Values=stopped. If you are finished with the machine, terminate it instead.
When not to use this
- Checking by hand does not scale. Teams tag every resource, create budgets per tag and define infrastructure as code so one destroy command removes a whole environment.
- A sweep only finds what you ask about. Keep reading the Bills page, because a service you forgot to list is exactly what will surprise you.
Check yourself
Which volume state means a disk is attached to nothing?
available. A volume that is in-use is attached to an instance.
Why does a stopped instance still cost money?
The compute charge stops, but its disk is still stored and billed, and so is any Elastic IP kept on it.
Why sweep every Region and not just your usual one?
Resources belong to a single Region, and the console shows one Region at a time, so something forgotten elsewhere is invisible but still billed.
Keep going
Lesson 62 of 92 in the DevOps path