Lesson 8 of 8 in Cloud fundamentals with AWS

Find and delete surprise AWS costs

Core10 minWritten for AWS CLI v2 (preinstalled in CloudShell)

Cleanup is the habit of listing what you created and deleting what you no longer need, because the cloud keeps billing for anything that exists. Surprise bills almost always come from a few forgotten resources.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Accounts and billing safety before you build, Virtual machines on EC2

The problem this solves

Most surprise bills have nothing to do with heavy usage. They come from a small thing left running for weeks: a VM, an unattached disk, a NAT gateway or an unused public IP address. The charge per hour is small, which is why nobody notices until the bill arrives.

The cloud console does not warn you when a lab is finished. Deleting is your job, and it is easy to look in the wrong place. Resources live in one Region, and a single forgotten Region can hold the problem.

A sweep is a short list of read-only commands you run after every lab and once a week. It costs nothing and turns the bill from a surprise into something you already knew.

How it works

Where surprise bills usually come from
Where surprise bills usually come fromForgotten virtual machines, NAT gateways, unattached volumes and unused public IP addresses all feed into your bill because each is charged by time or size.compute hourshours andtrafficstorageaddress hoursYour billForgotten VMbilled per hourNAT gatewayhourly plus dataUnattached volumebilled per GB-monthUnused public IPbilled per hour
  • Control
  • Runs code
  • Network
  • Data

Forgotten virtual machines, NAT gateways, unattached volumes and unused public IP addresses all feed into your bill because each is charged by time or size.

  1. Forgotten VM to Your bill: compute hours
  2. NAT gateway to Your bill: hours and traffic
  3. Unattached volume to Your bill: storage
  4. Unused public IP to Your bill: address hours
A safe cleanup loop
A safe cleanup loopYou list everything, check each item against your own notes, delete only what is yours, and then list again to prove it is gone.1 compare toyour notes2 only yourown3 re-run thelistListread-only commandsIs it mine?check names and tagsDeleteinside outList againconfirm it is gone
  • Control
  • People and clients
  • Runs code

You list everything, check each item against your own notes, delete only what is yours, and then list again to prove it is gone.

  1. List to Is it mine?: 1 compare to your notes
  2. Is it mine? to Delete: 2 only your own
  3. Delete to List again: 3 re-run the list

Four causes explain most surprises. A stopped or running instance bills for compute or for its disk. A volume left behind after its instance was deleted still bills for its size. A NAT gateway bills for every hour it exists and for the data it moves. A public IPv4 address bills hourly, including an Elastic IP that is not attached to anything.

The sweep lists each of those. A volume in the available state is not attached to anything, which makes it a prime suspect. An Elastic IP with no association is unused. A NAT gateway that is available is billing right now. Run the list again in every Region you have ever used, because the console only shows one at a time.

Delete from the inside out: instances first, then volumes, then addresses and gateways, and last the network. Azure has the same traps, with budgets in Cost Management and resource groups that you can delete in one go. Google Cloud has budgets and alerts under Billing and lets you delete a whole project to remove everything in it.

Try it yourself

Every command here is a read-only list, free to run in CloudShell. The final step shows delete commands as templates, with placeholders that you replace only for things you created.

  1. 1
    Read the bill by service

    In the console open Billing and Cost Management, then Bills. Expand the services with a charge and note the Region shown for each. Open the Free Tier page to see how much of each free allowance you have used. Use the console for this rather than the Cost Explorer API, because API calls to Cost Explorer are charged per request.

  2. 2
    Sweep the current Region
    aws ec2 describe-instances --filters Name=instance-state-name,Values=pending,running,stopping,stopped --query "Reservations[].Instances[].[InstanceId,InstanceType,State.Name]" --output table
    aws ec2 describe-volumes --query "Volumes[].[VolumeId,Size,State]" --output table
    aws ec2 describe-addresses --query "Addresses[].[PublicIp,AssociationId]" --output table
    aws ec2 describe-nat-gateways --filter Name=state,Values=pending,available --query "NatGateways[].[NatGatewayId,State]" --output table
    aws ec2 describe-snapshots --owner-ids self --query "Snapshots[].[SnapshotId,VolumeSize]" --output table
    aws elbv2 describe-load-balancers --query "LoadBalancers[].[LoadBalancerName,Type]" --output table
    aws rds describe-db-instances --query "DBInstances[].[DBInstanceIdentifier,DBInstanceStatus]" --output table

    Empty output means nothing of that kind exists. A volume in the available state is attached to nothing. An address row with no association is unused.

  3. 3
    Sweep every Region for the big three
    for r in $(aws ec2 describe-regions --query "Regions[].RegionName" --output text); do
      echo "== $r"
      aws ec2 describe-instances --region "$r" --filters Name=instance-state-name,Values=pending,running,stopping,stopped --query "Reservations[].Instances[].InstanceId" --output text
      aws ec2 describe-volumes --region "$r" --query "Volumes[].VolumeId" --output text
      aws ec2 describe-nat-gateways --region "$r" --filter Name=state,Values=pending,available --query "NatGateways[].NatGatewayId" --output text
    done

    Each Region should print its name and then nothing. Any ID that appears under a Region is something to look at. This checks the Regions enabled for your account. It does not cover every service, so keep reading the Bills page too.

  4. 4
    Delete what is yours, inside out
    aws ec2 terminate-instances --instance-ids <instance-id>
    aws ec2 delete-volume --volume-id <volume-id>
    aws ec2 delete-nat-gateway --nat-gateway-id <nat-gateway-id>
    aws ec2 release-address --allocation-id <allocation-id>

    Run a line only for something you recognise as your own. Replace the placeholder first, and add --region <name> if it is in another Region. A NAT gateway takes a short while to delete, and its Elastic IP is not released with it, so release that address too. After deleting, run the sweep again.

  5. 5
    Check again tomorrow

    Billing data lags, so a mistake may not show up for hours. Look at the Bills page again the next day and confirm that nothing new has appeared. Put a weekly reminder in your calendar to run the sweep.

What breaks, and where to look

  • The bill shows charges for a service you never touched, but the console shows nothing.

    WHY IT HAPPENS
    The resource is in a Region you are not looking at, often one a tutorial or template used by default.
    CHECK FIRST
    On the Bills page, expand the service and read its Region. Then run the sweep with --region <that-region> and look in the console with that Region selected.
  • You terminated the instance, but a disk or address is still on the bill.

    WHY IT HAPPENS
    A volume attached as a second disk, or one created on its own, is not deleted with the instance unless it was set to be. An Elastic IP stays allocated until you release it.
    CHECK FIRST
    Run aws ec2 describe-volumes --filters Name=status,Values=available for orphaned disks and aws ec2 describe-addresses for unattached addresses.
  • An instance shows as stopped, but you still see charges.

    WHY IT HAPPENS
    Stopping ends the compute charge but not the cost of the attached disk, and a public Elastic IP kept on the instance keeps billing.
    CHECK FIRST
    Run aws ec2 describe-instances --filters Name=instance-state-name,Values=stopped. If you are finished with the machine, terminate it instead.

When not to use this

  • Checking by hand does not scale. Teams tag every resource, create budgets per tag and define infrastructure as code so one destroy command removes a whole environment.
  • A sweep only finds what you ask about. Keep reading the Bills page, because a service you forgot to list is exactly what will surprise you.

Check yourself

Which volume state means a disk is attached to nothing?

available. A volume that is in-use is attached to an instance.

Why does a stopped instance still cost money?

The compute charge stops, but its disk is still stored and billed, and so is any Elastic IP kept on it.

Why sweep every Region and not just your usual one?

Resources belong to a single Region, and the console shows one Region at a time, so something forgotten elsewhere is invisible but still billed.

Lesson 62 of 92 in the DevOps path

Search

Search cheat sheets and projects