Lesson 6 of 8 in Cloud fundamentals with AWS

Object storage with S3

Core12 minWritten for AWS CLI v2 (preinstalled in CloudShell)

Amazon S3 stores files as objects inside buckets and serves them over an API. You do not manage disks, and a bucket grows as you add data. You pay for the storage, the requests and the data that leaves AWS.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: IAM: who is allowed to do what

The problem this solves

Websites, backups, logs and build output are all files that many machines need to read. A server's own disk dies with the server and is awkward to share. You want a place that holds files durably and can be reached by name.

S3 provides that, but it makes one mistake easy: a public bucket. Data left readable by the whole internet is one of the most common ways companies leak data, usually through a wrongly written policy.

S3 has features to stop this and features to protect against deleting your own files. You will use both: Block Public Access and versioning.

How it works

Overwriting a file keeps the old one
Overwriting a file keeps the old oneYou upload a file to a bucket with versioning on, and uploading it again creates a new version while the first version stays stored.1 uploadhello.txt2 first upload3 upload againYouBucketversioning onVersion 1kept after overwriteVersion 2the current one
  • People and clients
  • Data

You upload a file to a bucket with versioning on, and uploading it again creates a new version while the first version stays stored.

  1. You to Bucket: 1 upload hello.txt
  2. Bucket to Version 1: 2 first upload
  3. Bucket to Version 2: 3 upload again
Who can read an object
Who can read an objectA signed-in request from you is checked against IAM and allowed, while an anonymous request meets Block Public Access, which keeps the bucket from being made public.1 anonymousrequest2 public grants blocked3 signed request allowedAnonymous visitorBlock Public Accessignores public grantsBucketprivate by defaultYou, signed in
  • People and clients
  • Security
  • Data

A signed-in request from you is checked against IAM and allowed, while an anonymous request meets Block Public Access, which keeps the bucket from being made public.

  1. Anonymous visitor to Block Public Access: 1 anonymous request
  2. Block Public Access to Bucket: 2 public grants blocked
  3. You, signed in to Bucket: 3 signed request allowed

A bucket is a named container. Its name is unique across all of AWS, not just your account. An object is a file plus its metadata, found by its key, which is its full name such as reports/2026/hello.txt. S3 has no real folders. The slashes in keys are just part of the name, which the console displays as folders.

Block Public Access is a set of four switches, at the account and bucket level, that override any bucket policy or access list that would make data public. New buckets have public access turned off by default. Leave all four on unless you are deliberately hosting a public website. To share one file for a short time, use a presigned URL instead. It is a link with a signature and an expiry time, and it works as long as the credentials that signed it do.

Versioning keeps every version of an object. Deleting or overwriting does not destroy anything, which protects you from mistakes. It also means old versions keep costing storage until you remove them, and you cannot delete a versioned bucket until every version is gone.

Azure calls this Blob Storage, with containers inside a storage account. Google Cloud calls it Cloud Storage, with buckets.

Try it yourself

Run these in CloudShell. The bucket holds two tiny files, so the cost is effectively nothing, and the cleanup step removes every version.

  1. 1
    Create a bucket with a unique name
    BUCKET=learn-demo-$(date +%s)
    aws s3 mb "s3://$BUCKET"
    echo "$BUCKET"

    Bucket names are global, lowercase, and 3 to 63 characters. The timestamp makes yours unique. The variable lasts only while this CloudShell session does, so write the name down.

  2. 2
    Confirm the bucket is private
    aws s3api get-public-access-block --bucket "$BUCKET"

    On a new bucket you should see BlockPublicAcls, IgnorePublicAcls, BlockPublicPolicy and RestrictPublicBuckets all set to true. If you get NoSuchPublicAccessBlockConfiguration, turn the protection on with aws s3api put-public-access-block --bucket "$BUCKET" --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true.

  3. 3
    Turn on versioning and upload the same file twice
    aws s3api put-bucket-versioning --bucket "$BUCKET" --versioning-configuration Status=Enabled
    echo "version one" > hello.txt
    aws s3 cp hello.txt "s3://$BUCKET/hello.txt"
    echo "version two" > hello.txt
    aws s3 cp hello.txt "s3://$BUCKET/hello.txt"
    aws s3api list-object-versions --bucket "$BUCKET" --query "Versions[].[Key,VersionId,IsLatest]" --output table

    You should see two rows for hello.txt with different version IDs, and only one marked as the latest.

  4. 4
    Share the file for five minutes without making anything public
    curl -s "$(aws s3 presign "s3://$BUCKET/hello.txt" --expires-in 300)"

    You should see

    version two

    aws s3 presign prints a long signed URL, and curl fetches it. The plain object URL, which looks like https://<bucket>.s3.<region>.amazonaws.com/hello.txt, returns AccessDenied in a private browser window because the bucket is still private. Anyone holding the signed URL can read the file until it expires, so treat it like a password.

  5. 5
    Clean up: remove every version, then the bucket
    aws s3api delete-objects --bucket "$BUCKET" --delete "$(aws s3api list-object-versions --bucket "$BUCKET" --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}' --output json)"
    aws s3 rb "s3://$BUCKET"
    rm hello.txt
    aws s3 ls

    If the bucket still reports it is not empty, you probably have delete markers too. Open the bucket in the console and use Empty bucket, then Delete. The last command should not list your bucket.

What breaks, and where to look

  • An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system.

    WHY IT HAPPENS
    Bucket names are unique across all of AWS. Someone else already owns that name.
    CHECK FIRST
    Add something unique to the name, such as a timestamp or random number. BucketAlreadyOwnedByYou means the name is yours already.
  • An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: The bucket you tried to delete is not empty. You must delete all versions in the bucket.

    WHY IT HAPPENS
    Versioning keeps old versions and delete markers. aws s3 rb --force does not remove them, so the bucket still has contents.
    CHECK FIRST
    List what remains with aws s3api list-object-versions --bucket "$BUCKET", delete each version and delete marker, or use Empty bucket in the console.
  • An error occurred (AccessDenied) when calling the PutBucketPolicy operation: ... because public policies are prevented by the BlockPublicPolicy setting in S3 Block Public Access.

    WHY IT HAPPENS
    You tried to attach a policy that grants access to everyone, and Block Public Access refused. The safety net worked.
    CHECK FIRST
    Check whether the data really should be public. To share one file, use a presigned URL. For a public website, put a CDN such as CloudFront in front of a private bucket instead of turning Block Public Access off.

When not to use this

  • S3 is not a disk. You cannot edit part of an object in place, so it suits whole files. For a VM's disk use EBS, and for a shared filesystem use EFS.
  • Do not leave a bucket public to 'make it work'. A signed link, or a policy that names specific principals, solves nearly every sharing need.

Check yourself

Why can aws s3 rb --force fail on a bucket that has versioning on?

It removes current objects but not old versions or delete markers, so the bucket is still not empty.

How can you share one file without making the bucket public?

Create a presigned URL, which is a signed link that expires.

What do the S3 folders in the console really are?

Just slashes in the object's key. S3 has a flat list of keys and no real folders.

Lesson 60 of 92 in the DevOps path

Search

Search cheat sheets and projects