Lesson 6 of 8 in Cloud fundamentals with AWS
Object storage with S3
Amazon S3 stores files as objects inside buckets and serves them over an API. You do not manage disks, and a bucket grows as you add data. You pay for the storage, the requests and the data that leaves AWS.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: IAM: who is allowed to do what
The problem this solves
Websites, backups, logs and build output are all files that many machines need to read. A server's own disk dies with the server and is awkward to share. You want a place that holds files durably and can be reached by name.
S3 provides that, but it makes one mistake easy: a public bucket. Data left readable by the whole internet is one of the most common ways companies leak data, usually through a wrongly written policy.
S3 has features to stop this and features to protect against deleting your own files. You will use both: Block Public Access and versioning.
How it works
- People and clients
- Data
You upload a file to a bucket with versioning on, and uploading it again creates a new version while the first version stays stored.
- You to Bucket: 1 upload hello.txt
- Bucket to Version 1: 2 first upload
- Bucket to Version 2: 3 upload again
- People and clients
- Security
- Data
A signed-in request from you is checked against IAM and allowed, while an anonymous request meets Block Public Access, which keeps the bucket from being made public.
- Anonymous visitor to Block Public Access: 1 anonymous request
- Block Public Access to Bucket: 2 public grants blocked
- You, signed in to Bucket: 3 signed request allowed
A bucket is a named container. Its name is unique across all of AWS, not just your account. An object is a file plus its metadata, found by its key, which is its full name such as reports/2026/hello.txt. S3 has no real folders. The slashes in keys are just part of the name, which the console displays as folders.
Block Public Access is a set of four switches, at the account and bucket level, that override any bucket policy or access list that would make data public. New buckets have public access turned off by default. Leave all four on unless you are deliberately hosting a public website. To share one file for a short time, use a presigned URL instead. It is a link with a signature and an expiry time, and it works as long as the credentials that signed it do.
Versioning keeps every version of an object. Deleting or overwriting does not destroy anything, which protects you from mistakes. It also means old versions keep costing storage until you remove them, and you cannot delete a versioned bucket until every version is gone.
Azure calls this Blob Storage, with containers inside a storage account. Google Cloud calls it Cloud Storage, with buckets.
Try it yourself
Run these in CloudShell. The bucket holds two tiny files, so the cost is effectively nothing, and the cleanup step removes every version.
- 1Create a bucket with a unique name
BUCKET=learn-demo-$(date +%s) aws s3 mb "s3://$BUCKET" echo "$BUCKET"
Bucket names are global, lowercase, and 3 to 63 characters. The timestamp makes yours unique. The variable lasts only while this CloudShell session does, so write the name down.
- 2Confirm the bucket is private
aws s3api get-public-access-block --bucket "$BUCKET"
On a new bucket you should see
BlockPublicAcls,IgnorePublicAcls,BlockPublicPolicyandRestrictPublicBucketsall set totrue. If you getNoSuchPublicAccessBlockConfiguration, turn the protection on withaws s3api put-public-access-block --bucket "$BUCKET" --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true. - 3Turn on versioning and upload the same file twice
aws s3api put-bucket-versioning --bucket "$BUCKET" --versioning-configuration Status=Enabled echo "version one" > hello.txt aws s3 cp hello.txt "s3://$BUCKET/hello.txt" echo "version two" > hello.txt aws s3 cp hello.txt "s3://$BUCKET/hello.txt" aws s3api list-object-versions --bucket "$BUCKET" --query "Versions[].[Key,VersionId,IsLatest]" --output table
You should see two rows for
hello.txtwith different version IDs, and only one marked as the latest. - 4Share the file for five minutes without making anything public
curl -s "$(aws s3 presign "s3://$BUCKET/hello.txt" --expires-in 300)"
You should see
version two
aws s3 presignprints a long signed URL, andcurlfetches it. The plain object URL, which looks likehttps://<bucket>.s3.<region>.amazonaws.com/hello.txt, returns AccessDenied in a private browser window because the bucket is still private. Anyone holding the signed URL can read the file until it expires, so treat it like a password. - 5Clean up: remove every version, then the bucket
aws s3api delete-objects --bucket "$BUCKET" --delete "$(aws s3api list-object-versions --bucket "$BUCKET" --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}' --output json)" aws s3 rb "s3://$BUCKET" rm hello.txt aws s3 lsIf the bucket still reports it is not empty, you probably have delete markers too. Open the bucket in the console and use Empty bucket, then Delete. The last command should not list your bucket.
What breaks, and where to look
An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system.
- WHY IT HAPPENS
- Bucket names are unique across all of AWS. Someone else already owns that name.
- CHECK FIRST
- Add something unique to the name, such as a timestamp or random number.
BucketAlreadyOwnedByYoumeans the name is yours already.
An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: The bucket you tried to delete is not empty. You must delete all versions in the bucket.
- WHY IT HAPPENS
- Versioning keeps old versions and delete markers.
aws s3 rb --forcedoes not remove them, so the bucket still has contents. - CHECK FIRST
- List what remains with
aws s3api list-object-versions --bucket "$BUCKET", delete each version and delete marker, or use Empty bucket in the console.
An error occurred (AccessDenied) when calling the PutBucketPolicy operation: ... because public policies are prevented by the BlockPublicPolicy setting in S3 Block Public Access.
- WHY IT HAPPENS
- You tried to attach a policy that grants access to everyone, and Block Public Access refused. The safety net worked.
- CHECK FIRST
- Check whether the data really should be public. To share one file, use a presigned URL. For a public website, put a CDN such as CloudFront in front of a private bucket instead of turning Block Public Access off.
When not to use this
- S3 is not a disk. You cannot edit part of an object in place, so it suits whole files. For a VM's disk use EBS, and for a shared filesystem use EFS.
- Do not leave a bucket public to 'make it work'. A signed link, or a policy that names specific principals, solves nearly every sharing need.
Check yourself
Why can aws s3 rb --force fail on a bucket that has versioning on?
It removes current objects but not old versions or delete markers, so the bucket is still not empty.
How can you share one file without making the bucket public?
Create a presigned URL, which is a signed link that expires.
What do the S3 folders in the console really are?
Just slashes in the object's key. S3 has a flat list of keys and no real folders.
Keep going
Lesson 60 of 92 in the DevOps path