criticalIntermediateSymptom-first#s3#deletion#versioning#recovery

S3 objects accidentally deleted — recover from versioning

A production object/key range disappears after a cleanup script

Likely source:data-lossconfiguration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • A production object/key range disappears after a cleanup script
  • Website/asset references 404 where they returned 200 yesterday
  • Lifecycle rule unexpectedly expired a prefix

Recovery speed is decided years before the incident: versioning-enabled buckets are recoverable in minutes, everything else depends on what backups exist. Time spent arguing with a lifecycle rule is time data stays lost.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Versioning state

    aws s3api get-bucket-versioning --bucket <BUCKET>

  • Object versions and delete markers

    aws s3api list-object-versions --bucket <BUCKET> --prefix <KEY>

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • The key 404s but versions exist in list-object-versions
  • A DeleteMarker is the newest entry

Confirm before fixing

aws s3api list-object-versions --bucket <BUCKET> --prefix <KEY> --query 'DeleteMarkers[0]'
READ

Confirm the marker

Fix

aws s3api delete-object --bucket <BUCKET> --key <KEY> --version-id <MARKER-VERSION-ID>
CHANGE

Remove the delete marker to restore the last version (no copy needed)

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws s3api get-bucket-versioning --bucket <BUCKET>
READ

Confirm versioning

2
aws s3api list-object-versions --bucket <BUCKET> --prefix <KEY>
READ

Enumerate available versions

3
aws s3api copy-object --bucket <BUCKET> --key <RESTORED-KEY> --copy-source "<BUCKET>/<KEY>?versionId=<GOOD-VERSION-ID>"
CHANGE

Copy the known-good version to a restore target

Expected: ETag returned

4
aws s3api delete-object --bucket <BUCKET> --key <KEY> --version-id <DELETE-MARKER-VERSION-ID>
DESTRUCTIVE

If restoring in place over a versioned bucket, remove the marker after copying the version

Prevention

  • Enable versioning + object lock on every bucket holding user data.
  • Add a retention rule for delete markers, and bulletproof the deletion script with dry-runs.
  • Back up unversioned buckets with S3 Replication or a cross-account sync.

Related runbooks

Same stack, same failure family — worth a skim.

S3 returns 403 — bucket policy or public access blockhigh

All client types get 403 Forbidden on a bucket

AWSFoundation3 causes
Open runbook
S3 presigned URL expired or generated wrongmedium

Browser upload/download fails with AccessDenied immediately

AWSFoundation3 causes
Open runbook
Cross-account S3 access blocked — policy mismatchmedium

A role in account A cannot read a bucket owned by account B

AWSAdvanced3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.