highAdvancedSymptom-first#vpc sc#vpc service controls#cloud storage#restricted#perimeter

VPC Service Controls blocking Cloud Storage / API access

gsutil cp fails with 403 'denied by VPC Service Controls' or PERMISSION_DENIED

Likely source:connectivitysecurity

Focused incident mode — strips away everything but this runbook.

Symptoms

  • gsutil cp fails with 403 'denied by VPC Service Controls' or PERMISSION_DENIED
  • APIs inside the perimeter stop working for a client outside it
  • Access works from an in-perimeter VM but not from a developer laptop
  • Cloud Functions/Cloud Run in a restricted project cannot reach Storage

VPC SC blocks fail with 403 but are a policy decision, not an IAM bug. Confirm the caller's access level and the perimeter's ingress/egress rules before touching IAM bindings.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Offending command/error

    gsutil ls gs://<BUCKET>/

  • In-scope resources and perimeter

    gcloud access-context-manager perimeters list

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Works from an in-perimeter VM, fails from a laptop
  • Error body names VPC SC
  • Caller's IP/identity not in the perimeter access level

Confirm before fixing

gcloud access-context-manager levels describe <LEVEL> --organization <ORG> --format='value(name,basic)'
READ

Review the access level requirements

Expected: An IP subnetwork or identity requirement the caller misses

Fix

gcloud access-context-manager perimeters update <PERIMETER> --organization <ORG> --add-access-levels <LEVEL>
CHANGE

Add the caller's access level to the perimeter (or use a bridge)

Expected: Updated [<PERIMETER>].

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
gsutil ls gs://<BUCKET>/
READ

Reproduce and read the VPC SC error

2
gcloud access-context-manager perimeters list --organization <ORG> --format='table(name,status.accessLevels[])'
READ

Find the perimeter covering the resource

3
gcloud access-context-manager perimeters update <PERIMETER> --organization <ORG> --add-access-levels <LEVEL>
CHANGE

Add the caller's access level (correct of the analysis is an access level issue)

4
gsutil ls gs://<BUCKET>/
READ

Verify access is restored

Prevention

  • Rehearse perimeter access levels with a test client inside and outside the perimeter.
  • Keep perimeter changes reviewed and version-controlled; they are privacy/security controls.
  • Document which projects/services are in and out of scope to avoid surprise blocks.
  • Log VPC SC denials to Cloud Logging and alert on them as access-control events.

Related runbooks

Same stack, same failure family — worth a skim.

Cloud Storage 403 — permission denied on objects or buckethigh

gsutil cp/ls fails with AccessDeniedException: 403

Google CloudFoundation3 causes
Open runbook
GCP firewall / Cloud NAT — SNAT exhaustion or traffic blockedhigh

Outbound requests from VMs time out while inbound to the VPC works

Google CloudAdvancedUSE3 causes
Open runbook
Cloud Run ingress 'Requires Authentication' — 401 when invokedmedium

curl to the Cloud Run URL returns 401 Unauthorized

Google CloudFoundation3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.