AKS pull from ACR fails: 403, i/o timeout, TLS handshake, blocked data endpoint
When identity is fine but pulls still fail, the network path is wrong: private DNS, peering, an IP allow list, a firewall or the data endpoint.
What you see
failed to fetch anonymous token: unexpected status: 403 Forbiddendial tcp ACR_PRIVATE_IP:443: i/o timeoutnet/http: TLS handshake timeoutGet "https://REGION.data.mcr.microsoft.com/...": connection reset by peer
What it means
Every pull goes through three stages, and the fastest way to the cause is to find which stage fails. Stage 2 is the registry's login endpoint (REGISTRY.azurecr.io). Stage 3 is the data endpoint that serves layers, which is a separate host: *.blob.core.windows.net or REGISTRY.REGION.data.azurecr.io for ACR, and *.data.mcr.microsoft.com for MCR.
If the manifest resolves but layers fail, the login endpoint is open and the data endpoint is blocked. An unauthenticated 401 from curl -v https://REGISTRY.azurecr.io/v2/ is expected and proves the login endpoint is reachable.
How to confirm the cause
- 1Check the registry endpoint is reachable
curl -v https://REGISTRY.azurecr.io/v2/
A
401is a good sign: the endpoint is reachable. - 2Check how the name resolves from the cluster
nslookup REGISTRY.azurecr.io && nslookup REGISTRY.REGION.data.azurecr.io
Run from a debug pod or node. With a private endpoint, both should resolve to private addresses.
- 3Test TCP to the registry from a node or pod
telnet REGISTRY_IP 443
Get the address from nslookup. A failure points to routing, peering or a firewall.
Causes and fixes
403: private DNS zone not linked, or public access limited
- HOW TO TELL
403 Forbiddenfor an anonymous token.- ROOT CAUSE
- If the private endpoint and the cluster are in different virtual networks, the cluster's network needs a link to the
privatelink.azurecr.ioprivate DNS zone. If the registry allows only selected public networks, the cluster's egress IP must be on the allow list. - FIX
- Add the virtual network link to the private DNS zone, or add the AKS load balancer's public IP to the registry's allowed range.
az acr network-rule add --name REGISTRY --ip-address AKS_LB_PUBLIC_IP
i/o timeout on a private endpoint
- HOW TO TELL
i/o timeoutto a private IP, and the registry uses Private Link.- ROOT CAUSE
- The cluster and the private endpoint are in different virtual networks without working peering, or the route is blocked.
- FIX
- Use virtual network peering and check its status is
Connected. If it isDisconnected, delete the peering on both networks and recreate it. Alternatively route through Azure Firewall in a hub and spoke design.az network vnet peering list --resource-group RESOURCE_GROUP --vnet-name VNET_NAME --output table
A firewall or network appliance blocks the path
- HOW TO TELL
TLS handshake timeoutorconnection reset by peer.- ROOT CAUSE
- A firewall, proxy or network virtual appliance drops or mangles the connection.
- FIX
- Check the firewall and proxy rules and the validated routes for the appliance.
Data endpoint blocked or mis-resolved
- HOW TO TELL
- Authentication succeeds, then layer downloads fail.
- ROOT CAUSE
- Egress filtering allows
REGISTRY.azurecr.iobut not the separate data endpoint. With a private endpoint, theprivatelink.blob.core.windows.netzone may be missing. - FIX
- Allow the data endpoints on TCP 443 (
*.blob.core.windows.netfor ACR,*.data.mcr.microsoft.comfor MCR), or fix the private DNS zone and link it to the AKS virtual network.
How to stop it coming back
- Document every endpoint the cluster must reach, including the data endpoints, and test them after a network change.
- Keep private DNS zones and links in IaC.
- Test a pull from a new node after each firewall change.
Keep going
- Runbook: AKS ImagePullBackOff from ACR: 401 Unauthorized
- Runbook: AKS pulls: pull QPS exceeded, 429 rate limits and stale tags
- Cheat sheet: Azure CLI
- Cheat sheet: dig
- Lesson: the networking topic in Learn
Sources
Microsoft Learn (learn.microsoft.com), read 18 Nov 2026. The page's own metadata shows it was last updated 30 Oct 2026.