AzureSenior

AKS pull from ACR fails: 403, i/o timeout, TLS handshake, blocked data endpoint

When identity is fine but pulls still fail, the network path is wrong: private DNS, peering, an IP allow list, a firewall or the data endpoint.

What you see

  • failed to fetch anonymous token: unexpected status: 403 Forbidden
  • dial tcp ACR_PRIVATE_IP:443: i/o timeout
  • net/http: TLS handshake timeout
  • Get "https://REGION.data.mcr.microsoft.com/...": connection reset by peer

What it means

Every pull goes through three stages, and the fastest way to the cause is to find which stage fails. Stage 2 is the registry's login endpoint (REGISTRY.azurecr.io). Stage 3 is the data endpoint that serves layers, which is a separate host: *.blob.core.windows.net or REGISTRY.REGION.data.azurecr.io for ACR, and *.data.mcr.microsoft.com for MCR.

If the manifest resolves but layers fail, the login endpoint is open and the data endpoint is blocked. An unauthenticated 401 from curl -v https://REGISTRY.azurecr.io/v2/ is expected and proves the login endpoint is reachable.

How to confirm the cause

  1. 1
    Check the registry endpoint is reachable
    curl -v https://REGISTRY.azurecr.io/v2/

    A 401 is a good sign: the endpoint is reachable.

  2. 2
    Check how the name resolves from the cluster
    nslookup REGISTRY.azurecr.io && nslookup REGISTRY.REGION.data.azurecr.io

    Run from a debug pod or node. With a private endpoint, both should resolve to private addresses.

  3. 3
    Test TCP to the registry from a node or pod
    telnet REGISTRY_IP 443

    Get the address from nslookup. A failure points to routing, peering or a firewall.

Causes and fixes

  • 403: private DNS zone not linked, or public access limited

    HOW TO TELL
    403 Forbidden for an anonymous token.
    ROOT CAUSE
    If the private endpoint and the cluster are in different virtual networks, the cluster's network needs a link to the privatelink.azurecr.io private DNS zone. If the registry allows only selected public networks, the cluster's egress IP must be on the allow list.
    FIX
    Add the virtual network link to the private DNS zone, or add the AKS load balancer's public IP to the registry's allowed range.
    az acr network-rule add --name REGISTRY --ip-address AKS_LB_PUBLIC_IP
  • i/o timeout on a private endpoint

    HOW TO TELL
    i/o timeout to a private IP, and the registry uses Private Link.
    ROOT CAUSE
    The cluster and the private endpoint are in different virtual networks without working peering, or the route is blocked.
    FIX
    Use virtual network peering and check its status is Connected. If it is Disconnected, delete the peering on both networks and recreate it. Alternatively route through Azure Firewall in a hub and spoke design.
    az network vnet peering list --resource-group RESOURCE_GROUP --vnet-name VNET_NAME --output table
  • A firewall or network appliance blocks the path

    HOW TO TELL
    TLS handshake timeout or connection reset by peer.
    ROOT CAUSE
    A firewall, proxy or network virtual appliance drops or mangles the connection.
    FIX
    Check the firewall and proxy rules and the validated routes for the appliance.
  • Data endpoint blocked or mis-resolved

    HOW TO TELL
    Authentication succeeds, then layer downloads fail.
    ROOT CAUSE
    Egress filtering allows REGISTRY.azurecr.io but not the separate data endpoint. With a private endpoint, the privatelink.blob.core.windows.net zone may be missing.
    FIX
    Allow the data endpoints on TCP 443 (*.blob.core.windows.net for ACR, *.data.mcr.microsoft.com for MCR), or fix the private DNS zone and link it to the AKS virtual network.

How to stop it coming back

  • Document every endpoint the cluster must reach, including the data endpoints, and test them after a network change.
  • Keep private DNS zones and links in IaC.
  • Test a pull from a new node after each firewall change.

Keep going

Sources

Microsoft Learn (learn.microsoft.com), read 18 Nov 2026. The page's own metadata shows it was last updated 30 Oct 2026.

Search

Search cheat sheets and projects