Cloud Networking
Intermediate 16 min read Cloud Networking

VPN and Hybrid Networking

On-prem ↔ cloud, over tunnels or private lines

Hybrid connectivity joins on-prem and cloud: site-to-site VPN over IPsec, Direct Connect/ExpressRoute/Interconnect for dedicated circuits, plus transit gateways to route it all. Latency, bandwidth and SLA drive the choice.

VPNsite-to-siteDirect ConnectExpressRouteInterconnectBGPhybrid cloud
Prerequisites: Route Tables and Gateways VPC Peering and Transit
Before we start

What you will learn

  • Compare IPsec VPN vs dedicated circuits (DX/ExpressRoute/Interconnect)
  • Explain BGP between on-prem and cloud gateways
  • Design a resilient hybrid path
In 60 seconds

Site-to-site VPNs tunnel over the internet; Direct Connect / ExpressRoute / Interconnect provide private physical circuits with SLAs. BGP exchanges routes between your routers and cloud gateways. Use tunnels for cost-critical redundancy, private lines when latency/SLA matter.

Introduction

What is this?

Hybrid networking connects your on-premises network(s) to the cloud. Site-to-site VPN (AWS VPN, Azure VPN gateway, GCP Cloud VPN) runs IPsec over the internet — affordable, redundant if you run two tunnels. Dedicated circuits (AWS Direct Connect, Azure ExpressRoute, GCP Cloud Interconnect) traverse carrier-managed private paths with higher bandwidth and SLAs (down/up asymmetrical). Cloud gateways run BGP to exchange routes with on-prem routers; Transit Gateway / hub VNet route hybrid traffic centrally, and route propagation automates discovery.

Motivation

Why do you need it?

  • Hybrid is the reality of every migration: lift-and-shift before cloud-native.
  • Latency-critical apps (databases, storage) do better on private circuits than noisy internet.
  • BGP gives automatic failover: one tunnel down, routes re-advertise.
  • Compliance (data residency) often demands dedicated connectivity with SPI retentions.
Think about it

Mental model

Imagine

Paying postal mail vs a dedicated courier line

Think of

VPN vs Direct Connect

IPsec VPN is guaranteed-delivery postal mail — works, cheap, can be slow and variable. Direct Connect is your own courier team with a fixed-speed highway: consistent, premium, more setup.

Mechanics

How it works

  • IPsec tunnel: on-prem VPN device + cloud VPN gateway encrypt GRE/IPsec over the internet; two tunnels for HA.
  • BGP: cloud gateway peers BGP with on-prem router, exchanging prefixes; route propagation feeds VPC route tables.
  • Direct Connect: order a port, get LOA, cross-connect at carrier, establish virtual interfaces (private: to VPC; public: to AWS services).
  • ExpressRoute/Interconnect: similar physical circuits with provider-managed SFI and BGP VIFs.
  • Redundancy: multiple tunnels/circuits with distinct paths and AS path prepepend so failover is deterministic.

Hybrid cloud networking.

On-prem premises
IPsec VPN tunnel
Direct Connect / ExpressRoute
Cloud gateway / TGW
Cloud VPC
Real world

In production

  • A bank runs critical replication over Direct Connect and keeps the VPN as an emergency path.
  • An enterprise uses Azure ExpressRoute with a hub VNet gateway and spokes peered in for hybrid apps.
  • A startup uses two AWS Site-to-Site VPN tunnels with BGP to reach its office for a single app.
Guidelines

Best practices

  • Always run BGP with active/standby so cloud routes follow tunnel health.
  • Prepend AS paths on the backup path to pin failover behavior.
  • Size circuits to peak + headroom; provision a second provider path for resilience.
  • Keep IP address plans disjoint and document BGP communities/prefix filters.
Watch out

Common mistakes

  • Running only one tunnel and 'hoping' the ISP is reliable.
  • Advertising overlapping prefixes from on-prem and cloud.
  • Forgetting route propagation so BGP routes never reach VPC subnets.
  • Using internet VPN for latency-sensitive DB traffic and then blaming the cloud.
Root cause analysis

Common issues

Error
DHCP fails to cross the tunnel
Root cause

ACL at either end drops the protocol or the tunnel doesn't carry broadcast/multicast

Fix

Use routed (L3) connectivity and forward specific prefixes over the tunnel.

Error
Fails over slowly on tunnel loss
Root cause

BGP hold timers + no path pre-preference

Fix

Reduce hold timers and configure AS path prepends so the secondary path is preferred only if primary dies.

Error
DX virtual interface error lights but BGP never establishes
Root cause

LOA cross-connect not complete or VLAN/VNI mismatch

Fix

Verify carrier cross-connect, VIF state, and BGP authentication match (MD5 or keys).

When things break

Debugging tips

  • Tunnel down: check IKE/phase states, BGP connectivity, and both ends' ACLs.
  • Route flapping: look at BGP sessions, AS-path prepend, and site-to-site tunnel health.
  • Bandwidth: monitor DX/ExpressRoute utilization; plane traffic may exhaust QoS.
  • Latency: compare tunnel latency vs circuit latency from a representative client.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which option uses a dedicated physical circuit with SLAs?

2

BGP lets cloud gateways automatically discover routes from on-prem.

Go deeper

Official documentation

AWS — Site-to-Site VPN
docs
AWS — Direct Connect
docs
Azure — VPN gateway docs
docs