Overlay Networks Explained
Troubleshoot transport and tenant forwarding as separate layers of the same packet journey.
Underlay first
An overlay creates logical connectivity on shared transport. Start by identifying the tunnel endpoints and the path between their transport addresses. In the diagram below, the solid links and transit routers form that transport. The dashed relationship is a logical service between the edges; it is not an additional physical link.
Two views of one network
| View | What it must know |
|---|---|
| Underlay | IP routes to VTEP addresses, usable links, MTU and transport policy. |
| Overlay | Segment membership, tenant MAC reachability and replication rules. |
| Tenant host | Its own subnet, neighbor resolution and any default gateway. |
The underlay can use static routing or an appropriate routing protocol. OSPF or eBGP are design choices, not requirements imposed by the VXLAN header. Ethernet still exists on physical links; the distinction is that tenant bridging is not the transit fabric’s forwarding job.
Recognize the same idea in different technologies
| Example | Logical service | Transport or mechanism |
|---|---|---|
| 802.1Q VLAN | A separate Ethernet broadcast domain | A tag distinguishes traffic on shared Ethernet links; not an IP tunnel. |
| GRE | A virtual link between IP endpoints | An extra GRE and outer IP wrapper. |
| IPsec tunnel mode | A protected packet path | Authenticated/encrypted inner traffic inside outer IP transport. |
| DMVPN | Multipoint overlay connectivity | NHRP and multipoint GRE, commonly protected with IPsec. |
| VXLAN | Tenant Ethernet segments | Ethernet inside UDP/IP between VTEPs. |
| LISP | Separate endpoint identity from location | A mapping system relates EIDs to routing locators. |
| SD-WAN | Policy-driven WAN connectivity | Overlay paths and control/management functions; implementation varies. |
These technologies share an abstraction, not a common header or identical security guarantees. A VLAN is a logical segmentation example, while GRE and VXLAN explicitly tunnel traffic. A VPN service is not synonymous with any one encapsulation.
Check a known unicast path
VTEP1 underlay: 10.255.0.2/32 -> R1 or R2 (equal cost)
VTEP1 overlay: VNI 5012, H2 MAC -> VTEP 10.255.0.2
VTEP2 overlay: VNI 5012, H2 MAC -> local VLAN 20
Tenant result: H1 can resolve and reach H2 when all stages workThese are normalized teaching records, not executable CLI or a measured result. First the overlay selects a remote VTEP. Then the underlay routes the outer packet to it. Finally the receiving edge selects the local attachment. Transport reachability alone cannot prove any of the tenant lookups.
A redundant path is not a second overlay
R1 and R2 offer two physical routed paths between the same endpoint pair. The underlay can select a next hop using an ECMP hash. One tenant flow normally stays on one path under flow-based hashing; multiple flows may use different paths. A source-port entropy value helps distinguish them. Exact behavior, polarization and rehashing depend on the forwarding platform.
When a path fails, underlay convergence can restore reachability without changing VNI 5012 or host membership. There may still be loss during detection and repair. This is not a promise of lossless failover, nor does every pair of equal-looking links necessarily form ECMP.
Separate three independent faults
| Observation | Likely next checks | What not to assume |
|---|---|---|
| No route to remote VTEP | Underlay routing, source address, physical path | A correct VNI can repair missing IP reachability. |
| Small transport probes pass; larger overlay traffic fails | MTU along every selected path, ACLs, counters | A small ping proves full-size UDP forwarding. |
| Outer traffic reaches VTEP2; tenant delivery fails | VNI membership, MAC location, local VLAN/port, endpoint ARP | A reachable VTEP proves matching segment configuration. |
Use captures at access and transport points where available. An access capture shows the tenant frame; an underlay capture shows the outer VTEP addresses and VXLAN wrapper. Check both directions, not just H1’s request. A wrong learning or replication policy can break initial ARP even while a previously learned unicast path still works.
Where EVPN fits later
An EVPN deployment uses BGP to distribute overlay reachability and related information. That control plane is distinct from the IP routing that carries VXLAN packets. A fabric might use eBGP for the underlay and MP-BGP EVPN for the overlay, but those are different address families and responsibilities. A flood-and-learn fabric instead uses its selected data-plane learning and replication design.
The next lessons establish VTEP reachability, size the MTU and inspect ECMP before configuring VLAN-to-VNI mappings and replication. EVPN MAC/IP advertisements and routed overlays follow after the Layer 2 data path is understood.
Benefits and operational cost
| Benefit | Trade-off |
|---|---|
| Logical service changes need not recable transit | Endpoint and transport state must both be diagnosed. |
| Multiple tenants can share links | Segmentation policy, capacity and identifier scope still need validation. |
| Routing can repair transport paths | Detection/convergence can interrupt service. |
| Encapsulation hides tenant details from transit | Headers consume MTU; troubleshooting needs captures at both layers. |
The layers are operationally distinct but share bandwidth, queues, failure modes and hardware resources. Overlay changes can affect underlay load, and underlay faults can disrupt every overlay using the failed path. Independence does not mean isolation from physical capacity.
Further reading
Key Takeaways
- Underlay routing delivers outer packets; overlay state selects tenant delivery.
- VTEP reachability is necessary but not sufficient for a working tenant segment.
- ECMP offers transport alternatives without creating separate VNIs.
- Check MTU, transport policy and both traffic directions as well as routing.
- EVPN supplies overlay control-plane information and does not replace the underlay.