VXLAN · EVPN overlays

MP-BGP EVPN Layer 3 VNI

Route between two tenant subnets using a shared IP-VRF and a Layer 3 VNI.

Different subnets need routing

Example platform: Nexus 9300/9000v · NX-OS 9.3(9) command model. Commands are platform-specific. Use a release and hardware/image with the stated VXLAN, multicast and routing features. All outputs and packet excerpts below are constructed teaching examples, not lab recordings.

H1 is in VLAN 10 / 10.10.10.0/24 at LEAF1; H2 is in VLAN 20 / 10.10.20.0/24 at LEAF2. Each host uses its local .254 gateway. L2 VNIs 10010 and 10020 identify those access segments. Both leaves associate VRF CUSTOMER with L3 VNI 100000 through transit VLAN 2000. The L3 VNI is a tenant routing context, not a third host subnet.

Physical topology of evpn-l3; addressing and roles appear in the following table. — Solid lines are physical connections; crossings are not junctions. The tenant service terminates on leaves, not on transit spines. Port addresses and host settings are listed below.

Physical link	Address at first endpoint	Address at second endpoint
SPINE1 Ethernet1/1 ↔ LEAF1 Ethernet1/1	192.0.2.1/30	192.0.2.2/30
SPINE1 Ethernet1/2 ↔ LEAF2 Ethernet1/1	192.0.2.5/30	192.0.2.6/30

Device	Loopback0 / router ID
SPINE1	10.255.0.1/32
LEAF1	10.255.0.2/32
LEAF2	10.255.0.3/32

Device	Local L2 segment	Shared routing context
LEAF1	VLAN 10 / VNI 10010 / 10.10.10.254	CUSTOMER / L3 VNI 100000
LEAF2	VLAN 20 / VNI 10020 / 10.10.20.254	CUSTOMER / L3 VNI 100000

Associate the tenant VRF with an L3 VNI

Both leaves · shared tenant routing policy

feature interface-vlan
fabric forwarding anycast-gateway-mac 0000.2222.3333
vlan 2000
  vn-segment 100000
vrf context CUSTOMER
  vni 100000
  rd auto
  address-family ipv4 unicast
    route-target both 65000:100000
    route-target both 65000:100000 evpn
interface Vlan2000
  no shutdown
  vrf member CUSTOMER
  ip forward
interface nve1
  member vni 100000 associate-vrf

Vlan2000 is the traditional VLAN-backed L3-VNI model used in this release recipe. It has ip forward, not a tenant host IP. Newer platforms can offer different L3-VNI models; do not mix those syntaxes into this example.

Configure each local gateway and advertise the subnet

LEAF1 · local tenant gateway and BGP origin

interface Vlan10
  no shutdown
  vrf member CUSTOMER
  ip address 10.10.10.254/24
  fabric forwarding mode anycast-gateway
ip prefix-list TENANT-CONNECTED seq 10 permit 10.10.10.0/24
route-map TENANT-DIRECT permit 10
  match ip address prefix-list TENANT-CONNECTED
router bgp 65000
  vrf CUSTOMER
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map TENANT-DIRECT

LEAF2 uses Vlan20, 10.10.20.254/24 and the corresponding connected-prefix filter. Both use the same configured gateway MAC. The filter exports only this tenant’s intended subnet, not every connected underlay interface. Full recipes include the L2 VNI, NVE/BGP and OSPF prerequisites.

Verify tenant routes rather than only the global table

LEAF1 · illustrative output, selected fields

LEAF1# show ip route vrf CUSTOMER 10.10.20.0/24
10.10.20.0/24, ubest/mbest: 1/0
    *via 10.255.0.3, [200/0], bgp-65000, internal
     VNI: 100000, encapsulation: VXLAN

LEAF1 · illustrative output, selected fields

LEAF1# show nve vni
Interface VNI      State Mode Type
nve1      10010    Up    CP   L2 [10]
nve1      100000   Up    CP   L3 [CUSTOMER]

Selected fields highlight the tenant prefix, remote VTEP and L3 context; formatting differs by release. The global table still resolves 10.255.0.3. H1 must ARP for its local gateway, then route to H2; it should not ARP directly for a host in a different /24.

A symmetric routing walkthrough

Step	Forwarding decision
H1 → LEAF1	Send the off-subnet packet to gateway 10.10.10.254.
Ingress tenant route	Lookup H2’s subnet in CUSTOMER; select LEAF2 and L3 VNI 100000.
VXLAN transit	Outer IP is 10.255.0.2 → 10.255.0.3; inner Ethernet uses routing context/MACs, not an unchanged H1-to-H2 bridge frame.
LEAF2 → H2	Decapsulate into CUSTOMER, route to local VLAN 20 and resolve H2’s MAC.

This is symmetric IRB: both VTEPs perform routing in the tenant context. An asymmetric design instead carries the destination L2 segment across the tunnel and has different state requirements. The L3 VNI does not mean that the spines hold CUSTOMER routes.

Keep tenant policy and transport failures distinct

If each host reaches its gateway but not the remote subnet, inspect CUSTOMER RT import/export and advertised prefixes. If the tenant route is installed but its VTEP next hop is unresolved, repair global underlay reachability. Restore the common L3 RT 65000:100000 and associate-vrf binding if they were deliberately removed, then validate H1 ↔ H2 with the listed gateways. Both local L2 VNIs use BGP ingress replication in this independent scenario; only the routed tenant exchange is tested here.

Complete independent lab configurations

Start with clean isolated lab devices. These are the complete scenario configurations, not changes to apply on top of a different lesson. Physical ports must match the address table. Virtual-image data-plane capabilities can differ from physical switches; these examples have not been executed on live devices.

Download configuration reference

SPINE1

hostname SPINE1
feature ospf
router ospf UNDERLAY
  router-id 10.255.0.1
interface loopback0
  ip address 10.255.0.1/32
  ip router ospf UNDERLAY area 0.0.0.0
interface Ethernet1/1
  no switchport
  mtu 9216
  ip address 192.0.2.1/30
  ip ospf network point-to-point
  ip ospf cost 40
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown
interface Ethernet1/2
  no switchport
  mtu 9216
  ip address 192.0.2.5/30
  ip ospf network point-to-point
  ip ospf cost 40
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown
feature bgp
nv overlay evpn
router bgp 65000
  router-id 10.255.0.1
  address-family l2vpn evpn
  neighbor 10.255.0.2
    remote-as 65000
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
      route-reflector-client
  neighbor 10.255.0.3
    remote-as 65000
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
      route-reflector-client

LEAF1

hostname LEAF1
feature ospf
router ospf UNDERLAY
  router-id 10.255.0.2
interface loopback0
  ip address 10.255.0.2/32
  ip router ospf UNDERLAY area 0.0.0.0
interface Ethernet1/1
  no switchport
  mtu 9216
  ip address 192.0.2.2/30
  ip ospf network point-to-point
  ip ospf cost 40
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown
feature nv overlay
feature vn-segment-vlan-based
vlan 10
  vn-segment 10010
interface Ethernet1/2
  switchport
  switchport mode access
  switchport access vlan 10
  spanning-tree port type edge
  no shutdown
interface nve1
  no shutdown
  source-interface loopback0
  host-reachability protocol bgp
  member vni 10010
    ingress-replication protocol bgp
evpn
  vni 10010 l2
    rd auto
    route-target import 65000:10010
    route-target export 65000:10010
feature bgp
nv overlay evpn
router bgp 65000
  router-id 10.255.0.2
  address-family l2vpn evpn
  neighbor 10.255.0.1
    remote-as 65000
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
feature interface-vlan
fabric forwarding anycast-gateway-mac 0000.2222.3333
vlan 2000
  vn-segment 100000
vrf context CUSTOMER
  vni 100000
  rd auto
  address-family ipv4 unicast
    route-target both 65000:100000
    route-target both 65000:100000 evpn
interface Vlan2000
  no shutdown
  vrf member CUSTOMER
  ip forward
interface Vlan10
  no shutdown
  vrf member CUSTOMER
  ip address 10.10.10.254/24
  fabric forwarding mode anycast-gateway
interface nve1
  member vni 100000 associate-vrf
ip prefix-list TENANT-CONNECTED seq 10 permit 10.10.10.0/24
route-map TENANT-DIRECT permit 10
  match ip address prefix-list TENANT-CONNECTED
router bgp 65000
  vrf CUSTOMER
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map TENANT-DIRECT

LEAF2

hostname LEAF2
feature ospf
router ospf UNDERLAY
  router-id 10.255.0.3
interface loopback0
  ip address 10.255.0.3/32
  ip router ospf UNDERLAY area 0.0.0.0
interface Ethernet1/1
  no switchport
  mtu 9216
  ip address 192.0.2.6/30
  ip ospf network point-to-point
  ip ospf cost 40
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown
feature nv overlay
feature vn-segment-vlan-based
vlan 20
  vn-segment 10020
interface Ethernet1/2
  switchport
  switchport mode access
  switchport access vlan 20
  spanning-tree port type edge
  no shutdown
interface nve1
  no shutdown
  source-interface loopback0
  host-reachability protocol bgp
  member vni 10020
    ingress-replication protocol bgp
evpn
  vni 10020 l2
    rd auto
    route-target import 65000:10020
    route-target export 65000:10020
feature bgp
nv overlay evpn
router bgp 65000
  router-id 10.255.0.3
  address-family l2vpn evpn
  neighbor 10.255.0.1
    remote-as 65000
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
feature interface-vlan
fabric forwarding anycast-gateway-mac 0000.2222.3333
vlan 2000
  vn-segment 100000
vrf context CUSTOMER
  vni 100000
  rd auto
  address-family ipv4 unicast
    route-target both 65000:100000
    route-target both 65000:100000 evpn
interface Vlan2000
  no shutdown
  vrf member CUSTOMER
  ip forward
interface Vlan20
  no shutdown
  vrf member CUSTOMER
  ip address 10.10.20.254/24
  fabric forwarding mode anycast-gateway
interface nve1
  member vni 100000 associate-vrf
ip prefix-list TENANT-CONNECTED seq 10 permit 10.10.20.0/24
route-map TENANT-DIRECT permit 10
  match ip address prefix-list TENANT-CONNECTED
router bgp 65000
  vrf CUSTOMER
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map TENANT-DIRECT

Host	Address	Gateway
H1	10.10.10.11/24	10.10.10.254
H2	10.10.20.22/24	10.10.20.254

Key Takeaways

Different tenant subnets require routing, not just a common L2 VNI.
L2 VNIs describe access segments; the L3 VNI associates the tenant IP-VRF.
A gateway SVI has a tenant IP; the VLAN-backed transit L3 SVI uses ip forward.
Tenant route import and underlay next-hop resolution are separate checks.
Symmetric IRB routes at both VTEPs without making spines tenant gateways.