PacketLab
Protocol Library/Routing & Switching/VXLAN
VXLAN · Routed underlay deployments

eBGP Multi-AS Underlay

Use distinct leaf AS numbers for IPv4 transport and carry EVPN over separate loopback sessions.

Keep transport and EVPN sessions distinct

Example platform: Nexus 9300/9000v · NX-OS 10.4(5) command model. Commands are platform-specific. Use a release and hardware/image with the stated VXLAN, multicast and routing features. All outputs and packet excerpts below are constructed teaching examples, not lab recordings.

SPINE1/2 use AS 65000. LEAF1–4 use 65003, 65004, 65005 and 65006 respectively. This removes the shared-leaf-AS loop exception, but auto-derived RTs using different AS numbers would no longer automatically match across leaves.

Independent lab · ebgp-multi-asIllustrative topology
Physical topology of ebgp-multi-as; addressing and roles appear in the following table.
Solid lines are physical connections; crossings are not junctions. The tenant service terminates on leaves, not on transit spines. Port addresses and host settings are listed below.
Physical linkAddress at first endpointAddress at second endpoint
SPINE1 Ethernet1/1 ↔ LEAF1 Ethernet1/1192.0.2.0/31192.0.2.1/31
SPINE1 Ethernet1/2 ↔ LEAF2 Ethernet1/1192.0.2.2/31192.0.2.3/31
SPINE1 Ethernet1/3 ↔ LEAF3 Ethernet1/1192.0.2.4/31192.0.2.5/31
SPINE1 Ethernet1/4 ↔ LEAF4 Ethernet1/1192.0.2.6/31192.0.2.7/31
SPINE2 Ethernet1/1 ↔ LEAF1 Ethernet1/2192.0.2.8/31192.0.2.9/31
SPINE2 Ethernet1/2 ↔ LEAF2 Ethernet1/2192.0.2.10/31192.0.2.11/31
SPINE2 Ethernet1/3 ↔ LEAF3 Ethernet1/2192.0.2.12/31192.0.2.13/31
SPINE2 Ethernet1/4 ↔ LEAF4 Ethernet1/2192.0.2.14/31192.0.2.15/31
DeviceLoopback0 / router ID
SPINE110.255.0.1/32
SPINE210.255.0.2/32
LEAF110.255.0.3/32
LEAF210.255.0.4/32
LEAF310.255.0.5/32
LEAF410.255.0.6/32

Form direct /31 transport peers

LEAF1 · IPv4 direct peers

feature bgp
ip prefix-list INFRA seq 10 permit 10.255.0.0/24 ge 32 le 32
router bgp 65003
  router-id 10.255.0.3
  address-family ipv4 unicast
    network 10.255.0.3/32
    maximum-paths 2
  neighbor 192.0.2.0
    remote-as 65000
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
  neighbor 192.0.2.8
    remote-as 65000
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out

Only loopback /32s within 10.255.0.0/24 pass the infrastructure prefix filter; the network statement originates each device’s existing Loopback0 route. The filter is applied in both directions on direct IPv4 sessions. Overlay EVPN uses different loopback neighbor entries and has separate RT policy.

Resolve AS-path policy deliberately

RoleAS number
SPINE165000
SPINE265000
LEAF165003
LEAF265004
LEAF365005
LEAF465006
LEAF1 · illustrative output, selected fields
LEAF1# show bgp ipv4 unicast 10.255.0.4/32
Paths via both spines: 65000 65004
Local leaf AS: 65003 (absent from that path)

Remote leaf prefixes no longer contain the receiving leaf’s own AS, so this independent lab uses neither leaf allowas-in nor spine disable-peer-as-check. Both paths traverse the common spine AS and the same remote leaf AS, so ordinary equal-path eligibility can be satisfied. A future design with different spine AS paths needs its own multipath policy; do not add relax commands without a reason.

Install eligible paths, then inspect the IP table

LEAF1 · illustrative output, selected fields
LEAF1# show ip route 10.255.0.4
10.255.0.4/32, ubest/mbest: 2/0
    *via 192.0.2.0, [20/0], bgp, external
    *via 192.0.2.8, [20/0], bgp, external

The two candidate AS paths are 65000 65004. The IPv4 family sets maximum-paths 2; other best-path attributes must also be compatible. A BGP table with two received paths alone is not proof both entered the IP forwarding table. Both spines have routes to every leaf loopback, so they can carry outer VXLAN packets.

Retain routes and preserve the VTEP next hop

Both spines · EVPN transit policy

route-map KEEP-VTEP permit 10
  set ip next-hop unchanged
router bgp 65000
  address-family l2vpn evpn
    retain route-target all
    nexthop route-map KEEP-VTEP

SPINE1 · example LEAF1 loopback session

router bgp 65000
  neighbor 10.255.0.3
    remote-as 65003
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
      route-map KEEP-VTEP out

All leaves · deliberate common L2 route target

evpn
  vni 10010 l2
    rd auto
    route-target import 65000:10010
    route-target export 65000:10010

The RT is a common import/export policy value; it need not equal each leaf’s local BGP AS. The RD still uniquely identifies route origin. Alternatively a platform can offer RT-AS rewrite features, but this example uses explicit matching RTs to avoid relying on that behavior.

The complete recipe applies the matching policy to every leaf neighbor. retain route-target all lets a non-VTEP transit spine keep routes with no local matching tenant VNI. The outgoing route-map and address-family next-hop policy retain the remote originating VTEP. Ordinary IPv4 transport advertisements still use direct next-hop behavior.

Verify control import and tenant delivery

LEAF1 · illustrative output, selected fields
LEAF1# show bgp l2vpn evpn route-type 2
Remote MAC: 0200.0000.0012
  Next hop: 10.255.0.4
  Route target: 65000:10010
  Encapsulation: VXLAN, VNI 10010

The route must import into local VNI 10010 and resolve to LEAF2, not a non-VTEP spine. Send H1 → H2, where hosts use 10.10.10.11–14/24 on Ethernet1/3 of LEAF1–4. For a deliberate wrong-next-hop fault, restore KEEP-VTEP on the spine’s EVPN family and outgoing neighbor policy, then inspect the re-advertised route and actual traffic again.

Complete independent lab configurations

Start with clean isolated lab devices. These are the complete scenario configurations, not changes to apply on top of a different lesson. Physical ports must match the address table. Virtual-image data-plane capabilities can differ from physical switches; these examples have not been executed on live devices.

Download configuration reference

SPINE1

hostname SPINE1
interface loopback0
  ip address 10.255.0.1/32
interface Ethernet1/1
  no switchport
  mtu 9216
  ip address 192.0.2.0/31
  no shutdown
interface Ethernet1/2
  no switchport
  mtu 9216
  ip address 192.0.2.2/31
  no shutdown
interface Ethernet1/3
  no switchport
  mtu 9216
  ip address 192.0.2.4/31
  no shutdown
interface Ethernet1/4
  no switchport
  mtu 9216
  ip address 192.0.2.6/31
  no shutdown
feature bgp
ip prefix-list INFRA seq 10 permit 10.255.0.0/24 ge 32 le 32
router bgp 65000
  router-id 10.255.0.1
  address-family ipv4 unicast
    network 10.255.0.1/32
    maximum-paths 2
  neighbor 192.0.2.1
    remote-as 65003
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
  neighbor 192.0.2.3
    remote-as 65004
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
  neighbor 192.0.2.5
    remote-as 65005
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
  neighbor 192.0.2.7
    remote-as 65006
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
nv overlay evpn
route-map KEEP-VTEP permit 10
  set ip next-hop unchanged
router bgp 65000
  address-family l2vpn evpn
    retain route-target all
    nexthop route-map KEEP-VTEP
  neighbor 10.255.0.3
    remote-as 65003
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
      route-map KEEP-VTEP out
  neighbor 10.255.0.4
    remote-as 65004
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
      route-map KEEP-VTEP out
  neighbor 10.255.0.5
    remote-as 65005
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
      route-map KEEP-VTEP out
  neighbor 10.255.0.6
    remote-as 65006
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
      route-map KEEP-VTEP out

SPINE2

hostname SPINE2
interface loopback0
  ip address 10.255.0.2/32
interface Ethernet1/1
  no switchport
  mtu 9216
  ip address 192.0.2.8/31
  no shutdown
interface Ethernet1/2
  no switchport
  mtu 9216
  ip address 192.0.2.10/31
  no shutdown
interface Ethernet1/3
  no switchport
  mtu 9216
  ip address 192.0.2.12/31
  no shutdown
interface Ethernet1/4
  no switchport
  mtu 9216
  ip address 192.0.2.14/31
  no shutdown
feature bgp
ip prefix-list INFRA seq 10 permit 10.255.0.0/24 ge 32 le 32
router bgp 65000
  router-id 10.255.0.2
  address-family ipv4 unicast
    network 10.255.0.2/32
    maximum-paths 2
  neighbor 192.0.2.9
    remote-as 65003
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
  neighbor 192.0.2.11
    remote-as 65004
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
  neighbor 192.0.2.13
    remote-as 65005
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
  neighbor 192.0.2.15
    remote-as 65006
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
nv overlay evpn
route-map KEEP-VTEP permit 10
  set ip next-hop unchanged
router bgp 65000
  address-family l2vpn evpn
    retain route-target all
    nexthop route-map KEEP-VTEP
  neighbor 10.255.0.3
    remote-as 65003
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
      route-map KEEP-VTEP out
  neighbor 10.255.0.4
    remote-as 65004
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
      route-map KEEP-VTEP out
  neighbor 10.255.0.5
    remote-as 65005
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
      route-map KEEP-VTEP out
  neighbor 10.255.0.6
    remote-as 65006
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
      route-map KEEP-VTEP out

LEAF1

hostname LEAF1
interface loopback0
  ip address 10.255.0.3/32
interface Ethernet1/1
  no switchport
  mtu 9216
  ip address 192.0.2.1/31
  no shutdown
interface Ethernet1/2
  no switchport
  mtu 9216
  ip address 192.0.2.9/31
  no shutdown
feature bgp
ip prefix-list INFRA seq 10 permit 10.255.0.0/24 ge 32 le 32
router bgp 65003
  router-id 10.255.0.3
  address-family ipv4 unicast
    network 10.255.0.3/32
    maximum-paths 2
  neighbor 192.0.2.0
    remote-as 65000
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
  neighbor 192.0.2.8
    remote-as 65000
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
nv overlay evpn
route-map KEEP-VTEP permit 10
  set ip next-hop unchanged
router bgp 65003
  address-family l2vpn evpn
  neighbor 10.255.0.1
    remote-as 65000
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.255.0.2
    remote-as 65000
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
feature nv overlay
feature vn-segment-vlan-based
vlan 10
  vn-segment 10010
interface Ethernet1/3
  switchport
  switchport mode access
  switchport access vlan 10
  spanning-tree port type edge
  no shutdown
interface nve1
  no shutdown
  source-interface loopback0
  host-reachability protocol bgp
  member vni 10010
    ingress-replication protocol bgp
evpn
  vni 10010 l2
    rd auto
    route-target import 65000:10010
    route-target export 65000:10010

LEAF2

hostname LEAF2
interface loopback0
  ip address 10.255.0.4/32
interface Ethernet1/1
  no switchport
  mtu 9216
  ip address 192.0.2.3/31
  no shutdown
interface Ethernet1/2
  no switchport
  mtu 9216
  ip address 192.0.2.11/31
  no shutdown
feature bgp
ip prefix-list INFRA seq 10 permit 10.255.0.0/24 ge 32 le 32
router bgp 65004
  router-id 10.255.0.4
  address-family ipv4 unicast
    network 10.255.0.4/32
    maximum-paths 2
  neighbor 192.0.2.2
    remote-as 65000
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
  neighbor 192.0.2.10
    remote-as 65000
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
nv overlay evpn
route-map KEEP-VTEP permit 10
  set ip next-hop unchanged
router bgp 65004
  address-family l2vpn evpn
  neighbor 10.255.0.1
    remote-as 65000
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.255.0.2
    remote-as 65000
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
feature nv overlay
feature vn-segment-vlan-based
vlan 10
  vn-segment 10010
interface Ethernet1/3
  switchport
  switchport mode access
  switchport access vlan 10
  spanning-tree port type edge
  no shutdown
interface nve1
  no shutdown
  source-interface loopback0
  host-reachability protocol bgp
  member vni 10010
    ingress-replication protocol bgp
evpn
  vni 10010 l2
    rd auto
    route-target import 65000:10010
    route-target export 65000:10010

LEAF3

hostname LEAF3
interface loopback0
  ip address 10.255.0.5/32
interface Ethernet1/1
  no switchport
  mtu 9216
  ip address 192.0.2.5/31
  no shutdown
interface Ethernet1/2
  no switchport
  mtu 9216
  ip address 192.0.2.13/31
  no shutdown
feature bgp
ip prefix-list INFRA seq 10 permit 10.255.0.0/24 ge 32 le 32
router bgp 65005
  router-id 10.255.0.5
  address-family ipv4 unicast
    network 10.255.0.5/32
    maximum-paths 2
  neighbor 192.0.2.4
    remote-as 65000
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
  neighbor 192.0.2.12
    remote-as 65000
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
nv overlay evpn
route-map KEEP-VTEP permit 10
  set ip next-hop unchanged
router bgp 65005
  address-family l2vpn evpn
  neighbor 10.255.0.1
    remote-as 65000
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.255.0.2
    remote-as 65000
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
feature nv overlay
feature vn-segment-vlan-based
vlan 10
  vn-segment 10010
interface Ethernet1/3
  switchport
  switchport mode access
  switchport access vlan 10
  spanning-tree port type edge
  no shutdown
interface nve1
  no shutdown
  source-interface loopback0
  host-reachability protocol bgp
  member vni 10010
    ingress-replication protocol bgp
evpn
  vni 10010 l2
    rd auto
    route-target import 65000:10010
    route-target export 65000:10010

LEAF4

hostname LEAF4
interface loopback0
  ip address 10.255.0.6/32
interface Ethernet1/1
  no switchport
  mtu 9216
  ip address 192.0.2.7/31
  no shutdown
interface Ethernet1/2
  no switchport
  mtu 9216
  ip address 192.0.2.15/31
  no shutdown
feature bgp
ip prefix-list INFRA seq 10 permit 10.255.0.0/24 ge 32 le 32
router bgp 65006
  router-id 10.255.0.6
  address-family ipv4 unicast
    network 10.255.0.6/32
    maximum-paths 2
  neighbor 192.0.2.6
    remote-as 65000
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
  neighbor 192.0.2.14
    remote-as 65000
    address-family ipv4 unicast
      prefix-list INFRA in
      prefix-list INFRA out
nv overlay evpn
route-map KEEP-VTEP permit 10
  set ip next-hop unchanged
router bgp 65006
  address-family l2vpn evpn
  neighbor 10.255.0.1
    remote-as 65000
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.255.0.2
    remote-as 65000
    update-source loopback0
    ebgp-multihop 3
    address-family l2vpn evpn
      send-community
      send-community extended
feature nv overlay
feature vn-segment-vlan-based
vlan 10
  vn-segment 10010
interface Ethernet1/3
  switchport
  switchport mode access
  switchport access vlan 10
  spanning-tree port type edge
  no shutdown
interface nve1
  no shutdown
  source-interface loopback0
  host-reachability protocol bgp
  member vni 10010
    ingress-replication protocol bgp
evpn
  vni 10010 l2
    rd auto
    route-target import 65000:10010
    route-target export 65000:10010
HostAddressGateway
H110.10.10.11/24none
H210.10.10.12/24none
H310.10.10.13/24none
H410.10.10.14/24none

Further reading

Key Takeaways

  • Multi-AS assigns a distinct AS to each leaf or coordinated leaf pair.
  • The remote leaf’s AS differs from the local AS, avoiding this two-AS loop exception.
  • Explicit matching RTs avoid auto-RT mismatch between different leaf ASNs.
  • Transport and overlay next hops have different rewriting requirements.
  • Common-spine-AS paths can support IPv4 ECMP without blindly enabling multipath relax.

eBGP Multi-AS Underlay Quiz

Six questions · four options each · select one answer per question. Submit all answers to see your score and explanations.

1. Which AS does LEAF2 use here?
2. Why is allowas-in unnecessary in this independent baseline?
3. What can fail when each leaf auto-derives RTs from a different AS?
4. How does this recipe align import/export policy?
5. Does a manual RT have to equal the leaf’s local BGP AS?
6. Should a different future AS design automatically reuse all multipath settings?