I just wanted to share my lab experience setting up, configuring and verifying VXLAN EVPN using Cisco Nexus 9000v and CSR1000v virtual platforms. The Nexus NX-OSv was a great tool because it allows anyone to explore NX-OS technologies otherwise only reserved for those who had access to very expensive physical hardware Nexus 9K switches. The CSR1000v is also a great tool and can be used to setup VXLAN flood and learn, however the NX-OSv opened the door to learn the next gen of datacenter technologies, VXLAN EVPN.
The information here is not meant to be a detailed tutorial on VXLAN EVPN but a configuration and operational look at the technology. So please be aware that some previous knowledge of the technology is required. The references section does contains a collection of links to some good documentation and useful articles. I recommend anyone to read up on it.
Here’s a breakdown of the components used in my lab environment.
- VMWare ESXi 6.5 (standalone)
- (5) Cisco NX-OSv 9000: version 7.0(3)I6(1)
- Used 2 vCPUs instead of the default 4. It seems to run fine.
- Used default RAM at 8Gb
- (2) CSR1000v: version 16.4.2 [Everest]
- Used default vCPU setting of 1
- Used default RAM at 4Gb
- (8) Damn Small Linux (DSL) 4.x as test hosts
Diagram & Topology
- Spine/Leaf Clos architecture
- 2 Spines
- 2 Leafs, 1 Border Leaf
- Underlay Infrastructure
- IP Address Space: 10.0.0.0/24
- Each link addressed using /30's
- IGP: OSPF
- Multicast: Anycast RP PIM with RP configured on Spines
- Control Plane Protocol: iBGP
- BGP Route Reflectors configured on Spines
- Multi-Tenant Setup - 2 Tenants
- TENANT_A (Orange)
- User IP Address Space: 172.16.0.0/22
- TENANT_B (Gold)
- User IP Address Space: 192.168.0.0/22
- VLAN to VNI mapping
- TENANT_A
- VLAN 100 → L2 VNI 10000
- VLAN 200 → L2 VNI 20000
- VLAN 900 → L3 VNI 90000
- TENANT_B
- VLAN 300 → L2 VNI 30000
- VLAN 400 → L2 VNI 40000
- VLAN 910 → L3 VNI 91000
- WAN Edge Router
- eBGP to Border Leaf3
- eBGP to WAN
Configurations
Spine_01
version 7.0(3)I6(1)
hostname DC1_SPINE_01
vdc DC1_SPINE_01 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
ip pim rp-address 100.100.100.100 group-list 225.0.0.0/8
ip pim bsr rp-candidate loopback1 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
ip pim anycast-rp 100.100.100.100 1.1.1.1
ip pim anycast-rp 100.100.100.100 2.2.2.2
vlan 1
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Ethernet1/1
description TO LEAF_01
no switchport
mtu 9216
ip address 10.0.0.1/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO LEAF_02
no switchport
mtu 9216
ip address 10.0.0.5/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
description TO LEAF_03
no switchport
mtu 9216
ip address 10.0.0.17/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/4
shutdown
mtu 9216
interface Ethernet1/5
shutdown
mtu 9216
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.210/24
interface loopback0
description RID
ip address 1.1.1.1/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
interface loopback1
description MCAST ANYCAST RP
ip address 100.100.100.100/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
cli alias name ll sh ip route
cli alias name p ping
cli alias name tr traceroute
cli alias name q exit
cli alias name lo exit
cli alias name sr sh running-config
cli alias name sibsle show bgp l2vpn evpn summary
cli alias name sible sh bgp l2vpn evpn
cli alias name slema show l2route evpn mac all
cli alias name slemia show l2route evpn mac-ip all
cli alias name llm sh ip mroute
cli alias name sion sh ip ospf neighbors
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 100
router-id 1.1.1.1
passive-interface default
router bgp 65500
router-id 1.1.1.1
address-family l2vpn evpn
neighbor 3.3.3.3
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 4.4.4.4
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 5.5.5.5
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
hostname DC1_SPINE_01
vdc DC1_SPINE_01 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
ip pim rp-address 100.100.100.100 group-list 225.0.0.0/8
ip pim bsr rp-candidate loopback1 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
ip pim anycast-rp 100.100.100.100 1.1.1.1
ip pim anycast-rp 100.100.100.100 2.2.2.2
vlan 1
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Ethernet1/1
description TO LEAF_01
no switchport
mtu 9216
ip address 10.0.0.1/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO LEAF_02
no switchport
mtu 9216
ip address 10.0.0.5/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
description TO LEAF_03
no switchport
mtu 9216
ip address 10.0.0.17/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/4
shutdown
mtu 9216
interface Ethernet1/5
shutdown
mtu 9216
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.210/24
interface loopback0
description RID
ip address 1.1.1.1/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
interface loopback1
description MCAST ANYCAST RP
ip address 100.100.100.100/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
cli alias name ll sh ip route
cli alias name p ping
cli alias name tr traceroute
cli alias name q exit
cli alias name lo exit
cli alias name sr sh running-config
cli alias name sibsle show bgp l2vpn evpn summary
cli alias name sible sh bgp l2vpn evpn
cli alias name slema show l2route evpn mac all
cli alias name slemia show l2route evpn mac-ip all
cli alias name llm sh ip mroute
cli alias name sion sh ip ospf neighbors
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 100
router-id 1.1.1.1
passive-interface default
router bgp 65500
router-id 1.1.1.1
address-family l2vpn evpn
neighbor 3.3.3.3
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 4.4.4.4
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 5.5.5.5
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
Spine_02
version 7.0(3)I6(1)
hostname DC1_SPINE_02
vdc DC1_SPINE_02 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
ip pim rp-address 100.100.100.100 group-list 225.0.0.0/8
ip pim bsr rp-candidate loopback1 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
ip pim anycast-rp 100.100.100.100 1.1.1.1
ip pim anycast-rp 100.100.100.100 2.2.2.2
vlan 1
no cdp enable
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Ethernet1/1
description TO LEAF_02
no switchport
mtu 9216
ip address 10.0.0.9/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO LEAF_01
no switchport
mtu 9216
ip address 10.0.0.13/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
description TO LEAF_03
no switchport
mtu 9216
ip address 10.0.0.21/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/4
shutdown
mtu 9216
interface Ethernet1/5
shutdown
mtu 9216
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.211/24
interface loopback0
description RID
ip address 2.2.2.2/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
interface loopback1
description MCAST ANYCAST RP
ip address 100.100.100.100/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
cli alias name ll sh ip route
cli alias name p ping
cli alias name tr traceroute
cli alias name q exit
cli alias name lo exit
cli alias name sr sh running-config
cli alias name sibsle show bgp l2vpn evpn summary
cli alias name sible sh bgp l2vpn evpn
cli alias name llm sh ip mroute
cli alias name sion sh ip ospf neighbors
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 100
router-id 2.2.2.2
passive-interface default
router bgp 65500
router-id 2.2.2.2
neighbor 3.3.3.3
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 4.4.4.4
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 5.5.5.5
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
hostname DC1_SPINE_02
vdc DC1_SPINE_02 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
ip pim rp-address 100.100.100.100 group-list 225.0.0.0/8
ip pim bsr rp-candidate loopback1 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
ip pim anycast-rp 100.100.100.100 1.1.1.1
ip pim anycast-rp 100.100.100.100 2.2.2.2
vlan 1
no cdp enable
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Ethernet1/1
description TO LEAF_02
no switchport
mtu 9216
ip address 10.0.0.9/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO LEAF_01
no switchport
mtu 9216
ip address 10.0.0.13/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
description TO LEAF_03
no switchport
mtu 9216
ip address 10.0.0.21/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/4
shutdown
mtu 9216
interface Ethernet1/5
shutdown
mtu 9216
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.211/24
interface loopback0
description RID
ip address 2.2.2.2/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
interface loopback1
description MCAST ANYCAST RP
ip address 100.100.100.100/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
cli alias name ll sh ip route
cli alias name p ping
cli alias name tr traceroute
cli alias name q exit
cli alias name lo exit
cli alias name sr sh running-config
cli alias name sibsle show bgp l2vpn evpn summary
cli alias name sible sh bgp l2vpn evpn
cli alias name llm sh ip mroute
cli alias name sion sh ip ospf neighbors
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 100
router-id 2.2.2.2
passive-interface default
router bgp 65500
router-id 2.2.2.2
neighbor 3.3.3.3
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 4.4.4.4
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 5.5.5.5
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
Leaf_01
version 7.0(3)I6(1)
hostname DC1_LEAF_01
vdc DC1_LEAF_01 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
vlan 1,100,200,300,400,900,910
fabric forwarding anycast-gateway-mac 0000.dc01.dc01
ip pim rp-address 100.100.100.100 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 100
vn-segment 10000
vlan 200
vn-segment 20000
vlan 300
vn-segment 30000
vlan 400
vn-segment 40000
vlan 900
vn-segment 90000
vlan 910
vn-segment 91000
vrf context TENANT_A
vni 90000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context TENANT_B
vni 91000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Vlan1
interface Vlan100
description TENANT_A VL100
no shutdown
mtu 9216
vrf member TENANT_A
ip address 172.16.1.254/24
fabric forwarding mode anycast-gateway
interface Vlan200
description TENANT_A VL200
no shutdown
mtu 9216
vrf member TENANT_A
ip address 172.16.2.254/24
fabric forwarding mode anycast-gateway
interface Vlan300
description TENANT_B VL300
no shutdown
mtu 9216
vrf member TENANT_B
ip address 192.168.1.254/24
fabric forwarding mode anycast-gateway
interface Vlan400
description TENANT_B VL400
no shutdown
mtu 9216
vrf member TENANT_B
ip address 192.168.2.254/24
fabric forwarding mode anycast-gateway
interface Vlan900
description TENANT_A VL900 L3VNI
no shutdown
mtu 9216
vrf member TENANT_A
ip forward
interface Vlan910
description TENANT_B VL910 L3VNI
no shutdown
mtu 9216
vrf member TENANT_B
ip forward
interface nve1
no shutdown
description VTEP_01
source-interface loopback0
host-reachability protocol bgp
member vni 10000
suppress-arp
mcast-group 225.1.0.1
member vni 20000
suppress-arp
mcast-group 225.2.0.1
member vni 30000
suppress-arp
mcast-group 225.3.0.1
member vni 40000
suppress-arp
mcast-group 225.4.0.1
member vni 90000 associate-vrf
member vni 91000 associate-vrf
interface Ethernet1/1
description TO SPINE_01
no switchport
mtu 9216
ip address 10.0.0.2/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO SPINE_02
no switchport
mtu 9216
ip address 10.0.0.14/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
mtu 9216
interface Ethernet1/4
description TO HOST4 - VL100
switchport access vlan 100
mtu 9216
interface Ethernet1/5
description TO HOST5 - VL200
switchport access vlan 200
mtu 9216
interface Ethernet1/6
description TO HOST1 - VL300
switchport access vlan 300
mtu 9216
interface Ethernet1/7
description TO HOST12 - VL400
switchport access vlan 400
mtu 9216
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.212/24
interface loopback0
description RID - VTEP
ip address 3.3.3.3/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
cli alias name ll sh ip route
cli alias name p ping
cli alias name tr traceroute
cli alias name q exit
cli alias name lo exit
cli alias name sr sh running-config
cli alias name sibsle show bgp l2vpn evpn summary
cli alias name sible sh bgp l2vpn evpn
cli alias name snn sh nve vni
cli alias name snp sh nve peer
cli alias name slema show l2route evpn mac all
cli alias name slemia show l2route evpn mac-ip all
cli alias name lla sh ip route vrf TENANT_A
cli alias name svi show vxlan interface
cli alias name siasd show ip arp suppression-cache detail
cli alias name snipd sh nve internal platform interface nve1 detail
cli alias name snpd sh nve peer detail
cli alias name llm sh ip mroute
cli alias name sion sh ip ospf neighbors
cli alias name llb sh ip route vrf TENANT_B
cli alias name shmac show system internal l2fwder mac
cli alias name clms clear mac address-table datapath static
cli alias name clmd clear mac address-table datapath dynamic
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 100
router-id 3.3.3.3
passive-interface default
router bgp 65500
router-id 3.3.3.3
neighbor 1.1.1.1
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 2.2.2.2
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf TENANT_A
address-family ipv4 unicast
advertise l2vpn evpn
vrf TENANT_B
address-family ipv4 unicast
advertise l2vpn evpn
evpn
vni 10000 l2
rd auto
route-target import auto
route-target export auto
vni 20000 l2
rd auto
route-target import auto
route-target export auto
vni 30000 l2
rd auto
route-target import auto
route-target export auto
vni 40000 l2
rd auto
route-target import auto
route-target export auto
hostname DC1_LEAF_01
vdc DC1_LEAF_01 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
vlan 1,100,200,300,400,900,910
fabric forwarding anycast-gateway-mac 0000.dc01.dc01
ip pim rp-address 100.100.100.100 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 100
vn-segment 10000
vlan 200
vn-segment 20000
vlan 300
vn-segment 30000
vlan 400
vn-segment 40000
vlan 900
vn-segment 90000
vlan 910
vn-segment 91000
vrf context TENANT_A
vni 90000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context TENANT_B
vni 91000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Vlan1
interface Vlan100
description TENANT_A VL100
no shutdown
mtu 9216
vrf member TENANT_A
ip address 172.16.1.254/24
fabric forwarding mode anycast-gateway
interface Vlan200
description TENANT_A VL200
no shutdown
mtu 9216
vrf member TENANT_A
ip address 172.16.2.254/24
fabric forwarding mode anycast-gateway
interface Vlan300
description TENANT_B VL300
no shutdown
mtu 9216
vrf member TENANT_B
ip address 192.168.1.254/24
fabric forwarding mode anycast-gateway
interface Vlan400
description TENANT_B VL400
no shutdown
mtu 9216
vrf member TENANT_B
ip address 192.168.2.254/24
fabric forwarding mode anycast-gateway
interface Vlan900
description TENANT_A VL900 L3VNI
no shutdown
mtu 9216
vrf member TENANT_A
ip forward
interface Vlan910
description TENANT_B VL910 L3VNI
no shutdown
mtu 9216
vrf member TENANT_B
ip forward
interface nve1
no shutdown
description VTEP_01
source-interface loopback0
host-reachability protocol bgp
member vni 10000
suppress-arp
mcast-group 225.1.0.1
member vni 20000
suppress-arp
mcast-group 225.2.0.1
member vni 30000
suppress-arp
mcast-group 225.3.0.1
member vni 40000
suppress-arp
mcast-group 225.4.0.1
member vni 90000 associate-vrf
member vni 91000 associate-vrf
interface Ethernet1/1
description TO SPINE_01
no switchport
mtu 9216
ip address 10.0.0.2/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO SPINE_02
no switchport
mtu 9216
ip address 10.0.0.14/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
mtu 9216
interface Ethernet1/4
description TO HOST4 - VL100
switchport access vlan 100
mtu 9216
interface Ethernet1/5
description TO HOST5 - VL200
switchport access vlan 200
mtu 9216
interface Ethernet1/6
description TO HOST1 - VL300
switchport access vlan 300
mtu 9216
interface Ethernet1/7
description TO HOST12 - VL400
switchport access vlan 400
mtu 9216
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.212/24
interface loopback0
description RID - VTEP
ip address 3.3.3.3/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
cli alias name ll sh ip route
cli alias name p ping
cli alias name tr traceroute
cli alias name q exit
cli alias name lo exit
cli alias name sr sh running-config
cli alias name sibsle show bgp l2vpn evpn summary
cli alias name sible sh bgp l2vpn evpn
cli alias name snn sh nve vni
cli alias name snp sh nve peer
cli alias name slema show l2route evpn mac all
cli alias name slemia show l2route evpn mac-ip all
cli alias name lla sh ip route vrf TENANT_A
cli alias name svi show vxlan interface
cli alias name siasd show ip arp suppression-cache detail
cli alias name snipd sh nve internal platform interface nve1 detail
cli alias name snpd sh nve peer detail
cli alias name llm sh ip mroute
cli alias name sion sh ip ospf neighbors
cli alias name llb sh ip route vrf TENANT_B
cli alias name shmac show system internal l2fwder mac
cli alias name clms clear mac address-table datapath static
cli alias name clmd clear mac address-table datapath dynamic
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 100
router-id 3.3.3.3
passive-interface default
router bgp 65500
router-id 3.3.3.3
neighbor 1.1.1.1
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 2.2.2.2
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf TENANT_A
address-family ipv4 unicast
advertise l2vpn evpn
vrf TENANT_B
address-family ipv4 unicast
advertise l2vpn evpn
evpn
vni 10000 l2
rd auto
route-target import auto
route-target export auto
vni 20000 l2
rd auto
route-target import auto
route-target export auto
vni 30000 l2
rd auto
route-target import auto
route-target export auto
vni 40000 l2
rd auto
route-target import auto
route-target export auto
Leaf_02
version 7.0(3)I6(1)
hostname DC1_LEAF_02
vdc DC1_LEAF_02 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
vlan 1,10,100,200,300,400,900,910
fabric forwarding anycast-gateway-mac 0000.dc01.dc01
ip pim rp-address 100.100.100.100 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 100
vn-segment 10000
vlan 200
vn-segment 20000
vlan 300
vn-segment 30000
vlan 400
vn-segment 40000
vlan 900
vn-segment 90000
vlan 910
vn-segment 91000
no cdp enable
vrf context TENANT_A
vni 90000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context TENANT_B
vni 91000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Vlan1
interface Vlan100
description TENANT_A VL100
no shutdown
mtu 9216
vrf member TENANT_A
ip address 172.16.1.254/24
fabric forwarding mode anycast-gateway
interface Vlan200
description TENANT_A VL200
no shutdown
mtu 9216
vrf member TENANT_A
ip address 172.16.2.254/24
fabric forwarding mode anycast-gateway
interface Vlan300
description TENANT_B VL300
no shutdown
mtu 9216
vrf member TENANT_B
ip address 192.168.1.254/24
fabric forwarding mode anycast-gateway
interface Vlan400
description TENANT_B VL400
no shutdown
mtu 9216
vrf member TENANT_B
ip address 192.168.2.254/24
fabric forwarding mode anycast-gateway
interface Vlan900
description TENANT_A VL900 L3VNI
no shutdown
mtu 9216
vrf member TENANT_A
ip forward
interface Vlan910
description TENANT_B VL910 L3VNI
no shutdown
mtu 9216
vrf member TENANT_B
ip forward
interface nve1
no shutdown
description VTEP_02
source-interface loopback0
host-reachability protocol bgp
member vni 10000
suppress-arp
mcast-group 225.1.0.1
member vni 20000
suppress-arp
mcast-group 225.2.0.1
member vni 30000
suppress-arp
mcast-group 225.3.0.1
member vni 40000
suppress-arp
mcast-group 225.4.0.1
member vni 90000 associate-vrf
member vni 91000 associate-vrf
interface Ethernet1/1
description TO SPINE_02
no switchport
mtu 9216
ip address 10.0.0.10/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO SPINE_01
no switchport
mtu 9216
ip address 10.0.0.6/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
shutdown
mtu 9216
interface Ethernet1/4
description TO HOST6 - VL100
switchport access vlan 100
mtu 9216
interface Ethernet1/5
description TO HOST7 - VL200
switchport access vlan 200
mtu 9216
interface Ethernet1/6
description TO HOST2 - VL300
switchport access vlan 300
mtu 9216
interface Ethernet1/7
description TO HOST13 - VL400
switchport access vlan 400
mtu 9216
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.213/24
interface loopback0
description RID - VTEP
ip address 4.4.4.4/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
cli alias name ll sh ip route
cli alias name p ping
cli alias name tr traceroute
cli alias name q exit
cli alias name lo exit
cli alias name sr sh running-config
cli alias name sibsle show bgp l2vpn evpn summary
cli alias name sible sh bgp l2vpn evpn
cli alias name snn sh nve vni
cli alias name snp sh nve peer
cli alias name slema show l2route evpn mac all
cli alias name slemia show l2route evpn mac-ip all
cli alias name lla sh ip route vrf TENANT_A
cli alias name svi show vxlan interface
cli alias name siasd show ip arp suppression-cache detail
cli alias name snipd sh nve internal platform interface nve1 detail
cli alias name snpd sh nve peer detail
cli alias name llm sh ip mroute
cli alias name sion sh ip ospf neighbors
cli alias name llb sh ip route vrf TENANT_B
cli alias name shmac show system internal l2fwder mac
cli alias name clms clear mac address-table datapath static
cli alias name clmd clear mac address-table datapath dynamic
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 100
router-id 4.4.4.4
passive-interface default
router bgp 65500
router-id 4.4.4.4
neighbor 1.1.1.1
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 2.2.2.2
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf TENANT_A
address-family ipv4 unicast
advertise l2vpn evpn
vrf TENANT_B
address-family ipv4 unicast
advertise l2vpn evpn
evpn
vni 10000 l2
rd auto
route-target import auto
route-target export auto
vni 20000 l2
rd auto
route-target import auto
route-target export auto
vni 30000 l2
rd auto
route-target import auto
route-target export auto
vni 40000 l2
rd auto
route-target import auto
route-target export auto
hostname DC1_LEAF_02
vdc DC1_LEAF_02 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
vlan 1,10,100,200,300,400,900,910
fabric forwarding anycast-gateway-mac 0000.dc01.dc01
ip pim rp-address 100.100.100.100 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 100
vn-segment 10000
vlan 200
vn-segment 20000
vlan 300
vn-segment 30000
vlan 400
vn-segment 40000
vlan 900
vn-segment 90000
vlan 910
vn-segment 91000
no cdp enable
vrf context TENANT_A
vni 90000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context TENANT_B
vni 91000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Vlan1
interface Vlan100
description TENANT_A VL100
no shutdown
mtu 9216
vrf member TENANT_A
ip address 172.16.1.254/24
fabric forwarding mode anycast-gateway
interface Vlan200
description TENANT_A VL200
no shutdown
mtu 9216
vrf member TENANT_A
ip address 172.16.2.254/24
fabric forwarding mode anycast-gateway
interface Vlan300
description TENANT_B VL300
no shutdown
mtu 9216
vrf member TENANT_B
ip address 192.168.1.254/24
fabric forwarding mode anycast-gateway
interface Vlan400
description TENANT_B VL400
no shutdown
mtu 9216
vrf member TENANT_B
ip address 192.168.2.254/24
fabric forwarding mode anycast-gateway
interface Vlan900
description TENANT_A VL900 L3VNI
no shutdown
mtu 9216
vrf member TENANT_A
ip forward
interface Vlan910
description TENANT_B VL910 L3VNI
no shutdown
mtu 9216
vrf member TENANT_B
ip forward
interface nve1
no shutdown
description VTEP_02
source-interface loopback0
host-reachability protocol bgp
member vni 10000
suppress-arp
mcast-group 225.1.0.1
member vni 20000
suppress-arp
mcast-group 225.2.0.1
member vni 30000
suppress-arp
mcast-group 225.3.0.1
member vni 40000
suppress-arp
mcast-group 225.4.0.1
member vni 90000 associate-vrf
member vni 91000 associate-vrf
interface Ethernet1/1
description TO SPINE_02
no switchport
mtu 9216
ip address 10.0.0.10/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO SPINE_01
no switchport
mtu 9216
ip address 10.0.0.6/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
shutdown
mtu 9216
interface Ethernet1/4
description TO HOST6 - VL100
switchport access vlan 100
mtu 9216
interface Ethernet1/5
description TO HOST7 - VL200
switchport access vlan 200
mtu 9216
interface Ethernet1/6
description TO HOST2 - VL300
switchport access vlan 300
mtu 9216
interface Ethernet1/7
description TO HOST13 - VL400
switchport access vlan 400
mtu 9216
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.213/24
interface loopback0
description RID - VTEP
ip address 4.4.4.4/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
cli alias name ll sh ip route
cli alias name p ping
cli alias name tr traceroute
cli alias name q exit
cli alias name lo exit
cli alias name sr sh running-config
cli alias name sibsle show bgp l2vpn evpn summary
cli alias name sible sh bgp l2vpn evpn
cli alias name snn sh nve vni
cli alias name snp sh nve peer
cli alias name slema show l2route evpn mac all
cli alias name slemia show l2route evpn mac-ip all
cli alias name lla sh ip route vrf TENANT_A
cli alias name svi show vxlan interface
cli alias name siasd show ip arp suppression-cache detail
cli alias name snipd sh nve internal platform interface nve1 detail
cli alias name snpd sh nve peer detail
cli alias name llm sh ip mroute
cli alias name sion sh ip ospf neighbors
cli alias name llb sh ip route vrf TENANT_B
cli alias name shmac show system internal l2fwder mac
cli alias name clms clear mac address-table datapath static
cli alias name clmd clear mac address-table datapath dynamic
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 100
router-id 4.4.4.4
passive-interface default
router bgp 65500
router-id 4.4.4.4
neighbor 1.1.1.1
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 2.2.2.2
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf TENANT_A
address-family ipv4 unicast
advertise l2vpn evpn
vrf TENANT_B
address-family ipv4 unicast
advertise l2vpn evpn
evpn
vni 10000 l2
rd auto
route-target import auto
route-target export auto
vni 20000 l2
rd auto
route-target import auto
route-target export auto
vni 30000 l2
rd auto
route-target import auto
route-target export auto
vni 40000 l2
rd auto
route-target import auto
route-target export auto
Leaf_03
version 7.0(3)I6(1)
hostname DC1_LEAF_03
vdc DC1_LEAF_03 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
vlan 1,100,200,300,400,900,910
fabric forwarding anycast-gateway-mac 0000.dc01.dc01
ip pim rp-address 100.100.100.100 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 100
vn-segment 10000
vlan 200
vn-segment 20000
vlan 300
vn-segment 30000
vlan 400
vn-segment 40000
vlan 900
vn-segment 90000
vlan 910
vn-segment 91000
vrf context TENANT_A
vni 90000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context TENANT_B
vni 91000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Vlan1
interface Vlan100
description TENANT_A VL100
no shutdown
mtu 9216
vrf member TENANT_A
ip address 172.16.1.254/24
fabric forwarding mode anycast-gateway
interface Vlan200
description TENANT_A VL200
no shutdown
mtu 9216
vrf member TENANT_A
ip address 172.16.2.254/24
fabric forwarding mode anycast-gateway
interface Vlan300
description TENANT_B VL300
no shutdown
mtu 9216
vrf member TENANT_B
ip address 192.168.1.254/24
fabric forwarding mode anycast-gateway
interface Vlan400
description TENANT_B VL400
no shutdown
mtu 9216
vrf member TENANT_B
ip address 192.168.2.254/24
fabric forwarding mode anycast-gateway
interface Vlan900
description TENANT_A VL900 L3VNI
no shutdown
mtu 9216
vrf member TENANT_A
ip forward
interface Vlan910
description TENANT_B VL910 L3VNI
no shutdown
mtu 9216
vrf member TENANT_B
ip forward
interface nve1
no shutdown
description VTEP_03
source-interface loopback0
host-reachability protocol bgp
member vni 10000
suppress-arp
mcast-group 225.1.0.1
member vni 20000
suppress-arp
mcast-group 225.2.0.1
member vni 30000
suppress-arp
mcast-group 225.3.0.1
member vni 40000
suppress-arp
mcast-group 225.4.0.1
member vni 90000 associate-vrf
member vni 91000 associate-vrf
interface Ethernet1/1
description TO SPINE_01
no switchport
mtu 9216
ip address 10.0.0.18/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO SPINE_02
no switchport
mtu 9216
ip address 10.0.0.22/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
no switchport
mtu 9216
interface Ethernet1/4
description TO CE_R07 - OTV LAN
switchport mode trunk
switchport trunk allowed vlan 100,200,300,400
mtu 9216
interface Ethernet1/5
description TO CE7 - EXT L3
no switchport
mtu 9216
no shutdown
interface Ethernet1/5.90
description TO CE7 - TENANT A L3VNI EXT L3
mtu 9216
encapsulation dot1q 900
vrf member TENANT_A
ip address 172.16.3.253/30
no shutdown
interface Ethernet1/5.91
description TO CE7 - TENANT B L3VNI EXT L3
mtu 9216
encapsulation dot1q 910
vrf member TENANT_B
ip address 192.168.3.253/30
no shutdown
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.214/24
interface loopback0
description RID - VTEP
ip address 5.5.5.5/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
cli alias name ll sh ip route
cli alias name p ping
cli alias name tr traceroute
cli alias name q exit
cli alias name lo exit
cli alias name sr sh running-config
cli alias name sibsle show bgp l2vpn evpn summary
cli alias name sible sh bgp l2vpn evpn
cli alias name snn sh nve vni
cli alias name snp sh nve peer
cli alias name slema show l2route evpn mac all
cli alias name slemia show l2route evpn mac-ip all
cli alias name lla sh ip route vrf TENANT_A
cli alias name svi show vxlan interface
cli alias name siasd show ip arp suppression-cache detail
cli alias name snipd sh nve internal platform interface nve1 detail
cli alias name snpd sh nve peer detail
cli alias name llm sh ip mroute
cli alias name sion sh ip ospf neighbors
cli alias name llb sh ip route vrf TENANT_B
cli alias name shmac show system internal l2fwder mac
cli alias name clms clear mac address-table datapath static
cli alias name clmd clear mac address-table datapath dynamic
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 100
router-id 5.5.5.5
passive-interface default
router bgp 65500
router-id 5.5.5.5
neighbor 1.1.1.1
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 2.2.2.2
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf TENANT_A
timers bgp 7 21
address-family ipv4 unicast
advertise l2vpn evpn
neighbor 172.16.3.254
remote-as 65000
address-family ipv4 unicast
send-community
send-community extended
vrf TENANT_B
timers bgp 7 21
address-family ipv4 unicast
advertise l2vpn evpn
neighbor 192.168.3.254
remote-as 65000
address-family ipv4 unicast
send-community
send-community extended
evpn
vni 10000 l2
rd auto
route-target import auto
route-target export auto
vni 20000 l2
rd auto
route-target import auto
route-target export auto
vni 30000 l2
rd auto
route-target import auto
route-target export auto
vni 40000 l2
rd auto
route-target import auto
route-target export auto
hostname DC1_LEAF_03
vdc DC1_LEAF_03 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
vlan 1,100,200,300,400,900,910
fabric forwarding anycast-gateway-mac 0000.dc01.dc01
ip pim rp-address 100.100.100.100 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 100
vn-segment 10000
vlan 200
vn-segment 20000
vlan 300
vn-segment 30000
vlan 400
vn-segment 40000
vlan 900
vn-segment 90000
vlan 910
vn-segment 91000
vrf context TENANT_A
vni 90000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context TENANT_B
vni 91000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Vlan1
interface Vlan100
description TENANT_A VL100
no shutdown
mtu 9216
vrf member TENANT_A
ip address 172.16.1.254/24
fabric forwarding mode anycast-gateway
interface Vlan200
description TENANT_A VL200
no shutdown
mtu 9216
vrf member TENANT_A
ip address 172.16.2.254/24
fabric forwarding mode anycast-gateway
interface Vlan300
description TENANT_B VL300
no shutdown
mtu 9216
vrf member TENANT_B
ip address 192.168.1.254/24
fabric forwarding mode anycast-gateway
interface Vlan400
description TENANT_B VL400
no shutdown
mtu 9216
vrf member TENANT_B
ip address 192.168.2.254/24
fabric forwarding mode anycast-gateway
interface Vlan900
description TENANT_A VL900 L3VNI
no shutdown
mtu 9216
vrf member TENANT_A
ip forward
interface Vlan910
description TENANT_B VL910 L3VNI
no shutdown
mtu 9216
vrf member TENANT_B
ip forward
interface nve1
no shutdown
description VTEP_03
source-interface loopback0
host-reachability protocol bgp
member vni 10000
suppress-arp
mcast-group 225.1.0.1
member vni 20000
suppress-arp
mcast-group 225.2.0.1
member vni 30000
suppress-arp
mcast-group 225.3.0.1
member vni 40000
suppress-arp
mcast-group 225.4.0.1
member vni 90000 associate-vrf
member vni 91000 associate-vrf
interface Ethernet1/1
description TO SPINE_01
no switchport
mtu 9216
ip address 10.0.0.18/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO SPINE_02
no switchport
mtu 9216
ip address 10.0.0.22/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
no switchport
mtu 9216
interface Ethernet1/4
description TO CE_R07 - OTV LAN
switchport mode trunk
switchport trunk allowed vlan 100,200,300,400
mtu 9216
interface Ethernet1/5
description TO CE7 - EXT L3
no switchport
mtu 9216
no shutdown
interface Ethernet1/5.90
description TO CE7 - TENANT A L3VNI EXT L3
mtu 9216
encapsulation dot1q 900
vrf member TENANT_A
ip address 172.16.3.253/30
no shutdown
interface Ethernet1/5.91
description TO CE7 - TENANT B L3VNI EXT L3
mtu 9216
encapsulation dot1q 910
vrf member TENANT_B
ip address 192.168.3.253/30
no shutdown
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.214/24
interface loopback0
description RID - VTEP
ip address 5.5.5.5/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
cli alias name ll sh ip route
cli alias name p ping
cli alias name tr traceroute
cli alias name q exit
cli alias name lo exit
cli alias name sr sh running-config
cli alias name sibsle show bgp l2vpn evpn summary
cli alias name sible sh bgp l2vpn evpn
cli alias name snn sh nve vni
cli alias name snp sh nve peer
cli alias name slema show l2route evpn mac all
cli alias name slemia show l2route evpn mac-ip all
cli alias name lla sh ip route vrf TENANT_A
cli alias name svi show vxlan interface
cli alias name siasd show ip arp suppression-cache detail
cli alias name snipd sh nve internal platform interface nve1 detail
cli alias name snpd sh nve peer detail
cli alias name llm sh ip mroute
cli alias name sion sh ip ospf neighbors
cli alias name llb sh ip route vrf TENANT_B
cli alias name shmac show system internal l2fwder mac
cli alias name clms clear mac address-table datapath static
cli alias name clmd clear mac address-table datapath dynamic
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 100
router-id 5.5.5.5
passive-interface default
router bgp 65500
router-id 5.5.5.5
neighbor 1.1.1.1
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 2.2.2.2
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf TENANT_A
timers bgp 7 21
address-family ipv4 unicast
advertise l2vpn evpn
neighbor 172.16.3.254
remote-as 65000
address-family ipv4 unicast
send-community
send-community extended
vrf TENANT_B
timers bgp 7 21
address-family ipv4 unicast
advertise l2vpn evpn
neighbor 192.168.3.254
remote-as 65000
address-family ipv4 unicast
send-community
send-community extended
evpn
vni 10000 l2
rd auto
route-target import auto
route-target export auto
vni 20000 l2
rd auto
route-target import auto
route-target export auto
vni 30000 l2
rd auto
route-target import auto
route-target export auto
vni 40000 l2
rd auto
route-target import auto
route-target export auto
WAN_Edge (CE_R07)
version 16.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime show-timezone year
service timestamps log datetime localtime show-timezone year
service password-encryption
no platform punt-keepalive disable-kernel-core
platform console auto
!
hostname CE_R07
!
boot-start-marker
boot-end-marker
!
!
vrf definition TENANT_A
rd 65000:1
!
address-family ipv4
exit-address-family
!
vrf definition TENANT_B
rd 65000:2
!
address-family ipv4
exit-address-family
!
logging buffered 16000
no logging console
enable secret 5 <removed>
!
no aaa new-model
!
ip vrf MGMT
!
!
!
!
!
!
!
!
!
no ip domain lookup
ip domain name domain.local
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
!
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
!
!
!
license udi pid CSR1000V sn <removed>
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
description RID
ip address 70.70.70.70 255.255.255.255
!
interface GigabitEthernet1
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet1.3000
description WAN TO PE_R13 - TENANT A VPLS
encapsulation dot1Q 3000
vrf forwarding TENANT_A
ip address 172.16.3.1 255.255.255.248
!
interface GigabitEthernet1.3001
description WAN TO PE_R13 - TENANT B VPLS
encapsulation dot1Q 3001
vrf forwarding TENANT_B
ip address 192.168.3.1 255.255.255.248
!
interface GigabitEthernet2
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet2.90
description TO VXLAN EVPN DC1_LEAF_03 - L3VNI TENANT A
encapsulation dot1Q 900
vrf forwarding TENANT_A
ip address 172.16.3.254 255.255.255.252
!
interface GigabitEthernet2.91
description TO VXLAN EVPN DC1_LEAF_03 - L3VNI TENANT B
encapsulation dot1Q 910
vrf forwarding TENANT_B
ip address 192.168.3.254 255.255.255.252
!
interface GigabitEthernet3
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet4
description MANAGEMENT
ip vrf forwarding MGMT
ip address 10.1.1.227 255.255.255.0
negotiation auto
no mop enabled
no mop sysid
!
router bgp 65000
bgp router-id 70.70.70.70
bgp log-neighbor-changes
!
address-family ipv4
exit-address-family
!
address-family ipv4 vrf TENANT_A
neighbor DC1_LEAF3_TENANT_A peer-group
neighbor DC1_LEAF3_TENANT_A remote-as 65500
neighbor DC1_LEAF3_TENANT_A description TO DC1 VXLAN TENANT A
neighbor DC1_LEAF3_TENANT_A timers 7 21
neighbor DC1_LEAF3_TENANT_A send-community both
neighbor BRANCH_CE17_TENANT_A peer-group
neighbor BRANCH_CE17_TENANT_A remote-as 65100
neighbor BRANCH_CE17_TENANT_A description TO BRANCH CE17 TENANT A
neighbor BRANCH_CE17_TENANT_A timers 7 21
neighbor BRANCH_CE17_TENANT_A send-community both
neighbor 172.16.3.3 peer-group BRANCH_CE17_TENANT_A
neighbor 172.16.3.3 activate
neighbor 172.16.3.253 peer-group DC1_LEAF3_TENANT_A
neighbor 172.16.3.253 activate
exit-address-family
!
address-family ipv4 vrf TENANT_B
neighbor DC1_LEAF3_TENANT_B peer-group
neighbor DC1_LEAF3_TENANT_B remote-as 65500
neighbor DC1_LEAF3_TENANT_B description TO DC1 VXLAN TENANT B
neighbor DC1_LEAF3_TENANT_B timers 7 21
neighbor DC1_LEAF3_TENANT_B send-community both
neighbor BRANCH_CE17_TENANT_B peer-group
neighbor BRANCH_CE17_TENANT_B remote-as 65100
neighbor BRANCH_CE17_TENANT_B description TO BRANCH CE17 TENANT B
neighbor BRANCH_CE17_TENANT_B timers 7 21
neighbor BRANCH_CE17_TENANT_B send-community both
neighbor 192.168.3.3 peer-group BRANCH_CE17_TENANT_B
neighbor 192.168.3.3 activate
neighbor 192.168.3.253 peer-group DC1_LEAF3_TENANT_B
neighbor 192.168.3.253 activate
exit-address-family
!
!
virtual-service csr_mgmt
!
no ip forward-protocol nd
no ip http server
ip http authentication local
no ip http secure-server
!
ip bgp-community new-format
ip tftp source-interface GigabitEthernet4
ip route vrf MGMT 0.0.0.0 0.0.0.0 10.1.1.254 name MGMT_DEFAULT
!
!
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
alias exec ll sh ip route
alias exec sibs sh ip bgp summary
alias exec sib sh ip bgp
alias exec cibs clear ip bgp * soft
alias exec sibsa sh ip bgp vpnv4 vrf TENANT_A summary
alias exec siba sh ip bgp vpnv4 vrf TENANT_A
alias exec lla sh ip route vrf TENANT_A
alias exec pa ping vrf TENANT_A
alias exec sibsb sh ip bgp vpnv4 vrf TENANT_B summary
alias exec sibb sh ip bgp vpnv4 vrf TENANT_B
alias exec llb sh ip route vrf TENANT_B
alias exec pb ping vrf TENANT_B
!
line con 0
logging synchronous
stopbits 1
line vty 0
exec-timeout 60 0
logging synchronous
login local
transport input telnet ssh
line vty 1
exec-timeout 60 0
logging synchronous
login local
length 0
transport input telnet ssh
line vty 2 4
exec-timeout 60 0
logging synchronous
login local
transport input telnet ssh
!
ntp server pool.ntp.org
!
!
!
!
!
end
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime show-timezone year
service timestamps log datetime localtime show-timezone year
service password-encryption
no platform punt-keepalive disable-kernel-core
platform console auto
!
hostname CE_R07
!
boot-start-marker
boot-end-marker
!
!
vrf definition TENANT_A
rd 65000:1
!
address-family ipv4
exit-address-family
!
vrf definition TENANT_B
rd 65000:2
!
address-family ipv4
exit-address-family
!
logging buffered 16000
no logging console
enable secret 5 <removed>
!
no aaa new-model
!
ip vrf MGMT
!
!
!
!
!
!
!
!
!
no ip domain lookup
ip domain name domain.local
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
!
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
!
!
!
license udi pid CSR1000V sn <removed>
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
description RID
ip address 70.70.70.70 255.255.255.255
!
interface GigabitEthernet1
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet1.3000
description WAN TO PE_R13 - TENANT A VPLS
encapsulation dot1Q 3000
vrf forwarding TENANT_A
ip address 172.16.3.1 255.255.255.248
!
interface GigabitEthernet1.3001
description WAN TO PE_R13 - TENANT B VPLS
encapsulation dot1Q 3001
vrf forwarding TENANT_B
ip address 192.168.3.1 255.255.255.248
!
interface GigabitEthernet2
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet2.90
description TO VXLAN EVPN DC1_LEAF_03 - L3VNI TENANT A
encapsulation dot1Q 900
vrf forwarding TENANT_A
ip address 172.16.3.254 255.255.255.252
!
interface GigabitEthernet2.91
description TO VXLAN EVPN DC1_LEAF_03 - L3VNI TENANT B
encapsulation dot1Q 910
vrf forwarding TENANT_B
ip address 192.168.3.254 255.255.255.252
!
interface GigabitEthernet3
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet4
description MANAGEMENT
ip vrf forwarding MGMT
ip address 10.1.1.227 255.255.255.0
negotiation auto
no mop enabled
no mop sysid
!
router bgp 65000
bgp router-id 70.70.70.70
bgp log-neighbor-changes
!
address-family ipv4
exit-address-family
!
address-family ipv4 vrf TENANT_A
neighbor DC1_LEAF3_TENANT_A peer-group
neighbor DC1_LEAF3_TENANT_A remote-as 65500
neighbor DC1_LEAF3_TENANT_A description TO DC1 VXLAN TENANT A
neighbor DC1_LEAF3_TENANT_A timers 7 21
neighbor DC1_LEAF3_TENANT_A send-community both
neighbor BRANCH_CE17_TENANT_A peer-group
neighbor BRANCH_CE17_TENANT_A remote-as 65100
neighbor BRANCH_CE17_TENANT_A description TO BRANCH CE17 TENANT A
neighbor BRANCH_CE17_TENANT_A timers 7 21
neighbor BRANCH_CE17_TENANT_A send-community both
neighbor 172.16.3.3 peer-group BRANCH_CE17_TENANT_A
neighbor 172.16.3.3 activate
neighbor 172.16.3.253 peer-group DC1_LEAF3_TENANT_A
neighbor 172.16.3.253 activate
exit-address-family
!
address-family ipv4 vrf TENANT_B
neighbor DC1_LEAF3_TENANT_B peer-group
neighbor DC1_LEAF3_TENANT_B remote-as 65500
neighbor DC1_LEAF3_TENANT_B description TO DC1 VXLAN TENANT B
neighbor DC1_LEAF3_TENANT_B timers 7 21
neighbor DC1_LEAF3_TENANT_B send-community both
neighbor BRANCH_CE17_TENANT_B peer-group
neighbor BRANCH_CE17_TENANT_B remote-as 65100
neighbor BRANCH_CE17_TENANT_B description TO BRANCH CE17 TENANT B
neighbor BRANCH_CE17_TENANT_B timers 7 21
neighbor BRANCH_CE17_TENANT_B send-community both
neighbor 192.168.3.3 peer-group BRANCH_CE17_TENANT_B
neighbor 192.168.3.3 activate
neighbor 192.168.3.253 peer-group DC1_LEAF3_TENANT_B
neighbor 192.168.3.253 activate
exit-address-family
!
!
virtual-service csr_mgmt
!
no ip forward-protocol nd
no ip http server
ip http authentication local
no ip http secure-server
!
ip bgp-community new-format
ip tftp source-interface GigabitEthernet4
ip route vrf MGMT 0.0.0.0 0.0.0.0 10.1.1.254 name MGMT_DEFAULT
!
!
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
alias exec ll sh ip route
alias exec sibs sh ip bgp summary
alias exec sib sh ip bgp
alias exec cibs clear ip bgp * soft
alias exec sibsa sh ip bgp vpnv4 vrf TENANT_A summary
alias exec siba sh ip bgp vpnv4 vrf TENANT_A
alias exec lla sh ip route vrf TENANT_A
alias exec pa ping vrf TENANT_A
alias exec sibsb sh ip bgp vpnv4 vrf TENANT_B summary
alias exec sibb sh ip bgp vpnv4 vrf TENANT_B
alias exec llb sh ip route vrf TENANT_B
alias exec pb ping vrf TENANT_B
!
line con 0
logging synchronous
stopbits 1
line vty 0
exec-timeout 60 0
logging synchronous
login local
transport input telnet ssh
line vty 1
exec-timeout 60 0
logging synchronous
login local
length 0
transport input telnet ssh
line vty 2 4
exec-timeout 60 0
logging synchronous
login local
transport input telnet ssh
!
ntp server pool.ntp.org
!
!
!
!
!
end
Testing Results
As proof of concept, I only tested VXLAN connectivity with hosts in TENANT_A. The testing protocol used here is for Host4 to simply ping & trace the other hosts (5, 6 and 7) in the DC, and also to the external branch host outside the DC network to verify full reachability.
- Host4 (DC Source)
- IP: 172.16.1.1/24
- MAC: 000c.2977.ea21
- VLAN 100
- Location: Leaf1
- Host6 (DC Destination)
- IP: 172.16.1.2
- MAC: 000c.2998.c13c
- VLAN 100
- Location: Leaf2
- Host5 (DC Destination)
- IP: 172.16.2.1
- MAC: 000c.29a8.acde
- VLAN 200
- Location: Leaf1
- Host7 (DC Destination)
- IP: 172.16.2.2
- MAC: 000c.292d.161b
- VLAN 200
- Location: Leaf2
- Host3 (Branch Destination)
- IP: 172.16.100.1
- Location: External WAN
- Host4 pings to other DC Hosts
- Host4 ARP cache
- Default gateway MAC is the Distributed Anycast Gateway
- Host4 traceroutes to other DC Hosts
- Host4 ping and trace to branch host
VXLAN EVPN Infrastructure Verification
Peering Information and Verification
Spine underlay IGP peering (OSPF)
DC1_SPINE_01#
sh ip ospf neighbors
OSPF Process ID 100 VRF default
Total number of neighbors: 3
Neighbor ID Pri State Up Time Address Interface
3.3.3.3 1 FULL/ - 1w0d 10.0.0.2 Eth1/1
4.4.4.4 1 FULL/ - 1w0d 10.0.0.6 Eth1/2
5.5.5.5 1 FULL/ - 6d23h 10.0.0.18 Eth1/3
|
DC1_SPINE_02#
sh ip ospf neighbors
OSPF Process ID 100 VRF default
Total number of neighbors: 3
Neighbor ID Pri State Up Time Address Interface
4.4.4.4 1 FULL/ - 1w0d 10.0.0.10 Eth1/1
3.3.3.3 1 FULL/ - 1w0d 10.0.0.14 Eth1/2
5.5.5.5 1 FULL/ - 6d23h 10.0.0.22 Eth1/3
|
Leaf underlay IGP peering (OSPF)
DC1_LEAF_01#
sh ip ospf neighbors
OSPF Process ID 100 VRF default
Total number of neighbors: 2
Neighbor ID Pri State Up Time Address Interface
1.1.1.1 1 FULL/ - 1w0d 10.0.0.1 Eth1/1
2.2.2.2 1 FULL/ - 1w0d 10.0.0.13 Eth1/2
|
DC1_LEAF_02#
sh ip ospf neighbors
OSPF Process ID 100 VRF default
Total number of neighbors: 2
Neighbor ID Pri State Up Time Address Interface
2.2.2.2 1 FULL/ - 1w0d 10.0.0.9 Eth1/1
1.1.1.1 1 FULL/ - 1w0d 10.0.0.5 Eth1/2
|
DC1_LEAF_03#
sh ip ospf neighbors
OSPF Process ID 100 VRF default
Total number of neighbors: 2
Neighbor ID Pri State Up Time Address Interface
1.1.1.1 1 FULL/ - 6d23h 10.0.0.17 Eth1/1
2.2.2.2 1 FULL/ - 6d23h 10.0.0.21 Eth1/2
|
Spine to Leaf iBGP peering for address family L2VPN EVPN
DC1_SPINE_01#
sh bgp l2vpn evpn summary
BGP summary
information for VRF default, address family L2VPN EVPN
BGP router
identifier 1.1.1.1, local AS number 65500
BGP table
version is 109238, L2VPN EVPN config peers 3, capable peers 3
21 network
entries and 21 paths using 4536 bytes of memory
BGP attribute
entries [17/2652], BGP AS path entries [3/34]
BGP community
entries [0/0], BGP clusterlist entries [0/0]
Neighbor V
AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
3.3.3.3 4 65500 66854
92169 109238 0
0 1w0d 8
4.4.4.4 4 65500 68134
90511 109238 0
0 1w0d 8
5.5.5.5 4 65500 54529
96769 109238 0
0 6d23h 5
|
DC1_SPINE_02#
sh bgp l2vpn evpn summary
BGP summary
information for VRF default, address family L2VPN EVPN
BGP router
identifier 2.2.2.2, local AS number 65500
BGP table
version is 108948, L2VPN EVPN config peers 3, capable peers 3
21 network
entries and 21 paths using 4536 bytes of memory
BGP attribute
entries [17/2652], BGP AS path entries [3/34]
BGP community
entries [0/0], BGP clusterlist entries [0/0]
Neighbor V
AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
3.3.3.3 4 65500 66855
92166 108948 0
0 1w0d 8
4.4.4.4 4 65500 68141
90502 108948 0
0 1w0d 8
5.5.5.5 4 65500 54534
96773 108948 0
0 6d23h 5
|
Leaf to Spine iBGP peering for address family L2VPN EVPN
DC1_LEAF_01#
sh bgp l2vpn evpn summary
BGP summary
information for VRF default, address family L2VPN EVPN
BGP router
identifier 3.3.3.3, local AS number 65500
BGP table
version is 126312, L2VPN EVPN config peers 2, capable peers 2
38 network
entries and 51 paths using 7696 bytes of memory
BGP attribute
entries [32/4992], BGP AS path entries [3/34]
BGP community
entries [0/0], BGP clusterlist entries [4/16]
Neighbor V
AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
1.1.1.1 4 65500 39968
21311 126312 0
0 1w0d 13
2.2.2.2 4 65500 39967
21311 126312 0
0 1w0d 13
|
DC1_LEAF_02#
sh bgp l2vpn evpn summary
BGP summary
information for VRF default, address family L2VPN EVPN
BGP router
identifier 4.4.4.4, local AS number 65500
BGP table
version is 128780, L2VPN EVPN config peers 2, capable peers 2
38 network
entries and 51 paths using 7696 bytes of memory
BGP attribute
entries [32/4992], BGP AS path entries [3/34]
BGP community
entries [0/0], BGP clusterlist entries [4/16]
Neighbor V
AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
1.1.1.1 4 65500 38877
28637 128780 0
0 1w0d 13
2.2.2.2 4 65500 38868
28639 128780 0
0 1w0d 13
|
DC1_LEAF_03#
sh bgp l2vpn evpn summary
BGP summary
information for VRF default, address family L2VPN EVPN
BGP router
identifier 5.5.5.5, local AS number 65500
BGP table
version is 108507, L2VPN EVPN config peers 2, capable peers 2
45 network
entries and 61 paths using 8696 bytes of memory
BGP attribute
entries [36/5616], BGP AS path entries [3/34]
BGP community
entries [0/0], BGP clusterlist entries [4/16]
Neighbor V
AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
1.1.1.1 4 65500 35939
22995 108507 0
0 6d23h 16
2.2.2.2 4 65500
35939 22995 108507
0 0 6d23h 16
|
Leaf3 (Border Leaf) BGP peering to external router (CE_R07)
DC1_LEAF_03# sh ip bgp vrf TENANT_A summary
BGP summary information for VRF TENANT_A, address family IPv4 Unicast
BGP router identifier 172.16.2.254, local AS number 65500
BGP table version is 57, IPv4 Unicast config peers 1, capable peers 1
5 network entries and 5 paths using 568 bytes of memory
BGP attribute entries [5/780], BGP AS path entries [1/10]
BGP community entries [0/0], BGP clusterlist entries [4/16]
Neighbor V AS MsgRcvd MsgSent TblVer
InQ OutQ Up/Down State/PfxRcd
172.16.3.254 4 65000 89956
86926 57 0
0 1w0d 1
|
External WAN Edge Router (CE_R07)
BGP peering to Border Leaf3
CE_R07#sh ip bgp vpnv4 vrf TENANT_A summary
BGP router identifier 70.70.70.70, local AS number 65000
BGP table version is 79, main routing table version 79
5 network entries using 1280 bytes of memory
5 path entries using 640 bytes of memory
9/4 BGP path/bestpath attribute entries using 2592 bytes of memory
7 BGP AS-PATH entries using 232 bytes of memory
1 BGP extended community entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 4768 total bytes of memory
BGP activity 36/23 prefixes, 53/36 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer
InQ OutQ Up/Down State/PfxRcd
172.16.3.3 4 65100 5334
5345 79
0 0 10:01:26 1
172.16.3.253 4 65500 86942
89973 79 0
0 1w0d
|
External WAN Edge Router
(CE_R07) Route Table
CE_R07#sh ip route vrf TENANT_A
Routing Table: TENANT_A
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B -
BGP
D - EIGRP, EX - EIGRP
external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external
type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1,
E2 - OSPF external type 2
i - IS-IS, su - IS-IS
summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * -
candidate default, U - per-user static route
o - ODR, P - periodic
downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % -
next hop override, p - overrides from PfR
Gateway of last resort is not set
172.16.0.0/16 is variably
subnetted, 9 subnets, 4 masks
B 172.16.1.1/32 [20/0] via
172.16.3.253, 1d10h
B 172.16.1.2/32 [20/0] via
172.16.3.253, 1d10h
B 172.16.2.1/32 [20/0] via
172.16.3.253, 1d10h
B 172.16.2.2/32 [20/0] via
172.16.3.253, 1d10h
C 172.16.3.0/29 is directly
connected, GigabitEthernet1.3000
L 172.16.3.1/32 is directly
connected, GigabitEthernet1.3000
C 172.16.3.252/30 is
directly connected, GigabitEthernet2.90
L 172.16.3.254/32 is
directly connected, GigabitEthernet2.90
B 172.16.100.0/24 [20/0]
via 172.16.3.3, 10:04:42
|
Underlay Multicast Information & Verification
PIM RP on Spines
DC1_SPINE_01#
sh ip pim rp
PIM RP Status
Information for VRF "default"
BSR: Not
Operational
Auto-RP
disabled
BSR RP
Candidate policy: None
BSR RP
policy: None
Auto-RP
Announce policy: None
Auto-RP
Discovery policy: None
Anycast-RP
100.100.100.100 members:
1.1.1.1*
2.2.2.2
RP:
100.100.100.100*, (0), uptime: 2w5d, expires: never,
priority: 255, RP-source: (local), group
ranges:
225.0.0.0/8
|
DC1_SPINE_02#
sh ip pim rp
PIM RP Status
Information for VRF "default"
BSR: Not
Operational
Auto-RP
disabled
BSR RP
Candidate policy: None
BSR RP
policy: None
Auto-RP
Announce policy: None
Auto-RP
Discovery policy: None
Anycast-RP
100.100.100.100 members:
1.1.1.1
2.2.2.2*
RP:
100.100.100.100*, (0), uptime: 2w5d, expires: never,
priority: 255, RP-source: (local), group
ranges:
225.0.0.0/8
|
PIM Neighbors
DC1_SPINE_01# sh ip pim neighbor
PIM Neighbor Status for VRF "default"
Neighbor Interface Uptime Expires DR Bidir- BFD
Priority Capable State
10.0.0.2 Ethernet1/1 1w0d 00:01:39 1 yes n/a
10.0.0.6 Ethernet1/2 1w0d 00:01:35 1 yes n/a
10.0.0.18 Ethernet1/3 1w0d 00:01:26 1 yes n/a
|
DC1_SPINE_02# sh ip pim neighbor
PIM Neighbor Status for VRF "default"
Neighbor Interface Uptime Expires DR Bidir- BFD
Priority Capable State
10.0.0.10 Ethernet1/1 1w0d 00:01:39 1 yes n/a
10.0.0.14 Ethernet1/2 1w0d 00:01:31 1 yes n/a
10.0.0.22 Ethernet1/3 1w0d 00:01:28 1 yes n/a
|
DC1_LEAF_01# sh ip pim neighbor
PIM Neighbor Status for VRF "default"
Neighbor Interface Uptime Expires DR Bidir- BFD
Priority Capable State
10.0.0.1 Ethernet1/1 1w0d 00:01:20 1 yes n/a
10.0.0.13 Ethernet1/2 1w0d 00:01:34 1 yes n/a
|
DC1_LEAF_02# sh ip pim neighbor
PIM Neighbor Status for VRF "default"
Neighbor Interface Uptime Expires DR Bidir- BFD
Priority Capable State
10.0.0.9 Ethernet1/1 1w0d 00:01:43 1 yes n/a
10.0.0.5 Ethernet1/2 1w0d 00:01:44 1 yes n/a
|
DC1_LEAF_03# sh ip pim neighbor
PIM Neighbor Status for VRF "default"
Neighbor Interface Uptime Expires DR Bidir- BFD
Priority Capable State
10.0.0.17 Ethernet1/1 1w0d 00:01:43 1 yes n/a
10.0.0.21 Ethernet1/2 1w0d 00:01:21 1 yes n/a
|
Multicast Ping Verification (Spine_01)
(Used only one VTEP multicast group
225.1.0.1 for testing brevity)
DC1_SPINE_01# ping multicast 225.1.0.1 interface e1/1 count 1
PING 225.1.0.1 (225.1.0.1): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=254 time=3.117 ms
--- 225.1.0.1 ping multicast statistics ---
1 packets transmitted,
From member 10.0.0.2: 1 packet received, 0.00% packet loss
--- in total, 1 group member responded ---
|
DC1_SPINE_01# ping multicast 225.1.0.1 interface e1/2 count 1
PING 225.1.0.1 (225.1.0.1): 56 data bytes
64 bytes from 10.0.0.6: icmp_seq=0 ttl=254 time=3.07 ms
--- 225.1.0.1 ping multicast statistics ---
1 packets transmitted,
From member 10.0.0.6: 1 packet received, 0.00% packet loss
--- in total, 1 group member responded ---
|
DC1_SPINE_01# ping multicast 225.1.0.1 interface e1/3 count 1
PING 225.1.0.1 (225.1.0.1): 56 data bytes
64 bytes from 10.0.0.18: icmp_seq=0 ttl=254 time=2.665 ms
--- 225.1.0.1 ping multicast statistics ---
1 packets transmitted,
From member 10.0.0.18: 1 packet received, 0.00% packet loss
--- in total, 1 group member responded --
|
Multicast ping verification (Spine2)
(Used only one VTEP multicast group
225.1.0.1 for testing brevity)
DC1_SPINE_02# ping multicast 225.1.0.1 interface e1/1 count 1
PING 225.1.0.1 (225.1.0.1): 56 data bytes
64 bytes from 10.0.0.10: icmp_seq=0 ttl=254 time=3.121 ms
--- 225.1.0.1 ping multicast statistics ---
1 packets transmitted,
From member 10.0.0.10: 1 packet received, 0.00% packet loss
--- in total, 1 group member responded ---
|
DC1_SPINE_02# ping multicast 225.1.0.1 interface e1/2 count 1
PING 225.1.0.1 (225.1.0.1): 56 data bytes
64 bytes from 10.0.0.14: icmp_seq=0 ttl=254 time=2.806 ms
--- 225.1.0.1 ping multicast statistics ---
1 packets transmitted,
From member 10.0.0.14: 1 packet received, 0.00% packet loss
--- in total, 1 group member responded ---
|
DC1_SPINE_02# ping multicast 225.1.0.1 interface e1/3 count 1
PING 225.1.0.1 (225.1.0.1): 56 data bytes
64 bytes from 10.0.0.22: icmp_seq=0 ttl=254 time=3.082 ms
--- 225.1.0.1 ping multicast statistics ---
1 packets transmitted,
From member 10.0.0.22: 1 packet received, 0.00% packet loss
--- in total, 1 group member responded ---
|
VXLAN Overlay Verification
VLAN to VN-Segment (VNI)
DC1_LEAF_01#
sh vxlan
Vlan VN-Segment
==== ==========
100 10000
200 20000
300 30000
400 40000
900 90000
910 91000
|
DC1_LEAF_02#
sh vxlan
Vlan VN-Segment
==== ==========
100 10000
200 20000
300 30000
400 40000
900 90000
910 91000
|
DC1_LEAF_03#
sh vxlan
Vlan VN-Segment
==== ==========
100 10000
200 20000
300 30000
400 40000
900 90000
910 91000
|
VTEP peering between Leaf Switches
DC1_LEAF_01# sh nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 4.4.4.4 Up CP 1w0d 000c.2984.640e
nve1 5.5.5.5 Up CP 6d23h 000c.29b7.b1d3
|
DC1_LEAF_02# sh nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 3.3.3.3 Up CP 1w0d 000c.2910.ac2a
nve1 5.5.5.5 Up CP 6d23h 000c.29b7.b1d3
|
DC1_LEAF_03# sh nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 3.3.3.3 Up CP 6d23h 000c.2910.ac2a
nve1 4.4.4.4 Up CP 6d23h 000c.2984.640e
|
Displays associated VNI’s to multicast groups to bridge
domains
DC1_LEAF_01# sh nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 10000 225.1.0.1 Up CP L2 [100] SA
nve1 20000 225.2.0.1 Up CP L2 [200] SA
nve1 30000 225.3.0.1 Up CP L2 [300] SA
nve1 40000 225.4.0.1 Up CP L2 [400] SA
nve1 90000 n/a Up CP L3 [TENANT_A]
nve1 91000 n/a Up CP L3 [TENANT_B]
|
DC1_LEAF_02# sh nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 10000 225.1.0.1 Up CP L2 [100] SA
nve1 20000 225.2.0.1 Up CP L2 [200] SA
nve1 30000 225.3.0.1 Up CP L2 [300] SA
nve1 40000 225.4.0.1 Up CP L2 [400] SA
nve1 90000 n/a Up CP L3 [TENANT_A]
nve1 91000 n/a Up CP L3 [TENANT_B]
|
DC1_LEAF_03# sh nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 10000 225.1.0.1 Up CP L2 [100] SA
nve1 20000 225.2.0.1 Up CP L2 [200] SA
nve1 30000 225.3.0.1 Up CP L2 [300] SA
nve1 40000 225.4.0.1 Up CP L2 [400] SA
nve1 90000 n/a Up CP L3 [TENANT_A]
nve1 91000 n/a Up CP L3 [TENANT_B]
|
BGP EVPN Control-Plane Verification
Show IP Route
The route table on Leaf1 for TENANT_A shows a nice summary of
how VXLAN is routing the traffic. The
output here looks like your typical route table but remote hosts learned via BGP control-plane
has additional information related to VXLAN EVPN, such as Segment/VNI ID and
VTEP tunnel endpoint ID’s. Also, locally
attached hosts are learned via the NX-OS Host Mobility Manager (HMM), a
software process that’s responsible for host learning and cleanup as hosts move
within the fabric.
DC1_LEAF_01#
sh ip route vrf TENANT_A
IP Route
Table for VRF "TENANT_A"
'*' denotes
best ucast next-hop
'**' denotes
best mcast next-hop
'[x/y]'
denotes [preference/metric]
'%<string>'
in via output denotes VRF <string>
172.16.1.0/24,
ubest/mbest: 1/0, attached
*via 172.16.1.254, Vlan100,
[0/0], 08:01:03, direct
172.16.1.1/32,
ubest/mbest: 1/0, attached
*via 172.16.1.1, Vlan100,
[190/0], 08:01:03, hmm
172.16.1.2/32,
ubest/mbest: 1/0
*via 4.4.4.4%default,
[200/0], 08:01:03, bgp-65500, internal, tag 65500 (evpn) segid: 90000
tunnelid: 0x4040404 encap: VXLAN
172.16.1.254/32,
ubest/mbest: 1/0, attached
*via 172.16.1.254, Vlan100,
[0/0], 08:01:03, local
172.16.2.0/24,
ubest/mbest: 1/0, attached
*via 172.16.2.254, Vlan200, [0/0],
08:01:03, direct
172.16.2.1/32,
ubest/mbest: 1/0, attached
*via 172.16.2.1, Vlan200,
[190/0], 08:01:03, hmm
172.16.2.2/32,
ubest/mbest: 1/0
*via 4.4.4.4%default,
[200/0], 08:01:03, bgp-65500, internal, tag 65500 (evpn) segid: 90000
tunnelid: 0x4040404 encap: VXLAN
172.16.2.254/32,
ubest/mbest: 1/0, attached
*via 172.16.2.254, Vlan200,
[0/0], 08:01:03, local
172.16.100.0/24,
ubest/mbest: 1/0
*via 5.5.5.5%default,
[200/0], 07:37:12, bgp-65500, internal, tag 65000 (evpn) segid: 90000
tunnelid: 0x5050505 encap: VXLAN
|
Show BGP L2VPN EVPN
A deeper look at the BGP L2VPN EVPN table will reveal more, such as MAC to IP mapping, what VNI they are associated to and VTEP next-hop address. Below is a quick analysis from Leaf1’s perspective of the routes in each VNI and what it means.
- Route Distinguisher (RD) 3.3.3.3:32867 is attached to Leaf1’s VNI 10000/VLAN 100
- MAC/IP route type 2 for MAC 000c.2977.ea21, host IP 172.16.1.1 is a local route
- MAC/IP route type 2 for MAC 000c.2998.c13c, host IP 172.16.1.2 is reachable via 4.4.4.4 (Leaf2)
- Route Distinguisher (RD) 3.3.3.3:32967 is attached to Leaf1’s VNI 20000/VLAN 200
- MAC/IP route type 2 for MAC 000c.292d.161b, host IP 172.16.2.2 is reachable via 4.4.4.4 (Leaf2)
- MAC/IP route type 2 for MAC 000c.29a8.acde, host IP 172.16.2.1 is a local route
- Route Distinguisher (RD) 3.3.3.3:3 is attached to Leaf1’s VNI 90000/VLAN 900 or the L3 VNI.
- MAC/IP route type 2 for MAC 000c.292d.161b, host IP 172.16.2.2 is reachable via 4.4.4.4 (Leaf2)
- MAC/IP route type 2 for MAC 000c.2998.c13c, host IP 172.16.1.2 is reachable via 4.4.4.4 (Leaf2)
- IP subnet route type 5 for prefix 172.16.100.0/24 is reachable via 5.5.5.5 (Leaf3)
DC1_LEAF_01#
sh bgp l2vpn evpn
BGP routing
table information for VRF default, address family L2VPN EVPN
BGP table
version is 167764, local router ID is 3.3.3.3
Status:
s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type:
i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes:
i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 3.3.3.3:32867 (L2VNI 10000)
*>l[2]:[0]:[0]:[48]:[000c.2977.ea21]:[0]:[0.0.0.0]/216
3.3.3.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[000c.2998.c13c]:[0]:[0.0.0.0]/216
4.4.4.4 100 0 i
x i 4.4.4.4 100 0 i
*>l[2]:[0]:[0]:[48]:[000c.2977.ea21]:[32]:[172.16.1.1]/272
3.3.3.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[000c.2998.c13c]:[32]:[172.16.1.2]/272
4.4.4.4 100 0 i
Route Distinguisher: 3.3.3.3:32967 (L2VNI 20000)
*>i[2]:[0]:[0]:[48]:[000c.292d.161b]:[0]:[0.0.0.0]/216
4.4.4.4 100 0 i
*>l[2]:[0]:[0]:[48]:[000c.29a8.acde]:[0]:[0.0.0.0]/216
3.3.3.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[000c.292d.161b]:[32]:[172.16.2.2]/272
4.4.4.4
100 0 i
*>l[2]:[0]:[0]:[48]:[000c.29a8.acde]:[32]:[172.16.2.1]/272
3.3.3.3
100 32768 i
..snip..
Route Distinguisher: 3.3.3.3:3 (L3VNI 90000)
*>i[2]:[0]:[0]:[48]:[000c.292d.161b]:[32]:[172.16.2.2]/272
4.4.4.4
100 0 i
*>i[2]:[0]:[0]:[48]:[000c.2998.c13c]:[32]:[172.16.1.2]/272
4.4.4.4
100 0 i
*>i[5]:[0]:[0]:[24]:[172.16.100.0]:[0.0.0.0]/224
5.5.5.5
100 0 65000 65100 i
|
To explain how the information in the BGP L2VPN EVPN table is used, I’ll use some traffic scenarios as examples. We will assume the source host is directly connected off Leaf1.
- Host 172.16.1.1 (VLAN100, VNI10000) → Host 172.16.1.2 (VLAN100, VNI10000) will use the BGP entry in the L2VNI 10000 (RD 3.3.3.3:32867) since the destination host is in the same VLAN/VNI. It will be reachable via 4.4.4.4 (Leaf2) and bridged to destination host.
- Host 172.16.2.1 (VLAN200, VNI20000) → Host 172.16.2.2 (VLAN200, VNI20000) will use the BGP entry in the L2VNI 20000 (RD 3.3.3.3:32967) since the destination host is in the same VLAN/VNI. It will be reachable via 4.4.4.4 (Leaf2) and bridged to destination host.
- Host 172.16.1.1 (VLAN100, VNI10000) → Host 172.16.2.2 (VLAN200, VNI20000) will use the BGP entry in the L3VNI 90000 (RD 3.3.3.3:3). It will route using the L3VNI first to 4.4.4.4 (Leaf2) then bridged to the destination host in VLAN200, VNI20000.
- Host 172.16.1.1 (VLAN100, VNI10000) → Host 172.16.100.1 (External Network) will use the BGP entry in the L3VNI 90000 (RD 3.3.3.3:3). It will route using the L3VNI first to 5.5.5.5 (Leaf3) then handed off to the WAN edge router.
Show BGP L2VPN EVPN <IP Address> | MAC Address
Displaying the BGP L2VPN EVPN on a per-route/mac entry can offer a lot more information for troubleshooting purposes. However reading it can be very cryptic and overwhelming. It goes to show the wealth of information stored in the BGP control-plane.
Personally, part of the confusion I encountered while trying to read the information was that this single output contained multiple entries for all VNI's. I was wondering why there was so much repeated information for a single host? Below, I'll try to attempt to break this down. Hopefully I got this down correctly.
So the raw EVPN BGP NLRI below from Leaf1 for host 172.16.1.2 contains an entry that was learned from another Leaf with IP 4.4.4.4 with RD 4.4.4.4:32867. It learned two routes, one from Spine1 with IP 1.1.1.1 and the other from Spine2 with IP 2.2.2.2. Looking at the RR cluster list is a easy way to tell the source.
BGP chose the route from 1.1.1.1 as the best path and VNI10000 and VNI90000 imported that route because they have the same route target values (RT: 65500:10000 RT: 65500:90000). Then they slap on their own route distinguishers (RD) to make them unique within each VNI.
DC1_LEAF_01#
sh bgp l2vpn evpn 172.16.1.2
BGP routing
table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 3.3.3.3:32867 (L2VNI 10000)
BGP routing
table entry for [2]:[0]:[0]:[48]:[000c.2998.c13c]:[32]:[172.16.1.2]/272,
version 10523
Paths: (1
available, best #1)
Flags:
(0x000212) on xmit-list, is in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best
path, no labeled nexthop, in rib
Imported from
4.4.4.4:32867:[2]:[0]:[0]:[48]:[000c.2998.c13c]:[32]:[172.16.1.2]/144
AS-Path: NONE, path sourced internal to AS
4.4.4.4 (metric 81) from 1.1.1.1
(1.1.1.1)
Origin IGP, MED not set, localpref 100,
weight 0
Received label 10000 90000
Extcommunity: RT:65500:10000 RT:65500:90000 ENCAP:8
Router MAC:000c.2984.640e
Originator: 4.4.4.4 Cluster list:
1.1.1.1
Path-id 1 not advertised to any peer
Route Distinguisher: 4.4.4.4:32867
BGP routing table entry for
[2]:[0]:[0]:[48]:[000c.2998.c13c]:[32]:[172.16.1.2]/272, version 10518
Paths: (2 available, best #1)
Flags: (0x000202) on xmit-list, is not in
l2rib/evpn, is not in HW, is locked
Advertised path-id 1
Path
type: internal, path is valid, is best path, no labeled nexthop
AS-Path:
NONE, path sourced internal to AS
4.4.4.4 (metric 81) from 1.1.1.1 (1.1.1.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 90000
Extcommunity: RT:65500:10000
RT:65500:90000 ENCAP:8 Router MAC:000c.2984.640e
Originator: 4.4.4.4 Cluster list: 1.1.1.1
Path
type: internal, path is valid, not best reason: Neighbor Address, no labeled
nexthop
AS-Path:
NONE, path sourced internal to AS
4.4.4.4 (metric 81) from 2.2.2.2 (2.2.2.2)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 90000
Extcommunity: RT:65500:10000
RT:65500:90000 ENCAP:8 Router MAC:000c.2984.640e
Originator: 4.4.4.4 Cluster list: 2.2.2.2
Path-id 1 not advertised to any peer
Route Distinguisher: 3.3.3.3:3 (L3VNI 90000)
BGP routing
table entry for [2]:[0]:[0]:[48]:[000c.2998.c13c]:[32]:[172.16.1.2]/272,
version 10524
Paths: (1
available, best #1)
Flags:
(0x000202) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best
path, no labeled nexthop
Imported from
4.4.4.4:32867:[2]:[0]:[0]:[48]:[000c.2998.c13c]:[32]:[172.16.1.2]/144
AS-Path: NONE, path sourced internal to AS
4.4.4.4 (metric 81) from 1.1.1.1
(1.1.1.1)
Origin IGP, MED not set, localpref 100,
weight 0
Received label 10000 90000
Extcommunity: RT:65500:10000 RT:65500:90000 ENCAP:8
Router MAC:000c.2984.640e
Originator: 4.4.4.4 Cluster list:
1.1.1.1
Path-id 1 not advertised to any peer
|
Furthermore, the two slides below shows there are additional information embedded into the NLRI for the different route types.
- Route Type 2
- Route Type 5
Show MAC Address-Table
This command show L2 information, similarly to most switches. The output should be familiar but here it will contain additional information such as the destination VTEP and its address.
- Local Hosts
- Remote Hosts
DC1_LEAF_01#
show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC,
(R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ -
primary entry using vPC Peer-Link,
(T) - True, (F) - False, C -
ControlPlane MAC
VLAN
MAC Address Type age
Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
G 900
000c.2910.ac2a static -
F F sup-eth1(R)
* 100 000c.2977.ea21 dynamic
00:04:00 F F
Eth1/4
* 100 000c.2998.c13c static
- F F
(0x47000002) nve-peer2 4.4.4.4
* 200 000c.29a8.acde dynamic
00:01:08 F F
Eth1/5
G 200
000c.2910.ac2a static -
F F sup-eth1(R)
G 100
000c.2910.ac2a static -
F F sup-eth1(R)
* 200 000c.292d.161b static
- F F
(0x47000002) nve-peer2 4.4.4.4
1 1 -00:00:dc:01:dc:01 - 1
|
DC1_LEAF_02#
show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) -
Routed MAC, O - Overlay MAC
age - seconds since last seen,+ -
primary entry using vPC Peer-Link,
(T) - True, (F) - False, C -
ControlPlane MAC
VLAN
MAC Address Type age
Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 100 000c.2977.ea21 static
- F F
(0x47000002) nve-peer2 3.3.3.3
* 100 000c.2998.c13c dynamic
00:00:38 F F
Eth1/4
* 200 000c.29a8.acde static
- F F
(0x47000002) nve-peer2 3.3.3.3
G 900
000c.2984.640e static -
F F sup-eth1(R)
* 200 000c.292d.161b dynamic
00:00:14 F F
Eth1/5
G 100
000c.2984.640e static
- F F
sup-eth1(R)
G 200
000c.2984.640e static -
F F sup-eth1(R)
1 1 -00:00:dc:01:dc:01 - 1
|
- Border Leaf3 has no local hosts.
DC1_LEAF_03#
show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC,
(R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ -
primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane
MAC
VLAN
MAC Address Type age
Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 100 000c.2977.ea21 static
- F F
(0x47000002) nve-peer2 3.3.3.3
* 100 000c.2998.c13c static
- F F
(0x47000001) nve-peer1 4.4.4.4
* 200 000c.29a8.acde static
- F F
(0x47000002) nve-peer2 3.3.3.3
G 200
000c.29b7.b1d3 static -
F F sup-eth1(R)
* 200 000c.292d.161b static
- F F
(0x47000001) nve-peer1 4.4.4.4
G 100
000c.29b7.b1d3 static -
F F sup-eth1(R)
* 100
001e.bd7a.d8bc dynamic 00:04:43
F F Eth1/4
1
1 1 -00:00:dc:01:dc:01 - 1- 1
|
Note: Per the Cisco NX-OSv 9000 Guide, the virtual platform has a substitute command to show the mac address-table (see below).
Show L2ROUTE EVPN
DC1_LEAF_01#
sh l2route evpn mac all
Flags
-(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link
(Dup):Duplicate
(Spl):Split (Rcv):Recv (AD):Auto-Delete(D):Del Pending (S):Stale (C):Clear
(Ps):Peer
Sync (O):Re-Originated
Topology Mac Address Prod
Flags Seq No Next-Hops
-----------
-------------- ------ ------------- ---------- ----------------
100 000c.2977.ea21 Local L, 0 Eth1/4
100 000c.2998.c13c BGP SplRcv 0 4.4.4.4
100 001e.bd7a.d8bc BGP Rcv 0 5.5.5.5
200 000c.292d.161b BGP SplRcv 0 4.4.4.4
200 000c.29a8.acde Local L, 0 Eth1/5
200 001e.bd7a.d8bc BGP Rcv 0 5.5.5.5
900 000c.2984.640e VXLAN Rmac 0 4.4.4.4
900 000c.29b7.b1d3 VXLAN Rmac 0 5.5.5.5
|
DC1_LEAF_01#
sh l2route evpn mac-ip all
Flags
-(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link
(Dup):Duplicate
(Spl):Split (Rcv):Recv(D):Del Pending (S):Stale (C):Clear
(Ps):Peer
Sync (Ro):Re-Originated
Topology Mac Address Prod
Flags Seq No Host IP Next-Hops
-----------
-------------- ------ ---------- --------------- ---------------
100 000c.2977.ea21 HMM -- 0 172.16.1.1 Local
100 000c.2998.c13c BGP -- 0 172.16.1.2 4.4.4.4
200 000c.29a8.acde HMM -- 0 172.16.2.1 Local
200 000c.292d.161b BGP -- 0 172.16.2.2 4.4.4.4
|
Conclusion
So here you have it folks. Hopefully I got this all right but if not let me know in the comments. I had fun and learned a lot during this process and I wanted to give back to the community.
For my next lab, I'll take this further and build out another Nexus 9K DC and use Cisco OTV to stitch them together. This will give me an opportunity to learn some Datacenter Internconnect (DCI) technologies in the process.
References
Here's a collection of some really good Nexus 9K VXLAN EVPN configuration documentation. These articles, documents and presentations were instrumental during the study process and would like to personally thank everyone for sharing their experience.
INE Introduction to VXLAN EVPN
Cisco: Deploy a VXLAN Network with an MP-BGP EVPN Control Plane White Paper
Personally this Cisco Live 2016 (Berlin) presentation was really good and full of useful information. I recommend anyone to watch this.
Cisco Live: BRKDCT-3378 - Building DataCenter networks with VXLAN BGP-EVPN (2016 Berlin)
VXLAN EVPN Configuration
Cisco: Configuration and Verification VXLAN with MP-BGP EVPN Control Plane
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
VXLAN BGP EVPN Challenge Lab
VXLAN EVPN
VXLAN Fabric using EVPN with Cisco Nexus 9000 Switches
VXLAN MP-BGP EVPN Overview
VxLAN BGP EVPN on Nexus 9000v – Part 1
Lab Setup Caveats
- ESXi standard vSwitch setup caveat
- Make sure MTU is set to 9000 for the additional VXLAN overhead.
- Make sure the following security settings are to accept, otherwise traffic won’t pass over the vSwitch.
- Cisco NX-OSv 9000 Setup Information
- Serial port access on Cisco NX-OSv 9000 on VMWare ESXi
- Note: An important note here is that in order to configure the virtual serial port, you need Enterprise or better licensing.
- A standalone ESXi install will allow you to create a remote serial port in the VM, however it simply will not work. The VMWare logs will show this…
2017-06-29T01:07:33.786Z| vcpu-0| I125:
[msg.serial.uri.nolicense.basic] The currently available license does not
permit the use of remote serial ports.
2017-06-29T01:07:33.786Z| vcpu-0| I125:
[msg.device.startdisconnected] Virtual device 'serial0' will start
disconnected.
|
- If you only have a standalone version of ESXi, getting the console to work on the NX-OSv was kind of tricky. I first had to deploy the VM in an ESXi environment with Enterprise licensing first (having all the vnics in an isolated vSwitch for safety) then configure the remote serial port per the Cisco document. Once you have console access to the VM, assign an IP address to the “mgmt0” interface. Make sure that interface is not in shutdown, add a default route into the “management” vrf context, enable the telnet service (feature telnet) and configure a local username and password. Save the configuration and power off the VM. At this point, you can put the mgmt0 interface into a management vSwitch and can then telnet from a management station in the same subnet. You can then clone the VM for multiple instances and change the management IP as necessary.
