After learning quite a bit with VXLAN EVPN, I wanted to take
it a little further and understand how we could join 2 distributed datacenters (Multi-fabric)
using a Datacenter Interconnect (DCI) technology. I read that Cisco’s Overlay Transport
Virtualization (OTV) was available on IOS XE so was tempted to try it out on
the CSR1000v.
Lab
Environment
In this lab I built out another DC fabric, called DC2. It’s configured for VXLAN EVPN and operates
similarly to DC1. The CE devices
(CSR1000v’s) that were used as WAN edge will also serve as the OTV edge devices
to provide L2 extension between the two DC’s.
Note: The following resources were appended to the existing VXLAN
EVPN lab.
- (5) Cisco NX-OSv 9000: version 7.0(3)I6(1)
- Used 2 vCPUs instead of the default 4
- Used default RAM at 8Gb
- (1) CSR1000v: version 16.4.2 [Everest]
- Used default vCPU setting of 1
- Used default RAM at 4Gb
- (4) Damn Small Linux (DSL) 4.x as test hosts
Diagram
& Topology
- OTV Overview
 |
| Fig. 1.1 |
- DC1 with OTV related configurations and connections
 |
| Fig. 1.2 |
- DC2 with OTV related configurations and connections
 |
| Fig. 1.3 |
Infrastructure Information- (2) Customer Edge (CE Routers)
- OTV Internal Interface connected to E1/4 on each DC’s Leaf03 switch
- WAN type (L2 VPLS transport, CE’s connected to VPLS lab)
- OTV configured in Multicast mode
- TENANT_A
- WAN IP Address Space: 172.16.3.0/29
- Extended VLANs 100 & 200
- TENANT_B
- WAN IP Address Space: 192.168.3.0/29
- Extended VLANs 300 & 400
Technology
Overview
Datacenter Interconnect (DCI)
- A DCI is a technology that can provide both L3 and L2 LAN extensions across multiple distributed datacenters. The two slides I’ve found on the Internet below (Fig 2.1 and 2.2) are good representations of what a DCI is trying to achieve.
 |
| Fig 2.1 |
 |
| Fig 2.2 |
- Using a DCI between datacenters can solve some of these IT and business related goals (Fig 2.3).
 |
| Fig 2.3 |
- Traditional DCIs can use a number of different L1 & L2 transport options, such as DWDM/Dark Fiber, VPLS or VPWS (Fig 2.4). However, new technologies are emerging where L2 extensions can occur over any IP routed networks.
 |
| Fig 2.4 |
- One main use case for LAN extensions is VM mobility across distributed datacenters (Fig 2.5).
·
 |
| Fig 2.5 |
Cisco Overlay Transport Virtualization (OTV)
Cisco OTV is a propriety protocol that can provide L2 extensions over an Any Transport
network. Unlike other L2 technologies, OTV is specifically designed for DCI.
- Here are some informational slides that will explain the technology and terminology used throughout this post (Figs 3.1 - 3.8).
 |
| Fig 3.1 |
 |
| Fig 3.2 |
 |
| Fig 3.3 |
- Common terms used during the configuration and troubleshooting of OTV (Figs. 3.4-3.5).
 |
| Fig 3.4 |
 |
| Fig 3.5 |
- OTV on the data plane can run in both Unicast and Multicast (Fig 3.6). Its usage depends on how many datacenters are connected. Running multicast mode requires you to have an IP transport network that is multicast aware. Control plane protocol uses ISIS for MAC learning across the IP transport (Fig. 3.7).
 |
| Fig. 3.6 |
 |
| Fig. 3.7 |
- OTV components (Fig 3.8).
 |
| Fig 3.8 |
Configurations
version 16.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime show-timezone year
service timestamps log datetime localtime show-timezone year
service password-encryption
no platform punt-keepalive disable-kernel-core
platform console auto
!
hostname CE_R07
!
boot-start-marker
boot-end-marker
!
!
vrf definition TENANT_A
rd 65000:1
!
address-family ipv4
exit-address-family
!
vrf definition TENANT_B
rd 65000:2
!
address-family ipv4
exit-address-family
!
logging buffered 16000
no logging console
enable secret 5 <removed>
!
no aaa new-model
!
ip vrf MGMT
!
ip multicast-routing distributed
ip multicast-routing vrf TENANT_A distributed
ip multicast-routing vrf TENANT_B distributed
!
!
!
!
!
!
!
!
no ip domain lookup
ip domain name domain.local
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
otv site bridge-domain 7
!
otv site-identifier 0000.0000.0007
!
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
!
!
!
license udi pid CSR1000V sn 9BW82G58JFL
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
description RID
ip address 70.70.70.70 255.255.255.255
!
interface Overlay1
description OTV OVERLAY FOR TENANT_A
no ip address
otv control-group 225.0.0.1
otv data-group 232.0.0.0/8
otv join-interface GigabitEthernet1.3000
no otv suppress arp-nd
no mop enabled
no mop sysid
service instance 100 ethernet
encapsulation dot1q 100
bridge-domain 100
!
service instance 200 ethernet
encapsulation dot1q 200
bridge-domain 200
!
!
interface Overlay2
description OTV OVERLAY FOR TENANT_B
no ip address
otv control-group 225.0.0.2
otv data-group 233.0.0.0/8
otv join-interface GigabitEthernet1.3001
no otv suppress arp-nd
no mop enabled
no mop sysid
service instance 300 ethernet
encapsulation dot1q 300
bridge-domain 300
!
service instance 400 ethernet
encapsulation dot1q 400
bridge-domain 400
!
!
interface GigabitEthernet1
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet1.3000
description WAN TO PE_R13 - TENANT A VPLS
encapsulation dot1Q 3000
vrf forwarding TENANT_A
ip address 172.16.3.1 255.255.255.248
ip pim passive
ip igmp version 3
!
interface GigabitEthernet1.3001
description WAN TO PE_R13 - TENANT B VPLS
encapsulation dot1Q 3001
vrf forwarding TENANT_B
ip address 192.168.3.1 255.255.255.248
ip pim passive
ip igmp version 3
!
interface GigabitEthernet2
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet2.90
description TO VXLAN EVPN DC1_LEAF_03 - L3VNI TENANT A
encapsulation dot1Q 900
vrf forwarding TENANT_A
ip address 172.16.3.254 255.255.255.252
!
interface GigabitEthernet2.91
description TO VXLAN EVPN DC1_LEAF_03 - L3VNI TENANT B
encapsulation dot1Q 910
vrf forwarding TENANT_B
ip address 192.168.3.254 255.255.255.252
!
interface GigabitEthernet3
description OTV LAN - ALL TENANTS
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
service instance 7 ethernet
encapsulation untagged
snmp ifindex persist
bridge-domain 7
!
service instance 100 ethernet
encapsulation dot1q 100
bridge-domain 100
!
service instance 200 ethernet
encapsulation dot1q 200
snmp ifindex persist
bridge-domain 200
!
service instance 300 ethernet
encapsulation dot1q 300
bridge-domain 300
!
service instance 400 ethernet
encapsulation dot1q 400
snmp ifindex persist
bridge-domain 400
!
!
interface GigabitEthernet4
description MANAGEMENT
ip vrf forwarding MGMT
ip address 10.1.1.227 255.255.255.0
negotiation auto
no mop enabled
no mop sysid
!
router bgp 65000
bgp router-id 70.70.70.70
bgp log-neighbor-changes
!
address-family ipv4
exit-address-family
!
address-family ipv4 vrf TENANT_A
neighbor DC1_LEAF3_TENANT_A peer-group
neighbor DC1_LEAF3_TENANT_A remote-as 65500
neighbor DC1_LEAF3_TENANT_A description TO DC1 VXLAN TENANT A
neighbor DC1_LEAF3_TENANT_A timers 7 21
neighbor DC1_LEAF3_TENANT_A send-community both
neighbor DC2_CE9_TENANT_A peer-group
neighbor DC2_CE9_TENANT_A remote-as 65001
neighbor DC2_CE9_TENANT_A description TO DC2 VXLAN TENANT A
neighbor DC2_CE9_TENANT_A timers 7 21
neighbor DC2_CE9_TENANT_A send-community both
neighbor BRANCH_CE17_TENANT_A peer-group
neighbor BRANCH_CE17_TENANT_A remote-as 65100
neighbor BRANCH_CE17_TENANT_A description TO BRANCH CE17 TENANT A
neighbor BRANCH_CE17_TENANT_A timers 7 21
neighbor BRANCH_CE17_TENANT_A send-community both
neighbor 172.16.3.2 peer-group DC2_CE9_TENANT_A
neighbor 172.16.3.2 activate
neighbor 172.16.3.3 peer-group BRANCH_CE17_TENANT_A
neighbor 172.16.3.3 activate
neighbor 172.16.3.253 peer-group DC1_LEAF3_TENANT_A
neighbor 172.16.3.253 activate
exit-address-family
!
address-family ipv4 vrf TENANT_B
neighbor DC1_LEAF3_TENANT_B peer-group
neighbor DC1_LEAF3_TENANT_B remote-as 65500
neighbor DC1_LEAF3_TENANT_B description TO DC1 VXLAN TENANT B
neighbor DC1_LEAF3_TENANT_B timers 7 21
neighbor DC1_LEAF3_TENANT_B send-community both
neighbor DC2_CE9_TENANT_B peer-group
neighbor DC2_CE9_TENANT_B remote-as 65001
neighbor DC2_CE9_TENANT_B description TO DC2 VXLAN TENANT B
neighbor DC2_CE9_TENANT_B timers 7 21
neighbor DC2_CE9_TENANT_B send-community both
neighbor BRANCH_CE17_TENANT_B peer-group
neighbor BRANCH_CE17_TENANT_B remote-as 65100
neighbor BRANCH_CE17_TENANT_B description TO BRANCH CE17 TENANT B
neighbor BRANCH_CE17_TENANT_B timers 7 21
neighbor BRANCH_CE17_TENANT_B send-community both
neighbor 192.168.3.2 peer-group DC2_CE9_TENANT_B
neighbor 192.168.3.2 activate
neighbor 192.168.3.3 peer-group BRANCH_CE17_TENANT_B
neighbor 192.168.3.3 activate
neighbor 192.168.3.253 peer-group DC1_LEAF3_TENANT_B
neighbor 192.168.3.253 activate
exit-address-family
!
!
virtual-service csr_mgmt
!
no ip forward-protocol nd
ip pim ssm default
ip pim vrf TENANT_A ssm default
ip pim vrf TENANT_B ssm default
no ip http server
ip http authentication local
no ip http secure-server
!
ip bgp-community new-format
ip tftp source-interface GigabitEthernet4
ip route vrf MGMT 0.0.0.0 0.0.0.0 10.1.1.254 name MGMT_DEFAULT
!
!
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
line con 0
logging synchronous
stopbits 1
line vty 0 4
exec-timeout 60 0
logging synchronous
login local
transport input telnet ssh
!
ntp server pool.ntp.org
!
!
!
!
!
end
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime show-timezone year
service timestamps log datetime localtime show-timezone year
service password-encryption
no platform punt-keepalive disable-kernel-core
platform console auto
!
hostname CE_R07
!
boot-start-marker
boot-end-marker
!
!
vrf definition TENANT_A
rd 65000:1
!
address-family ipv4
exit-address-family
!
vrf definition TENANT_B
rd 65000:2
!
address-family ipv4
exit-address-family
!
logging buffered 16000
no logging console
enable secret 5 <removed>
!
no aaa new-model
!
ip vrf MGMT
!
ip multicast-routing distributed
ip multicast-routing vrf TENANT_A distributed
ip multicast-routing vrf TENANT_B distributed
!
!
!
!
!
!
!
!
no ip domain lookup
ip domain name domain.local
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
otv site bridge-domain 7
!
otv site-identifier 0000.0000.0007
!
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
!
!
!
license udi pid CSR1000V sn 9BW82G58JFL
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
description RID
ip address 70.70.70.70 255.255.255.255
!
interface Overlay1
description OTV OVERLAY FOR TENANT_A
no ip address
otv control-group 225.0.0.1
otv data-group 232.0.0.0/8
otv join-interface GigabitEthernet1.3000
no otv suppress arp-nd
no mop enabled
no mop sysid
service instance 100 ethernet
encapsulation dot1q 100
bridge-domain 100
!
service instance 200 ethernet
encapsulation dot1q 200
bridge-domain 200
!
!
interface Overlay2
description OTV OVERLAY FOR TENANT_B
no ip address
otv control-group 225.0.0.2
otv data-group 233.0.0.0/8
otv join-interface GigabitEthernet1.3001
no otv suppress arp-nd
no mop enabled
no mop sysid
service instance 300 ethernet
encapsulation dot1q 300
bridge-domain 300
!
service instance 400 ethernet
encapsulation dot1q 400
bridge-domain 400
!
!
interface GigabitEthernet1
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet1.3000
description WAN TO PE_R13 - TENANT A VPLS
encapsulation dot1Q 3000
vrf forwarding TENANT_A
ip address 172.16.3.1 255.255.255.248
ip pim passive
ip igmp version 3
!
interface GigabitEthernet1.3001
description WAN TO PE_R13 - TENANT B VPLS
encapsulation dot1Q 3001
vrf forwarding TENANT_B
ip address 192.168.3.1 255.255.255.248
ip pim passive
ip igmp version 3
!
interface GigabitEthernet2
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet2.90
description TO VXLAN EVPN DC1_LEAF_03 - L3VNI TENANT A
encapsulation dot1Q 900
vrf forwarding TENANT_A
ip address 172.16.3.254 255.255.255.252
!
interface GigabitEthernet2.91
description TO VXLAN EVPN DC1_LEAF_03 - L3VNI TENANT B
encapsulation dot1Q 910
vrf forwarding TENANT_B
ip address 192.168.3.254 255.255.255.252
!
interface GigabitEthernet3
description OTV LAN - ALL TENANTS
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
service instance 7 ethernet
encapsulation untagged
snmp ifindex persist
bridge-domain 7
!
service instance 100 ethernet
encapsulation dot1q 100
bridge-domain 100
!
service instance 200 ethernet
encapsulation dot1q 200
snmp ifindex persist
bridge-domain 200
!
service instance 300 ethernet
encapsulation dot1q 300
bridge-domain 300
!
service instance 400 ethernet
encapsulation dot1q 400
snmp ifindex persist
bridge-domain 400
!
!
interface GigabitEthernet4
description MANAGEMENT
ip vrf forwarding MGMT
ip address 10.1.1.227 255.255.255.0
negotiation auto
no mop enabled
no mop sysid
!
router bgp 65000
bgp router-id 70.70.70.70
bgp log-neighbor-changes
!
address-family ipv4
exit-address-family
!
address-family ipv4 vrf TENANT_A
neighbor DC1_LEAF3_TENANT_A peer-group
neighbor DC1_LEAF3_TENANT_A remote-as 65500
neighbor DC1_LEAF3_TENANT_A description TO DC1 VXLAN TENANT A
neighbor DC1_LEAF3_TENANT_A timers 7 21
neighbor DC1_LEAF3_TENANT_A send-community both
neighbor DC2_CE9_TENANT_A peer-group
neighbor DC2_CE9_TENANT_A remote-as 65001
neighbor DC2_CE9_TENANT_A description TO DC2 VXLAN TENANT A
neighbor DC2_CE9_TENANT_A timers 7 21
neighbor DC2_CE9_TENANT_A send-community both
neighbor BRANCH_CE17_TENANT_A peer-group
neighbor BRANCH_CE17_TENANT_A remote-as 65100
neighbor BRANCH_CE17_TENANT_A description TO BRANCH CE17 TENANT A
neighbor BRANCH_CE17_TENANT_A timers 7 21
neighbor BRANCH_CE17_TENANT_A send-community both
neighbor 172.16.3.2 peer-group DC2_CE9_TENANT_A
neighbor 172.16.3.2 activate
neighbor 172.16.3.3 peer-group BRANCH_CE17_TENANT_A
neighbor 172.16.3.3 activate
neighbor 172.16.3.253 peer-group DC1_LEAF3_TENANT_A
neighbor 172.16.3.253 activate
exit-address-family
!
address-family ipv4 vrf TENANT_B
neighbor DC1_LEAF3_TENANT_B peer-group
neighbor DC1_LEAF3_TENANT_B remote-as 65500
neighbor DC1_LEAF3_TENANT_B description TO DC1 VXLAN TENANT B
neighbor DC1_LEAF3_TENANT_B timers 7 21
neighbor DC1_LEAF3_TENANT_B send-community both
neighbor DC2_CE9_TENANT_B peer-group
neighbor DC2_CE9_TENANT_B remote-as 65001
neighbor DC2_CE9_TENANT_B description TO DC2 VXLAN TENANT B
neighbor DC2_CE9_TENANT_B timers 7 21
neighbor DC2_CE9_TENANT_B send-community both
neighbor BRANCH_CE17_TENANT_B peer-group
neighbor BRANCH_CE17_TENANT_B remote-as 65100
neighbor BRANCH_CE17_TENANT_B description TO BRANCH CE17 TENANT B
neighbor BRANCH_CE17_TENANT_B timers 7 21
neighbor BRANCH_CE17_TENANT_B send-community both
neighbor 192.168.3.2 peer-group DC2_CE9_TENANT_B
neighbor 192.168.3.2 activate
neighbor 192.168.3.3 peer-group BRANCH_CE17_TENANT_B
neighbor 192.168.3.3 activate
neighbor 192.168.3.253 peer-group DC1_LEAF3_TENANT_B
neighbor 192.168.3.253 activate
exit-address-family
!
!
virtual-service csr_mgmt
!
no ip forward-protocol nd
ip pim ssm default
ip pim vrf TENANT_A ssm default
ip pim vrf TENANT_B ssm default
no ip http server
ip http authentication local
no ip http secure-server
!
ip bgp-community new-format
ip tftp source-interface GigabitEthernet4
ip route vrf MGMT 0.0.0.0 0.0.0.0 10.1.1.254 name MGMT_DEFAULT
!
!
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
line con 0
logging synchronous
stopbits 1
line vty 0 4
exec-timeout 60 0
logging synchronous
login local
transport input telnet ssh
!
ntp server pool.ntp.org
!
!
!
!
!
end
version 7.0(3)I6(1)
hostname DC1_LEAF_03
vdc DC1_LEAF_03 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
vlan 1,100,200,300,400,900,910
ip pim rp-address 100.100.100.100 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 100
vn-segment 10000
vlan 200
vn-segment 20000
vlan 300
vn-segment 30000
vlan 400
vn-segment 40000
vlan 900
vn-segment 90000
vlan 910
vn-segment 91000
vrf context TENANT_A
vni 90000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context TENANT_B
vni 91000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Vlan1
interface Vlan900
description TENANT_A VL900 L3VNI
no shutdown
mtu 9216
vrf member TENANT_A
ip forward
interface Vlan910
description TENANT_B VL910 L3VNI
no shutdown
mtu 9216
vrf member TENANT_B
ip forward
interface nve1
no shutdown
description VTEP_03
source-interface loopback0
host-reachability protocol bgp
member vni 10000
suppress-arp
mcast-group 225.1.0.1
member vni 20000
suppress-arp
mcast-group 225.2.0.1
member vni 30000
suppress-arp
mcast-group 225.3.0.1
member vni 40000
suppress-arp
mcast-group 225.4.0.1
member vni 90000 associate-vrf
member vni 91000 associate-vrf
interface Ethernet1/1
description TO SPINE_01
no switchport
mtu 9216
ip address 10.0.0.18/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO SPINE_02
no switchport
mtu 9216
ip address 10.0.0.22/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
no switchport
mtu 9216
interface Ethernet1/4
description TO CE_R07 - OTV LAN
switchport mode trunk
switchport trunk allowed vlan 100,200,300,400
mtu 9216
interface Ethernet1/5
description TO CE7 - EXT L3
no switchport
mtu 9216
no shutdown
interface Ethernet1/5.90
description TO CE7 - TENANT A L3VNI EXT L3
mtu 9216
encapsulation dot1q 900
vrf member TENANT_A
ip address 172.16.3.253/30
no shutdown
interface Ethernet1/5.91
description TO CE7 - TENANT B L3VNI EXT L3
mtu 9216
encapsulation dot1q 910
vrf member TENANT_B
ip address 192.168.3.253/30
no shutdown
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.214/24
interface loopback0
description RID - VTEP
ip address 5.5.5.5/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 100
router-id 5.5.5.5
passive-interface default
router bgp 65500
router-id 5.5.5.5
neighbor 1.1.1.1
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 2.2.2.2
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf TENANT_A
timers bgp 7 21
address-family ipv4 unicast
network 172.16.1.0/24
advertise l2vpn evpn
neighbor 172.16.3.254
remote-as 65000
address-family ipv4 unicast
send-community
send-community extended
vrf TENANT_B
timers bgp 7 21
address-family ipv4 unicast
advertise l2vpn evpn
neighbor 192.168.3.254
remote-as 65000
address-family ipv4 unicast
send-community
send-community extended
evpn
vni 10000 l2
rd auto
route-target import auto
route-target export auto
vni 20000 l2
rd auto
route-target import auto
route-target export auto
vni 30000 l2
rd auto
route-target import auto
route-target export auto
vni 40000 l2
rd auto
route-target import auto
route-target export auto
hostname DC1_LEAF_03
vdc DC1_LEAF_03 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
vlan 1,100,200,300,400,900,910
ip pim rp-address 100.100.100.100 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 100
vn-segment 10000
vlan 200
vn-segment 20000
vlan 300
vn-segment 30000
vlan 400
vn-segment 40000
vlan 900
vn-segment 90000
vlan 910
vn-segment 91000
vrf context TENANT_A
vni 90000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context TENANT_B
vni 91000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Vlan1
interface Vlan900
description TENANT_A VL900 L3VNI
no shutdown
mtu 9216
vrf member TENANT_A
ip forward
interface Vlan910
description TENANT_B VL910 L3VNI
no shutdown
mtu 9216
vrf member TENANT_B
ip forward
interface nve1
no shutdown
description VTEP_03
source-interface loopback0
host-reachability protocol bgp
member vni 10000
suppress-arp
mcast-group 225.1.0.1
member vni 20000
suppress-arp
mcast-group 225.2.0.1
member vni 30000
suppress-arp
mcast-group 225.3.0.1
member vni 40000
suppress-arp
mcast-group 225.4.0.1
member vni 90000 associate-vrf
member vni 91000 associate-vrf
interface Ethernet1/1
description TO SPINE_01
no switchport
mtu 9216
ip address 10.0.0.18/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO SPINE_02
no switchport
mtu 9216
ip address 10.0.0.22/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
no switchport
mtu 9216
interface Ethernet1/4
description TO CE_R07 - OTV LAN
switchport mode trunk
switchport trunk allowed vlan 100,200,300,400
mtu 9216
interface Ethernet1/5
description TO CE7 - EXT L3
no switchport
mtu 9216
no shutdown
interface Ethernet1/5.90
description TO CE7 - TENANT A L3VNI EXT L3
mtu 9216
encapsulation dot1q 900
vrf member TENANT_A
ip address 172.16.3.253/30
no shutdown
interface Ethernet1/5.91
description TO CE7 - TENANT B L3VNI EXT L3
mtu 9216
encapsulation dot1q 910
vrf member TENANT_B
ip address 192.168.3.253/30
no shutdown
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.214/24
interface loopback0
description RID - VTEP
ip address 5.5.5.5/32
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 100
router-id 5.5.5.5
passive-interface default
router bgp 65500
router-id 5.5.5.5
neighbor 1.1.1.1
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 2.2.2.2
remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf TENANT_A
timers bgp 7 21
address-family ipv4 unicast
network 172.16.1.0/24
advertise l2vpn evpn
neighbor 172.16.3.254
remote-as 65000
address-family ipv4 unicast
send-community
send-community extended
vrf TENANT_B
timers bgp 7 21
address-family ipv4 unicast
advertise l2vpn evpn
neighbor 192.168.3.254
remote-as 65000
address-family ipv4 unicast
send-community
send-community extended
evpn
vni 10000 l2
rd auto
route-target import auto
route-target export auto
vni 20000 l2
rd auto
route-target import auto
route-target export auto
vni 30000 l2
rd auto
route-target import auto
route-target export auto
vni 40000 l2
rd auto
route-target import auto
route-target export auto
version 16.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime show-timezone year
service timestamps log datetime localtime show-timezone year
service password-encryption
no platform punt-keepalive disable-kernel-core
platform console auto
!
hostname CE_R09
!
boot-start-marker
boot-end-marker
!
!
vrf definition TENANT_A
rd 65001:1
!
address-family ipv4
exit-address-family
!
vrf definition TENANT_B
rd 65001:2
!
address-family ipv4
exit-address-family
!
logging buffered 16000
no logging console
enable secret 5 <removed>
!
no aaa new-model
!
!
ip vrf MGMT
!
ip multicast-routing distributed
ip multicast-routing vrf TENANT_A distributed
ip multicast-routing vrf TENANT_B distributed
!
!
!
!
!
!
!
!
ip name-server 8.8.8.8
!
no ip domain lookup
ip domain name domain.local
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
otv site bridge-domain 9
!
otv site-identifier 0000.0000.0009
!
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
!
!
!
license udi pid CSR1000V sn 9YQL5RHXFT5
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
description RID
ip address 90.90.90.90 255.255.255.255
!
interface Overlay1
description OTV OVERLAY
no ip address
otv control-group 225.0.0.1
otv data-group 232.0.0.0/8
otv join-interface GigabitEthernet1.3000
no otv suppress arp-nd
no mop enabled
no mop sysid
service instance 100 ethernet
encapsulation dot1q 100
bridge-domain 100
!
service instance 200 ethernet
encapsulation dot1q 200
bridge-domain 200
!
!
interface Overlay2
description OTV OVERLAY FOR TENANT_B
no ip address
otv control-group 225.0.0.2
otv data-group 233.0.0.0/8
otv join-interface GigabitEthernet1.3001
no otv suppress arp-nd
no mop enabled
no mop sysid
service instance 300 ethernet
encapsulation dot1q 300
bridge-domain 300
!
service instance 400 ethernet
encapsulation dot1q 400
bridge-domain 400
!
!
interface GigabitEthernet1
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet1.3000
description WAN TO PE_R15 - TENANT A VPLS
encapsulation dot1Q 3000
vrf forwarding TENANT_A
ip address 172.16.3.2 255.255.255.248
ip pim passive
ip igmp version 3
!
interface GigabitEthernet1.3001
description WAN TO PE_R15 - TENANT B VPLS
encapsulation dot1Q 3001
vrf forwarding TENANT_B
ip address 192.168.3.2 255.255.255.248
ip pim passive
ip igmp version 3
!
interface GigabitEthernet2
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet2.91
description TO VXLAN EVPN DC2_LEAF_03 - L3VNI TENANT B
encapsulation dot1Q 910
vrf forwarding TENANT_B
ip address 192.168.3.250 255.255.255.252
!
interface GigabitEthernet2.900
description TO VXLAN EVPN DC2_LEAF_03 - L3VNI TENANT A
encapsulation dot1Q 900
vrf forwarding TENANT_A
ip address 172.16.3.250 255.255.255.252
!
interface GigabitEthernet3
description OTV LAN - ALL TENANTS
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
service instance 9 ethernet
encapsulation untagged
snmp ifindex persist
bridge-domain 9
!
service instance 100 ethernet
encapsulation dot1q 100
snmp ifindex persist
bridge-domain 100
!
service instance 200 ethernet
encapsulation dot1q 200
snmp ifindex persist
bridge-domain 200
!
service instance 300 ethernet
encapsulation dot1q 300
bridge-domain 300
!
service instance 400 ethernet
encapsulation dot1q 400
snmp ifindex persist
bridge-domain 400
!
!
interface GigabitEthernet4
description MANAGEMENT
ip vrf forwarding MGMT
ip address 10.1.1.229 255.255.255.0
negotiation auto
no mop enabled
no mop sysid
!
!
router bgp 65001
bgp router-id 90.90.90.90
bgp log-neighbor-changes
!
address-family ipv4
exit-address-family
!
address-family ipv4 vrf TENANT_A
neighbor DC2_LEAF3_TENANT_A peer-group
neighbor DC2_LEAF3_TENANT_A remote-as 65501
neighbor DC2_LEAF3_TENANT_A description TO DC2 VXLAN TENANT A
neighbor DC2_LEAF3_TENANT_A timers 7 21
neighbor DC2_LEAF3_TENANT_A send-community both
neighbor DC1_CE7_TENANT_A peer-group
neighbor DC1_CE7_TENANT_A remote-as 65000
neighbor DC1_CE7_TENANT_A description TO DC1 VXLAN TENANT A
neighbor DC1_CE7_TENANT_A timers 7 21
neighbor DC1_CE7_TENANT_A send-community both
neighbor BRANCH_CE17_TENANT_A peer-group
neighbor BRANCH_CE17_TENANT_A remote-as 65100
neighbor BRANCH_CE17_TENANT_A description TO BRANCH CE17 TENANT A
neighbor BRANCH_CE17_TENANT_A timers 7 21
neighbor BRANCH_CE17_TENANT_A send-community both
neighbor 172.16.3.1 peer-group DC1_CE7_TENANT_A
neighbor 172.16.3.1 activate
neighbor 172.16.3.3 peer-group BRANCH_CE17_TENANT_A
neighbor 172.16.3.3 activate
neighbor 172.16.3.249 peer-group DC2_LEAF3_TENANT_A
neighbor 172.16.3.249 activate
exit-address-family
!
address-family ipv4 vrf TENANT_B
network 0.0.0.0
neighbor DC2_LEAF3_TENANT_B peer-group
neighbor DC2_LEAF3_TENANT_B remote-as 65501
neighbor DC2_LEAF3_TENANT_B description TO DC2 VXLAN TENANT B
neighbor DC2_LEAF3_TENANT_B timers 7 21
neighbor DC2_LEAF3_TENANT_B send-community both
neighbor DC1_CE7_TENANT_B peer-group
neighbor DC1_CE7_TENANT_B remote-as 65000
neighbor DC1_CE7_TENANT_B description TO DC1 VXLAN TENANT B
neighbor DC1_CE7_TENANT_B timers 7 21
neighbor DC1_CE7_TENANT_B send-community both
neighbor BRANCH_CE17_TENANT_B peer-group
neighbor BRANCH_CE17_TENANT_B remote-as 65100
neighbor BRANCH_CE17_TENANT_B description TO BRANCH CE17 TENANT B
neighbor BRANCH_CE17_TENANT_B timers 7 21
neighbor BRANCH_CE17_TENANT_B send-community both
neighbor 192.168.3.1 peer-group DC1_CE7_TENANT_B
neighbor 192.168.3.1 activate
neighbor 192.168.3.3 peer-group BRANCH_CE17_TENANT_B
neighbor 192.168.3.3 activate
neighbor 192.168.3.249 peer-group DC2_LEAF3_TENANT_B
neighbor 192.168.3.249 activate
exit-address-family
!
!
virtual-service csr_mgmt
!
no ip forward-protocol nd
ip pim ssm default
ip pim vrf TENANT_A ssm default
ip pim vrf TENANT_B ssm default
no ip http server
ip http authentication local
no ip http secure-server
!
ip bgp-community new-format
ip tftp source-interface GigabitEthernet4
ip route vrf MGMT 0.0.0.0 0.0.0.0 10.1.1.254 name MGMT_DEFAULT
!
!
!
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
line con 0
logging synchronous
stopbits 1
line vty 0 4
exec-timeout 60 0
logging synchronous
login local
transport input telnet ssh
!
ntp server pool.ntp.org
!
!
!
!
!
end
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime show-timezone year
service timestamps log datetime localtime show-timezone year
service password-encryption
no platform punt-keepalive disable-kernel-core
platform console auto
!
hostname CE_R09
!
boot-start-marker
boot-end-marker
!
!
vrf definition TENANT_A
rd 65001:1
!
address-family ipv4
exit-address-family
!
vrf definition TENANT_B
rd 65001:2
!
address-family ipv4
exit-address-family
!
logging buffered 16000
no logging console
enable secret 5 <removed>
!
no aaa new-model
!
!
ip vrf MGMT
!
ip multicast-routing distributed
ip multicast-routing vrf TENANT_A distributed
ip multicast-routing vrf TENANT_B distributed
!
!
!
!
!
!
!
!
ip name-server 8.8.8.8
!
no ip domain lookup
ip domain name domain.local
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
otv site bridge-domain 9
!
otv site-identifier 0000.0000.0009
!
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
!
!
!
license udi pid CSR1000V sn 9YQL5RHXFT5
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
description RID
ip address 90.90.90.90 255.255.255.255
!
interface Overlay1
description OTV OVERLAY
no ip address
otv control-group 225.0.0.1
otv data-group 232.0.0.0/8
otv join-interface GigabitEthernet1.3000
no otv suppress arp-nd
no mop enabled
no mop sysid
service instance 100 ethernet
encapsulation dot1q 100
bridge-domain 100
!
service instance 200 ethernet
encapsulation dot1q 200
bridge-domain 200
!
!
interface Overlay2
description OTV OVERLAY FOR TENANT_B
no ip address
otv control-group 225.0.0.2
otv data-group 233.0.0.0/8
otv join-interface GigabitEthernet1.3001
no otv suppress arp-nd
no mop enabled
no mop sysid
service instance 300 ethernet
encapsulation dot1q 300
bridge-domain 300
!
service instance 400 ethernet
encapsulation dot1q 400
bridge-domain 400
!
!
interface GigabitEthernet1
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet1.3000
description WAN TO PE_R15 - TENANT A VPLS
encapsulation dot1Q 3000
vrf forwarding TENANT_A
ip address 172.16.3.2 255.255.255.248
ip pim passive
ip igmp version 3
!
interface GigabitEthernet1.3001
description WAN TO PE_R15 - TENANT B VPLS
encapsulation dot1Q 3001
vrf forwarding TENANT_B
ip address 192.168.3.2 255.255.255.248
ip pim passive
ip igmp version 3
!
interface GigabitEthernet2
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet2.91
description TO VXLAN EVPN DC2_LEAF_03 - L3VNI TENANT B
encapsulation dot1Q 910
vrf forwarding TENANT_B
ip address 192.168.3.250 255.255.255.252
!
interface GigabitEthernet2.900
description TO VXLAN EVPN DC2_LEAF_03 - L3VNI TENANT A
encapsulation dot1Q 900
vrf forwarding TENANT_A
ip address 172.16.3.250 255.255.255.252
!
interface GigabitEthernet3
description OTV LAN - ALL TENANTS
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
service instance 9 ethernet
encapsulation untagged
snmp ifindex persist
bridge-domain 9
!
service instance 100 ethernet
encapsulation dot1q 100
snmp ifindex persist
bridge-domain 100
!
service instance 200 ethernet
encapsulation dot1q 200
snmp ifindex persist
bridge-domain 200
!
service instance 300 ethernet
encapsulation dot1q 300
bridge-domain 300
!
service instance 400 ethernet
encapsulation dot1q 400
snmp ifindex persist
bridge-domain 400
!
!
interface GigabitEthernet4
description MANAGEMENT
ip vrf forwarding MGMT
ip address 10.1.1.229 255.255.255.0
negotiation auto
no mop enabled
no mop sysid
!
!
router bgp 65001
bgp router-id 90.90.90.90
bgp log-neighbor-changes
!
address-family ipv4
exit-address-family
!
address-family ipv4 vrf TENANT_A
neighbor DC2_LEAF3_TENANT_A peer-group
neighbor DC2_LEAF3_TENANT_A remote-as 65501
neighbor DC2_LEAF3_TENANT_A description TO DC2 VXLAN TENANT A
neighbor DC2_LEAF3_TENANT_A timers 7 21
neighbor DC2_LEAF3_TENANT_A send-community both
neighbor DC1_CE7_TENANT_A peer-group
neighbor DC1_CE7_TENANT_A remote-as 65000
neighbor DC1_CE7_TENANT_A description TO DC1 VXLAN TENANT A
neighbor DC1_CE7_TENANT_A timers 7 21
neighbor DC1_CE7_TENANT_A send-community both
neighbor BRANCH_CE17_TENANT_A peer-group
neighbor BRANCH_CE17_TENANT_A remote-as 65100
neighbor BRANCH_CE17_TENANT_A description TO BRANCH CE17 TENANT A
neighbor BRANCH_CE17_TENANT_A timers 7 21
neighbor BRANCH_CE17_TENANT_A send-community both
neighbor 172.16.3.1 peer-group DC1_CE7_TENANT_A
neighbor 172.16.3.1 activate
neighbor 172.16.3.3 peer-group BRANCH_CE17_TENANT_A
neighbor 172.16.3.3 activate
neighbor 172.16.3.249 peer-group DC2_LEAF3_TENANT_A
neighbor 172.16.3.249 activate
exit-address-family
!
address-family ipv4 vrf TENANT_B
network 0.0.0.0
neighbor DC2_LEAF3_TENANT_B peer-group
neighbor DC2_LEAF3_TENANT_B remote-as 65501
neighbor DC2_LEAF3_TENANT_B description TO DC2 VXLAN TENANT B
neighbor DC2_LEAF3_TENANT_B timers 7 21
neighbor DC2_LEAF3_TENANT_B send-community both
neighbor DC1_CE7_TENANT_B peer-group
neighbor DC1_CE7_TENANT_B remote-as 65000
neighbor DC1_CE7_TENANT_B description TO DC1 VXLAN TENANT B
neighbor DC1_CE7_TENANT_B timers 7 21
neighbor DC1_CE7_TENANT_B send-community both
neighbor BRANCH_CE17_TENANT_B peer-group
neighbor BRANCH_CE17_TENANT_B remote-as 65100
neighbor BRANCH_CE17_TENANT_B description TO BRANCH CE17 TENANT B
neighbor BRANCH_CE17_TENANT_B timers 7 21
neighbor BRANCH_CE17_TENANT_B send-community both
neighbor 192.168.3.1 peer-group DC1_CE7_TENANT_B
neighbor 192.168.3.1 activate
neighbor 192.168.3.3 peer-group BRANCH_CE17_TENANT_B
neighbor 192.168.3.3 activate
neighbor 192.168.3.249 peer-group DC2_LEAF3_TENANT_B
neighbor 192.168.3.249 activate
exit-address-family
!
!
virtual-service csr_mgmt
!
no ip forward-protocol nd
ip pim ssm default
ip pim vrf TENANT_A ssm default
ip pim vrf TENANT_B ssm default
no ip http server
ip http authentication local
no ip http secure-server
!
ip bgp-community new-format
ip tftp source-interface GigabitEthernet4
ip route vrf MGMT 0.0.0.0 0.0.0.0 10.1.1.254 name MGMT_DEFAULT
!
!
!
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
line con 0
logging synchronous
stopbits 1
line vty 0 4
exec-timeout 60 0
logging synchronous
login local
transport input telnet ssh
!
ntp server pool.ntp.org
!
!
!
!
!
end
version 7.0(3)I6(1)
hostname DC2_LEAF_03
vdc DC2_LEAF_03 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
vlan 1,100,200,300,400,900,910
ip pim rp-address 101.101.101.101 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 100
vn-segment 10000
vlan 200
vn-segment 20000
vlan 300
vn-segment 30000
vlan 400
vn-segment 40000
vlan 900
vn-segment 90000
vlan 910
vn-segment 91000
vrf context TENANT_A
vni 90000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context TENANT_B
vni 91000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Vlan1
interface Vlan900
description TENANT_A VL900 L3VNI
no shutdown
mtu 9216
vrf member TENANT_A
ip forward
interface Vlan910
description TENANT_B VL910 L3VNI
no shutdown
mtu 9216
vrf member TENANT_B
ip forward
interface nve1
no shutdown
description VTEP_03
source-interface loopback0
host-reachability protocol bgp
member vni 10000
suppress-arp
mcast-group 225.1.0.1
member vni 20000
suppress-arp
mcast-group 225.2.0.1
member vni 30000
suppress-arp
mcast-group 225.3.0.1
member vni 40000
suppress-arp
mcast-group 225.4.0.1
member vni 90000 associate-vrf
member vni 91000 associate-vrf
interface Ethernet1/1
description TO SPINE_01
no switchport
mtu 9216
ip address 10.1.0.18/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 101 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO SPINE_02
no switchport
mtu 9216
ip address 10.1.0.22/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 101 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
shutdown
mtu 9216
interface Ethernet1/4
description TO CE_R09 - OTV LAN
switchport mode trunk
switchport trunk allowed vlan 100,200,300,400
mtu 9216
interface Ethernet1/5
description TO CE9 - EXT L3
no switchport
mtu 9216
no shutdown
interface Ethernet1/5.90
description TO CE9 - TENANT A L3VNI EXT L3
mtu 9216
encapsulation dot1q 900
vrf member TENANT_A
ip address 172.16.3.249/30
no shutdown
interface Ethernet1/5.91
description TO CE9 - TENANT B L3VNI EXT L3
mtu 9216
encapsulation dot1q 910
vrf member TENANT_B
ip address 192.168.3.249/30
no shutdown
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.219/24
interface loopback0
description RID - VTEP
ip address 10.10.10.10/32
ip router ospf 101 area 0.0.0.0
ip pim sparse-mode
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 101
router-id 10.10.10.10
passive-interface default
router bgp 65501
router-id 10.10.10.10
neighbor 6.6.6.6
remote-as 65501
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 7.7.7.7
remote-as 65501
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf TENANT_A
timers bgp 7 21
address-family ipv4 unicast
network 172.16.1.0/24
advertise l2vpn evpn
neighbor 172.16.3.250
remote-as 65001
address-family ipv4 unicast
send-community
send-community extended
vrf TENANT_B
address-family ipv4 unicast
advertise l2vpn evpn
neighbor 192.168.3.250
remote-as 65001
address-family ipv4 unicast
send-community
send-community extended
evpn
vni 10000 l2
rd auto
route-target import auto
route-target export auto
vni 20000 l2
rd auto
route-target import auto
route-target export auto
vni 30000 l2
rd auto
route-target import auto
route-target export auto
vni 40000 l2
rd auto
route-target import auto
route-target export auto
hostname DC2_LEAF_03
vdc DC2_LEAF_03 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
no password strength-check
username admin password 5 <removed> role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
vlan 1,100,200,300,400,900,910
ip pim rp-address 101.101.101.101 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 100
vn-segment 10000
vlan 200
vn-segment 20000
vlan 300
vn-segment 30000
vlan 400
vn-segment 40000
vlan 900
vn-segment 90000
vlan 910
vn-segment 91000
vrf context TENANT_A
vni 90000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context TENANT_B
vni 91000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT
interface Vlan1
interface Vlan900
description TENANT_A VL900 L3VNI
no shutdown
mtu 9216
vrf member TENANT_A
ip forward
interface Vlan910
description TENANT_B VL910 L3VNI
no shutdown
mtu 9216
vrf member TENANT_B
ip forward
interface nve1
no shutdown
description VTEP_03
source-interface loopback0
host-reachability protocol bgp
member vni 10000
suppress-arp
mcast-group 225.1.0.1
member vni 20000
suppress-arp
mcast-group 225.2.0.1
member vni 30000
suppress-arp
mcast-group 225.3.0.1
member vni 40000
suppress-arp
mcast-group 225.4.0.1
member vni 90000 associate-vrf
member vni 91000 associate-vrf
interface Ethernet1/1
description TO SPINE_01
no switchport
mtu 9216
ip address 10.1.0.18/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 101 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
description TO SPINE_02
no switchport
mtu 9216
ip address 10.1.0.22/30
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 101 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
shutdown
mtu 9216
interface Ethernet1/4
description TO CE_R09 - OTV LAN
switchport mode trunk
switchport trunk allowed vlan 100,200,300,400
mtu 9216
interface Ethernet1/5
description TO CE9 - EXT L3
no switchport
mtu 9216
no shutdown
interface Ethernet1/5.90
description TO CE9 - TENANT A L3VNI EXT L3
mtu 9216
encapsulation dot1q 900
vrf member TENANT_A
ip address 172.16.3.249/30
no shutdown
interface Ethernet1/5.91
description TO CE9 - TENANT B L3VNI EXT L3
mtu 9216
encapsulation dot1q 910
vrf member TENANT_B
ip address 192.168.3.249/30
no shutdown
interface mgmt0
description MANAGEMENT
vrf member management
ip address 10.1.1.219/24
interface loopback0
description RID - VTEP
ip address 10.10.10.10/32
ip router ospf 101 area 0.0.0.0
ip pim sparse-mode
line console
exec-timeout 60
line vty
exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin
router ospf 101
router-id 10.10.10.10
passive-interface default
router bgp 65501
router-id 10.10.10.10
neighbor 6.6.6.6
remote-as 65501
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 7.7.7.7
remote-as 65501
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf TENANT_A
timers bgp 7 21
address-family ipv4 unicast
network 172.16.1.0/24
advertise l2vpn evpn
neighbor 172.16.3.250
remote-as 65001
address-family ipv4 unicast
send-community
send-community extended
vrf TENANT_B
address-family ipv4 unicast
advertise l2vpn evpn
neighbor 192.168.3.250
remote-as 65001
address-family ipv4 unicast
send-community
send-community extended
evpn
vni 10000 l2
rd auto
route-target import auto
route-target export auto
vni 20000 l2
rd auto
route-target import auto
route-target export auto
vni 30000 l2
rd auto
route-target import auto
route-target export auto
vni 40000 l2
rd auto
route-target import auto
route-target export auto
OTV Lab Specific Configurations
Enable multicast on OTV edge device (if using Multicast mode).
ip
multicast-routing vrf TENANT_A distributed
!
ip pim ssm
default
ip pim vrf TENANT_A
ssm default
|
Define the OTV site identifier and bridge domain.
otv site
bridge-domain 7
!
otv
site-identifier 0000.0000.0007
|
Enable multicast on OTV join interface.
interface
GigabitEthernet1.3000
description WAN TO PE_R13 - TENANT A VPLS
encapsulation dot1Q 3000
vrf forwarding TENANT_A
ip address 172.16.3.1 255.255.255.248
ip pim passive
ip igmp version 3
|
On the OTV internal interface, create the service instance for
both the site ID
and extended VLANs.
interface
GigabitEthernet3
description OTV LAN - ALL TENANTS
mtu 9216
no ip address
negotiation auto
no mop enabled
no mop sysid
service instance 7 ethernet
encapsulation untagged
bridge-domain 7
!
service instance 100 ethernet
encapsulation dot1q 100
bridge-domain 100
!
service instance 200 ethernet
encapsulation dot1q 200
bridge-domain 200
|
Create the Overlay interface with multicast enabled. Configure the join interface and the service instances
for the extended VLANs.
interface
Overlay1
description OTV OVERLAY FOR TENANT_A
no ip address
otv control-group 225.0.0.1
otv data-group 232.0.0.0/8
otv join-interface GigabitEthernet1.3000
no otv suppress arp-nd
no mop enabled
no mop sysid
service instance 100 ethernet
encapsulation dot1q 100
bridge-domain 100
!
service instance 200 ethernet
encapsulation dot1q 200
bridge-domain 200
|
Testing
Results
As proof of concept, testing was performed from within a
single tenant only (TENANT_A). The test
protocol was for Host4 in DC1 to ping and trace hosts in its own VXLAN fabric as well as hosts in the remote DC2 fabric.
- Host4's IP and MAC Address.
 |
| Fig. 4.1 |
- Host4’s ping results to all members of DC1 and DC2.
 |
| Fig. 4.2 |
- Host4 ping to branch site.
 |
| Fig. 4.3 |
- Arp cache for Host4.
 |
| Fig. 4.4 |
- Host4 traceroutes to hosts in DC1.
 |
| Fig. 4.5 |
- Host4 traceroutes to hosts in DC2.
 |
| Fig. 4.6 |
Verification
Tasks
The list below outlined the OTV verification tasks for only
TENANT_A.
- OTV Overlay Interface Status
- OTV Adjacency
- OTV Control Plane Protocol Status
- OTV Multicast
- OTV VLANs
- OTV Routes
- OTV Bridge-Domain
OTV Overlay Interface Status
CE_R07#sh otv
Overlay Interface
Overlay1
VPN name : None
VPN ID : 1
State : UP
Fwd-capable : Yes
Fwd-ready : Yes
AED-Server : Yes
Backup AED-Server : No
AED Capable : Yes
IPv4 control group : 225.0.0.1
Mcast data group range(s): 232.0.0.0/8
Join interface(s) : GigabitEthernet1.3000
Join IPv4 address : 172.16.3.1
Tunnel interface(s) : Tunnel0
Encapsulation format : GRE/IPv4
Site Bridge-Domain : 7
Capability : Multicast-reachable
Is Adjacency Server : No
Adj Server Configured : No
Prim/Sec Adj Svr(s) : None
|
OTV
Adjacency
CE_R07#sh otv
adjacency
Overlay
Adjacency Database for overlay 1
Hostname System-ID Dest Addr Site-ID Up Time State
CE_R09 001e.bdcd.c000
172.16.3.2 0000.0000.0009
2d23h UP
|
OTV Control Plane Protocol Status
CE_R07#sh otv
isis neighbors detail
Tag Overlay1:
System
Id Type Interface IP Address State Holdtime Circuit Id
CE_R09 L1
Ov1 172.16.3.2 UP
7 CE_R09.01
Area Address(es): 00
SNPA: 001e.bdcd.c0bc
State Changed: 2d23h
LAN Priority: 64
Format: Phase V
Remote TID: 0
Local TID:
0
Interface name: Overlay1
Site ID: 0000.0000.0009
Forward Ready
Version: 1
Version Mismatch: 0 AED Server Info:
Capability: 1
Priority: 0
AED Server Elected Value: 001E.BDCD.C000
AED State: 0
AED Elected Operational: 4
Backup AED Server Info:
Backup AED Server Elected Value:
0000.0000.0000
Backup AED State: 0
Backup AED Elected Operational: 0
Tag Site:
System
Id Type Interface IP Address State Holdtime Circuit Id
|
OTV Multicast
CE_R07#sh ip
mroute vrf TENANT_A
IP Multicast
Routing Table
Flags: D -
Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set,
F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP
created entry, E - Extranet,
X - Proxy Join Timer Running, A -
Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific
Host Report,
Z - Multicast Tunnel, z - MDT-data
group sender,
Y - Joined MDT-data group, y - Sending
to MDT-data group,
G - Received BGP C-Mroute, g - Sent
BGP C-Mroute,
N - Received BGP Shared-Tree Prune, n
- BGP C-Mroute suppressed,
Q - Received BGP S-A Route, q - Sent
BGP S-A Route,
V - RD & Vector, v - Vector, p -
PIM Joins on route,
x - VxLAN group
Outgoing
interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD,
State/Mode
(*,
225.0.0.1), 3w0d/stopped, RP 0.0.0.0, flags: DC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel0, Forward/Sparse-Dense,
3w0d/00:02:19
GigabitEthernet1.3000,
Forward/Sparse-Dense, 3w0d/Proxy
(172.16.3.2,
225.0.0.1), 3d01h/00:01:34, flags: T
Incoming interface: GigabitEthernet1.3000,
RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel0, Forward/Sparse-Dense,
3d01h/00:02:19
(172.16.3.1,
225.0.0.1), 3w0d/00:02:37, flags: T
Incoming interface: GigabitEthernet1.3000,
RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel0, Forward/Sparse-Dense,
3w0d/00:02:19
GigabitEthernet1.3000,
Forward/Sparse-Dense, 3w0d/Proxy
|
CE_R07#sh ip
mroute vrf TENANT_A count
Use
"show ip mfib count" to get better response time for a large number
of mroutes.
IP Multicast
Statistics
3 routes
using 3060 bytes of memory
1 groups,
2.00 average sources per group
Forwarding
Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per second
Other counts:
Total/RPF failed/Other drops(OIF-null, rate-limit etc)
Group:
225.0.0.1, Source count: 2, Packets forwarded: 436026, Packets received:
436027
Source: 172.16.3.2/32, Forwarding:
197757/0/862/5, Other: 197757/0/0
Source: 172.16.3.1/32, Forwarding:
238269/0/1327/2, Other: 238269/0/0
|
OTV VLANs
CE_R07#sh otv
vlan
Key: SI - Service Instance, NA - Non AED, NFC -
Not Forward Capable.
Overlay 1
VLAN Configuration Information
Inst VLAN BD Auth ED State Site If(s)
0
100 100 *CE_R07 active Gi3:SI100
0
200 200 *CE_R07 active Gi3:SI200
Total VLAN(s): 2
|
OTV Routes
CE_R07#sh otv
route
Codes: BD -
Bridge-Domain, AD - Admin-Distance,
SI - Service Instance, * - Backup
Route
OTV Unicast
MAC Routing Table for Overlay1
Inst VLAN BD MAC Address AD
Owner Next Hops(s)
----------------------------------------------------------
0
100 100 000c.2941.dc19 50 ISIS
CE_R09
0
100 100 000c.2977.ea21 40 BD Eng Gi3:SI100
0
100 100 000c.29d2.4ebe 50 ISIS
CE_R09
0
200 200 0000.ed01.ed01 50 ISIS
CE_R09
4 unicast
routes displayed in Overlay1
----------------------------------------------------------
4 Total
Unicast Routes Displayed
|
OTV Bridge-Domain
CE_R07#sh
bridge-domain
Bridge-domain
7 (1 ports in all)
State:
UP Mac learning:
Enabled
Aging-Timer:
300 second(s)
GigabitEthernet3 service instance 7
AED MAC address Policy
Tag Age Pseudoport
Bridge-domain
100 (2 ports in all)
State:
UP Mac learning:
Enabled
Aging-Timer:
1800 second(s)
GigabitEthernet3 service instance 100
Overlay1 service instance 100
AED MAC address Policy
Tag Age Pseudoport
1
000C.2941.DC19 forward static_r
0 OCE_PTR:0xe929d000
1
000C.2977.EA21 forward dynamic_c 1225 GigabitEthernet3.EFP100
1
000C.29D2.4EBE forward static_r
0 OCE_PTR:0xe929d000
1
0000.ED01.ED01 forward dynamic_c 1775 GigabitEthernet3.EFP100
Bridge-domain
200 (2 ports in all)
State:
UP Mac learning:
Enabled
Aging-Timer:
1800 second(s)
GigabitEthernet3 service instance 200
Overlay1 service instance 200
AED MAC address Policy
Tag Age Pseudoport
1
000C.29A8.ACDE forward dynamic_c 1434 GigabitEthernet3.EFP200
1
000C.29AD.8611 forward static_r
0 OCE_PTR:0xe929d020
1
0000.ED01.ED01 forward static_r
0 OCE_PTR:0xe929d020
1
000C.29B7.B1D3 forward dynamic_c 1745 GigabitEthernet3.EFP200
|
Packet
Walk
- A technical step by step packet walk diagram.
 |
| Fig. 5.1 |
In this example packet walk, we will look at host connectivity
within a single tenant (TENANT_A) but will cross datacenters fabrics using
either OTV (L2) or external routing (L3).
The walkthroughs will show MAC, IP and OTV route lookups for the
following scenarios.
- Host4 172.16.1.1 in DC1 sends data to Host8 172.16.1.3 in DC2 in the same VLAN (100 to 100)
- Host4 MAC: 000c.2977.ea21
- Host8 MAC: 000c.29d2.4ebe
- Host4 172.16.1.1 in DC1 sends data to Host10 172.16.2.3 in DC2 in a different VLAN (100 to 200)
Host4 to Host8 (VLAN 100 to 100)
- From DC1 Leaf01, Host4 is locally attached to interface E1/4
- From DC1 Leaf01, to reach Host8 use VXLAN to 5.5.5.5 (L2VNI to Leaf03)
DC1_LEAF_01#
show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC,
(R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ -
primary entry using vPC Peer-Link,
(T) - True, (F) - False, C -
ControlPlane MAC
VLAN
MAC Address Type age
Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 400
001e.bd7a.d8bc static -
F F (0x47000002) nve-peer2 5.5.5.5
* 300
001e.bd7a.d8bc static -
F F (0x47000002) nve-peer2 5.5.5.5
* 100 000c.2977.ea21 dynamic
00:06:04 F F
Eth1/4
* 100
000c.2998.c13c static -
F F (0x47000001) nve-peer1 4.4.4.4
* 200
000c.29a8.acde dynamic 00:06:21
F F Eth1/5
G 200
000c.2910.ac2a static -
F F sup-eth1(R)
* 100 000c.29d2.4ebe static
- F
F (0x47000002) nve-peer2
5.5.5.5
* 200
000c.29ad.8611 static -
F F (0x47000002) nve-peer2 5.5.5.5
* 200
001e.bd7a.d8bc static -
F F (0x47000002) nve-peer2 5.5.5.5
G 100
000c.2910.ac2a static
- F F
sup-eth1(R)
* 200
000c.292d.161b static -
F F (0x47000001) nve-peer1 4.4.4.4
G 400
000c.2910.ac2a static -
F F sup-eth1(R)
* 100
001e.bd7a.d8bc static -
F F (0x47000002) nve-peer2 5.5.5.5
G 300
000c.2910.ac2a static -
F F sup-eth1(R)
1 1 -00:00:ed:01:ed:01 - 1
|
- From DC1 Leaf03, to reach Host8 use OTV internal interface E1/4 (to CE_R07)
DC1_LEAF_03#
show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC,
(R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ -
primary entry using vPC Peer-Link,
(T) - True, (F) - False, C -
ControlPlane MAC
VLAN
MAC Address Type age
Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 400
001e.bd7a.d8bc dynamic 00:02:21
F F Eth1/4
* 300
001e.bd7a.d8bc dynamic 00:03:21
F F Eth1/4
* 100
000c.2977.ea21 static -
F F (0x47000002) nve-peer2 3.3.3.3
* 100
000c.2998.c13c static -
F F (0x47000001) nve-peer1 4.4.4.4
* 200
000c.29a8.acde static -
F F (0x47000002) nve-peer2 3.3.3.3
* 100 000c.29d2.4ebe dynamic
00:00:42 F F
Eth1/4
* 200
000c.29ad.8611 dynamic 00:04:16
F F Eth1/4
* 200
001e.bd7a.d8bc dynamic 00:02:51
F F Eth1/4
* 200
000c.292d.161b static -
F F (0x47000001) nve-peer1 4.4.4.4
* 100
001e.bd7a.d8bc dynamic 00:02:21
F F Eth1/4
|
- From CE_R07, to reach Host8 use OTV overlay interface to CE_R09
CE_R07#sh otv
route
Codes: BD -
Bridge-Domain, AD - Admin-Distance,
SI - Service Instance, * - Backup
Route
OTV Unicast
MAC Routing Table for Overlay1
Inst VLAN BD MAC Address AD
Owner Next Hops(s)
----------------------------------------------------------
0
100 100 000c.2977.ea21 40 BD Eng Gi3:SI100
0 100 100
000c.29d2.4ebe 50 ISIS CE_R09
0
200 200 000c.292d.161b 40 BD Eng Gi3:SI200
0
200 200 000c.29a8.acde 40 BD Eng Gi3:SI200
0
200 200 000c.29ad.8611 50 ISIS
CE_R09
5 unicast
routes displayed in Overlay1
----------------------------------------------------------
5 Total
Unicast Routes Displayed
|
- From CE_R09, to reach Host8 use OTV internal interface Service Instance 100 (to DC2 Leaf03)
CE_R09#sh otv
route
Codes: BD -
Bridge-Domain, AD - Admin-Distance,
SI - Service Instance, * - Backup
Route
OTV Unicast
MAC Routing Table for Overlay1
Inst VLAN BD MAC Address AD
Owner Next Hops(s)
----------------------------------------------------------
0
100 100 000c.2977.ea21 50 ISIS
CE_R07
0
100 100 000c.2998.c13c 50 ISIS
CE_R07
0 100 100
000c.29d2.4ebe 40 BD Eng
Gi3:SI100
0
200 200 000c.292d.161b 50 ISIS
CE_R07
0
200 200 000c.29a8.acde 50 ISIS
CE_R07
0
200 200 000c.29ad.8611 40 BD Eng Gi3:SI200
6 unicast
routes displayed in Overlay1
----------------------------------------------------------
6 Total
Unicast Routes Displayed
|
- From DC2’s Leaf03, to reach Host8 use VXLAN to 8.8.8.8 (L2VNI to DC2’s Leaf01)
DC2_LEAF_03#
show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC,
(R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ -
primary entry using vPC Peer-Link,
(T) - True, (F) - False, C -
ControlPlane MAC
VLAN
MAC Address Type age
Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 400
001e.bdcd.c0bc dynamic 00:00:53
F F Eth1/4
* 100
000c.2977.ea21 dynamic 00:03:54
F F Eth1/4
* 300
001e.bdcd.c0bc dynamic 00:00:23
F F Eth1/4
* 200
000c.29a8.acde dynamic 00:07:25
F F Eth1/4
* 100 000c.29d2.4ebe static
- F F
(0x47000001) nve-peer1 8.8.8.8
* 200
000c.29ad.8611 static -
F F (0x47000002) nve-peer2 9.9.9.9
* 200
000c.292d.161b dynamic 00:07:23
F F Eth1/4
|
- From DC2’s Leaf01, Host8 is locally attached to interface E1/4
DC2_LEAF_01#
show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC,
(R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ -
primary entry using vPC Peer-Link,
(T) - True, (F) - False, C -
ControlPlane MAC
VLAN
MAC Address Type age
Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 200
001e.bdcd.c0bc static -
F F (0x47000002) nve-peer2 10.10.1
* 100
001e.bdcd.c0bc static -
F F (0x47000002) nve-peer2 10.10.1
* 400
001e.bdcd.c0bc static -
F F (0x47000002) nve-peer2 10.10.1
G 400
000c.29de.a659 static -
F F sup-eth1(R)
* 100
000c.2977.ea21 static -
F F (0x47000002) nve-peer2 10.10.1
G 300
000c.29de.a659 static -
F F sup-eth1(R)
* 300
001e.bdcd.c0bc static -
F F (0x47000002) nve-peer2 10.10.1
* 200
000c.29a8.acde static -
F F (0x47000002) nve-peer2 10.10.1
* 100 000c.29d2.4ebe dynamic
00:12:14 F F
Eth1/4
* 200
000c.29ad.8611 static -
F F (0x47000001) nve-peer1 9.9.9.9
G 200
000c.29de.a659 static -
F F sup-eth1(R)
* 200
000c.292d.161b static -
F F (0x47000002) nve-peer2 10.10.1
G 100
000c.29de.a659 static -
F F sup-eth1(R)
G 900
000c.29de.a659 static -
F F sup-eth1(R)
G 910
000c.29de.a659 static -
F F sup-eth1(R)
1 1 -00:00:ed:01:ed:01 - 1
|
Host4 to Host10 (VLAN 100 to 200)
- From DC1 Leaf01, Host4 is locally attached to VLAN 100
- From DC1 Leaf01, to reach Host10 use VXLAN to 5.5.5.5 (L3VNI to Leaf03)
DC1_LEAF_01#
sh ip route vrf TENANT_A 172.16.1.1
IP Route
Table for VRF "TENANT_A"
'*' denotes
best ucast next-hop
'**' denotes
best mcast next-hop
'[x/y]'
denotes [preference/metric]
'%<string>'
in via output denotes VRF <string>
172.16.1.1/32, ubest/mbest: 1/0, attached
*via 172.16.1.1, Vlan100, [190/0], 4d16h, hmm
|
DC1_LEAF_01#
sh ip route vrf TENANT_A 172.16.2.3
IP Route
Table for VRF "TENANT_A"
'*' denotes
best ucast next-hop
'**' denotes
best mcast next-hop
'[x/y]'
denotes [preference/metric]
'%<string>'
in via output denotes VRF <string>
172.16.2.3/32, ubest/mbest: 1/0
*via 5.5.5.5%default, [200/0], 02:03:57, bgp-65500, internal, tag 65000
(evpn) segid: 90000 tunnelid: 0x5050505 encap: VXLAN
|
- From DC1 Leaf03, to reach Host10 use L3 routing (to CE_R07)
DC1_LEAF_03#
sh ip route vrf TENANT_A 172.16.2.3
IP Route
Table for VRF "TENANT_A"
'*' denotes
best ucast next-hop
'**' denotes
best mcast next-hop
'[x/y]'
denotes [preference/metric]
'%<string>'
in via output denotes VRF <string>
172.16.2.3/32, ubest/mbest: 1/0
*via 172.16.3.254, [20/0], 01:31:49, bgp-65500, external, tag 65000
|
- From CE_R07, to reach Host10 use L3 routing (to CE_R09)
CE_R07#sh ip
route vrf TENANT_A 172.16.2.3
Routing
Table: TENANT_A
Routing entry for 172.16.2.3/32
Known via "bgp 65000", distance
20, metric 0
Tag 65001, type external
Last update from 172.16.3.2 02:01:50 ago
Routing Descriptor Blocks:
* 172.16.3.2, from 172.16.3.2, 02:01:50 ago
Route metric is 0, traffic share count
is 1
AS Hops 2
Route tag 65001
MPLS label: none
|
- From CE_R09, to reach Host10 use L3 routing (to DC2 Leaf03)
CE_R09#sh ip
route vrf TENANT_A 172.16.2.3
Routing
Table: TENANT_A
Routing entry for 172.16.2.3/32
Known via "bgp 65001", distance
20, metric 0
Tag 65501, type external
Last update from 172.16.3.249 02:00:31 ago
Routing Descriptor Blocks:
* 172.16.3.249, from 172.16.3.249, 02:00:31 ago
Route metric is 0, traffic share count
is 1
AS Hops 1
Route tag 65501
MPLS label: none
|
- From DC2 Leaf03, to reach Host10 use VXLAN to 9.9.9.9 (L3VNI to Leaf02)
DC2_LEAF_03#
sh ip route vrf TENANT_A 172.16.2.3
IP Route
Table for VRF "TENANT_A"
'*' denotes
best ucast next-hop
'**' denotes
best mcast next-hop
'[x/y]'
denotes [preference/metric]
'%<string>'
in via output denotes VRF <string>
172.16.2.3/32, ubest/mbest: 1/0
*via 9.9.9.9%default, [200/0], 4d16h, bgp-65501, internal, tag 65501
(evpn) segid: 90000 tunnelid: 0x9090909 encap: VXLAN
|
- From DC2 Leaf02, Host10 is locally attached to VLAN 200
DC2_LEAF_02#
sh ip route vrf TENANT_A 172.16.2.3
IP Route
Table for VRF "TENANT_A"
'*' denotes
best ucast next-hop
'**' denotes
best mcast next-hop
'[x/y]'
denotes [preference/metric]
'%<string>'
in via output denotes VRF <string>
172.16.2.3/32, ubest/mbest: 1/0, attached
*via 172.16.2.3, Vlan200, [190/0], 4d16h, hmm
|
Issues
OTV over MPLS L3VPN
When I initially tried to connect the CE devices to a MPLS
L3VPN network as IP transport, I could not get the OTV adjacency to work in multicast
mode. When I switched the WAN service to
VPLS (L2), it worked fine.
Based on what I been reading, I believe the underlying IP
transport network must have IP Multicast enabled first (see link). When the OTV documentation specifies that you can
use “any transport”, I thought any IP routed network at a minimum would suffice,
not requiring anything extra such as the complexities of multicast. However, one can use OTV in unicast mode,
which is far simpler to deploy but can limit the number of datacenters that can
be interconnected.
InterVLAN routing across OTV
During my testing, I’ve noticed that I had no problems pinging
every host from DC1 to DC2 in the same VLAN over OTV (e.g., VL100 to VL100). It worked as expected. However when I pinged a host in another VLAN
(e.g., VL100 to VL200), it failed. The inter-vlan routing over OTV wasn’t
working. MAC learning wasn’t taking
place as I thought it should. The only
way to make this work was to have a host in DC1 VL200 ping across OTV to
another host in DC2 in the same VLAN (or vice versa). Once that was initiated, DC1’s VXLAN fabric
created a locally attached route (since its MAC address was learned via OTV)
and then MP-BGP would generate a type 2 route and share this to the rest of the
fabric. The problem would arise again
when that ARP entry ages out and the MP-BGP route would disappear.
After doing some research, I found an article that
specifically addresses this issue. It
states that inter-vlan routing in a multi-fabric setup using OTV should use the
MP-BGP route type 5 that was learned from the other DC fabric.
So I reconfigured my lab to ensure the
EVPN MP-BGP /32 routes were being shared across the two fabrics. However when I did that, I observed a strange routing issue even though there wasn’t any apparent connectivity issues.
What I saw was that a host in DC1 (172.16.1.1) could possibly take 2 different paths to a host in DC2 (172.16.2.3) that were not in the same VLAN. One path was via the external L3 (not using OTV) and the other was via the L2 extended OTV path.
Taking one or the other would depend on OTV's MAC learning for the DC2 host. If the DC2's host MAC hasn't been learned over OTV, the DC1 host (172.16.1.1) would take the L3 path to reach the host in DC2 (172.16.2.3). This would seem like normal operation based on the Cisco article. However, that same host would take the L2 path if OTV did learn of DC2' host MAC address.
At this point I’m not entirely sure if this is normal behavior for a multi-fabric
setup but the traceroute below will show this behavior. The top is the L3 path while the bottom is the L2 path.
Here’s a look at DC1 fabric’s border leaf route table before
OTV has learned 172.16.2.3’s MAC address.
It’s clearly going out the external router path, using the L3VNI and the BGP route type 5.
DC1_LEAF_03# sh ip route vrf TENANT_A
IP Route
Table for VRF "TENANT_A"
'*' denotes
best ucast next-hop
'**' denotes
best mcast next-hop
'[x/y]'
denotes [preference/metric]
'%<string>'
in via output denotes VRF <string>
0.0.0.0/0,
ubest/mbest: 1/0
*via 172.16.3.254, [20/0], 4d01h,
bgp-65500, external, tag 65000
172.16.1.0/24,
ubest/mbest: 1/0, attached
*via 172.16.1.254, Vlan100, [0/0], 3w1d,
direct
172.16.1.1/32,
ubest/mbest: 1/0
*via 3.3.3.3%default, [200/0], 4d00h,
bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x3030303 encap:
VXLAN
172.16.1.2/32,
ubest/mbest: 1/0
*via 4.4.4.4%default, [200/0], 3w0d,
bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x4040404 encap:
VXLAN
172.16.1.3/32,
ubest/mbest: 1/0
*via 172.16.3.254, [20/0], 02:09:55,
bgp-65500, external, tag 65000
172.16.1.254/32,
ubest/mbest: 1/0, attached
*via 172.16.1.254, Vlan100, [0/0], 3w1d,
local
172.16.2.0/24,
ubest/mbest: 1/0, attached
*via 172.16.2.254, Vlan200, [0/0], 3w1d,
direct
172.16.2.1/32,
ubest/mbest: 1/0
*via 3.3.3.3%default, [200/0], 3w0d,
bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x3030303 encap:
VXLAN
172.16.2.2/32,
ubest/mbest: 1/0
*via 4.4.4.4%default, [200/0], 3w0d,
bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x4040404 encap:
VXLAN
172.16.2.3/32, ubest/mbest: 1/0
*via
172.16.3.254, [20/0], 02:09:55, bgp-65500, external, tag 65000
172.16.2.254/32,
ubest/mbest: 1/0, attached
*via 172.16.2.254, Vlan200, [0/0], 3w1d,
local
172.16.3.252/30,
ubest/mbest: 1/0, attached
*via 172.16.3.253, Eth1/5.90, [0/0],
3w1d, direct
172.16.3.253/32,
ubest/mbest: 1/0, attached
*via 172.16.3.253, Eth1/5.90, [0/0],
3w1d, local
172.16.100.0/24,
ubest/mbest: 1/0
*via 172.16.3.254, [20/0], 1w1d,
bgp-65500, external, tag 65000
|
BGP Route Type 5
DC1_LEAF_03#
sh bgp l2vpn evpn 172.16.2.3
BGP routing
table information for VRF default, address family L2VPN EVPN
Route
Distinguisher: 5.5.5.5:3 (L3VNI 90000)
BGP routing
table entry for [5]:[0]:[0]:[32]:[172.16.2.3]:[0.0.0.0]/224,
version 211518
Paths: (1
available, best #1)
Flags:
(0x000002) on xmit-list, is not in l2rib/evpn
Advertised path-id 1
Path type: local, path is valid, is best
path, no labeled nexthop
AS-Path: 65000 65001 65501 , path sourced
external to AS
5.5.5.5 (metric 0) from 0.0.0.0 (5.5.5.5)
Origin IGP, MED not set, localpref 100,
weight 0
Received label 90000
Extcommunity: RT:65500:90000 ENCAP:8 Router
MAC:000c.29b7.b1d3
Path-id 1 advertised to peers:
1.1.1.1 2.2.2.2
|
On the other hand, if another host in the same VLAN (say 172.16.2.1)
initiates traffic to 172.16.2.3, MAC learning via OTV will take
place and will populate DC1’s fabric with a locally attached route. MP-BGP would
create a type 2 route and share it within that fabric. Any further traffic will now follow the L2VNI
to the OTV overlay.
DC1_LEAF_03# sh ip route vrf TENANT_A
IP Route
Table for VRF "TENANT_A"
'*' denotes
best ucast next-hop
'**' denotes
best mcast next-hop
'[x/y]'
denotes [preference/metric]
'%<string>'
in via output denotes VRF <string>
0.0.0.0/0,
ubest/mbest: 1/0
*via 172.16.3.254, [20/0], 00:03:45,
bgp-65500, external, tag 65000
172.16.1.0/24,
ubest/mbest: 1/0, attached
*via 172.16.1.254, Vlan100, [0/0],
00:03:47, direct
172.16.1.1/32,
ubest/mbest: 1/0
*via 3.3.3.3%default, [200/0], 00:03:46,
bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x3030303 encap:
VXLAN
172.16.1.2/32,
ubest/mbest: 1/0
*via 4.4.4.4%default, [200/0], 00:03:46,
bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x4040404 encap:
VXLAN
172.16.1.3/32,
ubest/mbest: 1/0
*via 172.16.3.254, [20/0], 00:03:45,
bgp-65500, external, tag 65000
172.16.1.254/32,
ubest/mbest: 1/0, attached
*via 172.16.1.254, Vlan100, [0/0],
00:03:47, local
172.16.2.0/24,
ubest/mbest: 1/0, attached
*via 172.16.2.254, Vlan200, [0/0],
00:03:47, direct
172.16.2.1/32,
ubest/mbest: 1/0
*via 3.3.3.3%default, [200/0], 00:03:46,
bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x3030303 encap:
VXLAN
172.16.2.2/32,
ubest/mbest: 1/0
*via 4.4.4.4%default, [200/0], 00:03:46,
bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x4040404 encap:
VXLAN
172.16.2.3/32, ubest/mbest: 1/0, attached
*via
172.16.2.3, Vlan200, [190/0], 00:00:13, hmm
172.16.2.254/32,
ubest/mbest: 1/0, attached
*via 172.16.2.254, Vlan200, [0/0],
00:03:47, local
172.16.3.252/30,
ubest/mbest: 1/0, attached
*via 172.16.3.253, Eth1/5.90, [0/0],
00:03:47, direct
172.16.3.253/32,
ubest/mbest: 1/0, attached
*via 172.16.3.253, Eth1/5.90, [0/0],
00:03:47, local
172.16.100.0/24,
ubest/mbest: 1/0
*via 172.16.3.254, [20/0], 00:03:45,
bgp-65500, external, tag 65000
|
Now it’s a route type 2
DC1_LEAF_03#
sh bgp l2vpn evpn 172.16.2.3
BGP routing
table information for VRF default, address family L2VPN EVPN
Route
Distinguisher: 5.5.5.5:32967 (L2VNI 20000)
BGP routing
table entry for [2]:[0]:[0]:[48]:[000c.29ad.8611]:[32]:[172.16.2.3]/272,
version 217874
Paths: (1
available, best #1)
Flags:
(0x000102) on xmit-list, is not in l2rib/evpn
Advertised path-id 1
Path type: local, path is valid, is best
path, no labeled nexthop
AS-Path: NONE, path locally originated
5.5.5.5 (metric 0) from 0.0.0.0 (5.5.5.5)
Origin IGP, MED not set, localpref 100,
weight 32768
Received label 20000 90000
Extcommunity: RT:65500:20000 RT:65500:90000 ENCAP:8 MAC
Mobility Sequence:00:1 Router MAC:000c.29b7.b1d3
Path-id 1 advertised to peers:
1.1.1.1 2.2.2.2
Route
Distinguisher: 5.5.5.5:3 (L3VNI
90000)
BGP routing
table entry for [5]:[0]:[0]:[32]:[172.16.2.3]:[0.0.0.0]/224, version 211518
Paths: (1
available, best #1)
Flags:
(0x000002) on xmit-list, is not in l2rib/evpn
Advertised path-id 1
Path type: local, path is valid, is best
path, no labeled nexthop
AS-Path: 65000 65001 65501 , path sourced
external to AS
5.5.5.5 (metric 0) from 0.0.0.0 (5.5.5.5)
Origin IGP, MED not set, localpref 100,
weight 0
Received label 90000
Extcommunity: RT:65500:90000 ENCAP:8 Router MAC:000c.29b7.b1d3
Path-id 1 advertised to peers:
1.1.1.1 2.2.2.2
|
Again, I don’t know if this is an actual problem because we
are still achieving reachability to all hosts.
It just seems strange to see traffic taking different paths depending on
the state of the MAC table. To me this doesn’t
seem right and will have to take a deeper look.
UPDATE (08/25/17): I believe I solved
the inter-vlan problem. I read that the
server SVI interfaces and the anycast gateway MAC address do not need to be configured
on the border leaf nodes. I guess that
makes sense. Technically, we don’t need
those since the border leaf is a dedicated node for external services.
After removing those pieces off both DC’s border leafs, the
ping and trace results looked consistent. In other words, any inter-vlan L3 traffic went
over the L3 path and L2 traffic went over OTV.
The post below does an excellent job doing a detailed packet walk for L2 and L3 traffic across multi-fabric VXLAN over OTV.
http://yves-louis.com/DCI/?p=1487
References
Cisco Live presentation on OTV
A great write up on the inter-workings of OTV
Another good Blog on OTV
Cisco references on OTV
