Monday, August 21, 2017

Multi-Fabric VXLAN EVPN using Cisco OTV as DCI

After learning quite a bit with VXLAN EVPN, I wanted to take it a little further and understand how we could join 2 distributed datacenters (Multi-fabric) using a Datacenter Interconnect (DCI) technology.  I read that Cisco’s Overlay Transport Virtualization (OTV) was available on IOS XE so was tempted to try it out on the CSR1000v.


Lab Environment

In this lab I built out another DC fabric, called DC2.  It’s configured for VXLAN EVPN and operates similarly to DC1.  The CE devices (CSR1000v’s) that were used as WAN edge will also serve as the OTV edge devices to provide L2 extension between the two DC’s.

Note: The following resources were appended to the existing VXLAN EVPN lab


  • (5) Cisco NX-OSv 9000: version 7.0(3)I6(1)
    • Used 2 vCPUs instead of the default 4
    • Used default RAM at 8Gb
  • (1) CSR1000v: version 16.4.2 [Everest]
    • Used default vCPU setting of 1
    • Used default RAM at 4Gb
  • (4) Damn Small Linux (DSL) 4.x as test hosts


Diagram & Topology

  • OTV Overview
Fig. 1.1

  • DC1 with OTV related configurations and connections
Fig. 1.2

  • DC2 with OTV related configurations and connections
Fig. 1.3

  • Infrastructure Information
    • (2) Customer Edge (CE Routers)
      • OTV Internal Interface connected to E1/4 on each DC’s Leaf03 switch
    • WAN type (L2 VPLS transport, CE’s connected to VPLS lab)
    • OTV configured in Multicast mode
    • TENANT_A
      • WAN IP Address Space: 172.16.3.0/29
      • Extended VLANs 100 & 200
    • TENANT_B
      • WAN IP Address Space: 192.168.3.0/29
      • Extended VLANs 300 & 400


Technology Overview


Datacenter Interconnect (DCI)
  • A DCI is a technology that can provide both L3 and L2 LAN extensions across multiple distributed datacenters.  The two slides I’ve found on the Internet below  (Fig 2.1 and 2.2) are good representations of what a DCI is trying to achieve.

Fig 2.1



Fig 2.2
  • Using a DCI between datacenters can solve some of these IT and business related goals (Fig 2.3).
Fig 2.3
  • Traditional DCIs can use a number of different L1 & L2 transport options, such as DWDM/Dark Fiber, VPLS or VPWS (Fig 2.4).  However, new technologies are emerging where L2 extensions can occur over any IP routed networks.

Fig 2.4
  • One main use case for LAN extensions is VM mobility across distributed datacenters (Fig 2.5).
·

Fig 2.5

Cisco Overlay Transport Virtualization (OTV)

Cisco OTV is a propriety protocol that can provide L2 extensions over an Any Transport network. Unlike other L2 technologies, OTV is specifically designed for DCI.
  • Here are some informational slides that will explain the technology and terminology used throughout this post (Figs 3.1 - 3.8).

Fig 3.1


Fig 3.2


Fig 3.3
  • Common terms used during the configuration and troubleshooting of OTV (Figs. 3.4-3.5).

Fig 3.4


Fig 3.5
  • OTV on the data plane can run in both Unicast and Multicast (Fig 3.6).  Its usage depends on how many datacenters are connected.  Running multicast mode requires you to have an IP transport network that is multicast aware.  Control plane protocol uses ISIS for MAC learning across the IP transport (Fig. 3.7).

Fig. 3.6


Fig. 3.7
  • OTV components (Fig 3.8). 


Fig 3.8


Configurations


CE_R07

version 16.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime show-timezone year
service timestamps log datetime localtime show-timezone year
service password-encryption
no platform punt-keepalive disable-kernel-core
platform console auto
!
hostname CE_R07
!
boot-start-marker
boot-end-marker
!
!
vrf definition TENANT_A
 rd 65000:1
 !
 address-family ipv4
 exit-address-family
!
vrf definition TENANT_B
 rd 65000:2
 !
 address-family ipv4
 exit-address-family
!
logging buffered 16000
no logging console
enable secret 5 <removed>
!
no aaa new-model
!
ip vrf MGMT
!
ip multicast-routing distributed
ip multicast-routing vrf TENANT_A distributed
ip multicast-routing vrf TENANT_B distributed
!
!
!
!
!
!
!
!
no ip domain lookup
ip domain name domain.local
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
otv site bridge-domain 7
!
otv site-identifier 0000.0000.0007
!
!
multilink bundle-name authenticated
!
!
!
!
!
!         
!
!
!
!
!
!
!
!
license udi pid CSR1000V sn 9BW82G58JFL
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
!

!
!
!
!
!
!
!
!
!
!
!
!






!
!
interface Loopback0
 description RID
 ip address 70.70.70.70 255.255.255.255
!
interface Overlay1
 description OTV OVERLAY FOR TENANT_A
 no ip address
 otv control-group 225.0.0.1
 otv data-group 232.0.0.0/8
 otv join-interface GigabitEthernet1.3000
 no otv suppress arp-nd
 no mop enabled
 no mop sysid
 service instance 100 ethernet
  encapsulation dot1q 100
  bridge-domain 100
 !
 service instance 200 ethernet
  encapsulation dot1q 200
  bridge-domain 200
 !
!
interface Overlay2
 description OTV OVERLAY FOR TENANT_B
 no ip address
 otv control-group 225.0.0.2
 otv data-group 233.0.0.0/8
 otv join-interface GigabitEthernet1.3001
 no otv suppress arp-nd
 no mop enabled
 no mop sysid
 service instance 300 ethernet
  encapsulation dot1q 300
  bridge-domain 300
 !
 service instance 400 ethernet
  encapsulation dot1q 400
  bridge-domain 400
 !
!
interface GigabitEthernet1
 mtu 9216
 no ip address
 negotiation auto
 no mop enabled
 no mop sysid
!
interface GigabitEthernet1.3000
 description WAN TO PE_R13 - TENANT A VPLS
 encapsulation dot1Q 3000
 vrf forwarding TENANT_A
 ip address 172.16.3.1 255.255.255.248
 ip pim passive
 ip igmp version 3
!
interface GigabitEthernet1.3001
 description WAN TO PE_R13 - TENANT B VPLS
 encapsulation dot1Q 3001
 vrf forwarding TENANT_B
 ip address 192.168.3.1 255.255.255.248
 ip pim passive
 ip igmp version 3
!
interface GigabitEthernet2
 mtu 9216
 no ip address
 negotiation auto
 no mop enabled
 no mop sysid
!
interface GigabitEthernet2.90
 description TO VXLAN EVPN DC1_LEAF_03 - L3VNI TENANT A
 encapsulation dot1Q 900
 vrf forwarding TENANT_A
 ip address 172.16.3.254 255.255.255.252
!
interface GigabitEthernet2.91
 description TO VXLAN EVPN DC1_LEAF_03 - L3VNI TENANT B
 encapsulation dot1Q 910
 vrf forwarding TENANT_B
 ip address 192.168.3.254 255.255.255.252
!
interface GigabitEthernet3
 description OTV LAN - ALL TENANTS
 mtu 9216
 no ip address
 negotiation auto
 no mop enabled
 no mop sysid
 service instance 7 ethernet
  encapsulation untagged
  snmp ifindex persist
  bridge-domain 7
 !
 service instance 100 ethernet
  encapsulation dot1q 100
  bridge-domain 100
 !
 service instance 200 ethernet
  encapsulation dot1q 200
  snmp ifindex persist
  bridge-domain 200
 !
 service instance 300 ethernet
  encapsulation dot1q 300
  bridge-domain 300
 !
 service instance 400 ethernet
  encapsulation dot1q 400
  snmp ifindex persist
  bridge-domain 400
 !        
!
interface GigabitEthernet4
 description MANAGEMENT
 ip vrf forwarding MGMT
 ip address 10.1.1.227 255.255.255.0
 negotiation auto
 no mop enabled
 no mop sysid
!
router bgp 65000
 bgp router-id 70.70.70.70
 bgp log-neighbor-changes
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv4 vrf TENANT_A
  neighbor DC1_LEAF3_TENANT_A peer-group
  neighbor DC1_LEAF3_TENANT_A remote-as 65500
  neighbor DC1_LEAF3_TENANT_A description TO DC1 VXLAN TENANT A
  neighbor DC1_LEAF3_TENANT_A timers 7 21
  neighbor DC1_LEAF3_TENANT_A send-community both
  neighbor DC2_CE9_TENANT_A peer-group
  neighbor DC2_CE9_TENANT_A remote-as 65001
  neighbor DC2_CE9_TENANT_A description TO DC2 VXLAN TENANT A
  neighbor DC2_CE9_TENANT_A timers 7 21
  neighbor DC2_CE9_TENANT_A send-community both
  neighbor BRANCH_CE17_TENANT_A peer-group
  neighbor BRANCH_CE17_TENANT_A remote-as 65100
  neighbor BRANCH_CE17_TENANT_A description TO BRANCH CE17 TENANT A
  neighbor BRANCH_CE17_TENANT_A timers 7 21
  neighbor BRANCH_CE17_TENANT_A send-community both
  neighbor 172.16.3.2 peer-group DC2_CE9_TENANT_A
  neighbor 172.16.3.2 activate
  neighbor 172.16.3.3 peer-group BRANCH_CE17_TENANT_A
  neighbor 172.16.3.3 activate
  neighbor 172.16.3.253 peer-group DC1_LEAF3_TENANT_A
  neighbor 172.16.3.253 activate
 exit-address-family
 !
 address-family ipv4 vrf TENANT_B
  neighbor DC1_LEAF3_TENANT_B peer-group
  neighbor DC1_LEAF3_TENANT_B remote-as 65500
  neighbor DC1_LEAF3_TENANT_B description TO DC1 VXLAN TENANT B
  neighbor DC1_LEAF3_TENANT_B timers 7 21
  neighbor DC1_LEAF3_TENANT_B send-community both
  neighbor DC2_CE9_TENANT_B peer-group
  neighbor DC2_CE9_TENANT_B remote-as 65001
  neighbor DC2_CE9_TENANT_B description TO DC2 VXLAN TENANT B
  neighbor DC2_CE9_TENANT_B timers 7 21
  neighbor DC2_CE9_TENANT_B send-community both
  neighbor BRANCH_CE17_TENANT_B peer-group
  neighbor BRANCH_CE17_TENANT_B remote-as 65100
  neighbor BRANCH_CE17_TENANT_B description TO BRANCH CE17 TENANT B
  neighbor BRANCH_CE17_TENANT_B timers 7 21
  neighbor BRANCH_CE17_TENANT_B send-community both
  neighbor 192.168.3.2 peer-group DC2_CE9_TENANT_B
  neighbor 192.168.3.2 activate
  neighbor 192.168.3.3 peer-group BRANCH_CE17_TENANT_B
  neighbor 192.168.3.3 activate
  neighbor 192.168.3.253 peer-group DC1_LEAF3_TENANT_B
  neighbor 192.168.3.253 activate
 exit-address-family
!
!
virtual-service csr_mgmt
!
no ip forward-protocol nd
ip pim ssm default
ip pim vrf TENANT_A ssm default
ip pim vrf TENANT_B ssm default
no ip http server
ip http authentication local
no ip http secure-server
!
ip bgp-community new-format
ip tftp source-interface GigabitEthernet4
ip route vrf MGMT 0.0.0.0 0.0.0.0 10.1.1.254 name MGMT_DEFAULT
!
!
!
!
!
!
control-plane
!
 !
 !
 !
 !
!
!
!
!
!
line con 0
 logging synchronous
 stopbits 1
line vty 0 4
 exec-timeout 60 0
 logging synchronous
 login local
 transport input telnet ssh
!
ntp server pool.ntp.org
!
!
!
!
!
end


DC1_LEAF_03

version 7.0(3)I6(1)
hostname DC1_LEAF_03
vdc DC1_LEAF_03 id 1
  limit-resource vlan minimum 16 maximum 4094
  limit-resource vrf minimum 2 maximum 4096
  limit-resource port-channel minimum 0 maximum 511
  limit-resource u4route-mem minimum 248 maximum 248
  limit-resource u6route-mem minimum 96 maximum 96
  limit-resource m4route-mem minimum 58 maximum 58
  limit-resource m6route-mem minimum 8 maximum 8

feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay

no password strength-check
username admin password 5 <removed>  role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO

vlan 1,100,200,300,400,900,910
ip pim rp-address 100.100.100.100 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 100
  vn-segment 10000
vlan 200
  vn-segment 20000
vlan 300
  vn-segment 30000
vlan 400
  vn-segment 40000
vlan 900
  vn-segment 90000
vlan 910
  vn-segment 91000

vrf context TENANT_A
  vni 90000
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
vrf context TENANT_B
  vni 91000
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
vrf context management
  ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT

interface Vlan1

interface Vlan900
  description TENANT_A VL900 L3VNI
  no shutdown
  mtu 9216
  vrf member TENANT_A
  ip forward

interface Vlan910
  description TENANT_B VL910 L3VNI
  no shutdown
  mtu 9216
  vrf member TENANT_B
  ip forward

interface nve1
  no shutdown
  description VTEP_03
  source-interface loopback0
  host-reachability protocol bgp
  member vni 10000
    suppress-arp
    mcast-group 225.1.0.1
  member vni 20000
    suppress-arp
    mcast-group 225.2.0.1
  member vni 30000
    suppress-arp
    mcast-group 225.3.0.1
  member vni 40000
    suppress-arp
    mcast-group 225.4.0.1
  member vni 90000 associate-vrf
  member vni 91000 associate-vrf

interface Ethernet1/1
  description TO SPINE_01
  no switchport
  mtu 9216
  ip address 10.0.0.18/30
  ip ospf network point-to-point
  no ip ospf passive-interface
  ip router ospf 100 area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet1/2
  description TO SPINE_02
  no switchport
  mtu 9216
  ip address 10.0.0.22/30
  ip ospf network point-to-point
  no ip ospf passive-interface
  ip router ospf 100 area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet1/3
  no switchport
  mtu 9216

interface Ethernet1/4
  description TO CE_R07 - OTV LAN
  switchport mode trunk
  switchport trunk allowed vlan 100,200,300,400
  mtu 9216

interface Ethernet1/5
  description TO CE7 - EXT L3
  no switchport
  mtu 9216
  no shutdown

interface Ethernet1/5.90
  description TO CE7 - TENANT A L3VNI EXT L3
  mtu 9216
  encapsulation dot1q 900
  vrf member TENANT_A
  ip address 172.16.3.253/30
  no shutdown

interface Ethernet1/5.91
  description TO CE7 - TENANT B L3VNI EXT L3
  mtu 9216
  encapsulation dot1q 910
  vrf member TENANT_B
  ip address 192.168.3.253/30
  no shutdown

interface mgmt0
  description MANAGEMENT
  vrf member management
  ip address 10.1.1.214/24

interface loopback0
  description RID - VTEP
  ip address 5.5.5.5/32
  ip router ospf 100 area 0.0.0.0
  ip pim sparse-mode

line console
  exec-timeout 60
line vty
  exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin 
router ospf 100
  router-id 5.5.5.5
  passive-interface default
router bgp 65500
  router-id 5.5.5.5
  neighbor 1.1.1.1
    remote-as 65500
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 2.2.2.2
    remote-as 65500
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  vrf TENANT_A
    timers bgp 7 21
    address-family ipv4 unicast
      network 172.16.1.0/24
      advertise l2vpn evpn
    neighbor 172.16.3.254
      remote-as 65000
      address-family ipv4 unicast
        send-community
        send-community extended
  vrf TENANT_B
    timers bgp 7 21
    address-family ipv4 unicast
      advertise l2vpn evpn
    neighbor 192.168.3.254
      remote-as 65000
      address-family ipv4 unicast
        send-community
        send-community extended
evpn
  vni 10000 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 20000 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 30000 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 40000 l2
    rd auto
    route-target import auto
    route-target export auto


CE_R09

version 16.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime show-timezone year
service timestamps log datetime localtime show-timezone year
service password-encryption
no platform punt-keepalive disable-kernel-core
platform console auto
!
hostname CE_R09
!
boot-start-marker
boot-end-marker
!
!
vrf definition TENANT_A
 rd 65001:1
 !
 address-family ipv4
 exit-address-family
!
vrf definition TENANT_B
 rd 65001:2
 !
 address-family ipv4
 exit-address-family
!
logging buffered 16000
no logging console
enable secret 5 <removed>
!
no aaa new-model
!
!
ip vrf MGMT
!
ip multicast-routing distributed
ip multicast-routing vrf TENANT_A distributed
ip multicast-routing vrf TENANT_B distributed
!
!
!
!
!
!
!
!
ip name-server 8.8.8.8
!
no ip domain lookup
ip domain name domain.local
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
otv site bridge-domain 9
!
otv site-identifier 0000.0000.0009
!
!
multilink bundle-name authenticated
!         
!
!
!
!
!
!
!
!
!
!
!
!
!
license udi pid CSR1000V sn 9YQL5RHXFT5
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
!

!
!
!
!
!
!
!
!
!
!
!
!






!
!
interface Loopback0
 description RID
 ip address 90.90.90.90 255.255.255.255
!
interface Overlay1
 description OTV OVERLAY
 no ip address
 otv control-group 225.0.0.1
 otv data-group 232.0.0.0/8
 otv join-interface GigabitEthernet1.3000
 no otv suppress arp-nd
 no mop enabled
 no mop sysid
 service instance 100 ethernet
  encapsulation dot1q 100
  bridge-domain 100
 !
 service instance 200 ethernet
  encapsulation dot1q 200
  bridge-domain 200
 !
!
interface Overlay2
 description OTV OVERLAY FOR TENANT_B
 no ip address
 otv control-group 225.0.0.2
 otv data-group 233.0.0.0/8
 otv join-interface GigabitEthernet1.3001
 no otv suppress arp-nd
 no mop enabled
 no mop sysid
 service instance 300 ethernet
  encapsulation dot1q 300
  bridge-domain 300
 !
 service instance 400 ethernet
  encapsulation dot1q 400
  bridge-domain 400
 !
!
interface GigabitEthernet1
 mtu 9216
 no ip address
 negotiation auto
 no mop enabled
 no mop sysid
!
interface GigabitEthernet1.3000
 description WAN TO PE_R15 - TENANT A VPLS
 encapsulation dot1Q 3000
 vrf forwarding TENANT_A
 ip address 172.16.3.2 255.255.255.248
 ip pim passive
 ip igmp version 3
!
interface GigabitEthernet1.3001
 description WAN TO PE_R15 - TENANT B VPLS
 encapsulation dot1Q 3001
 vrf forwarding TENANT_B
 ip address 192.168.3.2 255.255.255.248
 ip pim passive
 ip igmp version 3
!
interface GigabitEthernet2
 mtu 9216
 no ip address
 negotiation auto
 no mop enabled
 no mop sysid
!
interface GigabitEthernet2.91
 description TO VXLAN EVPN DC2_LEAF_03 - L3VNI TENANT B
 encapsulation dot1Q 910
 vrf forwarding TENANT_B
 ip address 192.168.3.250 255.255.255.252
!
interface GigabitEthernet2.900
 description TO VXLAN EVPN DC2_LEAF_03 - L3VNI TENANT A
 encapsulation dot1Q 900
 vrf forwarding TENANT_A
 ip address 172.16.3.250 255.255.255.252
!
interface GigabitEthernet3
 description OTV LAN - ALL TENANTS
 mtu 9216
 no ip address
 negotiation auto
 no mop enabled
 no mop sysid
 service instance 9 ethernet
  encapsulation untagged
  snmp ifindex persist
  bridge-domain 9
 !
 service instance 100 ethernet
  encapsulation dot1q 100
  snmp ifindex persist
  bridge-domain 100
 !
 service instance 200 ethernet
  encapsulation dot1q 200
  snmp ifindex persist
  bridge-domain 200
 !
 service instance 300 ethernet
  encapsulation dot1q 300
  bridge-domain 300
 !
 service instance 400 ethernet
  encapsulation dot1q 400
  snmp ifindex persist
  bridge-domain 400
 !
!
interface GigabitEthernet4
 description MANAGEMENT
 ip vrf forwarding MGMT
 ip address 10.1.1.229 255.255.255.0
 negotiation auto
 no mop enabled
 no mop sysid
!
!
router bgp 65001
 bgp router-id 90.90.90.90
 bgp log-neighbor-changes
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv4 vrf TENANT_A
  neighbor DC2_LEAF3_TENANT_A peer-group
  neighbor DC2_LEAF3_TENANT_A remote-as 65501
  neighbor DC2_LEAF3_TENANT_A description TO DC2 VXLAN TENANT A
  neighbor DC2_LEAF3_TENANT_A timers 7 21
  neighbor DC2_LEAF3_TENANT_A send-community both
  neighbor DC1_CE7_TENANT_A peer-group
  neighbor DC1_CE7_TENANT_A remote-as 65000
  neighbor DC1_CE7_TENANT_A description TO DC1 VXLAN TENANT A
  neighbor DC1_CE7_TENANT_A timers 7 21
  neighbor DC1_CE7_TENANT_A send-community both
  neighbor BRANCH_CE17_TENANT_A peer-group
  neighbor BRANCH_CE17_TENANT_A remote-as 65100
  neighbor BRANCH_CE17_TENANT_A description TO BRANCH CE17 TENANT A
  neighbor BRANCH_CE17_TENANT_A timers 7 21
  neighbor BRANCH_CE17_TENANT_A send-community both
  neighbor 172.16.3.1 peer-group DC1_CE7_TENANT_A
  neighbor 172.16.3.1 activate
  neighbor 172.16.3.3 peer-group BRANCH_CE17_TENANT_A
  neighbor 172.16.3.3 activate
  neighbor 172.16.3.249 peer-group DC2_LEAF3_TENANT_A
  neighbor 172.16.3.249 activate
 exit-address-family
 !
 address-family ipv4 vrf TENANT_B
  network 0.0.0.0
  neighbor DC2_LEAF3_TENANT_B peer-group
  neighbor DC2_LEAF3_TENANT_B remote-as 65501
  neighbor DC2_LEAF3_TENANT_B description TO DC2 VXLAN TENANT B
  neighbor DC2_LEAF3_TENANT_B timers 7 21
  neighbor DC2_LEAF3_TENANT_B send-community both
  neighbor DC1_CE7_TENANT_B peer-group
  neighbor DC1_CE7_TENANT_B remote-as 65000
  neighbor DC1_CE7_TENANT_B description TO DC1 VXLAN TENANT B
  neighbor DC1_CE7_TENANT_B timers 7 21
  neighbor DC1_CE7_TENANT_B send-community both
  neighbor BRANCH_CE17_TENANT_B peer-group
  neighbor BRANCH_CE17_TENANT_B remote-as 65100
  neighbor BRANCH_CE17_TENANT_B description TO BRANCH CE17 TENANT B
  neighbor BRANCH_CE17_TENANT_B timers 7 21
  neighbor BRANCH_CE17_TENANT_B send-community both
  neighbor 192.168.3.1 peer-group DC1_CE7_TENANT_B
  neighbor 192.168.3.1 activate
  neighbor 192.168.3.3 peer-group BRANCH_CE17_TENANT_B
  neighbor 192.168.3.3 activate
  neighbor 192.168.3.249 peer-group DC2_LEAF3_TENANT_B
  neighbor 192.168.3.249 activate
 exit-address-family
!
!
virtual-service csr_mgmt
!
no ip forward-protocol nd
ip pim ssm default
ip pim vrf TENANT_A ssm default
ip pim vrf TENANT_B ssm default
no ip http server
ip http authentication local
no ip http secure-server
!
ip bgp-community new-format
ip tftp source-interface GigabitEthernet4
ip route vrf MGMT 0.0.0.0 0.0.0.0 10.1.1.254 name MGMT_DEFAULT
!
!
!
!
!
!
!
control-plane
!
 !
 !
 !
 !
!
!
!
!
!
line con 0
 logging synchronous
 stopbits 1
line vty 0 4
 exec-timeout 60 0
 logging synchronous
 login local
 transport input telnet ssh
!
ntp server pool.ntp.org
!
!
!
!
!
end


DC2_LEAF_03

version 7.0(3)I6(1)
hostname DC2_LEAF_03
vdc DC2_LEAF_03 id 1
  limit-resource vlan minimum 16 maximum 4094
  limit-resource vrf minimum 2 maximum 4096
  limit-resource port-channel minimum 0 maximum 511
  limit-resource u4route-mem minimum 248 maximum 248
  limit-resource u6route-mem minimum 96 maximum 96
  limit-resource m4route-mem minimum 58 maximum 58
  limit-resource m6route-mem minimum 8 maximum 8

feature telnet
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay

no password strength-check
username admin password 5 <removed>  role network-admin
no ip domain-lookup
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:12:41:178:194:165
snmp-server user admin auth md5 <removed> priv <removed> localizedkey engineID 128
:0:0:9:3:0:80:86:160:176:48
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO

vlan 1,100,200,300,400,900,910
ip pim rp-address 101.101.101.101 group-list 225.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 100
  vn-segment 10000
vlan 200
  vn-segment 20000
vlan 300
  vn-segment 30000
vlan 400
  vn-segment 40000
vlan 900
  vn-segment 90000
vlan 910
  vn-segment 91000

vrf context TENANT_A
  vni 90000
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
vrf context TENANT_B
  vni 91000
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
vrf context management
  ip route 0.0.0.0/0 10.1.1.254 name MGMT_DEFAULT

interface Vlan1

interface Vlan900
  description TENANT_A VL900 L3VNI
  no shutdown
  mtu 9216
  vrf member TENANT_A
  ip forward

interface Vlan910
  description TENANT_B VL910 L3VNI
  no shutdown
  mtu 9216
  vrf member TENANT_B
  ip forward

interface nve1
  no shutdown
  description VTEP_03
  source-interface loopback0
  host-reachability protocol bgp
  member vni 10000
    suppress-arp
    mcast-group 225.1.0.1
  member vni 20000
    suppress-arp
    mcast-group 225.2.0.1
  member vni 30000
    suppress-arp
    mcast-group 225.3.0.1
  member vni 40000
    suppress-arp
    mcast-group 225.4.0.1
  member vni 90000 associate-vrf
  member vni 91000 associate-vrf

interface Ethernet1/1
  description TO SPINE_01
  no switchport
  mtu 9216
  ip address 10.1.0.18/30
  ip ospf network point-to-point
  no ip ospf passive-interface
  ip router ospf 101 area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet1/2
  description TO SPINE_02
  no switchport
  mtu 9216
  ip address 10.1.0.22/30
  ip ospf network point-to-point
  no ip ospf passive-interface
  ip router ospf 101 area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet1/3
  shutdown
  mtu 9216

interface Ethernet1/4
  description TO CE_R09 - OTV LAN
  switchport mode trunk
  switchport trunk allowed vlan 100,200,300,400
  mtu 9216

interface Ethernet1/5
  description TO CE9 - EXT L3
  no switchport
  mtu 9216
  no shutdown

interface Ethernet1/5.90
  description TO CE9 - TENANT A L3VNI EXT L3
  mtu 9216
  encapsulation dot1q 900
  vrf member TENANT_A
  ip address 172.16.3.249/30
  no shutdown

interface Ethernet1/5.91
  description TO CE9 - TENANT B L3VNI EXT L3
  mtu 9216
  encapsulation dot1q 910
  vrf member TENANT_B
  ip address 192.168.3.249/30
  no shutdown

interface mgmt0
  description MANAGEMENT
  vrf member management
  ip address 10.1.1.219/24

interface loopback0
  description RID - VTEP
  ip address 10.10.10.10/32
  ip router ospf 101 area 0.0.0.0
  ip pim sparse-mode

line console
  exec-timeout 60
line vty
  exec-timeout 60
boot nxos bootflash:/nxos.7.0.3.I6.1.bin 
router ospf 101
  router-id 10.10.10.10
  passive-interface default
router bgp 65501
  router-id 10.10.10.10
  neighbor 6.6.6.6
    remote-as 65501
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 7.7.7.7
    remote-as 65501
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  vrf TENANT_A
    timers bgp 7 21
    address-family ipv4 unicast
      network 172.16.1.0/24
      advertise l2vpn evpn
    neighbor 172.16.3.250
      remote-as 65001
      address-family ipv4 unicast
        send-community
        send-community extended
  vrf TENANT_B
    address-family ipv4 unicast
      advertise l2vpn evpn
    neighbor 192.168.3.250
      remote-as 65001
      address-family ipv4 unicast
        send-community
        send-community extended
evpn
  vni 10000 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 20000 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 30000 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 40000 l2
    rd auto
    route-target import auto
    route-target export auto



OTV Lab Specific Configurations

Enable multicast on OTV edge device (if using Multicast mode).

ip multicast-routing vrf TENANT_A distributed
!
ip pim ssm default
ip pim vrf TENANT_A ssm default


Define the OTV site identifier and bridge domain.

otv site bridge-domain 7
!
otv site-identifier 0000.0000.0007


Enable multicast on OTV join interface.

interface GigabitEthernet1.3000
 description WAN TO PE_R13 - TENANT A VPLS
 encapsulation dot1Q 3000
 vrf forwarding TENANT_A
 ip address 172.16.3.1 255.255.255.248
 ip pim passive
 ip igmp version 3


On the OTV internal interface, create the service instance for both the site ID and extended VLANs.

interface GigabitEthernet3
 description OTV LAN - ALL TENANTS
 mtu 9216
 no ip address
 negotiation auto
 no mop enabled
 no mop sysid
 service instance 7 ethernet
  encapsulation untagged
  bridge-domain 7
 !
 service instance 100 ethernet
  encapsulation dot1q 100
  bridge-domain 100
 !
 service instance 200 ethernet
  encapsulation dot1q 200
  bridge-domain 200


Create the Overlay interface with multicast enabled.  Configure the join interface and the service instances for the extended VLANs.

interface Overlay1
 description OTV OVERLAY FOR TENANT_A
 no ip address
 otv control-group 225.0.0.1
 otv data-group 232.0.0.0/8
 otv join-interface GigabitEthernet1.3000
 no otv suppress arp-nd
 no mop enabled
 no mop sysid
 service instance 100 ethernet
  encapsulation dot1q 100
  bridge-domain 100
 !
 service instance 200 ethernet
  encapsulation dot1q 200
  bridge-domain 200



Testing Results

As proof of concept, testing was performed from within a single tenant only (TENANT_A).  The test protocol was for Host4 in DC1 to ping and trace hosts in its own VXLAN fabric as well as hosts in the remote DC2 fabric.

  • Host4's IP and MAC Address.
Fig. 4.1
  • Host4’s ping results to all members of DC1 and DC2.
Fig. 4.2

  • Host4 ping to branch site.

Fig. 4.3
  • Arp cache for Host4.

Fig. 4.4

  • Host4 traceroutes to hosts in DC1.

Fig. 4.5

  • Host4 traceroutes to hosts in DC2.
Fig. 4.6


Verification Tasks

The list below outlined the OTV verification tasks for only TENANT_A.
  • OTV Overlay Interface Status
  • OTV Adjacency
  • OTV Control Plane Protocol Status
  • OTV Multicast
  • OTV VLANs
  • OTV Routes
  • OTV Bridge-Domain

OTV Overlay Interface Status

CE_R07#sh otv
Overlay Interface Overlay1
 VPN name                 : None
 VPN ID                   : 1
 State                    : UP
 Fwd-capable              : Yes
 Fwd-ready                : Yes
 AED-Server               : Yes
 Backup AED-Server        : No
 AED Capable              : Yes
 IPv4 control group       : 225.0.0.1
 Mcast data group range(s): 232.0.0.0/8
 Join interface(s)        : GigabitEthernet1.3000
 Join IPv4 address        : 172.16.3.1
 Tunnel interface(s)      : Tunnel0
 Encapsulation format     : GRE/IPv4
 Site Bridge-Domain       : 7
 Capability               : Multicast-reachable
 Is Adjacency Server      : No
 Adj Server Configured    : No
 Prim/Sec Adj Svr(s)      : None


OTV Adjacency

CE_R07#sh otv adjacency
Overlay Adjacency Database for overlay 1
Hostname                       System-ID      Dest Addr       Site-ID        Up Time   State
CE_R09                         001e.bdcd.c000 172.16.3.2      0000.0000.0009 2d23h     UP  


OTV Control Plane Protocol Status

CE_R07#sh otv isis neighbors detail

Tag Overlay1:
System Id       Type Interface     IP Address      State Holdtime Circuit Id
CE_R09          L1   Ov1           172.16.3.2      UP    7        CE_R09.01         
  Area Address(es): 00
  SNPA: 001e.bdcd.c0bc     
  State Changed: 2d23h
  LAN Priority: 64
  Format: Phase V
  Remote TID: 0
  Local TID:  0
  Interface name: Overlay1
  Site ID: 0000.0000.0009
   Forward Ready
  Version: 1
  Version Mismatch: 0  AED Server Info:
  Capability: 1
  Priority: 0
  AED Server Elected Value: 001E.BDCD.C000
  AED State: 0
  AED Elected Operational: 4

  Backup AED Server Info:
  Backup AED Server Elected Value: 0000.0000.0000
  Backup AED State: 0
  Backup AED Elected Operational: 0

Tag Site:
System Id       Type Interface     IP Address      State Holdtime Circuit Id


OTV Multicast

CE_R07#sh ip mroute vrf TENANT_A
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group,
       G - Received BGP C-Mroute, g - Sent BGP C-Mroute,
       N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed,
       Q - Received BGP S-A Route, q - Sent BGP S-A Route,
       V - RD & Vector, v - Vector, p - PIM Joins on route,
       x - VxLAN group
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 225.0.0.1), 3w0d/stopped, RP 0.0.0.0, flags: DC
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Tunnel0, Forward/Sparse-Dense, 3w0d/00:02:19
    GigabitEthernet1.3000, Forward/Sparse-Dense, 3w0d/Proxy

(172.16.3.2, 225.0.0.1), 3d01h/00:01:34, flags: T
  Incoming interface: GigabitEthernet1.3000, RPF nbr 0.0.0.0
  Outgoing interface list:
    Tunnel0, Forward/Sparse-Dense, 3d01h/00:02:19

(172.16.3.1, 225.0.0.1), 3w0d/00:02:37, flags: T
  Incoming interface: GigabitEthernet1.3000, RPF nbr 0.0.0.0
  Outgoing interface list:
    Tunnel0, Forward/Sparse-Dense, 3w0d/00:02:19
    GigabitEthernet1.3000, Forward/Sparse-Dense, 3w0d/Proxy


CE_R07#sh ip mroute vrf TENANT_A count
Use "show ip mfib count" to get better response time for a large number of mroutes.

IP Multicast Statistics
3 routes using 3060 bytes of memory
1 groups, 2.00 average sources per group
Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per second
Other counts: Total/RPF failed/Other drops(OIF-null, rate-limit etc)

Group: 225.0.0.1, Source count: 2, Packets forwarded: 436026, Packets received: 436027
  Source: 172.16.3.2/32, Forwarding: 197757/0/862/5, Other: 197757/0/0
  Source: 172.16.3.1/32, Forwarding: 238269/0/1327/2, Other: 238269/0/0


OTV VLANs

CE_R07#sh otv vlan
Key:  SI - Service Instance, NA - Non AED, NFC - Not Forward Capable.

Overlay 1 VLAN Configuration Information
 Inst VLAN BD   Auth ED              State                Site If(s)         
 0    100  100  *CE_R07              active              Gi3:SI100
 0    200  200  *CE_R07              active              Gi3:SI200
 Total VLAN(s): 2


OTV Routes

CE_R07#sh otv route

Codes: BD - Bridge-Domain, AD - Admin-Distance,
       SI - Service Instance, * - Backup Route

OTV Unicast MAC Routing Table for Overlay1

 Inst VLAN BD     MAC Address    AD    Owner  Next Hops(s)
----------------------------------------------------------
 0    100  100    000c.2941.dc19 50    ISIS   CE_R09
 0    100  100    000c.2977.ea21 40    BD Eng Gi3:SI100
 0    100  100    000c.29d2.4ebe 50    ISIS   CE_R09
 0    200  200    0000.ed01.ed01 50    ISIS   CE_R09

4 unicast routes displayed in Overlay1

----------------------------------------------------------
4 Total Unicast Routes Displayed


OTV Bridge-Domain

CE_R07#sh bridge-domain
Bridge-domain 7 (1 ports in all)
State: UP                    Mac learning: Enabled
Aging-Timer: 300 second(s)
    GigabitEthernet3 service instance 7
   AED MAC address    Policy  Tag       Age  Pseudoport

Bridge-domain 100 (2 ports in all)
State: UP                    Mac learning: Enabled
Aging-Timer: 1800 second(s)
    GigabitEthernet3 service instance 100
    Overlay1 service instance 100
   AED MAC address    Policy  Tag       Age  Pseudoport
   1   000C.2941.DC19 forward static_r  0    OCE_PTR:0xe929d000
   1   000C.2977.EA21 forward dynamic_c 1225 GigabitEthernet3.EFP100
   1   000C.29D2.4EBE forward static_r  0    OCE_PTR:0xe929d000
   1   0000.ED01.ED01 forward dynamic_c 1775 GigabitEthernet3.EFP100

Bridge-domain 200 (2 ports in all)
State: UP                    Mac learning: Enabled
Aging-Timer: 1800 second(s)
    GigabitEthernet3 service instance 200
    Overlay1 service instance 200
   AED MAC address    Policy  Tag       Age  Pseudoport
   1   000C.29A8.ACDE forward dynamic_c 1434 GigabitEthernet3.EFP200
   1   000C.29AD.8611 forward static_r  0    OCE_PTR:0xe929d020
   1   0000.ED01.ED01 forward static_r  0    OCE_PTR:0xe929d020
   1   000C.29B7.B1D3 forward dynamic_c 1745 GigabitEthernet3.EFP200



Packet Walk
  • A technical step by step packet walk diagram.


Fig. 5.1

In this example packet walk, we will look at host connectivity within a single tenant (TENANT_A) but will cross datacenters fabrics using either OTV (L2) or external routing (L3).  The walkthroughs will show MAC, IP and OTV route lookups for the following scenarios.

  • Host4 172.16.1.1 in DC1 sends data to Host8 172.16.1.3 in DC2 in the same VLAN (100 to 100)
    • Host4 MAC: 000c.2977.ea21
    • Host8 MAC: 000c.29d2.4ebe
  • Host4 172.16.1.1 in DC1 sends data to Host10 172.16.2.3 in DC2 in a different VLAN (100 to 200)


Host4 to Host8 (VLAN 100 to 100)
  • From DC1 Leaf01, Host4 is locally attached to interface E1/4
  • From DC1 Leaf01, to reach Host8 use VXLAN to 5.5.5.5 (L2VNI to Leaf03)

DC1_LEAF_01# show system internal l2fwder mac
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*   400    001e.bd7a.d8bc    static   -          F     F  (0x47000002) nve-peer2 5.5.5.5 
*   300    001e.bd7a.d8bc    static   -          F     F  (0x47000002) nve-peer2 5.5.5.5 
*   100    000c.2977.ea21   dynamic   00:06:04   F     F     Eth1/4 
*   100    000c.2998.c13c    static   -          F     F  (0x47000001) nve-peer1 4.4.4.4 
*   200    000c.29a8.acde   dynamic   00:06:21   F     F     Eth1/5 
G   200    000c.2910.ac2a    static   -          F     F   sup-eth1(R)
*   100    000c.29d2.4ebe    static   -          F     F  (0x47000002) nve-peer2 5.5.5.5 
*   200    000c.29ad.8611    static   -          F     F  (0x47000002) nve-peer2 5.5.5.5 
*   200    001e.bd7a.d8bc    static   -          F     F  (0x47000002) nve-peer2 5.5.5.5 
G   100    000c.2910.ac2a    static   -          F     F   sup-eth1(R)
*   200    000c.292d.161b    static   -          F     F  (0x47000001) nve-peer1 4.4.4.4 
G   400    000c.2910.ac2a    static   -          F     F   sup-eth1(R)
*   100    001e.bd7a.d8bc    static   -          F     F  (0x47000002) nve-peer2 5.5.5.5 
G   300    000c.2910.ac2a    static   -          F     F   sup-eth1(R)
    1           1         -00:00:ed:01:ed:01         -             1

  • From DC1 Leaf03, to reach Host8 use OTV internal interface E1/4 (to CE_R07)
DC1_LEAF_03# show system internal l2fwder mac
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*   400    001e.bd7a.d8bc   dynamic   00:02:21   F     F     Eth1/4 
*   300    001e.bd7a.d8bc   dynamic   00:03:21   F     F     Eth1/4 
*   100    000c.2977.ea21    static   -          F     F  (0x47000002) nve-peer2 3.3.3.3 
*   100    000c.2998.c13c    static   -          F     F  (0x47000001) nve-peer1 4.4.4.4 
*   200    000c.29a8.acde    static   -          F     F  (0x47000002) nve-peer2 3.3.3.3 
*   100    000c.29d2.4ebe   dynamic   00:00:42   F     F     Eth1/4 
*   200    000c.29ad.8611   dynamic   00:04:16   F     F     Eth1/4 
*   200    001e.bd7a.d8bc   dynamic   00:02:51   F     F     Eth1/4 
*   200    000c.292d.161b    static   -          F     F  (0x47000001) nve-peer1 4.4.4.4 
*   100    001e.bd7a.d8bc   dynamic   00:02:21   F     F     Eth1/4 

  • From CE_R07, to reach Host8 use OTV overlay interface to CE_R09
CE_R07#sh otv route

Codes: BD - Bridge-Domain, AD - Admin-Distance,
       SI - Service Instance, * - Backup Route

OTV Unicast MAC Routing Table for Overlay1

 Inst VLAN BD     MAC Address    AD    Owner  Next Hops(s)
----------------------------------------------------------
 0    100  100    000c.2977.ea21 40    BD Eng Gi3:SI100
 0    100  100    000c.29d2.4ebe 50    ISIS   CE_R09
 0    200  200    000c.292d.161b 40    BD Eng Gi3:SI200
 0    200  200    000c.29a8.acde 40    BD Eng Gi3:SI200
 0    200  200    000c.29ad.8611 50    ISIS   CE_R09

5 unicast routes displayed in Overlay1

----------------------------------------------------------
5 Total Unicast Routes Displayed

  • From CE_R09, to reach Host8 use OTV internal interface Service Instance 100 (to DC2 Leaf03)
CE_R09#sh otv route

Codes: BD - Bridge-Domain, AD - Admin-Distance,
       SI - Service Instance, * - Backup Route

OTV Unicast MAC Routing Table for Overlay1

 Inst VLAN BD     MAC Address    AD    Owner  Next Hops(s)
----------------------------------------------------------
 0    100  100    000c.2977.ea21 50    ISIS   CE_R07
 0    100  100    000c.2998.c13c 50    ISIS   CE_R07
 0    100  100    000c.29d2.4ebe 40    BD Eng Gi3:SI100
 0    200  200    000c.292d.161b 50    ISIS   CE_R07
 0    200  200    000c.29a8.acde 50    ISIS   CE_R07
 0    200  200    000c.29ad.8611 40    BD Eng Gi3:SI200

6 unicast routes displayed in Overlay1

----------------------------------------------------------
6 Total Unicast Routes Displayed

  • From DC2’s Leaf03, to reach Host8 use VXLAN to 8.8.8.8 (L2VNI to DC2’s Leaf01)
DC2_LEAF_03# show system internal l2fwder mac
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*   400    001e.bdcd.c0bc   dynamic   00:00:53   F     F     Eth1/4 
*   100    000c.2977.ea21   dynamic   00:03:54   F     F     Eth1/4 
*   300    001e.bdcd.c0bc   dynamic   00:00:23   F     F     Eth1/4 
*   200    000c.29a8.acde   dynamic   00:07:25   F     F     Eth1/4 
*   100    000c.29d2.4ebe    static   -          F     F  (0x47000001) nve-peer1 8.8.8.8 
*   200    000c.29ad.8611    static   -          F     F  (0x47000002) nve-peer2 9.9.9.9 
*   200    000c.292d.161b   dynamic   00:07:23   F     F     Eth1/4

  • From DC2’s Leaf01, Host8 is locally attached to interface E1/4
DC2_LEAF_01# show system internal l2fwder mac
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*   200    001e.bdcd.c0bc    static   -          F     F  (0x47000002) nve-peer2 10.10.1 
*   100    001e.bdcd.c0bc    static   -          F     F  (0x47000002) nve-peer2 10.10.1 
*   400    001e.bdcd.c0bc    static   -          F     F  (0x47000002) nve-peer2 10.10.1 
G   400    000c.29de.a659    static   -          F     F   sup-eth1(R)
*   100    000c.2977.ea21    static   -          F     F  (0x47000002) nve-peer2 10.10.1 
G   300    000c.29de.a659    static   -          F     F   sup-eth1(R)
*   300    001e.bdcd.c0bc    static   -          F     F  (0x47000002) nve-peer2 10.10.1 
*   200    000c.29a8.acde    static   -          F     F  (0x47000002) nve-peer2 10.10.1 
*   100    000c.29d2.4ebe   dynamic   00:12:14   F     F     Eth1/4 
*   200    000c.29ad.8611    static   -          F     F  (0x47000001) nve-peer1 9.9.9.9 
G   200    000c.29de.a659    static   -          F     F   sup-eth1(R)
*   200    000c.292d.161b    static   -          F     F  (0x47000002) nve-peer2 10.10.1 
G   100    000c.29de.a659    static   -          F     F   sup-eth1(R)
G   900    000c.29de.a659    static   -          F     F   sup-eth1(R)
G   910    000c.29de.a659    static   -          F     F   sup-eth1(R)
    1           1         -00:00:ed:01:ed:01         -             1



Host4 to Host10 (VLAN 100 to 200)
  • From DC1 Leaf01, Host4 is locally attached to VLAN 100
  • From DC1 Leaf01, to reach Host10 use VXLAN to 5.5.5.5 (L3VNI to Leaf03)
DC1_LEAF_01# sh ip route vrf TENANT_A 172.16.1.1
IP Route Table for VRF "TENANT_A"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

172.16.1.1/32, ubest/mbest: 1/0, attached
    *via 172.16.1.1, Vlan100, [190/0], 4d16h, hmm

DC1_LEAF_01# sh ip route vrf TENANT_A 172.16.2.3
IP Route Table for VRF "TENANT_A"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

172.16.2.3/32, ubest/mbest: 1/0
    *via 5.5.5.5%default, [200/0], 02:03:57, bgp-65500, internal, tag 65000 (evpn) segid: 90000 tunnelid: 0x5050505 encap: VXLAN

  • From DC1 Leaf03, to reach Host10 use L3 routing (to CE_R07)
DC1_LEAF_03# sh ip route vrf TENANT_A 172.16.2.3
IP Route Table for VRF "TENANT_A"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

172.16.2.3/32, ubest/mbest: 1/0
    *via 172.16.3.254, [20/0], 01:31:49, bgp-65500, external, tag 65000

  • From CE_R07, to reach Host10 use L3 routing (to CE_R09)
CE_R07#sh ip route vrf TENANT_A 172.16.2.3

Routing Table: TENANT_A
Routing entry for 172.16.2.3/32
  Known via "bgp 65000", distance 20, metric 0
  Tag 65001, type external
  Last update from 172.16.3.2 02:01:50 ago
  Routing Descriptor Blocks:
  * 172.16.3.2, from 172.16.3.2, 02:01:50 ago
      Route metric is 0, traffic share count is 1
      AS Hops 2
      Route tag 65001
      MPLS label: none

  • From CE_R09, to reach Host10 use L3 routing (to DC2 Leaf03)
CE_R09#sh ip route vrf TENANT_A 172.16.2.3

Routing Table: TENANT_A
Routing entry for 172.16.2.3/32
  Known via "bgp 65001", distance 20, metric 0
  Tag 65501, type external
  Last update from 172.16.3.249 02:00:31 ago
  Routing Descriptor Blocks:
  * 172.16.3.249, from 172.16.3.249, 02:00:31 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 65501
      MPLS label: none

  • From DC2 Leaf03, to reach Host10 use VXLAN to 9.9.9.9 (L3VNI to Leaf02)
DC2_LEAF_03# sh ip route vrf TENANT_A 172.16.2.3
IP Route Table for VRF "TENANT_A"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

172.16.2.3/32, ubest/mbest: 1/0
    *via 9.9.9.9%default, [200/0], 4d16h, bgp-65501, internal, tag 65501 (evpn) segid: 90000 tunnelid: 0x9090909 encap: VXLAN

  • From DC2 Leaf02, Host10 is locally attached to VLAN 200
DC2_LEAF_02# sh ip route vrf TENANT_A 172.16.2.3
IP Route Table for VRF "TENANT_A"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

172.16.2.3/32, ubest/mbest: 1/0, attached
    *via 172.16.2.3, Vlan200, [190/0], 4d16h, hmm



Issues

OTV over MPLS L3VPN

When I initially tried to connect the CE devices to a MPLS L3VPN network as IP transport, I could not get the OTV adjacency to work in multicast mode.  When I switched the WAN service to VPLS (L2), it worked fine.

Based on what I been reading, I believe the underlying IP transport network must have IP Multicast enabled first (see link).  When the OTV documentation specifies that you can use “any transport”, I thought any IP routed network at a minimum would suffice, not requiring anything extra such as the complexities of multicast.  However, one can use OTV in unicast mode, which is far simpler to deploy but can limit the number of datacenters that can be interconnected.



InterVLAN routing across OTV

During my testing, I’ve noticed that I had no problems pinging every host from DC1 to DC2 in the same VLAN over OTV (e.g., VL100 to VL100).  It worked as expected.  However when I pinged a host in another VLAN (e.g., VL100 to VL200), it failed. The inter-vlan routing over OTV wasn’t working.  MAC learning wasn’t taking place as I thought it should.  The only way to make this work was to have a host in DC1 VL200 ping across OTV to another host in DC2 in the same VLAN (or vice versa).  Once that was initiated, DC1’s VXLAN fabric created a locally attached route (since its MAC address was learned via OTV) and then MP-BGP would generate a type 2 route and share this to the rest of the fabric.  The problem would arise again when that ARP entry ages out and the MP-BGP route would disappear.

After doing some research, I found an article that specifically addresses this issue.  It states that inter-vlan routing in a multi-fabric setup using OTV should use the MP-BGP route type 5 that was learned from the other DC fabric.


So I reconfigured my lab to ensure the EVPN MP-BGP /32 routes were being shared across the two fabrics.  However when I did that, I observed a strange routing issue even though there wasn’t any apparent connectivity issues.

What I saw was that a host in DC1 (172.16.1.1) could possibly take 2 different paths to a host in DC2 (172.16.2.3) that were not in the same VLAN.  One path was via the external L3 (not using OTV) and the other was via the L2 extended OTV path.

Taking one or the other would depend on OTV's MAC learning for the DC2 host.  If the DC2's host MAC hasn't been learned over OTV, the DC1 host (172.16.1.1) would take the L3 path to reach the host in DC2 (172.16.2.3).  This would seem like normal operation based on the Cisco article. However, that same host would take the L2 path if OTV did learn of DC2' host MAC address. 

At this point I’m not entirely sure if this is normal behavior for a multi-fabric setup but the traceroute below will show this behavior.  The top is the L3 path while the bottom is the L2 path.



Here’s a look at DC1 fabric’s border leaf route table before OTV has learned 172.16.2.3’s MAC address.  It’s clearly going out the external router path, using the L3VNI and the BGP route type 5.


DC1_LEAF_03# sh ip route vrf TENANT_A
IP Route Table for VRF "TENANT_A"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

0.0.0.0/0, ubest/mbest: 1/0
    *via 172.16.3.254, [20/0], 4d01h, bgp-65500, external, tag 65000
172.16.1.0/24, ubest/mbest: 1/0, attached
    *via 172.16.1.254, Vlan100, [0/0], 3w1d, direct
172.16.1.1/32, ubest/mbest: 1/0
    *via 3.3.3.3%default, [200/0], 4d00h, bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x3030303 encap: VXLAN

172.16.1.2/32, ubest/mbest: 1/0
    *via 4.4.4.4%default, [200/0], 3w0d, bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x4040404 encap: VXLAN

172.16.1.3/32, ubest/mbest: 1/0
    *via 172.16.3.254, [20/0], 02:09:55, bgp-65500, external, tag 65000
172.16.1.254/32, ubest/mbest: 1/0, attached
    *via 172.16.1.254, Vlan100, [0/0], 3w1d, local
172.16.2.0/24, ubest/mbest: 1/0, attached
    *via 172.16.2.254, Vlan200, [0/0], 3w1d, direct
172.16.2.1/32, ubest/mbest: 1/0
    *via 3.3.3.3%default, [200/0], 3w0d, bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x3030303 encap: VXLAN

172.16.2.2/32, ubest/mbest: 1/0
    *via 4.4.4.4%default, [200/0], 3w0d, bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x4040404 encap: VXLAN

172.16.2.3/32, ubest/mbest: 1/0
    *via 172.16.3.254, [20/0], 02:09:55, bgp-65500, external, tag 65000
172.16.2.254/32, ubest/mbest: 1/0, attached
    *via 172.16.2.254, Vlan200, [0/0], 3w1d, local
172.16.3.252/30, ubest/mbest: 1/0, attached
    *via 172.16.3.253, Eth1/5.90, [0/0], 3w1d, direct
172.16.3.253/32, ubest/mbest: 1/0, attached
    *via 172.16.3.253, Eth1/5.90, [0/0], 3w1d, local
172.16.100.0/24, ubest/mbest: 1/0
    *via 172.16.3.254, [20/0], 1w1d, bgp-65500, external, tag 65000


BGP Route Type 5

DC1_LEAF_03# sh bgp l2vpn evpn 172.16.2.3
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 5.5.5.5:3    (L3VNI 90000)
BGP routing table entry for [5]:[0]:[0]:[32]:[172.16.2.3]:[0.0.0.0]/224, version 211518
Paths: (1 available, best #1)
Flags: (0x000002) on xmit-list, is not in l2rib/evpn

  Advertised path-id 1
  Path type: local, path is valid, is best path, no labeled nexthop
  AS-Path: 65000 65001 65501 , path sourced external to AS
    5.5.5.5 (metric 0) from 0.0.0.0 (5.5.5.5)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 90000
      Extcommunity:  RT:65500:90000 ENCAP:8 Router MAC:000c.29b7.b1d3

  Path-id 1 advertised to peers:
    1.1.1.1            2.2.2.2 


On the other hand, if another host in the same VLAN (say 172.16.2.1) initiates traffic to 172.16.2.3, MAC learning via OTV will take place and will populate DC1’s fabric with a locally attached route. MP-BGP would create a type 2 route and share it within that fabric.  Any further traffic will now follow the L2VNI to the OTV overlay.

DC1_LEAF_03# sh ip route vrf TENANT_A
IP Route Table for VRF "TENANT_A"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

0.0.0.0/0, ubest/mbest: 1/0
    *via 172.16.3.254, [20/0], 00:03:45, bgp-65500, external, tag 65000
172.16.1.0/24, ubest/mbest: 1/0, attached
    *via 172.16.1.254, Vlan100, [0/0], 00:03:47, direct
172.16.1.1/32, ubest/mbest: 1/0
    *via 3.3.3.3%default, [200/0], 00:03:46, bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x3030303 encap: VXLAN

172.16.1.2/32, ubest/mbest: 1/0
    *via 4.4.4.4%default, [200/0], 00:03:46, bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x4040404 encap: VXLAN

172.16.1.3/32, ubest/mbest: 1/0
    *via 172.16.3.254, [20/0], 00:03:45, bgp-65500, external, tag 65000
172.16.1.254/32, ubest/mbest: 1/0, attached
    *via 172.16.1.254, Vlan100, [0/0], 00:03:47, local
172.16.2.0/24, ubest/mbest: 1/0, attached
    *via 172.16.2.254, Vlan200, [0/0], 00:03:47, direct
172.16.2.1/32, ubest/mbest: 1/0
    *via 3.3.3.3%default, [200/0], 00:03:46, bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x3030303 encap: VXLAN

172.16.2.2/32, ubest/mbest: 1/0
    *via 4.4.4.4%default, [200/0], 00:03:46, bgp-65500, internal, tag 65500 (evpn) segid: 90000 tunnelid: 0x4040404 encap: VXLAN

172.16.2.3/32, ubest/mbest: 1/0, attached
    *via 172.16.2.3, Vlan200, [190/0], 00:00:13, hmm
172.16.2.254/32, ubest/mbest: 1/0, attached
    *via 172.16.2.254, Vlan200, [0/0], 00:03:47, local
172.16.3.252/30, ubest/mbest: 1/0, attached
    *via 172.16.3.253, Eth1/5.90, [0/0], 00:03:47, direct
172.16.3.253/32, ubest/mbest: 1/0, attached
    *via 172.16.3.253, Eth1/5.90, [0/0], 00:03:47, local
172.16.100.0/24, ubest/mbest: 1/0
    *via 172.16.3.254, [20/0], 00:03:45, bgp-65500, external, tag 65000



Now it’s a route type 2

DC1_LEAF_03# sh bgp l2vpn evpn 172.16.2.3
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 5.5.5.5:32967    (L2VNI 20000)
BGP routing table entry for [2]:[0]:[0]:[48]:[000c.29ad.8611]:[32]:[172.16.2.3]/272, version 217874
Paths: (1 available, best #1)
Flags: (0x000102) on xmit-list, is not in l2rib/evpn

  Advertised path-id 1
  Path type: local, path is valid, is best path, no labeled nexthop
  AS-Path: NONE, path locally originated
    5.5.5.5 (metric 0) from 0.0.0.0 (5.5.5.5)
      Origin IGP, MED not set, localpref 100, weight 32768
      Received label 20000 90000
      Extcommunity:  RT:65500:20000 RT:65500:90000 ENCAP:8 MAC Mobility Sequence:00:1 Router MAC:000c.29b7.b1d3

  Path-id 1 advertised to peers:
    1.1.1.1            2.2.2.2       

Route Distinguisher: 5.5.5.5:3    (L3VNI 90000)
BGP routing table entry for [5]:[0]:[0]:[32]:[172.16.2.3]:[0.0.0.0]/224, version 211518
Paths: (1 available, best #1)
Flags: (0x000002) on xmit-list, is not in l2rib/evpn

  Advertised path-id 1
  Path type: local, path is valid, is best path, no labeled nexthop
  AS-Path: 65000 65001 65501 , path sourced external to AS
    5.5.5.5 (metric 0) from 0.0.0.0 (5.5.5.5)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 90000
      Extcommunity:  RT:65500:90000 ENCAP:8 Router MAC:000c.29b7.b1d3

  Path-id 1 advertised to peers:
    1.1.1.1            2.2.2.2 


Again, I don’t know if this is an actual problem because we are still achieving reachability to all hosts.  It just seems strange to see traffic taking different paths depending on the state of the MAC table.  To me this doesn’t seem right and will have to take a deeper look.


UPDATE (08/25/17):  I believe I solved the inter-vlan problem.  I read that the server SVI interfaces and the anycast gateway MAC address do not need to be configured on the border leaf nodes.  I guess that makes sense.  Technically, we don’t need those since the border leaf is a dedicated node for external services.


After removing those pieces off both DC’s border leafs, the ping and trace results looked consistent. In other words, any inter-vlan L3 traffic went over the L3 path and L2 traffic went over OTV.


The post below does an excellent job doing a detailed packet walk for L2 and L3 traffic across multi-fabric VXLAN over OTV.

http://yves-louis.com/DCI/?p=1487




References


Cisco Live presentation on OTV



A great write up on the inter-workings of OTV



Another good Blog on OTV



Cisco references on OTV