Sunday, November 19, 2017

MPLS L2VPN (PBB-EVPN)

After getting myself some exposure to EVPN by testing technologies such as VXLAN EVPN and EVPN VPWS, I wanted to take it a bit further to learn more about the EVPN with MPLS.  Studying EVPN VPWS did help me understand EVPN’s BGP control plane operation and behavior, however it was a bit simplified and therefore felt a bit cheated since it didn't fully utilize the features of BGP.  With that in mind, my next obvious step was to build and test out a multi-point EVPN solution following RFC 7432.  I’d figured going through the process of building this lab would allow me to maximize my understanding of the BGP control plane and MPLS data plane operations.

According to my research, it appeared that Cisco never released EVPN from a traditional RFC 7432 perspective.  The only EVPN documentation that I found in the Cisco world was EVPN with Provider Backbone Bridging (RFC 7623), otherwise known as PBB-EVPN.  I guess Cisco found a better use case to go with PBB-EVPN rather than to stick with the traditional EVPN and decided to only integrate that into their code?  In any case, I’m always up for learning new technologies, so I took some time to lab this up.

As with my other posts, my focus was to look at the overall configuration and operational side of things.  I’ll do my best to explain the gist of the technology but since there are so many well written technical articles on PBB-EVPN, I would suggest reading those to get a good understanding of the technology.  I have some of these articles linked throughout the post and also have a list in the references section, so please take a look.


Lab Environment

Note: From my research, it doesn’t look like IOS XE has any support for PBB-EVPN.  However IOS XR does starting as of release 4.3.x.  For this lab, I provisioned IOS XRv and attached these resources to the existing IP/MPLS core network.

Also as a very important note, it appeared that IOS XRv (as of version tested) does NOT have data plane support for L2VPN technologies, including PBB-EVPN.  Please see this post for further info. This meant that data plane verification could NOT be performed (although control plane could be verified).  This was a very big disappointment to say the least.  For true data plane support, I would need to run actual ASR 9K hardware, and I simply didn’t have access to that.

  • (3) IOS XRv: Version 6.1.2 (Demo)

Diagram and Topology

Three additional PEs (PE_XR01, PE_XR02 and PE_XR03) were added to existing IP/MPLS core network and three additional CEs (CE_R27, CE_R28 and CE_R29) were attached to those PEs respectively.  The customer ELAN was then created between the three PEs and using the CE devices as end hosts.

Unfortunately, I was unable to test a key component of EVPN, which was the ability to multi-home between the CE and PE.  The IOS XRv images tested did not support the Multi-Chassis Link Aggregation protocol (MC-LAG), which was key in getting that configured.  A quick discussion can be found here regarding MC-LAG support in IOS XRv.

  • Infrastructure Information
    • PE_XR01, PE_XR02 and PE_XR03 iBGP to Route Reflector (R11)
    • CUSTOMER F
      • CE IP Address: 172.16.4.0/24
      • ELAN Name: CUSTOMER_F_ELAN
      • VPN ID (EVI): 40
      • Customer VLAN: 400
      • CE AC Interface: Gig1
      • PE AC Interface: Gig0/0/0/1
      • PBB Customer Backbone Service Instance Identifier (I-SID): 40400


Technology Overview

Here were some good informational slides I found on the technology.  It will cover common terms and acronyms that are used throughout this post. 


PBB-EVPN Summary
  • Next generation of multipoint L2VPN Ethernet services.
  • Uses BGP as control plane protocol for increased scalability.
  • Continues to use the existing MPLS network for data plane forwarding.
  • Primary use case for Datacenter Interconnects (DCI) due to the number of MAC addresses learning requirements.


PBB-EVPN Benefits
  • Advanced Multi-Homing capabilities: All Active & Per Flow Load Balancing.
  • Uses a MAC-in-MAC encapsulation to increase core scalability for MAC advertisements.
  • Promotes fast convergence and lower overhead.
  • Familiar operations using BGP/MPLS/IP data plane forwarding in the core.
PBB-EVPN Key Operational Concepts

Combined two different technologies to get the benefits of each. Provider Backbone Bridging (IEEE 802.1ah) + Ethernet VPN for MPLS (RFC 7432) = PBB-EVPN (RFC 7623).

  • PBB
    • Two Bridge domain concept to create a clear separation between the core and edge. Customer Bridge Domain (I-Component) & Core Bridge Domain (B-Component). See “Bridge Domain” diagram below.
    • Learns Customer MACs (C-MAC) on the data plane of provider edge routers (PE) in Customer Bridge domain.
    • Maps/binds multiple C-MACs to single PE’s Backbone MAC (B-MACs) in the Core Bridge Domain to reduce the overall number of MAC addresses required of the core to learn and advertise.  B-MACs forwards within the Core Bridge Domain.  Also known as MAC-in-MAC encapsulation due to C-MACs being aggregated to a B-MAC.
  • EVPN
    • Uses MP-BGP control plane protocol to learn and advertise the PE’s B-MACs.
    • BGP only learns/advertises B-MACs in the core (i.e., no C-MACs are seen here), whereas traditional EVPN learns all C-MACs in the core.
    • Continues to use the existing MPLS network for data plane forwarding.  Service providers do not need to modify their core networks to add PBB-EVPN services.

PBB-EVPN Key Components Overview

  • I-SID (Instance Service Identifier or also referred to as the Backbone Service Identifier )
    • Unique and global on the PBB backbone which identifies service instance (i.e., customers).
    • Maps the I-Component to its associated B-Component.
    • Uses 24 bits, so can have over 16 million service instances.
      • As traditional 802.1ad (Q-in-Q or Provider Bridges) networks used a 12 bit Service VLAN ID (S-VID) as service instance identifier, it was only limited to 4096 service instances.  However, with the introduction of the 24 bit I-SID in PBB, it now allows over 16 million service instances to be defined.
  • I-Component (Instance Component)
    • Known as the “customer side” edge bridge.
    • Data plane MAC learning and forwarding.
    • Maps customer traffic to I-SID.
    • C-MAC to B-MAC mapping and forwarding.
  • B-Component (Backbone Component)
    • Known as the “backbone side” edge bridge.
    • Control plane (BGP) learning and forwarding.
    • Forwarding only between B-MACs.
    • Maps I-SID to MPLS labels.


  • EVI (EVPN Instance)
    • Identifies the VPN in the IP/MPLS network and spans on all participating PE routers.  Analogous to a VRF used in a L3VPN and sometimes referred to as a L2 VRF.


EVPN BGP Routes Overview

EVPN introduced many new BGP route types as explained in this article.  However for the purposes of PBB-EVPN, only a subset of these routes were used.


EVPN BGP Route Types
  • Type 1: EAD routes were technically not required in PBB-EVPN.  See RFC 7623 Sec 5.1 for more details.
  • Type 2: MAC Advertisement routes were used to advertise the MAC to IP association.  However, in PBB-EVPN, B-MACs from all the PEs were advertised as type 2 routes.
  • Type 3: Inclusive Multicast routes were used for Broadcast, Unknown Unicast and Multicast (BUM) traffic delivery in EVPN networks.  It's also used for PE auto discovery so it could setup a LSP based multicast tunnel (MP2P, P2P, P2MP, or MP2MP) for the I-SID.
  • Type 4: ES routes were used in a multi-homing scenario.  It was used to identify each “leg” or connection to the different PEs and also used to elect a designated forwarder.


EVPN BGP Extended Communities

Extended communities provided a mechanism to embed additional information into a MP-BGP route. In PBB-EVPN, the following new extended communities were used.
  • The ESI label extended community was not used in PBB-EVPN per RFC 7623 (Section 5.5).  It was only relevant with Type 1 EAD routes, which PBB-EVPN didn't use.
  • The ES-Import extended community are carried in Type 4 Ethernet Segment routes.  It was used in a multi-homing scenario which allowed the importing of routes from all PEs with the same Ethernet Segment Identifier (ESI).
  • The MAC Mobility extended community was used to communicate to the PEs that a MAC move from one PE to another took place.  It was used to prevent traffic black-holing between PEs when multiple PEs advertised the same MAC route.  With PBB-EVPN, the MAC mobility extended community sent from the new PE signaled the original PE to flush all C-MAC addresses associated with the B-MAC in the corresponding I-SID.
  • The Default Gateway extended community was not used in PBB-EVPN per RFC 7623 (Section 5.8).


Configurations

Configurations below are from the three new PEs.  The MPLS core configurations has not changed.


PE_XR01

hostname PE_XR01
logging console debugging
logging buffered debugging
telnet vrf MGMT ipv4 server max-servers 10
telnet vrf default ipv4 server max-servers 10
domain name domain.local
domain lookup disable
vrf MGMT
 description MANAGEMENT
 vpn id 1:1
 address-family ipv4 unicast
 !
!
line default
 exec-timeout 60 0
 transport input telnet ssh
!
interface Loopback0
 description RID
 ipv4 address 101.101.101.101 255.255.255.255
!
interface MgmtEth0/0/CPU0/0
 description MANAGEMENT
 vrf MGMT
 ipv4 address 10.1.1.209 255.255.255.0
!
interface GigabitEthernet0/0/0/0
 bandwidth 100
 mtu 9000
!
interface GigabitEthernet0/0/0/0.38
 description TO P_R01
 bandwidth 100
 mtu 9000
 ipv4 address 10.1.1.58 255.255.255.252
 encapsulation dot1q 38
!
interface GigabitEthernet0/0/0/0.39
 description TO P_R03
 bandwidth 100
 mtu 9000
 ipv4 address 10.1.1.62 255.255.255.252
 encapsulation dot1q 39
!
interface GigabitEthernet0/0/0/1
 description TO CE_R27
!
interface GigabitEthernet0/0/0/1.400 l2transport
 description TO CE_R27 - ELAN VL400
 encapsulation dot1q 400
 rewrite ingress tag pop 1 symmetric
!
interface GigabitEthernet0/0/0/2
 shutdown
!
interface GigabitEthernet0/0/0/3
 shutdown
!
router static
 vrf MGMT
  address-family ipv4 unicast
   0.0.0.0/0 10.1.1.254
  !
 !
!
router ospf 100
 log adjacency changes detail
 router-id 101.101.101.101
 passive enable
 area 0
  interface Loopback0
  !
  interface GigabitEthernet0/0/0/0.38
   cost 2
   network point-to-point
   passive disable
   mtu-ignore enable
  !
  interface GigabitEthernet0/0/0/0.39
   cost 1
   network point-to-point
   passive disable
   mtu-ignore enable
  !
 !
!
router bgp 2345
 timers bgp 7 21
 bgp router-id 101.101.101.101
 address-family l2vpn evpn
 !
 neighbor 11.11.11.11
  remote-as 2345
  description TO ROUTE REFLECTOR - R11
  update-source Loopback0
  address-family l2vpn evpn
  !
 !
!
evpn
 evi 40
  description CUSTOMER_F_ELAN
 !
!
l2vpn
 pbb
  backbone-source-mac 0001.0001.0001
 !
 bridge group PBB_EVPN_CORE_BG
  bridge-domain PBB_EVPN_CORE_BD_EVI40
   pbb core
    evi 40
    !
   !
  !
 !
 bridge group PBB_EVPN_EDGE_BG
  bridge-domain PBB_EVPN_EDGE_BD_EVI40
   interface GigabitEthernet0/0/0/1.400
   !
   pbb edge i-sid 40400 core-bridge PBB_EVPN_CORE_BD_EVI40
   !
  !
 !
!
mpls ldp
 router-id 101.101.101.101
 interface GigabitEthernet0/0/0/0.38
 !
 interface GigabitEthernet0/0/0/0.39
 !
!
mpls label range table 0 100100 100199
end


PE_XR02

hostname PE_XR02
logging console debugging
logging buffered debugging
telnet vrf MGMT ipv4 server max-servers 10
telnet vrf default ipv4 server max-servers 10
domain name domain.local
domain lookup disable
vrf MGMT
 description MANAGEMENT
 vpn id 1:1
 address-family ipv4 unicast
 !
!
line default
 exec-timeout 60 0
 transport input telnet ssh
!
interface Loopback0
 description RID
 ipv4 address 102.102.102.102 255.255.255.255
!
interface MgmtEth0/0/CPU0/0
 description MANAGEMENT
 vrf MGMT
 ipv4 address 10.1.1.208 255.255.255.0
!
interface GigabitEthernet0/0/0/0
 bandwidth 100
 mtu 9000
!
interface GigabitEthernet0/0/0/0.40
 description TO P_R01
 bandwidth 100
 mtu 9000
 ipv4 address 10.1.1.66 255.255.255.252
 encapsulation dot1q 40
!
interface GigabitEthernet0/0/0/0.41
 description TO P_R02
 bandwidth 100
 mtu 9000
 ipv4 address 10.1.1.70 255.255.255.252
 encapsulation dot1q 41
!
interface GigabitEthernet0/0/0/1
 description TO CE_R28
!
interface GigabitEthernet0/0/0/1.400 l2transport
 description TO CE_R28 - ELAN VL400
 encapsulation dot1q 400
 rewrite ingress tag pop 1 symmetric
!
interface GigabitEthernet0/0/0/2
 shutdown
!
interface GigabitEthernet0/0/0/3
 shutdown
!
router static
 vrf MGMT
  address-family ipv4 unicast
   0.0.0.0/0 10.1.1.254
  !
 !
!
router ospf 100
 log adjacency changes detail
 router-id 102.102.102.102
 passive enable
 area 0
  interface Loopback0
  !
  interface GigabitEthernet0/0/0/0.40
   cost 4
   network point-to-point
   passive disable
   mtu-ignore enable
  !
  interface GigabitEthernet0/0/0/0.41
   cost 1
   network point-to-point
   passive disable
   mtu-ignore enable
  !
 !
!
router bgp 2345
 timers bgp 7 21
 bgp router-id 102.102.102.102
 address-family l2vpn evpn
 !
 neighbor 11.11.11.11
  remote-as 2345
  description TO ROUTE REFLECTOR - R11
  update-source Loopback0
  address-family l2vpn evpn
  !
 !
!
evpn
 evi 40
  description CUSTOMER_F_ELAN
 !
!
l2vpn
 pbb
  backbone-source-mac 0002.0002.0002
 !
 bridge group PBB_EVPN_CORE_BG
  bridge-domain PBB_EVPN_CORE_BD_EVI40
   pbb core
    evi 40
    !
   !
  !
 !
 bridge group PBB_EVPN_EDGE_BG
  bridge-domain PBB_EVPN_EDGE_BD_EVI40
   interface GigabitEthernet0/0/0/1.400
   !
   pbb edge i-sid 40400 core-bridge PBB_EVPN_CORE_BD_EVI40
   !
  !
 !
!
mpls ldp
 router-id 102.102.102.102
 interface GigabitEthernet0/0/0/0.40
 !
 interface GigabitEthernet0/0/0/0.41
 !
!
mpls label range table 0 100200 100299
end


PE_XR03

hostname PE_XR03
logging console debugging
logging buffered debugging
telnet vrf MGMT ipv4 server max-servers 10
telnet vrf default ipv4 server max-servers 10
domain name domain.local
domain lookup disable
vrf MGMT
 description MANAGEMENT
 vpn id 1:1
 address-family ipv4 unicast
 !
!
line default
 exec-timeout 60 0
 transport input telnet ssh
!
interface Loopback0
 description RID
 ipv4 address 103.103.103.103 255.255.255.255
!
interface MgmtEth0/0/CPU0/0
 description MANAGEMENT
 vrf MGMT
 ipv4 address 10.1.1.207 255.255.255.0
!
interface GigabitEthernet0/0/0/0
 bandwidth 100
 mtu 9000
!
interface GigabitEthernet0/0/0/0.42
 description TO P_R02
 bandwidth 100
 mtu 9000
 ipv4 address 10.1.1.74 255.255.255.252
 encapsulation dot1q 42
!
interface GigabitEthernet0/0/0/0.43
 description TO P_R04
 bandwidth 100
 mtu 9000
 ipv4 address 10.1.1.78 255.255.255.252
 encapsulation dot1q 43
!
interface GigabitEthernet0/0/0/1
 description TO CE_R29
!
interface GigabitEthernet0/0/0/1.400 l2transport
 description TO CE_R29 - ELAN VL400
 encapsulation dot1q 400
!
interface GigabitEthernet0/0/0/2
 shutdown
!
interface GigabitEthernet0/0/0/3
 shutdown
!
router static
 vrf MGMT
  address-family ipv4 unicast
   0.0.0.0/0 10.1.1.254
  !
 !
!
router ospf 100
 log adjacency changes detail
 router-id 103.103.103.103
 passive enable
 area 0
  interface Loopback0
  !
  interface GigabitEthernet0/0/0/0.42
   cost 3
   network point-to-point
   passive disable
   mtu-ignore enable
  !
  interface GigabitEthernet0/0/0/0.43
   cost 1
   network point-to-point
   passive disable
   mtu-ignore enable
  !
 !
!
router bgp 2345
 timers bgp 7 21
 bgp router-id 103.103.103.103
 address-family l2vpn evpn
 !
 neighbor 11.11.11.11
  remote-as 2345
  description TO ROUTE REFLECTOR - R11
  update-source Loopback0
  address-family l2vpn evpn
  !
 !
!
evpn
 evi 40
  description CUSTOMER_F_ELAN
 !
!
l2vpn
 pbb
  backbone-source-mac 0003.0003.0003
 !
 bridge group PBB_EVPN_CORE_BG
  bridge-domain PBB_EVPN_CORE_BD_EVI40
   pbb core
    evi 40
    !
   !
  !
 !
 bridge group PBB_EVPN_EDGE_BG
  bridge-domain PBB_EVPN_EDGE_BD_EVI40
   interface GigabitEthernet0/0/0/1.400
   !
   pbb edge i-sid 40400 core-bridge PBB_EVPN_CORE_BD_EVI40
   !
  !
 !
!
mpls ldp
 router-id 103.103.103.103
 interface GigabitEthernet0/0/0/0.42
 !
 interface GigabitEthernet0/0/0/0.43
 !
!
mpls label range table 0 100300 100399
end



PBB-EVPN Specific Configurations

Configuration from only PE_XR01 were used since the other PEs will have similar configurations.

EVPN Instance

The following configuration builds the EVPN instance by defining the EVI ID.  The ID value can be an arbitrary number between 1 – 65534.

evpn
 evi 40
  description CUSTOMER_F_ELAN


PBB Bridge Group/Domain

The following configuration defines the PBB parameters, such as the PE backbone MAC address, core & edge bridge groups and domains.

Per the PBB architecture, we created the 2 bridge domains.  The core bridge domain (B-Component) specified the EVPN instance while the edge bridge domain (I-Component) defined the customer facing attachment circuit and unique backbone identifier (I-SID).

As a side note, many examples I’ve seen used a 2 bridge group configuration, separating the core bridge domain with the edge bridge domain.  However, a single bridge group configuration seems to work as well, just as long as we have created 2 different bridge domains.  I believe the bridge group was simply a configuration container to make things easier to organize the bridge domains.

l2vpn
 pbb
  backbone-source-mac 0001.0001.0001
 !
 bridge group PBB_EVPN_CORE_BG
  bridge-domain PBB_EVPN_CORE_BD_EVI40
   pbb core
    evi 40
    !
   !
  !
 !
 bridge group PBB_EVPN_EDGE_BG
  bridge-domain PBB_EVPN_EDGE_BD_EVI40
   interface GigabitEthernet0/0/0/1.400
   !
   pbb edge i-sid 40400 core-bridge PBB_EVPN_CORE_BD_EVI40


Attachment Circuit

The attachment circuit that connected the CE was configured to match VLAN 400 to classify Customer F’s traffic and placed it into the PBB edge bridge domain.  It then popped the tag off as the DOT1q header was no longer required (or relevant) for MPLS forwarding.

interface GigabitEthernet0/0/0/1
 description TO CE_R27
!
interface GigabitEthernet0/0/0/1.400 l2transport
 description TO CE_R27 - ELAN VL400
 encapsulation dot1q 400
 rewrite ingress tag pop 1 symmetric


MP-BGP

MP-BGP peering was configured between the PEs to a Route Reflector (RR) under AFI 25 (L2VPN) & SAFI 70 (EVPN).

PE_XR01's BGP configuration to RR.

router bgp 2345
 timers bgp 7 21
 bgp router-id 101.101.101.101
 address-family l2vpn evpn
 !
 neighbor 11.11.11.11
  remote-as 2345
  description TO ROUTE REFLECTOR - R11
  update-source Loopback0
  address-family l2vpn evpn

RR's BGP configuration to the PEs.

router bgp 2345
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor PE_XR01 peer-group
 neighbor PE_XR01 remote-as 2345
 neighbor PE_XR01 description TO PE XR01
 neighbor PE_XR01 update-source Loopback0
 neighbor PE_XR01 timers 7 21
 neighbor PE_XR02 peer-group
 neighbor PE_XR02 remote-as 2345
 neighbor PE_XR02 description TO PE XR02
 neighbor PE_XR02 update-source Loopback0
 neighbor PE_XR02 timers 7 21
 neighbor PE_XR03 peer-group
 neighbor PE_XR03 remote-as 2345
 neighbor PE_XR03 description TO PE XR03
 neighbor PE_XR03 update-source Loopback0
 neighbor PE_XR03 timers 7 21
 neighbor 101.101.101.101 peer-group PE_XR01
 neighbor 102.102.102.102 peer-group PE_XR02
 neighbor 103.103.103.103 peer-group PE_XR03
 !
 address-family ipv4
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor PE_XR01 send-community both
  neighbor PE_XR01 route-reflector-client
  neighbor PE_XR02 send-community both
  neighbor PE_XR02 route-reflector-client
  neighbor PE_XR03 send-community both
  neighbor PE_XR03 route-reflector-client
  neighbor 101.101.101.101 activate
  neighbor 102.102.102.102 activate
  neighbor 103.103.103.103 activate
 exit-address-family


CE Interface

The CE interface was configured to use VLAN 400 as transport (configuration from CE27).

interface GigabitEthernet1
 description TO PE_XR01
 no ip address
 negotiation auto
 no mop enabled
 no mop sysid
!
interface GigabitEthernet1.400
 description TO PE_XR01 - PBB-EVPN ELAN400
 encapsulation dot1Q 400
 ip address 172.16.4.1 255.255.255.0



Testing Results

As noted earlier, due to a lack of IOS XRv data plane support for L2VPN services, verification using ping tests could not be performed.

CE27 to CE28

CE_R27#ping 172.16.4.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.4.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

CE27 to CE29

CE_R27#ping 172.16.4.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.4.3, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

CE27’s ARP Table

CE_R27#sh arp        
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  172.16.4.1              -   000c.299a.09c8  ARPA   GigabitEthernet1.400
Internet  172.16.4.2              0   Incomplete      ARPA  
Internet  172.16.4.3              0   Incomplete      ARPA 



Verification Tasks

The list below outlined commands used to verify PBB-EVPN operation.  Outputs were only from PE_XR01 for brevity.

  • OSPF Neighbors to Core
  • LDP Neighbors to Core 
  • PBB Bridge Domains
  • PBB Backbone MACs (all PEs)
  • PBB Bridge Domain Forwarding
  • EVPN EVI
  • EVPN EVI MAC
  • EVPN EVI Neighbors
  • EVPN EVI Inclusive-Multicast
  • BGP L2VPN EVPN Summary
  • BGP L2VPN EVPN
  • BGP L2VPN EVPN NLRI [Type 2 & 3 Routes]

OSPF Neighbors

Verified the OSPF neighbors from PE_XR01 to MPLS core network (P1 and P3).

RP/0/0/CPU0:PE_XR01#sh ospf neighbor 
Tue Oct 24 05:41:55.026 UTC

* Indicates MADJ interface
# Indicates Neighbor awaiting BFD session up

Neighbors for OSPF 100

Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1         1     FULL/  -        00:00:35    10.1.1.57       GigabitEthernet0/0/0/0.38
    Neighbor is up for 1d03h
3.3.3.3         1     FULL/  -        00:00:35    10.1.1.61       GigabitEthernet0/0/0/0.39
    Neighbor is up for 1d03h

Total neighbor count: 2


LDP Neighbors

Verified the LDP neighbors from PE_XR01 to MPLS core network (P1 and P3).

RP/0/0/CPU0:PE_XR01#sh mpls ldp neighbor
Tue Oct 24 05:42:33.524 UTC

Peer LDP Identifier: 3.3.3.3:0
  TCP connection: 3.3.3.3:646 - 101.101.101.101:43929
  Graceful Restart: No
  Session Holdtime: 180 sec
  State: Oper; Msgs sent/rcvd: 1986/1976; Downstream-Unsolicited
  Up time: 1d04h
  LDP Discovery Sources:
    IPv4: (1)
      GigabitEthernet0/0/0/0.39
    IPv6: (0)
  Addresses bound to this peer:
    IPv4: (9)
      3.3.3.3        10.0.0.6       10.0.0.18      10.0.0.21     
      10.0.0.101     10.1.1.5       10.1.1.33      10.1.1.41     
      10.1.1.61     
    IPv6: (0)

Peer LDP Identifier: 1.1.1.1:0
  TCP connection: 1.1.1.1:646 - 101.101.101.101:13052
  Graceful Restart: No
  Session Holdtime: 180 sec
  State: Oper; Msgs sent/rcvd: 1986/1977; Downstream-Unsolicited
  Up time: 1d04h
  LDP Discovery Sources:
    IPv4: (1)
      GigabitEthernet0/0/0/0.38
    IPv6: (0)
  Addresses bound to this peer:
    IPv4: (9)
      1.1.1.1        10.0.0.1       10.0.0.5       10.0.0.9      
      10.1.1.1       10.1.1.17      10.1.1.29      10.1.1.57     
      10.1.1.65     
    IPv6: (0)


PBB Bridge Domains

This command displays detailed information about the PBB bridge domains, such as:
  • Bridge Groups
  • Bridge Domains
  • Bridge Domain Type (core or edge)
  • I-SID
  • Bridge Domain, EVPN and AC operational status
RP/0/0/CPU0:PE_XR01#sh l2vpn bridge-domain
Tue Oct 24 05:49:59.893 UTC
Legend: pp = Partially Programmed.
Bridge group: PBB_EVPN_CORE_BG, bridge-domain: PBB_EVPN_CORE_BD_EVI40, id: 2, state: up, ShgId: 0, MSTi: 0
  Type: pbb-core
  Number of associated pbb-edge BDs: 1
  Aging: 300 s, MAC limit: 4000, Action: none, Notification: syslog
  Filter MAC addresses: 0
  ACs: 0 (0 up), VFIs: 0, PWs: 0 (0 up), PBBs: 1 (1 up), VNIs: 0 (0 up)
  List of PBBs:
    PBB Core, state: up
  List of EVPNs:
    EVPN, state: up
  List of ACs:
  List of Access PWs:
  List of VFIs:
Bridge group: PBB_EVPN_EDGE_BG, bridge-domain: PBB_EVPN_EDGE_BD_EVI40, id: 3, state: up, ShgId: 0, MSTi: 0
  Type: pbb-edge, I-SID: 40400
  Aging: 300 s, MAC limit: 4000, Action: none, Notification: syslog
  Filter MAC addresses: 0
  ACs: 1 (1 up), VFIs: 0, PWs: 0 (0 up), PBBs: 1 (1 up), VNIs: 0 (0 up)
  List of PBBs:
    PBB Edge, state: up, Static MAC addresses: 0
  List of ACs:
    Gi0/0/0/1.400, state: up, Static MAC addresses: 0, MSTi: 9
  List of Access PWs:
  List of VFIs:


PBB Backbone MACs

This command displays the configured PBB Backbone MAC addresses (from all PEs).

RP/0/0/CPU0:PE_XR01#sh l2vpn pbb backbone-source-mac
Tue Oct 24 06:23:10.357 UTC
Backbone Source MAC: 0001.0001.0001
Chassis MAC        : 02b4.a7bf.0c05

RP/0/0/CPU0:PE_XR02#sh l2vpn pbb backbone-source-mac
Tue Oct 24 06:24:04.194 UTC
Backbone Source MAC: 0002.0002.0002
Chassis MAC        : 0228.89f6.f405

RP/0/0/CPU0:PE_XR03#sh l2vpn pbb backbone-source-mac
Tue Oct 24 06:24:07.007 UTC
Backbone Source MAC: 0003.0003.0003
Chassis MAC        : 024f.54bd.0c05


PBB Bridge Domain Forwarding

This command displays the PBB Bridge Domain’s forwarding table (i.e., PE’s data plane).

Note: Unfortunately my lab output did not show any C-MAC learning since IOS XRv does not have data plane support.

RP/0/0/CPU0:PE_XR01#sh l2vpn forwarding bridge-domain mac-address location 0/0/CPU0
Tue Oct 24 05:57:19.773 UTC
Mac Address    Type    Learned from/Filtered on    LC learned Age                    Mapped to      
-------------- ------- --------------------------- ---------- ---------------------- ----------
0002.0002.0002 BMAC    BD id: 2                    N/A        N/A                    N/A         
0003.0003.0003 BMAC    BD id: 2                    N/A        N/A                    N/A    

As a reference, the following output was a working example.  Notice the MAC addresses starting with 2a22 and 2b22 being mapped to a single B-MAC bbbb.bbbb.bbbb.


EVPN EVI

This command displays EVPN EVI details such as:
  • EVI ID
  • Bridge Domain name and type
  • MPLS labels for Unicast and Multicast (BUM) traffic
  • RD’s and RT’s
RP/0/0/CPU0:PE_XR01#sh evpn evi vpn-id 40 detail
Tue Oct 24 05:54:32.164 UTC

EVI        Bridge Domain                Type  
---------- ---------------------------- -------
40         PBB_EVPN_CORE_BD_EVI40       PBB   
   Description    : CUSTOMER_F_ELAN
   Unicast Label  : 100144
   Multicast Label: 100145
   Flow Label: N
   Control-Word: Enabled
   Forward-class: 0
   RD Config: none
   RD Auto  : (auto) 101.101.101.101:40
   RT Auto  : 2345:40
   Route Targets in Use           Type                
   ------------------------------ ---------------------
   2345:40                        Import              
   2345:40                        Export  


EVPN EVI MAC

This command displays B-MACs, next-hop information, ESI and MPLS VPN labels.

RP/0/0/CPU0:PE_XR01#sh evpn evi vpn-id 40 mac detail
Tue Oct 24 05:55:05.132 UTC

MAC address    Nexthop                                 Label    vpn-id 
-------------- --------------------------------------- -------- --------
0001.0001.0001 N/A                                     100144   40     
   Ether.Segment: 0000.0000.0000.0000.0000
   ESI port key : 0x0000
   Source       : Local
   Multi-paths resolved: FALSE
   Multi-paths local label: 0        
   Flush Count  : 3
0002.0002.0002 102.102.102.102                         100244   40     
   Ether.Segment: 0000.0000.0000.0000.0000
   ESI port key : 0x0000
   Source       : Remote, MPLS
   Multi-paths resolved: FALSE
   Multi-paths local label: 0        
   Flush Count  : 1
0003.0003.0003 103.103.103.103                         100344   40     
   Ether.Segment: 0000.0000.0000.0000.0000
   ESI port key : 0x0000
   Source       : Remote, MPLS
   Multi-paths resolved: FALSE
   Multi-paths local label: 0        
   Flush Count  : 1


EVPN EVI Neighbors

This command displays all its neighbors in a particular EVI.

RP/0/0/CPU0:PE_XR01#sh evpn evi neighbor
Tue Oct 24 05:54:09.686 UTC

Neighbor IP                              vpn-id 
---------------------------------------- --------
102.102.102.102                          40     
103.103.103.103                          40 


EVPN EVI Inclusive-Multicast

This command displays the endpoints in the multicast flood list for BUM forwarding.  It contains next-hop and forwarding labels.

RP/0/0/CPU0:PE_XR01#show evpn evi inclusive-multicast detail
Tue Oct 24 08:06:24.222 UTC
Ethernet Tag: 40400, Originating IP: 101.101.101.101, vpn-id: 40
    Nexthop: ::
    Label  : 100145
    Source : Local
Ethernet Tag: 40400, Originating IP: 102.102.102.102, vpn-id: 40
    Nexthop: 102.102.102.102
    Label  : 100245
    Source : Remote
Ethernet Tag: 40400, Originating IP: 103.103.103.103, vpn-id: 40
    Nexthop: 103.103.103.103
    Label  : 100345
    Source : Remote


BGP L2VPN EVPN Summary

This command displays the BGP neighbor peering status.

RP/0/0/CPU0:PE_XR01#sh bgp l2vpn evpn summary
Tue Oct 24 06:05:37.699 UTC
BGP router identifier 101.101.101.101, local AS number 2345
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0   RD version: 0
BGP main routing table version 86
BGP NSR Initial initsync version 1 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker              86         86         86         86          86           0

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
11.11.11.11       0  2345    5224    5030       86    0    0 09:48:03          4

This was the BGP neighbor peering status from the RR’s perspective.

Note: Neighbors with 25.25.25.25 and 26.26.26.26 were PEs from the previous EVPN VPWS lab that used the same AFI/SAFI.

RR_R11#sh bgp l2vpn evpn summary
BGP router identifier 11.11.11.11, local AS number 2345
BGP table version is 100, main routing table version 100
8 network entries using 2752 bytes of memory
8 path entries using 1312 bytes of memory
6/6 BGP path/bestpath attribute entries using 1584 bytes of memory
7 BGP extended community entries using 668 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 6316 total bytes of memory
BGP activity 33/20 prefixes, 45/32 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
25.25.25.25     4         2345  119786  119839      100    0    0 1w2d            1
26.26.26.26     4         2345  119833  119831      100    0    0 1w2d            1
101.101.101.101 4         2345    5079    5275      100    0    0 09:53:48        2
102.102.102.102 4         2345    5079    5270      100    0    0 09:53:44        2
103.103.103.103 4         2345    5081    5279      100    0    0 09:53:59        2


BGP L2VPN EVPN

This command displays the BGP table for the L2VPN EVPN address family.

This output contained Type 2 and Type 3 routes from each PE (from three different RDs).  The routes highlighted were routes only related to PE_XR01’s RD.

RP/0/0/CPU0:PE_XR01#sh bgp l2vpn evpn
Tue Oct 24 06:05:18.270 UTC
BGP router identifier 101.101.101.101, local AS number 2345
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0   RD version: 0
BGP main routing table version 86
BGP NSR Initial initsync version 1 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 101.101.101.101:40 (default for vrf PBB_EVPN_CORE_BD_EVI40)
*> [2][0][48][0001.0001.0001][0]/104
                      0.0.0.0                                0 i
*>i[2][0][48][0002.0002.0002][0]/104
                      102.102.102.102               100      0 i
*>i[2][0][48][0003.0003.0003][0]/104
                      103.103.103.103               100      0 i
*> [3][40400][32][101.101.101.101]/80
                      0.0.0.0                                0 i
*>i[3][40400][32][102.102.102.102]/80
                      102.102.102.102               100      0 i
*>i[3][40400][32][103.103.103.103]/80
                      103.103.103.103               100      0 i
Route Distinguisher: 102.102.102.102:40
*>i[2][0][48][0002.0002.0002][0]/104
                      102.102.102.102               100      0 i
*>i[3][40400][32][102.102.102.102]/80
                      102.102.102.102               100      0 i
Route Distinguisher: 103.103.103.103:40
*>i[2][0][48][0003.0003.0003][0]/104
                      103.103.103.103               100      0 i
*>i[3][40400][32][103.103.103.103]/80
                      103.103.103.103               100      0 i

Processed 10 prefixes, 10 paths

The output below was a look at the RR's L2VPN EVPN BGP table.

RR_R11#sh bgp l2vpn evpn all
BGP table version is 100, local router ID is 11.11.11.11
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 101.101.101.101:40
 *>i [2][101.101.101.101:40][00000000000000000000][0][48][000100010001][0][*][100144]/33
                       101.101.101.101               100      0 i
Route Distinguisher: 102.102.102.102:40
 *>i [2][102.102.102.102:40][00000000000000000000][0][48][000200020002][0][*][100244]/33
                       102.102.102.102               100      0 i
Route Distinguisher: 103.103.103.103:40
 *>i [2][103.103.103.103:40][00000000000000000000][0][48][000300030003][0][*][100344]/33
                       103.103.103.103               100      0 i
Route Distinguisher: 101.101.101.101:40
 *>i [3][101.101.101.101:40][40400][32][662634496.3060.812b.277f]/17
                       101.101.101.101               100      0 i
Route Distinguisher: 102.102.102.102:40
 *>i [3][102.102.102.102:40][40400][32][662634496.3060.812b.277f]/17
                       102.102.102.102               100      0 i
Route Distinguisher: 103.103.103.103:40
 *>i [3][103.103.103.103:40][40400][32][662634496.3060.812b.277f]/17
                       103.103.103.103               100      0 i


BGP EVPN NLRI

Here’s a deeper look into the EVPN BGP Type 2 NLRI (MAC Advertisement).

Deciphering the route entry for: [2][0][48][0002.0002.0002][0]/104
  • BGP Route Type [2]
  • ESI [0]
  • MAC Address Length [48]
  • MAC Address (B-MAC) [0002.0002.0002]
  • IP Address (No IP address) [0]
  • Max Bits [/140]
  • Route Distinguisher [101.101.101.101:40]
  • MPLS VPN Label
  • Extended Community
  • Route Target
  • Ethernet Segment Identifier

RP/0/0/CPU0:PE_XR01#show bgp l2vpn evpn bdomain PBB_EVPN_CORE_BD_EVI40 [2][0][48][0002.0002.0002][0]/104
Tue Oct 24 06:08:01.089 UTC
BGP routing table entry for [2][0][48][0002.0002.0002][0]/104, Route Distinguisher: 101.101.101.101:40
Versions:
  Process           bRIB/RIB  SendTblVer
  Speaker                 85          85
Last Modified: Oct 23 20:17:42.538 for 09:50:18
Paths: (1 available, best #1)
  Not advertised to any peer
  Path #1: Received by speaker 0
  Not advertised to any peer
  Local
    102.102.102.102 (metric 6) from 11.11.11.11 (102.102.102.102)
      Received Label 100244
      Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported, rib-install
      Received Path ID 0, Local Path ID 0, version 85
      Extended community: EVPN MAC Mobility:3 RT:2345:40
      Originator: 102.102.102.102, Cluster list: 11.11.11.11
      EVPN ESI: 0000.0000.0000.0000.0000
      Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 102.102.102.102:40


Here’s a deeper look into the EVPN BGP Type 3 NLRI (Inclusive-Multicast).

Deciphering the route entry for: [3][40400][32][102.102.102.102]/80
  • BGP Route Type [3]
  • Backbone Service ID (I-SID) [40400]
  • IP Address length [32]
  • IP Address of originating router [102.102.102.102]
  • Max Bits [/80]
  • Route Distinguisher [101.101.101.101:40]
  • Extended Community (RT)
  • Provider Multicast Service Interface (PMSI) Tunnel attributes:
    • Tunnel Type 6 (Ingress Replication)
    • MPLS VPN Label
RP/0/0/CPU0:PE_XR01#show bgp l2vpn evpn bdomain PBB_EVPN_CORE_BD_EVI40 [3][40400][32][102.102.102.102]/80
Tue Oct 24 06:08:42.516 UTC
BGP routing table entry for [3][40400][32][102.102.102.102]/80, Route Distinguisher: 101.101.101.101:40
Versions:
  Process           bRIB/RIB  SendTblVer
  Speaker                 86          86
Last Modified: Oct 23 20:17:42.538 for 09:51:00
Paths: (1 available, best #1)
  Not advertised to any peer
  Path #1: Received by speaker 0
  Not advertised to any peer
  Local
    102.102.102.102 (metric 6) from 11.11.11.11 (102.102.102.102)
      Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported
      Received Path ID 0, Local Path ID 0, version 86
      Extended community: RT:2345:40
      Originator: 102.102.102.102, Cluster list: 11.11.11.11
      PMSI: flags 0x00, type 6, label 100245, ID 0x66666666
      Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 102.102.102.102:40



MPLS Forwarding Trace

This was an example MPLS forwarding trace from PE_XR01 to PE_XR02’s B-MAC of 0002.0002.0002.

PE_XR01 performed a lookup of PE_XR02’s Type 2 B-MAC route and has received VPN label of 100244.

RP/0/0/CPU0:PE_XR01#show bgp l2vpn evpn bdomain PBB_EVPN_CORE_BD_EVI40 [2][0][48][0002.0002.0002][0]/104
Tue Oct 24 07:45:12.909 UTC
BGP routing table entry for [2][0][48][0002.0002.0002][0]/104, Route Distinguisher: 101.101.101.101:40
Versions:
  Process           bRIB/RIB  SendTblVer
  Speaker                 85          85
Last Modified: Oct 23 20:17:42.538 for 11:27:30
Paths: (1 available, best #1)
  Not advertised to any peer
  Path #1: Received by speaker 0
  Not advertised to any peer
  Local
    102.102.102.102 (metric 6) from 11.11.11.11 (102.102.102.102)
      Received Label 100244
      Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported, rib-install
      Received Path ID 0, Local Path ID 0, version 85
      Extended community: EVPN MAC Mobility:3 RT:2345:40
      Originator: 102.102.102.102, Cluster list: 11.11.11.11
      EVPN ESI: 0000.0000.0000.0000.0000
      Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 102.102.102.102:40


PE_XR01 performed a MPLS forwarding prefix lookup for 102.102.102.102, then imposed an inner VPN label of 100244 and an outer LSP label of 334 then forwarded to P3.

RP/0/0/CPU0:PE_XR01#sh mpls forwarding prefix 102.102.102.102/32 detail
Tue Oct 24 07:46:03.626 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes      
Label  Label       or ID              Interface                    Switched   
------ ----------- ------------------ ------------ --------------- ------------
100143 334         102.102.102.102/32 Gi0/0/0/0.39 10.1.1.61       974625     
     Updated: Oct 23 02:36:45.216
     Version: 142, Priority: 3
     Label Stack (Top -> Bottom): { 334 }
     NHID: 0x0, Encap-ID: N/A, Path idx: 0, Backup path idx: 0, Weight: 0
     MAC/Encaps: 18/22, MTU: 8982
     Packets Switched: 18184

P3 swapped label 334 with 235 and forwarded to P2.

P_R03#sh mpls forwarding-table labels 334 detail
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop   
Label      Label      or Tunnel Id     Switched      interface             
334        235        102.102.102.102/32   \
                                       1301588       Gi1.14     10.0.0.17  
        MAC/Encaps=18/22, MRU=9216, Label Stack{235}
        000C294A72B4000C291EB42A8100000E8847 000EB000
        No output feature configured

P2 popped label 235, then forwarded to PE_XR02.

P_R02#sh mpls forwarding-table labels 235 detail
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop   
Label      Label      or Tunnel Id     Switched      interface             
235        Pop Label  102.102.102.102/32   \
                                       2278385       Gi1.41     10.1.1.70  
        MAC/Encaps=18/18, MRU=9220, Label Stack{}
        000C29692DBA000C294A72B4810000298847
        No output feature configured

PE_XR02 looked up the VPN label of 100244 and forwarded out its configured AC interface.

RP/0/0/CPU0:PE_XR02#sh mpls forwarding labels 100244 detail
Tue Oct 24 07:51:21.145 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes      
Label  Label       or ID              Interface                    Switched   
------ ----------- ------------------ ------------ --------------- ------------
100244 Pop         EVPN:0xc0000028    BD=2 PE      point2point     0          
     Updated: Oct 23 02:02:18.209
     PW Flow Label   : Enabled

     PW Flow Label: Enabled
     Label Stack (Top -> Bottom): { }
     MAC/Encaps: 0/0, MTU: 0
     Packets Switched: 0


References

Provider Backbone Bridging (PBB):






BGP/MPLS EVPN:







PBB-EVPN:





Cisco Live Sessions and Other Presentations





Other Blogs and Labs

(Personal thanks for all who posted and shared their lab experience and knowledge.)