the peering xchange : [a network engineer’s blog]: MPLS L2VPN (PBB-EVPN)

After getting myself some exposure to EVPN by testing technologies such as VXLAN EVPN and EVPN VPWS, I wanted to take it a bit further to learn more about the EVPN with MPLS. Studying EVPN VPWS did help me understand EVPN’s BGP control plane operation and behavior, however it was a bit simplified and therefore felt a bit cheated since it didn't fully utilize the features of BGP. With that in mind, my next obvious step was to build and test out a multi-point EVPN solution following RFC 7432. I’d figured going through the process of building this lab would allow me to maximize my understanding of the BGP control plane and MPLS data plane operations.

According to my research, it appeared that Cisco never released EVPN from a traditional RFC 7432 perspective. The only EVPN documentation that I found in the Cisco world was EVPN with Provider Backbone Bridging (RFC 7623), otherwise known as PBB-EVPN. I guess Cisco found a better use case to go with PBB-EVPN rather than to stick with the traditional EVPN and decided to only integrate that into their code? In any case, I’m always up for learning new technologies, so I took some time to lab this up.

As with my other posts, my focus was to look at the overall configuration and operational side of things. I’ll do my best to explain the gist of the technology but since there are so many well written technical articles on PBB-EVPN, I would suggest reading those to get a good understanding of the technology. I have some of these articles linked throughout the post and also have a list in the references section, so please take a look.

Lab Environment

Note: From my research, it doesn’t look like IOS XE has any support for PBB-EVPN. However IOS XR does starting as of release 4.3.x. For this lab, I provisioned IOS XRv and attached these resources to the existing IP/MPLS core network.

Also as a very important note, it appeared that IOS XRv (as of version tested) does NOT have data plane support for L2VPN technologies, including PBB-EVPN. Please see this post for further info. This meant that data plane verification could NOT be performed (although control plane could be verified). This was a very big disappointment to say the least. For true data plane support, I would need to run actual ASR 9K hardware, and I simply didn’t have access to that.

(3) IOS XRv: Version 6.1.2 (Demo)

Diagram and Topology

Three additional PEs (PE_XR01, PE_XR02 and PE_XR03) were added to existing IP/MPLS core network and three additional CEs (CE_R27, CE_R28 and CE_R29) were attached to those PEs respectively. The customer ELAN was then created between the three PEs and using the CE devices as end hosts.

Unfortunately, I was unable to test a key component of EVPN, which was the ability to multi-home between the CE and PE. The IOS XRv images tested did not support the Multi-Chassis Link Aggregation protocol (MC-LAG), which was key in getting that configured. A quick discussion can be found here regarding MC-LAG support in IOS XRv.

Infrastructure Information

PE_XR01, PE_XR02 and PE_XR03 iBGP to Route Reflector (R11)
CUSTOMER F

CE IP Address: 172.16.4.0/24
ELAN Name: CUSTOMER_F_ELAN
VPN ID (EVI): 40
Customer VLAN: 400
CE AC Interface: Gig1
PE AC Interface: Gig0/0/0/1
PBB Customer Backbone Service Instance Identifier (I-SID): 40400

Technology Overview

Here were some good informational slides I found on the technology. It will cover common terms and acronyms that are used throughout this post.

PBB-EVPN Summary

Next generation of multipoint L2VPN Ethernet services.
Uses BGP as control plane protocol for increased scalability.
Continues to use the existing MPLS network for data plane forwarding.
Primary use case for Datacenter Interconnects (DCI) due to the number of MAC addresses learning requirements.

https://www.slideshare.net/getyourbuildon/pbbevpn-cisco-ckn-techadvantage-webinar

PBB-EVPN Benefits

Advanced Multi-Homing capabilities: All Active & Per Flow Load Balancing.
Uses a MAC-in-MAC encapsulation to increase core scalability for MAC advertisements.
Promotes fast convergence and lower overhead.
Familiar operations using BGP/MPLS/IP data plane forwarding in the core.

https://www.slideshare.net/getyourbuildon/pbbevpn-cisco-ckn-techadvantage-webinar

PBB-EVPN Key Operational Concepts

Combined two different technologies to get the benefits of each. Provider Backbone Bridging (IEEE 802.1ah) + Ethernet VPN for MPLS (RFC 7432) = PBB-EVPN (RFC 7623).

PBB

Two Bridge domain concept to create a clear separation between the core and edge. Customer Bridge Domain (I-Component) & Core Bridge Domain (B-Component). See “Bridge Domain” diagram below.
Learns Customer MACs (C-MAC) on the data plane of provider edge routers (PE) in Customer Bridge domain.
Maps/binds multiple C-MACs to single PE’s Backbone MAC (B-MACs) in the Core Bridge Domain to reduce the overall number of MAC addresses required of the core to learn and advertise. B-MACs forwards within the Core Bridge Domain. Also known as MAC-in-MAC encapsulation due to C-MACs being aggregated to a B-MAC.

EVPN

Uses MP-BGP control plane protocol to learn and advertise the PE’s B-MACs.
BGP only learns/advertises B-MACs in the core (i.e., no C-MACs are seen here), whereas traditional EVPN learns all C-MACs in the core.
Continues to use the existing MPLS network for data plane forwarding. Service providers do not need to modify their core networks to add PBB-EVPN services.

https://www.slideshare.net/getyourbuildon/pbbevpn-cisco-ckn-techadvantage-webinar

https://www.broadband-forum.org/marketing/download/mktgdocs/BBF_EVPN_tutorial.pdf

https://netquirks.co.uk/category/pbb-evpn/

PBB-EVPN Key Components Overview

I-SID (Instance Service Identifier or also referred to as the Backbone Service Identifier )

Unique and global on the PBB backbone which identifies service instance (i.e., customers).
Maps the I-Component to its associated B-Component.
Uses 24 bits, so can have over 16 million service instances.

As traditional 802.1ad (Q-in-Q or Provider Bridges) networks used a 12 bit Service VLAN ID (S-VID) as service instance identifier, it was only limited to 4096 service instances. However, with the introduction of the 24 bit I-SID in PBB, it now allows over 16 million service instances to be defined.

http://www.brocade.com/content/html/en/configuration-guide/NI_05800a_SWITCHING/GUID-39750C1A-4994-4B7C-B988-1254C87F859E.html

I-Component (Instance Component)

Known as the “customer side” edge bridge.
Data plane MAC learning and forwarding.
Maps customer traffic to I-SID.
C-MAC to B-MAC mapping and forwarding.

https://tools.ietf.org/html/rfc7080

https://www.juniper.net/documentation/en_US/junos/topics/concept/pbb-evpn-integration-for-dci-overview.html

B-Component (Backbone Component)

Known as the “backbone side” edge bridge.
Control plane (BGP) learning and forwarding.
Forwarding only between B-MACs.
Maps I-SID to MPLS labels.

https://tools.ietf.org/html/rfc7080

https://www.juniper.net/documentation/en_US/junos/topics/concept/pbb-evpn-integration-for-dci-overview.html

EVI (EVPN Instance)

Identifies the VPN in the IP/MPLS network and spans on all participating PE routers. Analogous to a VRF used in a L3VPN and sometimes referred to as a L2 VRF.

https://www.juniper.net/documentation/en_US/junos/topics/concept/evpn-bgp-multihoming-overview.html

EVPN BGP Routes Overview

EVPN introduced many new BGP route types as explained in this article. However for the purposes of PBB-EVPN, only a subset of these routes were used.

New BGP NLRI for PBB-EVPN and will use route types 2 – 4.
New BGP Subsequent Address Family Identifiers (SAFI) 70 to define BGP EVPN.

EVPN BGP Route Types

Type 1: EAD routes were technically not required in PBB-EVPN. See RFC 7623 Sec 5.1 for more details.
Type 2: MAC Advertisement routes were used to advertise the MAC to IP association. However, in PBB-EVPN, B-MACs from all the PEs were advertised as type 2 routes.
Type 3: Inclusive Multicast routes were used for Broadcast, Unknown Unicast and Multicast (BUM) traffic delivery in EVPN networks. It's also used for PE auto discovery so it could setup a LSP based multicast tunnel (MP2P, P2P, P2MP, or MP2MP) for the I-SID.
Type 4: ES routes were used in a multi-homing scenario. It was used to identify each “leg” or connection to the different PEs and also used to elect a designated forwarder.

EVPN BGP Extended Communities

Extended communities provided a mechanism to embed additional information into a MP-BGP route. In PBB-EVPN, the following new extended communities were used.

The ESI label extended community was not used in PBB-EVPN per RFC 7623 (Section 5.5). It was only relevant with Type 1 EAD routes, which PBB-EVPN didn't use.
The ES-Import extended community are carried in Type 4 Ethernet Segment routes. It was used in a multi-homing scenario which allowed the importing of routes from all PEs with the same Ethernet Segment Identifier (ESI).
The MAC Mobility extended community was used to communicate to the PEs that a MAC move from one PE to another took place. It was used to prevent traffic black-holing between PEs when multiple PEs advertised the same MAC route. With PBB-EVPN, the MAC mobility extended community sent from the new PE signaled the original PE to flush all C-MAC addresses associated with the B-MAC in the corresponding I-SID.
The Default Gateway extended community was not used in PBB-EVPN per RFC 7623 (Section 5.8).

Configurations

Configurations below are from the three new PEs. The MPLS core configurations has not changed.

PE_XR01

hostname PE_XR01
logging console debugging
logging buffered debugging
telnet vrf MGMT ipv4 server max-servers 10
telnet vrf default ipv4 server max-servers 10
domain name domain.local
domain lookup disable
vrf MGMT
description MANAGEMENT
vpn id 1:1
address-family ipv4 unicast
!
!
line default
exec-timeout 60 0
transport input telnet ssh
!
interface Loopback0
description RID
ipv4 address 101.101.101.101 255.255.255.255
!
interface MgmtEth0/0/CPU0/0
description MANAGEMENT
vrf MGMT
ipv4 address 10.1.1.209 255.255.255.0
!
interface GigabitEthernet0/0/0/0
bandwidth 100
mtu 9000
!
interface GigabitEthernet0/0/0/0.38
description TO P_R01
bandwidth 100
mtu 9000
ipv4 address 10.1.1.58 255.255.255.252
encapsulation dot1q 38
!
interface GigabitEthernet0/0/0/0.39
description TO P_R03
bandwidth 100
mtu 9000
ipv4 address 10.1.1.62 255.255.255.252
encapsulation dot1q 39
!
interface GigabitEthernet0/0/0/1
description TO CE_R27
!
interface GigabitEthernet0/0/0/1.400 l2transport
description TO CE_R27 - ELAN VL400
encapsulation dot1q 400
rewrite ingress tag pop 1 symmetric
!
interface GigabitEthernet0/0/0/2
shutdown
!
interface GigabitEthernet0/0/0/3
shutdown
!
router static
vrf MGMT
address-family ipv4 unicast
0.0.0.0/0 10.1.1.254
!
!
!
router ospf 100
log adjacency changes detail
router-id 101.101.101.101
passive enable
area 0
interface Loopback0
!
interface GigabitEthernet0/0/0/0.38
cost 2
network point-to-point
passive disable
mtu-ignore enable
!
interface GigabitEthernet0/0/0/0.39
cost 1
network point-to-point
passive disable
mtu-ignore enable
!
!
!
router bgp 2345
timers bgp 7 21
bgp router-id 101.101.101.101
address-family l2vpn evpn
!
neighbor 11.11.11.11
remote-as 2345
description TO ROUTE REFLECTOR - R11
update-source Loopback0
address-family l2vpn evpn
!
!
!
evpn
evi 40
description CUSTOMER_F_ELAN
!
!
l2vpn
pbb
backbone-source-mac 0001.0001.0001
!
bridge group PBB_EVPN_CORE_BG
bridge-domain PBB_EVPN_CORE_BD_EVI40
pbb core
evi 40
!
!
!
!
bridge group PBB_EVPN_EDGE_BG
bridge-domain PBB_EVPN_EDGE_BD_EVI40
interface GigabitEthernet0/0/0/1.400
!
pbb edge i-sid 40400 core-bridge PBB_EVPN_CORE_BD_EVI40
!
!
!
!
mpls ldp
router-id 101.101.101.101
interface GigabitEthernet0/0/0/0.38
!
interface GigabitEthernet0/0/0/0.39
!
!
mpls label range table 0 100100 100199
end

PE_XR02

hostname PE_XR02
logging console debugging
logging buffered debugging
telnet vrf MGMT ipv4 server max-servers 10
telnet vrf default ipv4 server max-servers 10
domain name domain.local
domain lookup disable
vrf MGMT
description MANAGEMENT
vpn id 1:1
address-family ipv4 unicast
!
!
line default
exec-timeout 60 0
transport input telnet ssh
!
interface Loopback0
description RID
ipv4 address 102.102.102.102 255.255.255.255
!
interface MgmtEth0/0/CPU0/0
description MANAGEMENT
vrf MGMT
ipv4 address 10.1.1.208 255.255.255.0
!
interface GigabitEthernet0/0/0/0
bandwidth 100
mtu 9000
!
interface GigabitEthernet0/0/0/0.40
description TO P_R01
bandwidth 100
mtu 9000
ipv4 address 10.1.1.66 255.255.255.252
encapsulation dot1q 40
!
interface GigabitEthernet0/0/0/0.41
description TO P_R02
bandwidth 100
mtu 9000
ipv4 address 10.1.1.70 255.255.255.252
encapsulation dot1q 41
!
interface GigabitEthernet0/0/0/1
description TO CE_R28
!
interface GigabitEthernet0/0/0/1.400 l2transport
description TO CE_R28 - ELAN VL400
encapsulation dot1q 400
rewrite ingress tag pop 1 symmetric
!
interface GigabitEthernet0/0/0/2
shutdown
!
interface GigabitEthernet0/0/0/3
shutdown
!
router static
vrf MGMT
address-family ipv4 unicast
0.0.0.0/0 10.1.1.254
!
!
!
router ospf 100
log adjacency changes detail
router-id 102.102.102.102
passive enable
area 0
interface Loopback0
!
interface GigabitEthernet0/0/0/0.40
cost 4
network point-to-point
passive disable
mtu-ignore enable
!
interface GigabitEthernet0/0/0/0.41
cost 1
network point-to-point
passive disable
mtu-ignore enable
!
!
!
router bgp 2345
timers bgp 7 21
bgp router-id 102.102.102.102
address-family l2vpn evpn
!
neighbor 11.11.11.11
remote-as 2345
description TO ROUTE REFLECTOR - R11
update-source Loopback0
address-family l2vpn evpn
!
!
!
evpn
evi 40
description CUSTOMER_F_ELAN
!
!
l2vpn
pbb
backbone-source-mac 0002.0002.0002
!
bridge group PBB_EVPN_CORE_BG
bridge-domain PBB_EVPN_CORE_BD_EVI40
pbb core
evi 40
!
!
!
!
bridge group PBB_EVPN_EDGE_BG
bridge-domain PBB_EVPN_EDGE_BD_EVI40
interface GigabitEthernet0/0/0/1.400
!
pbb edge i-sid 40400 core-bridge PBB_EVPN_CORE_BD_EVI40
!
!
!
!
mpls ldp
router-id 102.102.102.102
interface GigabitEthernet0/0/0/0.40
!
interface GigabitEthernet0/0/0/0.41
!
!
mpls label range table 0 100200 100299
end

PE_XR03

hostname PE_XR03
logging console debugging
logging buffered debugging
telnet vrf MGMT ipv4 server max-servers 10
telnet vrf default ipv4 server max-servers 10
domain name domain.local
domain lookup disable
vrf MGMT
description MANAGEMENT
vpn id 1:1
address-family ipv4 unicast
!
!
line default
exec-timeout 60 0
transport input telnet ssh
!
interface Loopback0
description RID
ipv4 address 103.103.103.103 255.255.255.255
!
interface MgmtEth0/0/CPU0/0
description MANAGEMENT
vrf MGMT
ipv4 address 10.1.1.207 255.255.255.0
!
interface GigabitEthernet0/0/0/0
bandwidth 100
mtu 9000
!
interface GigabitEthernet0/0/0/0.42
description TO P_R02
bandwidth 100
mtu 9000
ipv4 address 10.1.1.74 255.255.255.252
encapsulation dot1q 42
!
interface GigabitEthernet0/0/0/0.43
description TO P_R04
bandwidth 100
mtu 9000
ipv4 address 10.1.1.78 255.255.255.252
encapsulation dot1q 43
!
interface GigabitEthernet0/0/0/1
description TO CE_R29
!
interface GigabitEthernet0/0/0/1.400 l2transport
description TO CE_R29 - ELAN VL400
encapsulation dot1q 400
!
interface GigabitEthernet0/0/0/2
shutdown
!
interface GigabitEthernet0/0/0/3
shutdown
!
router static
vrf MGMT
address-family ipv4 unicast
0.0.0.0/0 10.1.1.254
!
!
!
router ospf 100
log adjacency changes detail
router-id 103.103.103.103
passive enable
area 0
interface Loopback0
!
interface GigabitEthernet0/0/0/0.42
cost 3
network point-to-point
passive disable
mtu-ignore enable
!
interface GigabitEthernet0/0/0/0.43
cost 1
network point-to-point
passive disable
mtu-ignore enable
!
!
!
router bgp 2345
timers bgp 7 21
bgp router-id 103.103.103.103
address-family l2vpn evpn
!
neighbor 11.11.11.11
remote-as 2345
description TO ROUTE REFLECTOR - R11
update-source Loopback0
address-family l2vpn evpn
!
!
!
evpn
evi 40
description CUSTOMER_F_ELAN
!
!
l2vpn
pbb
backbone-source-mac 0003.0003.0003
!
bridge group PBB_EVPN_CORE_BG
bridge-domain PBB_EVPN_CORE_BD_EVI40
pbb core
evi 40
!
!
!
!
bridge group PBB_EVPN_EDGE_BG
bridge-domain PBB_EVPN_EDGE_BD_EVI40
interface GigabitEthernet0/0/0/1.400
!
pbb edge i-sid 40400 core-bridge PBB_EVPN_CORE_BD_EVI40
!
!
!
!
mpls ldp
router-id 103.103.103.103
interface GigabitEthernet0/0/0/0.42
!
interface GigabitEthernet0/0/0/0.43
!
!
mpls label range table 0 100300 100399
end

PBB-EVPN Specific Configurations

Configuration from only PE_XR01 were used since the other PEs will have similar configurations.

EVPN Instance

The following configuration builds the EVPN instance by defining the EVI ID. The ID value can be an arbitrary number between 1 – 65534.

evpn

evi 40

description CUSTOMER_F_ELAN

PBB Bridge Group/Domain

The following configuration defines the PBB parameters, such as the PE backbone MAC address, core & edge bridge groups and domains.

Per the PBB architecture, we created the 2 bridge domains. The core bridge domain (B-Component) specified the EVPN instance while the edge bridge domain (I-Component) defined the customer facing attachment circuit and unique backbone identifier (I-SID).

As a side note, many examples I’ve seen used a 2 bridge group configuration, separating the core bridge domain with the edge bridge domain. However, a single bridge group configuration seems to work as well, just as long as we have created 2 different bridge domains. I believe the bridge group was simply a configuration container to make things easier to organize the bridge domains.

l2vpn

pbb

backbone-source-mac 0001.0001.0001

bridge group PBB_EVPN_CORE_BG

bridge-domain PBB_EVPN_CORE_BD_EVI40

pbb core

evi 40

bridge group PBB_EVPN_EDGE_BG

bridge-domain PBB_EVPN_EDGE_BD_EVI40

interface GigabitEthernet0/0/0/1.400

pbb edge i-sid 40400 core-bridge PBB_EVPN_CORE_BD_EVI40

Attachment Circuit

The attachment circuit that connected the CE was configured to match VLAN 400 to classify Customer F’s traffic and placed it into the PBB edge bridge domain. It then popped the tag off as the DOT1q header was no longer required (or relevant) for MPLS forwarding.

interface GigabitEthernet0/0/0/1

description TO CE_R27

interface GigabitEthernet0/0/0/1.400 l2transport

description TO CE_R27 - ELAN VL400

encapsulation dot1q 400

rewrite ingress tag pop 1 symmetric

MP-BGP

MP-BGP peering was configured between the PEs to a Route Reflector (RR) under AFI 25 (L2VPN) & SAFI 70 (EVPN).

PE_XR01's BGP configuration to RR.

router bgp 2345

timers bgp 7 21

bgp router-id 101.101.101.101

address-family l2vpn evpn

neighbor 11.11.11.11

remote-as 2345

description TO ROUTE REFLECTOR - R11

update-source Loopback0

address-family l2vpn evpn

RR's BGP configuration to the PEs.

router bgp 2345

bgp log-neighbor-changes

no bgp default ipv4-unicast

neighbor PE_XR01 peer-group

neighbor PE_XR01 remote-as 2345

neighbor PE_XR01 description TO PE XR01

neighbor PE_XR01 update-source Loopback0

neighbor PE_XR01 timers 7 21

neighbor PE_XR02 peer-group

neighbor PE_XR02 remote-as 2345

neighbor PE_XR02 description TO PE XR02

neighbor PE_XR02 update-source Loopback0

neighbor PE_XR02 timers 7 21

neighbor PE_XR03 peer-group

neighbor PE_XR03 remote-as 2345

neighbor PE_XR03 description TO PE XR03

neighbor PE_XR03 update-source Loopback0

neighbor PE_XR03 timers 7 21

neighbor 101.101.101.101 peer-group PE_XR01

neighbor 102.102.102.102 peer-group PE_XR02

neighbor 103.103.103.103 peer-group PE_XR03

address-family ipv4

exit-address-family

address-family l2vpn evpn

neighbor PE_XR01 send-community both

neighbor PE_XR01 route-reflector-client

neighbor PE_XR02 send-community both

neighbor PE_XR02 route-reflector-client

neighbor PE_XR03 send-community both

neighbor PE_XR03 route-reflector-client

neighbor 101.101.101.101 activate

neighbor 102.102.102.102 activate

neighbor 103.103.103.103 activate

exit-address-family

CE Interface

The CE interface was configured to use VLAN 400 as transport (configuration from CE27).

interface GigabitEthernet1

description TO PE_XR01

no ip address

negotiation auto

no mop enabled

no mop sysid

interface GigabitEthernet1.400

description TO PE_XR01 - PBB-EVPN ELAN400

encapsulation dot1Q 400

ip address 172.16.4.1 255.255.255.0

Testing Results

As noted earlier, due to a lack of IOS XRv data plane support for L2VPN services, verification using ping tests could not be performed.

CE27 to CE28

CE_R27#ping 172.16.4.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.4.2, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

CE27 to CE29

CE_R27#ping 172.16.4.3

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.4.3, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

CE27’s ARP Table

CE_R27#sh arp

Protocol Address Age (min) Hardware Addr Type Interface

Internet 172.16.4.1 - 000c.299a.09c8 ARPA GigabitEthernet1.400

Internet 172.16.4.2 0 Incomplete ARPA

Internet 172.16.4.3 0 Incomplete ARPA

Verification Tasks

The list below outlined commands used to verify PBB-EVPN operation. Outputs were only from PE_XR01 for brevity.

OSPF Neighbors to Core
LDP Neighbors to Core
PBB Bridge Domains
PBB Backbone MACs (all PEs)
PBB Bridge Domain Forwarding
EVPN EVI
EVPN EVI MAC
EVPN EVI Neighbors
EVPN EVI Inclusive-Multicast
BGP L2VPN EVPN Summary
BGP L2VPN EVPN
BGP L2VPN EVPN NLRI [Type 2 & 3 Routes]

OSPF Neighbors

Verified the OSPF neighbors from PE_XR01 to MPLS core network (P1 and P3).

RP/0/0/CPU0:PE_XR01#sh ospf neighbor

Tue Oct 24 05:41:55.026 UTC

* Indicates MADJ interface

# Indicates Neighbor awaiting BFD session up

Neighbors for OSPF 100

Neighbor ID Pri State Dead Time Address Interface

1.1.1.1 1 FULL/ - 00:00:35 10.1.1.57 GigabitEthernet0/0/0/0.38

Neighbor is up for 1d03h

3.3.3.3 1 FULL/ - 00:00:35 10.1.1.61 GigabitEthernet0/0/0/0.39

Neighbor is up for 1d03h

Total neighbor count: 2

LDP Neighbors

Verified the LDP neighbors from PE_XR01 to MPLS core network (P1 and P3).

RP/0/0/CPU0:PE_XR01#sh mpls ldp neighbor

Tue Oct 24 05:42:33.524 UTC

Peer LDP Identifier: 3.3.3.3:0

TCP connection: 3.3.3.3:646 - 101.101.101.101:43929

Graceful Restart: No

Session Holdtime: 180 sec

State: Oper; Msgs sent/rcvd: 1986/1976; Downstream-Unsolicited

Up time: 1d04h

LDP Discovery Sources:

IPv4: (1)

GigabitEthernet0/0/0/0.39

IPv6: (0)

Addresses bound to this peer:

IPv4: (9)

3.3.3.3 10.0.0.6 10.0.0.18 10.0.0.21

10.0.0.101 10.1.1.5 10.1.1.33 10.1.1.41

10.1.1.61

IPv6: (0)

Peer LDP Identifier: 1.1.1.1:0

TCP connection: 1.1.1.1:646 - 101.101.101.101:13052

Graceful Restart: No

Session Holdtime: 180 sec

State: Oper; Msgs sent/rcvd: 1986/1977; Downstream-Unsolicited

Up time: 1d04h

LDP Discovery Sources:

IPv4: (1)

GigabitEthernet0/0/0/0.38

IPv6: (0)

Addresses bound to this peer:

IPv4: (9)

1.1.1.1 10.0.0.1 10.0.0.5 10.0.0.9

10.1.1.1 10.1.1.17 10.1.1.29 10.1.1.57

10.1.1.65

IPv6: (0)

PBB Bridge Domains

This command displays detailed information about the PBB bridge domains, such as:

Bridge Groups
Bridge Domains
Bridge Domain Type (core or edge)
I-SID
Bridge Domain, EVPN and AC operational status

RP/0/0/CPU0:PE_XR01#sh l2vpn bridge-domain

Tue Oct 24 05:49:59.893 UTC

Legend: pp = Partially Programmed.

Bridge group: PBB_EVPN_CORE_BG, bridge-domain: PBB_EVPN_CORE_BD_EVI40, id: 2, state: up, ShgId: 0, MSTi: 0

Type: pbb-core

Number of associated pbb-edge BDs: 1

Aging: 300 s, MAC limit: 4000, Action: none, Notification: syslog

Filter MAC addresses: 0

ACs: 0 (0 up), VFIs: 0, PWs: 0 (0 up), PBBs: 1 (1 up), VNIs: 0 (0 up)

List of PBBs:

PBB Core, state: up

List of EVPNs:

EVPN, state: up

List of ACs:

List of Access PWs:

List of VFIs:

Bridge group: PBB_EVPN_EDGE_BG, bridge-domain: PBB_EVPN_EDGE_BD_EVI40, id: 3, state: up, ShgId: 0, MSTi: 0

Type: pbb-edge, I-SID: 40400

Aging: 300 s, MAC limit: 4000, Action: none, Notification: syslog

Filter MAC addresses: 0

ACs: 1 (1 up), VFIs: 0, PWs: 0 (0 up), PBBs: 1 (1 up), VNIs: 0 (0 up)

List of PBBs:

PBB Edge, state: up, Static MAC addresses: 0

List of ACs:

Gi0/0/0/1.400, state: up, Static MAC addresses: 0, MSTi: 9

List of Access PWs:

List of VFIs:

PBB Backbone MACs

This command displays the configured PBB Backbone MAC addresses (from all PEs).

RP/0/0/CPU0:PE_XR01#sh l2vpn pbb backbone-source-mac

Tue Oct 24 06:23:10.357 UTC

Backbone Source MAC: 0001.0001.0001

Chassis MAC : 02b4.a7bf.0c05

RP/0/0/CPU0:PE_XR02#sh l2vpn pbb backbone-source-mac

Tue Oct 24 06:24:04.194 UTC

Backbone Source MAC: 0002.0002.0002

Chassis MAC : 0228.89f6.f405

RP/0/0/CPU0:PE_XR03#sh l2vpn pbb backbone-source-mac

Tue Oct 24 06:24:07.007 UTC

Backbone Source MAC: 0003.0003.0003

Chassis MAC : 024f.54bd.0c05

PBB Bridge Domain Forwarding

This command displays the PBB Bridge Domain’s forwarding table (i.e., PE’s data plane).

Note: Unfortunately my lab output did not show any C-MAC learning since IOS XRv does not have data plane support.

RP/0/0/CPU0:PE_XR01#sh l2vpn forwarding bridge-domain mac-address location 0/0/CPU0

Tue Oct 24 05:57:19.773 UTC

Mac Address Type Learned from/Filtered on LC learned Age Mapped to

-------------- ------- --------------------------- ---------- ---------------------- ----------

0002.0002.0002 BMAC BD id: 2 N/A N/A N/A

0003.0003.0003 BMAC BD id: 2 N/A N/A N/A

As a reference, the following output was a working example. Notice the MAC addresses starting with 2a22 and 2b22 being mapped to a single B-MAC bbbb.bbbb.bbbb.

https://netquirks.co.uk/category/pbb-evpn/

EVPN EVI

This command displays EVPN EVI details such as:

EVI ID
Bridge Domain name and type
MPLS labels for Unicast and Multicast (BUM) traffic
RD’s and RT’s

RP/0/0/CPU0:PE_XR01#sh evpn evi vpn-id 40 detail

Tue Oct 24 05:54:32.164 UTC

EVI Bridge Domain Type

---------- ---------------------------- -------

40 PBB_EVPN_CORE_BD_EVI40 PBB

Description : CUSTOMER_F_ELAN

Unicast Label : 100144

Multicast Label: 100145

Flow Label: N

Control-Word: Enabled

Forward-class: 0

RD Config: none

RD Auto : (auto) 101.101.101.101:40

RT Auto : 2345:40

Route Targets in Use Type

------------------------------ ---------------------

2345:40 Import

2345:40 Export

EVPN EVI MAC

This command displays B-MACs, next-hop information, ESI and MPLS VPN labels.

RP/0/0/CPU0:PE_XR01#sh evpn evi vpn-id 40 mac detail

Tue Oct 24 05:55:05.132 UTC

MAC address Nexthop Label vpn-id

-------------- --------------------------------------- -------- --------

0001.0001.0001 N/A 100144 40

Ether.Segment: 0000.0000.0000.0000.0000

ESI port key : 0x0000

Source : Local

Multi-paths resolved: FALSE

Multi-paths local label: 0

Flush Count : 3

0002.0002.0002 102.102.102.102 100244 40

Ether.Segment: 0000.0000.0000.0000.0000

ESI port key : 0x0000

Source : Remote, MPLS

Multi-paths resolved: FALSE

Multi-paths local label: 0

Flush Count : 1

0003.0003.0003 103.103.103.103 100344 40

Ether.Segment: 0000.0000.0000.0000.0000

ESI port key : 0x0000

Source : Remote, MPLS

Multi-paths resolved: FALSE

Multi-paths local label: 0

Flush Count : 1

EVPN EVI Neighbors

This command displays all its neighbors in a particular EVI.

RP/0/0/CPU0:PE_XR01#sh evpn evi neighbor

Tue Oct 24 05:54:09.686 UTC

Neighbor IP vpn-id

---------------------------------------- --------

102.102.102.102 40

103.103.103.103 40

EVPN EVI Inclusive-Multicast

This command displays the endpoints in the multicast flood list for BUM forwarding. It contains next-hop and forwarding labels.

RP/0/0/CPU0:PE_XR01#show evpn evi inclusive-multicast detail

Tue Oct 24 08:06:24.222 UTC

Ethernet Tag: 40400, Originating IP: 101.101.101.101, vpn-id: 40

Nexthop: ::

Label : 100145

Source : Local

Ethernet Tag: 40400, Originating IP: 102.102.102.102, vpn-id: 40

Nexthop: 102.102.102.102

Label : 100245

Source : Remote

Ethernet Tag: 40400, Originating IP: 103.103.103.103, vpn-id: 40

Nexthop: 103.103.103.103

Label : 100345

Source : Remote

BGP L2VPN EVPN Summary

This command displays the BGP neighbor peering status.

RP/0/0/CPU0:PE_XR01#sh bgp l2vpn evpn summary

Tue Oct 24 06:05:37.699 UTC

BGP router identifier 101.101.101.101, local AS number 2345

BGP generic scan interval 60 secs

Non-stop routing is enabled

BGP table state: Active

Table ID: 0x0 RD version: 0

BGP main routing table version 86

BGP NSR Initial initsync version 1 (Reached)

BGP NSR/ISSU Sync-Group versions 0/0

BGP scan interval 60 secs

BGP is operating in STANDALONE mode.

Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer

Speaker 86 86 86 86 86 0

Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd

11.11.11.11 0 2345 5224 5030 86 0 0 09:48:03 4

This was the BGP neighbor peering status from the RR’s perspective.

Note: Neighbors with 25.25.25.25 and 26.26.26.26 were PEs from the previous EVPN VPWS lab that used the same AFI/SAFI.

RR_R11#sh bgp l2vpn evpn summary

BGP router identifier 11.11.11.11, local AS number 2345

BGP table version is 100, main routing table version 100

8 network entries using 2752 bytes of memory

8 path entries using 1312 bytes of memory

6/6 BGP path/bestpath attribute entries using 1584 bytes of memory

7 BGP extended community entries using 668 bytes of memory

0 BGP route-map cache entries using 0 bytes of memory

0 BGP filter-list cache entries using 0 bytes of memory

BGP using 6316 total bytes of memory

BGP activity 33/20 prefixes, 45/32 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

25.25.25.25 4 2345 119786 119839 100 0 0 1w2d 1

26.26.26.26 4 2345 119833 119831 100 0 0 1w2d 1

101.101.101.101 4 2345 5079 5275 100 0 0 09:53:48 2

102.102.102.102 4 2345 5079 5270 100 0 0 09:53:44 2

103.103.103.103 4 2345 5081 5279 100 0 0 09:53:59 2

BGP L2VPN EVPN

This command displays the BGP table for the L2VPN EVPN address family.

This output contained Type 2 and Type 3 routes from each PE (from three different RDs). The routes highlighted were routes only related to PE_XR01’s RD.

RP/0/0/CPU0:PE_XR01#sh bgp l2vpn evpn

Tue Oct 24 06:05:18.270 UTC

BGP router identifier 101.101.101.101, local AS number 2345

BGP generic scan interval 60 secs

Non-stop routing is enabled

BGP table state: Active

Table ID: 0x0 RD version: 0

BGP main routing table version 86

BGP NSR Initial initsync version 1 (Reached)

BGP NSR/ISSU Sync-Group versions 0/0

BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best

i - internal, r RIB-failure, S stale, N Nexthop-discard

Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path

Route Distinguisher: 101.101.101.101:40 (default for vrf PBB_EVPN_CORE_BD_EVI40)

*> [2][0][48][0001.0001.0001][0]/104

0.0.0.0 0 i

*>i[2][0][48][0002.0002.0002][0]/104

102.102.102.102 100 0 i

*>i[2][0][48][0003.0003.0003][0]/104

103.103.103.103 100 0 i

*> [3][40400][32][101.101.101.101]/80

0.0.0.0 0 i

*>i[3][40400][32][102.102.102.102]/80

102.102.102.102 100 0 i

*>i[3][40400][32][103.103.103.103]/80

103.103.103.103 100 0 i

Route Distinguisher: 102.102.102.102:40

*>i[2][0][48][0002.0002.0002][0]/104

102.102.102.102 100 0 i

*>i[3][40400][32][102.102.102.102]/80

102.102.102.102 100 0 i

Route Distinguisher: 103.103.103.103:40

*>i[2][0][48][0003.0003.0003][0]/104

103.103.103.103 100 0 i

*>i[3][40400][32][103.103.103.103]/80

103.103.103.103 100 0 i

Processed 10 prefixes, 10 paths

The output below was a look at the RR's L2VPN EVPN BGP table.

RR_R11#sh bgp l2vpn evpn all

BGP table version is 100, local router ID is 11.11.11.11

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,

x best-external, a additional-path, c RIB-compressed,

Origin codes: i - IGP, e - EGP, ? - incomplete

RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path

Route Distinguisher: 101.101.101.101:40

*>i [2][101.101.101.101:40][00000000000000000000][0][48][000100010001][0][*][100144]/33

101.101.101.101 100 0 i

Route Distinguisher: 102.102.102.102:40

*>i [2][102.102.102.102:40][00000000000000000000][0][48][000200020002][0][*][100244]/33

102.102.102.102 100 0 i

Route Distinguisher: 103.103.103.103:40

*>i [2][103.103.103.103:40][00000000000000000000][0][48][000300030003][0][*][100344]/33

103.103.103.103 100 0 i

Route Distinguisher: 101.101.101.101:40

*>i [3][101.101.101.101:40][40400][32][662634496.3060.812b.277f]/17

101.101.101.101 100 0 i

Route Distinguisher: 102.102.102.102:40

*>i [3][102.102.102.102:40][40400][32][662634496.3060.812b.277f]/17

102.102.102.102 100 0 i

Route Distinguisher: 103.103.103.103:40

*>i [3][103.103.103.103:40][40400][32][662634496.3060.812b.277f]/17

103.103.103.103 100 0 i

BGP EVPN NLRI

Here’s a deeper look into the EVPN BGP Type 2 NLRI (MAC Advertisement).

Deciphering the route entry for: [2][0][48][0002.0002.0002][0]/104

BGP Route Type [2]
ESI [0]
MAC Address Length [48]
MAC Address (B-MAC) [0002.0002.0002]
IP Address (No IP address) [0]
Max Bits [/140]
Route Distinguisher [101.101.101.101:40]
MPLS VPN Label
Extended Community
Route Target
Ethernet Segment Identifier

RP/0/0/CPU0:PE_XR01#show bgp l2vpn evpn bdomain PBB_EVPN_CORE_BD_EVI40 [2][0][48][0002.0002.0002][0]/104

Tue Oct 24 06:08:01.089 UTC

BGP routing table entry for [2][0][48][0002.0002.0002][0]/104, Route Distinguisher: 101.101.101.101:40

Versions:

Process bRIB/RIB SendTblVer

Speaker 85 85

Last Modified: Oct 23 20:17:42.538 for 09:50:18

Paths: (1 available, best #1)

Not advertised to any peer

Path #1: Received by speaker 0

Not advertised to any peer

Local

102.102.102.102 (metric 6) from 11.11.11.11 (102.102.102.102)

Received Label 100244

Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported, rib-install

Received Path ID 0, Local Path ID 0, version 85

Extended community: EVPN MAC Mobility:3 RT:2345:40

Originator: 102.102.102.102, Cluster list: 11.11.11.11

EVPN ESI: 0000.0000.0000.0000.0000

Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 102.102.102.102:40

Here’s a deeper look into the EVPN BGP Type 3 NLRI (Inclusive-Multicast).

Deciphering the route entry for: [3][40400][32][102.102.102.102]/80

BGP Route Type [3]
Backbone Service ID (I-SID) [40400]
IP Address length [32]
IP Address of originating router [102.102.102.102]
Max Bits [/80]
Route Distinguisher [101.101.101.101:40]
Extended Community (RT)
Provider Multicast Service Interface (PMSI) Tunnel attributes:

Tunnel Type 6 (Ingress Replication)
MPLS VPN Label

RP/0/0/CPU0:PE_XR01#show bgp l2vpn evpn bdomain PBB_EVPN_CORE_BD_EVI40 [3][40400][32][102.102.102.102]/80

Tue Oct 24 06:08:42.516 UTC

BGP routing table entry for [3][40400][32][102.102.102.102]/80, Route Distinguisher: 101.101.101.101:40

Versions:

Process bRIB/RIB SendTblVer

Speaker 86 86

Last Modified: Oct 23 20:17:42.538 for 09:51:00

Paths: (1 available, best #1)

Not advertised to any peer

Path #1: Received by speaker 0

Not advertised to any peer

Local

102.102.102.102 (metric 6) from 11.11.11.11 (102.102.102.102)

Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported

Received Path ID 0, Local Path ID 0, version 86

Extended community: RT:2345:40

Originator: 102.102.102.102, Cluster list: 11.11.11.11

PMSI: flags 0x00, type 6, label 100245, ID 0x66666666

Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 102.102.102.102:40

MPLS Forwarding Trace

This was an example MPLS forwarding trace from PE_XR01 to PE_XR02’s B-MAC of 0002.0002.0002.

PE_XR01 performed a lookup of PE_XR02’s Type 2 B-MAC route and has received VPN label of 100244.

RP/0/0/CPU0:PE_XR01#show bgp l2vpn evpn bdomain PBB_EVPN_CORE_BD_EVI40 [2][0][48][0002.0002.0002][0]/104

Tue Oct 24 07:45:12.909 UTC

BGP routing table entry for [2][0][48][0002.0002.0002][0]/104, Route Distinguisher: 101.101.101.101:40

Versions:

Process bRIB/RIB SendTblVer

Speaker 85 85

Last Modified: Oct 23 20:17:42.538 for 11:27:30

Paths: (1 available, best #1)

Not advertised to any peer

Path #1: Received by speaker 0

Not advertised to any peer

Local

102.102.102.102 (metric 6) from 11.11.11.11 (102.102.102.102)

Received Label 100244

Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported, rib-install

Received Path ID 0, Local Path ID 0, version 85

Extended community: EVPN MAC Mobility:3 RT:2345:40

Originator: 102.102.102.102, Cluster list: 11.11.11.11

EVPN ESI: 0000.0000.0000.0000.0000

Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 102.102.102.102:40

PE_XR01 performed a MPLS forwarding prefix lookup for 102.102.102.102, then imposed an inner VPN label of 100244 and an outer LSP label of 334 then forwarded to P3.

RP/0/0/CPU0:PE_XR01#sh mpls forwarding prefix 102.102.102.102/32 detail

Tue Oct 24 07:46:03.626 UTC

Local Outgoing Prefix Outgoing Next Hop Bytes

Label Label or ID Interface Switched

------ ----------- ------------------ ------------ --------------- ------------

100143 334 102.102.102.102/32 Gi0/0/0/0.39 10.1.1.61 974625

Updated: Oct 23 02:36:45.216

Version: 142, Priority: 3

Label Stack (Top -> Bottom): { 334 }

NHID: 0x0, Encap-ID: N/A, Path idx: 0, Backup path idx: 0, Weight: 0

MAC/Encaps: 18/22, MTU: 8982

Packets Switched: 18184

P3 swapped label 334 with 235 and forwarded to P2.

P_R03#sh mpls forwarding-table labels 334 detail

Local Outgoing Prefix Bytes Label Outgoing Next Hop

Label Label or Tunnel Id Switched interface

334 235 102.102.102.102/32 \

1301588 Gi1.14 10.0.0.17

MAC/Encaps=18/22, MRU=9216, Label Stack{235}

000C294A72B4000C291EB42A8100000E8847 000EB000

No output feature configured

P2 popped label 235, then forwarded to PE_XR02.

P_R02#sh mpls forwarding-table labels 235 detail

Local Outgoing Prefix Bytes Label Outgoing Next Hop

Label Label or Tunnel Id Switched interface

235 Pop Label 102.102.102.102/32 \

2278385 Gi1.41 10.1.1.70

MAC/Encaps=18/18, MRU=9220, Label Stack{}

000C29692DBA000C294A72B4810000298847

No output feature configured

PE_XR02 looked up the VPN label of 100244 and forwarded out its configured AC interface.

RP/0/0/CPU0:PE_XR02#sh mpls forwarding labels 100244 detail

Tue Oct 24 07:51:21.145 UTC

Local Outgoing Prefix Outgoing Next Hop Bytes

Label Label or ID Interface Switched

------ ----------- ------------------ ------------ --------------- ------------

100244 Pop EVPN:0xc0000028 BD=2 PE point2point 0

Updated: Oct 23 02:02:18.209

PW Flow Label : Enabled

PW Flow Label: Enabled

Label Stack (Top -> Bottom): { }

MAC/Encaps: 0/0, MTU: 0

Packets Switched: 0

References

Provider Backbone Bridging (PBB):

https://sites.google.com/site/amitsciscozone/home/pbb/understanding-pbb

https://www.juniper.net/documentation/en_US/junos12.3/topics/concept/pbb-understanding.html

http://www.brocade.com/content/html/en/configuration-guide/NI_05800a_SWITCHING/GUID-39750C1A-4994-4B7C-B988-1254C87F859E.html

https://www.tatacommunications.com/vpn/PBBknowledgeCenter/BRKSPG-2203.pdf

BGP/MPLS EVPN:

http://bgphelp.com/2017/06/12/evpn-faq/

http://wiki.kemot-net.com/vpn-evpn-overview

https://tools.ietf.org/html/rfc7432

https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5500/vpn/61x/b-ncs5500-l2vpn-configuration-guide-61x/b-ncs5500-l2vpn-configuration-guide-61x_chapter_01000.html

https://www.broadband-forum.org/marketing/download/mktgdocs/BBF_EVPN_tutorial.pdf

PBB-EVPN:

https://www.cisco.com/c/en/us/products/collateral/routers/asr-9000-series-aggregation-services-routers/whitepaper_c11-731864.html

https://www.juniper.net/documentation/en_US/junos/topics/concept/pbb-evpn-integration-for-dci-overview.html

https://tools.ietf.org/html/rfc7623

Cisco Live Sessions and Other Presentations

https://www.ciscolive.com/global/on-demand-library/?search=pbb%20evpn#/session/14545296381430016dUA

https://www.ciscolive.com/global/on-demand-library/?search=e-vpn#/session/1421636379562001lvYZ

https://www.slideshare.net/getyourbuildon/pbbevpn-cisco-ckn-techadvantage-webinar

Other Blogs and Labs

(Personal thanks for all who posted and shared their lab experience and knowledge.)

http://theriker.blogspot.com/2017/01/ccie-spv4-mpls-l2vpn-evpn-and-pbb-evpn.html

https://netquirks.co.uk/category/pbb-evpn/

https://ccie49534.com/2017/04/23/pbb-evpn-a-new-flexible-l2vpn-service/

https://tgregory.org/2016/11/15/evpn-vs-pbb-evpn-and-massive-scale/

http://www.networkers.fi/blog/dci-with-pbb-evpn-and-cisco-asr9000/

http://eng-mssk.blogspot.com/2017/01/pbb-evpn.html

https://robertalvianus.wordpress.com/2015/06/25/provider-backbone-bridging-ethernet-vpn-pbb-evpn-part-2/

the peering xchange : [a network engineer’s blog]

Pages

Sunday, November 19, 2017

MPLS L2VPN (PBB-EVPN)

No comments:

Post a Comment

Report Abuse