the peering xchange : [a network engineer’s blog]

As a continuation of my work with EVPN technologies using Juniper vMX routers, I reconfigured my previous EVPN lab to test Provider Backbone Bridging with Ethernet VPN (PBB-EVPN, RFC 7623). The aim here was to learn the operational differences between PBB-EVPN vs. RFC 7432 EVPN, and how PBB-EVPN's control plane scaling can benefit very large L2VPN use cases, such as with Datacenter Interconnects (DCI).

Lab Environment

Note: These resources were appended to the existing IP/MPLS core network running 4 Cisco IOS XE Version 03.16.05.S.

(3) Juniper vMX: Version 17.2R1.13
(6) Cisco CSR1000v: Version 16.4.2

Diagram & Topology

No changes to the existing Juniper PEs (PE_MXR01, PE_MXR02 and PE_MXR03) were made from the EVPN Lab. An additional Cisco CSR1000v router was attached to each PE, to have 2 routers per PE acting as end hosts.

Infrastructure Information

PE_MXR01, PE_MXR02 and PE_MXR03 iBGP Full Mesh Peering
CUSTOMER G ELAN Information

EVPN Instance Names:

PBB_EVPN_B_COMP_CUSTOMER_G
PBB_EVPN_I_COMP_CUSTOMER_G

Customer VLAN:

CE Network IP Addresses:

172.16.50.0/24

CE AC Interface:

GigabitEthernet1

PE AC Interface:

GE-0/0/2

PBB Customer Backbone Service Instance Identifier (I-SID):

10500

Singled Homed

Technology Overview

For a PBB-EVPN technology overview, please look at my previous PBB-EVPN post. This blog post also does an excellent job explaining the technical details and operation of PBB-EVPN in a Juniper environment.

Configurations

Full configurations from all PEs are shown below.

PE_MXR01

version 17.2R1.13;
system {
    host-name PE_MXR01;
    root-authentication {
        encrypted-password "<removed>"; ## SECRET-DATA
    }
    login {
        user admin {
            uid 2000;
            class super-user;
            authentication {
                encrypted-password "<removed>"; ## SECRET-DATA
            }
        }
    }
    services {
        ssh {
            root-login allow;
        }
    }
    syslog {
        user * {
            any emergency;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
    processes {
        dhcp-service {
            traceoptions {
                file dhcp_logfile size 10m;
                level all;
                flag all;
            }
        }
    }
}
chassis {
    fpc 0 {
        lite-mode;
    }
    network-services enhanced-ip;
}
interfaces {
    ge-0/0/0 {
        unit 0 {
            description MGMT;
            family inet {
                address 10.1.1.204/24;
            }
        }
    }
    ge-0/0/1 {
        vlan-tagging;
        mtu 9234;
        unit 44 {
            description "TO P_R03";
            vlan-id 44;
            family inet {
                address 10.1.1.82/30;
            }
            family mpls;
        }
        unit 45 {
            description "TO P_R01";
            vlan-id 45;
            family inet {
                address 10.1.1.86/30;
            }
            family mpls;
        }
    }
    ge-0/0/2 {
        description "TO CUSTOMER_G ELAN 500 VSWITCH_24";
        flexible-vlan-tagging;
        encapsulation flexible-ethernet-services;
        unit 500 {
            encapsulation vlan-bridge;
            vlan-id 500;
        }
    }
    cbp0 {
        unit 500 {
            family bridge {
                interface-mode trunk;
                bridge-domain-type bvlan;
                isid-list all;
            }
        }
    }
    fxp0 {
        description "RE MGMT";
        unit 0 {
            disable;
            family inet {
                dhcp {
                    vendor-id Juniper-vmx;
                }
            }
        }
    }
    lo0 {
        unit 111 {
            description RID;
            family inet {
                address 111.111.111.111/32;
            }
        }
    }
    pip0 {
        unit 500 {
            family bridge {
                interface-mode trunk;
                bridge-domain-type svlan;
                isid-list all-service-groups;
            }
        }
    }
}
routing-options {
    router-id 111.111.111.111;
    autonomous-system 2345;
}
protocols {
    mpls {
        label-range {
            dynamic-label-range 111000 111999;
        }
    }
    bgp {
        group PE {
            type internal;
            local-address 111.111.111.111;
            hold-time 21;
            family evpn {
                signaling;
            }
            neighbor 112.112.112.112;
            neighbor 113.113.113.113;
        }
    }
    ospf {
        area 0.0.0.0 {
            interface ge-0/0/1.44 {
                interface-type p2p;
                metric 1;
            }
            interface lo0.111 {
                passive;
            }
            interface ge-0/0/1.45 {
                interface-type p2p;
                metric 4;
            }
        }
    }
    ldp {
        interface ge-0/0/1.44;
        interface ge-0/0/1.45;
    }
}
routing-instances {
    MGMT {
        instance-type virtual-router;
        interface ge-0/0/0.0;
        routing-options {
            static {
                route 0.0.0.0/0 next-hop 10.1.1.254;
            }
        }
    }
    PBB_EVPN_B_COMP_CUSTOMER_G {
        instance-type virtual-switch;
        interface cbp0.500;
        route-distinguisher 111.111.111.111:50;
        vrf-target target:2345:50;
        protocols {
            evpn {
                pbb-evpn-core;
                extended-isid-list 10500;
            }
        }
        bridge-domains {
            BD_500 {
                vlan-id 500;
                isid-list 10500;
                vlan-id-scope-local;
            }
        }
    }
    PBB_EVPN_I_COMP_CUSTOMER_G {
        instance-type virtual-switch;
        interface pip0.500;
        bridge-domains {
            BD_500 {
                domain-type bridge;
                vlan-id 500;
                interface ge-0/0/2.500;
            }
        }
        pbb-options {
            peer-instance PBB_EVPN_B_COMP_CUSTOMER_G;
        }
        service-groups {
            SG_500 {
                service-type elan;
                pbb-service-options {
                    isid 10500 vlan-id-list 500;
                    source-bmac 00:01:01:01:01:01;
                }
            }
        }
    }
}

PE_MXR02

version 17.2R1.13;
system {
    host-name PE_MXR02;
    root-authentication {
        encrypted-password "<removed>"; ## SECRET-DATA
    }
    login {
        user admin {
            uid 2000;
            class super-user;
            authentication {
                encrypted-password "<removed>"; ## SECRET-DATA
            }
        }
    }
    services {
        ssh {
            root-login allow;
        }
    }
    syslog {
        user * {
            any emergency;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
    processes {
        dhcp-service {
            traceoptions {
                file dhcp_logfile size 10m;
                level all;
                flag all;
            }
        }
    }
}
chassis {
    fpc 0 {
        lite-mode;
    }
    network-services enhanced-ip;
}
interfaces {
    ge-0/0/0 {
        unit 0 {
            description MGMT;
            family inet {
                address 10.1.1.205/24;
            }
        }
    }
    ge-0/0/1 {
        vlan-tagging;
        mtu 9234;
        unit 46 {
            description "TO P_R03";
            vlan-id 46;
            family inet {
                address 10.1.1.90/30;
            }
            family mpls;
        }
        unit 47 {
            description "TO P_R04";
            vlan-id 47;
            family inet {
                address 10.1.1.94/30;
            }
            family mpls;
        }
    }
    ge-0/0/2 {
        description "TO CUSTOMER_G ELAN 500 VSWITCH";
        flexible-vlan-tagging;
        encapsulation flexible-ethernet-services;
        unit 500 {
            encapsulation vlan-bridge;
            vlan-id 500;
        }
    }
    cbp0 {
        unit 500 {
            family bridge {
                interface-mode trunk;
                bridge-domain-type bvlan;
                isid-list all;
            }
        }
    }
    fxp0 {
        description "RE MGMT";
        unit 0 {
            disable;
            family inet {
                dhcp {
                    vendor-id Juniper-vmx;
                }
            }
        }
    }
    lo0 {
        unit 112 {
            description RID;
            family inet {
                address 112.112.112.112/32;
            }
        }
    }
    pip0 {
        unit 500 {
            family bridge {
                interface-mode trunk;
                bridge-domain-type svlan;
                isid-list all-service-groups;
            }
        }
    }
}
routing-options {
    router-id 112.112.112.112;
    autonomous-system 2345;
}
protocols {
    mpls {
        label-range {
            dynamic-label-range 112000 112999;
        }
    }
    bgp {
        group PE {
            type internal;
            local-address 112.112.112.112;
            hold-time 21;
            family evpn {
                signaling;
            }
            neighbor 111.111.111.111;
            neighbor 113.113.113.113;
        }
    }
    ospf {
        area 0.0.0.0 {
            interface ge-0/0/1.46 {
                interface-type p2p;
                metric 1;
            }
            interface ge-0/0/1.47 {
                interface-type p2p;
                metric 5;
            }
            interface lo0.112 {
                passive;
            }
        }
    }
    ldp {
        interface ge-0/0/1.46;
        interface ge-0/0/1.47;
    }
}
routing-instances {
    MGMT {
        instance-type virtual-router;
        interface ge-0/0/0.0;
        routing-options {
            static {
                route 0.0.0.0/0 next-hop 10.1.1.254;
            }
        }
    }
    PBB_EVPN_B_COMP_CUSTOMER_G {
        instance-type virtual-switch;
        interface cbp0.500;
        route-distinguisher 112.112.112.112:50;
        vrf-target target:2345:50;
        protocols {
            evpn {
                pbb-evpn-core;
                extended-isid-list 10500;
            }
        }
        bridge-domains {
            BD_500 {
                vlan-id 500;
                isid-list 10500;
                vlan-id-scope-local;
            }
        }
    }
    PBB_EVPN_I_COMP_CUSTOMER_G {
        instance-type virtual-switch;
        interface pip0.500;
        bridge-domains {
            BD_500 {
                domain-type bridge;
                vlan-id 500;
                interface ge-0/0/2.500;
            }
        }
        pbb-options {
            peer-instance PBB_EVPN_B_COMP_CUSTOMER_G;
        }
        service-groups {
            SG_500 {
                service-type elan;
                pbb-service-options {
                    isid 10500 vlan-id-list 500;
                    source-bmac 00:02:02:02:02:02;
                }
            }
        }
    }
}

PE_MXR03

version 17.2R1.13;
system {
    host-name PE_MXR03;
    root-authentication {
        encrypted-password "<removed>"; ## SECRET-DATA
    }
    login {
        user admin {
            uid 2000;
            class super-user;
            authentication {
                encrypted-password "<removed>"; ## SECRET-DATA
            }
        }
    }
    services {
        ssh {
            root-login allow;
        }
    }
    syslog {
        user * {
            any emergency;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
    processes {
        dhcp-service {
            traceoptions {
                file dhcp_logfile size 10m;
                level all;
                flag all;
            }
        }
    }
}
chassis {
    fpc 0 {
        lite-mode;
    }
    network-services enhanced-ip;
}
interfaces {
    ge-0/0/0 {
        unit 0 {
            description MGMT;
            family inet {
                address 10.1.1.206/24;
            }
        }
    }
    ge-0/0/1 {
        vlan-tagging;
        mtu 9234;
        unit 48 {
            description "TO P_R02";
            vlan-id 48;
            family inet {
                address 10.1.1.98/30;
            }
            family mpls;
        }
        unit 49 {
            description "TO P_R01";
            vlan-id 49;
            family inet {
                address 10.1.1.102/30;
            }
            family mpls;
        }
    }
    ge-0/0/2 {
        description "TO CUSTOMER_G ELAN 500 VSWITCH_26";
        flexible-vlan-tagging;
        encapsulation flexible-ethernet-services;
        unit 500 {
            encapsulation vlan-bridge;
            vlan-id 500;
        }
    }
    ge-0/0/3 {
        description "TO CUSTOMER_G ELAN 500 VSWITCH_27";
        disable;
        flexible-vlan-tagging;
        encapsulation flexible-ethernet-services;
    }
    cbp0 {
        unit 500 {
            family bridge {
                interface-mode trunk;
                bridge-domain-type bvlan;
                isid-list all;
            }
        }
    }
    fxp0 {
        description "RE MGMT";
        unit 0 {
            disable;
            family inet {
                dhcp {
                    vendor-id Juniper-vmx;
                }
            }
        }
    }
    lo0 {
        unit 113 {
            description RID;
            family inet {
                address 113.113.113.113/32;
            }
        }
    }
    pip0 {
        unit 500 {
            family bridge {
                interface-mode trunk;
                bridge-domain-type svlan;
                isid-list all-service-groups;
            }
        }
    }
}
routing-options {
    router-id 113.113.113.113;
    autonomous-system 2345;
}
protocols {
    mpls {
        label-range {
            dynamic-label-range 113000 113999;
        }
    }
    bgp {
        group PE {
            type internal;
            local-address 113.113.113.113;
            hold-time 21;
            family evpn {
                signaling;
            }
            neighbor 112.112.112.112;
            neighbor 111.111.111.111;
        }
    }
    ospf {
        area 0.0.0.0 {
            interface ge-0/0/1.48 {
                interface-type p2p;
                metric 1;
            }
            interface ge-0/0/1.49 {
                interface-type p2p;
                metric 5;
            }
            interface lo0.113 {
                passive;
            }
        }
    }
    ldp {
        interface ge-0/0/1.48;
        interface ge-0/0/1.49;
    }
}
routing-instances {
    MGMT {
        instance-type virtual-router;
        interface ge-0/0/0.0;
        routing-options {
            static {
                route 0.0.0.0/0 next-hop 10.1.1.254;
            }
        }
    }
    PBB_EVPN_B_COMP_CUSTOMER_G {
        instance-type virtual-switch;
        interface cbp0.500;
        route-distinguisher 113.113.113.113:50;
        vrf-target target:2345:50;
        protocols {
            evpn {
                pbb-evpn-core;
                extended-isid-list 10500;
            }
        }
        bridge-domains {
            BD_500 {
                vlan-id 500;
                isid-list 10500;
                vlan-id-scope-local;
            }
        }
    }
    PBB_EVPN_I_COMP_CUSTOMER_G {
        instance-type virtual-switch;
        interface pip0.500;
        bridge-domains {
            BD_500 {
                domain-type bridge;
                vlan-id 500;
                interface ge-0/0/2.500;
            }
        }
        pbb-options {
            peer-instance PBB_EVPN_B_COMP_CUSTOMER_G;
        }
        service-groups {
            SG_500 {
                service-type elan;
                pbb-service-options {
                    isid 10500 vlan-id-list 500;
                    source-bmac 00:03:03:03:03:03;
                }
            }
        }
    }
}

PBB-EVPN Specific Configurations

In this section, PE_MXR01's configuration was used as the primary example since the other PEs have similar configuration.

EVPN Instance

PBB-EVPN requires two routing instances to segment the customer and backbone traffic. The I-COMP instance defines the customer facing bridge domain, the AC interface and I-SID. It’s responsible for mapping the customer Ethernet traffic to the I-SID and maintains the C-MAC to B-MAC table. The B-COMP, or backbone instance, defines the core bridge domain and is responsible for the forwarding of traffic based on B-MACs.

routing-instances {

PBB_EVPN_B_COMP_CUSTOMER_G {

instance-type virtual-switch;

interface cbp0.500;

route-distinguisher 111.111.111.111:50;

vrf-target target:2345:50;

protocols {

evpn {

pbb-evpn-core;

extended-isid-list 10500;

}

bridge-domains {

BD_500 {

vlan-id 500;

isid-list 10500;

vlan-id-scope-local;

}

PBB_EVPN_I_COMP_CUSTOMER_G {

instance-type virtual-switch;

interface pip0.500;

bridge-domains {

BD_500 {

domain-type bridge;

vlan-id 500;

interface ge-0/0/2.500;

}

pbb-options {

peer-instance PBB_EVPN_B_COMP_CUSTOMER_G;

}

service-groups {

SG_500 {

service-type elan;

pbb-service-options {

isid 10500 vlan-id-list 500;

source-bmac 00:01:01:01:01:01;

}

PBB-EVPN Interfaces

As with the separate instances, PBB-EVPN requires a couple of logical interfaces to join the I-COMP and B-COMP instances together. The Provider Instance Port (PIP) and the Customer Backbone Port (CBP) accomplish this.

The PIP performs the PBB encapsulation/decapsulation, such as pushing & popping of the I-SID and mapping the source/destination B-MACs to customer traffic. The CBP performs the transmission/receiving of traffic customer traffic in the core bridge.

interfaces {

..snip..

cbp0 {

unit 500 {

family bridge {

interface-mode trunk;

bridge-domain-type bvlan;

isid-list all;

}

..snip..

pip0 {

unit 500 {

family bridge {

interface-mode trunk;

bridge-domain-type svlan;

isid-list all-service-groups;

}

Attachment Circuit (PE)

The PE’s attachment circuit is configured to match VLAN 500 to classify the customer’s traffic.

interfaces {

ge-0/0/2 {

description "TO CUSTOMER_G ELAN 500 VSWITCH_24";

flexible-vlan-tagging;

encapsulation flexible-ethernet-services;

unit 500 {

encapsulation vlan-bridge;

vlan-id 500;

}

Attachment Circuit (CE)

The CE’s interface is configured to tag VLAN 500.

interface GigabitEthernet1

no ip address

load-interval 30

negotiation auto

interface GigabitEthernet1.500

description TO PE_MXR01

encapsulation dot1Q 500

ip address 172.16.50.1 255.255.255.0

MP-BGP

A full mesh of MP-iBGP peers are configured between the PEs under AFI 25 (L2VPN) & SAFI 70 (EVPN).

routing-options {

router-id 111.111.111.111;

autonomous-system 2345;

}

protocols {

bgp {

group PE {

type internal;

local-address 111.111.111.111;

hold-time 21;

family evpn {

signaling;

}

neighbor 112.112.112.112;

neighbor 113.113.113.113;

}

Testing Results

CE_R27 → CE_R29 – 32

Basic ping tests were used to validate connectivity from CE_R27 → CE_R29 – R32.

CE_R27#ping 172.16.50.3

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.50.3, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 3/10/25 ms

CE_R27#ping 172.16.50.4

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.50.4, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 3/10/26 ms

CE_R27#ping 172.16.50.5

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.50.5, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 4/11/29 ms

CE_R27#ping 172.16.50.6

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.50.6, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 4/12/25 ms

CE_R31 → CE_R27 & CE_R28

Connectivity was validated to CE_R27 and CE_R28 from CE_R31. Since R27/R28 were connected off the same vSwitch, the validation test was performed from a host located off another PE.

CE_R31#ping 172.16.50.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.50.1, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 5/32/50 ms

CE_R31#ping 172.16.50.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.50.2, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 4/14/37 ms

CE_R27’s ARP Table

CE_R27#sh arp

Protocol Address Age (min) Hardware Addr Type Interface

Internet 172.16.50.1 - 000c.299a.09c8 ARPA GigabitEthernet1.500

Internet 172.16.50.2 23 000c.2949.aa8c ARPA GigabitEthernet1.500

Internet 172.16.50.3 16 000c.29e4.3ecd ARPA GigabitEthernet1.500

Internet 172.16.50.4 25 000c.29db.195d ARPA GigabitEthernet1.500

Internet 172.16.50.5 6 000c.299a.1a48 ARPA GigabitEthernet1.500

Internet 172.16.50.6 23 000c.29cb.1db0 ARPA GigabitEthernet1.500

CE_R31’s ARP Table

CE_R31#sh arp

Protocol Address Age (min) Hardware Addr Type Interface

Internet 172.16.50.1 23 000c.299a.09c8 ARPA GigabitEthernet1.500

Internet 172.16.50.2 6 000c.2949.aa8c ARPA GigabitEthernet1.500

Internet 172.16.50.3 16 000c.29e4.3ecd ARPA GigabitEthernet1.500

Internet 172.16.50.4 6 000c.29db.195d ARPA GigabitEthernet1.500

Internet 172.16.50.5 - 000c.299a.1a48 ARPA GigabitEthernet1.500

Internet 172.16.50.6 6 000c.29cb.1db0 ARPA GigabitEthernet1.500

Verification Tasks

The following commands were used to verify PBB-EVPN operation. Most outputs seen below were from PE_MXR01 for brevity, unless otherwise indicated.

Show EVPN Instance (extensive)
Show EVPN Database
Show EVPN Database (extensive)
Show Bridge MAC Table Count
Show Bridge MAC Table
Show BGP Summary
Show Route Advertising-Protocol BGP
Show Route Receive-Protocol BGP
Show Route Table EVPN
Show Route Table EVPN Extensive (Type 2 NLRI)
Show Route Table EVPN Extensive (Type 3 NLRI)

EVPN Instance (extensive)

Displays detailed information about the PBB-EVPN core instance (B-COMP).

EVPN Instance Name
Instance Status
Customer Backbone Port Status
I-SID
EVPN Neighbors and learned MACs

admin@PE_MXR01> show evpn instance extensive

Instance: PBB_EVPN_B_COMP_CUSTOMER_G

Route Distinguisher: 111.111.111.111:50

Per-instance MAC route label: 111031

Per-instance multicast route label: 111032

PBB EVPN Core enabled

MAC database status Local Remote

MAC advertisements: 1 2

MAC+IP advertisements: 0 0

Default gateway MAC advertisements: 0 0

Number of local interfaces: 1 (1 up)

Interface name ESI Mode Status AC-Role

cbp0.500 00:00:00:00:00:00:00:00:00:00 single-homed Up Root

Number of IRB interfaces: 0 (0 up)

Number of bridge domains: 1

VLAN Domain ID Intfs / up IRB intf Mode MAC sync IM route label SG sync IM core nexthop

10500 0 0 Extended Enabled 111032 Disabled

Number of Bundle bridge domains: 0

Number of neighbors: 2

Address MAC MAC+IP AD IM ES Leaf-label

112.112.112.112 1 0 0 1 0

113.113.113.113 1 0 0 1 0

Number of ethernet segments: 0

Instance: __default_evpn__

Route Distinguisher: 111.111.111.111:0

Number of bridge domains: 0

Number of neighbors: 0

EVPN Database

Displays the MAC addresses in the EVPN database. With PBB-EVPN, the database maps the backbone MAC (B-MAC) vs. the customer MAC (C-MAC) as seen in EVPN.

admin@PE_MXR01> show evpn database

Instance: PBB_EVPN_B_COMP_CUSTOMER_G

VLAN DomainId MAC address Active source Timestamp IP address

10500 00:01:01:01:01:01 Local Jun 14 20:54:06

10500 00:02:02:02:02:02 112.112.112.112 Jun 14 20:58:17

10500 00:03:03:03:03:03 113.113.113.113 Jun 14 21:01:29

admin@PE_MXR02> show evpn database

Instance: PBB_EVPN_B_COMP_CUSTOMER_G

VLAN DomainId MAC address Active source Timestamp IP address

10500 00:01:01:01:01:01 111.111.111.111 Jun 14 20:54:06

10500 00:02:02:02:02:02 Local Jun 14 20:58:16

10500 00:03:03:03:03:03 113.113.113.113 Jun 14 21:01:29

admin@PE_MXR03> show evpn database

Instance: PBB_EVPN_B_COMP_CUSTOMER_G

VLAN DomainId MAC address Active source Timestamp IP address

10500 00:01:01:01:01:01 111.111.111.111 Jun 14 20:54:06

10500 00:02:02:02:02:02 112.112.112.112 Jun 14 20:58:17

10500 00:03:03:03:03:03 Local Jun 14 21:01:28

EVPN Database (extensive)

Displays a detailed output of the EVPN database, including the MPLS labels allocated for each B-MAC.

admin@PE_MXR01> show evpn database extensive

Instance: PBB_EVPN_B_COMP_CUSTOMER_G

MAC address:: 00:01:01:01:01:01, ISID: 10500, Active source: Local

State: <Adv Local>

MAC address:: 00:02:02:02:02:02, ISID: 10500, Active source: 112.112.112.112

State: <Adv>, Nexthop ID: 1048574

Source: Single home, Rank: 1, Status: Active

Remote origin: 112.112.112.112 MAC label: 112050

MAC address:: 00:03:03:03:03:03, ISID: 10500, Active source: 113.113.113.113

State: <Adv>, Nexthop ID: 1048575

Source: Single home, Rank: 1, Status: Active

Remote origin: 113.113.113.113 MAC label: 113067

Bridge MAC Table Count

Displays the number of MAC address learned in both the customer facing (I-COMP) and backbone facing (B-COMP) bridge domains.

In this lab, the B-COMP has learned 3 B-MACs while the I-COMP has learned 6 C-MACs (2 from the local interface and 4 from the other PEs). Outputs from all PEs are shown below.

admin@PE_MXR01> show bridge mac-table count

3 MAC address learned in routing instance PBB_EVPN_B_COMP_CUSTOMER_G bridge domain BD_500

MAC address count per learn VLAN within routing instance:

Learn VLAN ID MAC count

500 3

6 MAC address learned in routing instance PBB_EVPN_I_COMP_CUSTOMER_G bridge domain BD_500

MAC address count per interface within routing instance:

Logical interface MAC count

ge-0/0/2.500:500 2

rbeb.32768 2

rbeb.32769 2

MAC address count per learn VLAN within routing instance:

Learn VLAN ID MAC count

500 6

admin@PE_MXR02> show bridge mac-table count

3 MAC address learned in routing instance PBB_EVPN_B_COMP_CUSTOMER_G bridge domain BD_500

MAC address count per learn VLAN within routing instance:

Learn VLAN ID MAC count

500 3

6 MAC address learned in routing instance PBB_EVPN_I_COMP_CUSTOMER_G bridge domain BD_500

MAC address count per interface within routing instance:

Logical interface MAC count

ge-0/0/2.500:500 2

rbeb.32768 2

rbeb.32769 2

MAC address count per learn VLAN within routing instance:

Learn VLAN ID MAC count

500 6

admin@PE_MXR03> show bridge mac-table count

3 MAC address learned in routing instance PBB_EVPN_B_COMP_CUSTOMER_G bridge domain BD_500

MAC address count per learn VLAN within routing instance:

Learn VLAN ID MAC count

500 3

6 MAC address learned in routing instance PBB_EVPN_I_COMP_CUSTOMER_G bridge domain BD_500

MAC address count per interface within routing instance:

Logical interface MAC count

ge-0/0/2.500:500 2

rbeb.32768 2

rbeb.32769 2

MAC address count per learn VLAN within routing instance:

Learn VLAN ID MAC count

500 6

Bridge MAC Table

Displays the indivdual MAC addresses learned in the routing instance’s bridge domains. The B-COMP bridge domain contains the B-MACs learned from the other PEs. PBB introduces an additional MAC address (01:1e:83:00:29:04) called the Group B-MAC. It is a special destination address for BUM traffic delivery. It consists of the standard group OUI of 01:1e:83 plus the I-SID translated into HEX. So in other words, with an I-SID of 100500 used in this lab, the last 3 octect of this address will be translated to 00:29:04. The article below explains this concept in more detail.

https://infoproducts.alcatel-lucent.com/html/0_add-h-f/93-0076-10-01/7750_SR_OS_Services_Guide/services_PBB.html

The I-COMP bridge domain contains the C-MACs learned over the local AC interface and from the control plane via the other PEs. The 2 local end host’s C-MACs were learned over the customer facing AC interface and 4 C-MACs were learned over the control plane and mapped to the backbone addresses.

admin@PE_MXR01> show bridge mac-table

MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC

O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)

Routing instance : PBB_EVPN_B_COMP_CUSTOMER_G

Bridging domain : BD_500, VLAN : 500

MAC MAC Logical NH MAC

address flags interface Index property

00:02:02:02:02:02 DC 1048574

00:03:03:03:03:03 DC 1048575

01:1e:83:00:29:04 DC 1048578

MAC flags (S -static MAC, D -dynamic MAC,

SE -Statistics enabled, NM -Non configured MAC, P -Pinned MAC)

Routing instance : PBB_EVPN_I_COMP_CUSTOMER_G

Bridging domain : BD_500, ISID : 10500, VLAN : 500

MAC MAC Logical Remote

address flags interface BEB address

00:0c:29:49:aa:8c D ge-0/0/2.500

00:0c:29:9a:09:c8 D ge-0/0/2.500

00:0c:29:9a:1a:48 D rbeb.32769 00:03:03:03:03:03

00:0c:29:cb:1d:b0 D rbeb.32769 00:03:03:03:03:03

00:0c:29:db:19:5d D rbeb.32768 00:02:02:02:02:02

00:0c:29:e4:3e:cd D rbeb.32768 00:02:02:02:02:02

admin@PE_MXR02> show bridge mac-table

MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC

O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)

Routing instance : PBB_EVPN_B_COMP_CUSTOMER_G

Bridging domain : BD_500, VLAN : 500

MAC MAC Logical NH MAC

address flags interface Index property

00:01:01:01:01:01 DC 1048578

00:03:03:03:03:03 DC 1048574

01:1e:83:00:29:04 DC 1048577

MAC flags (S -static MAC, D -dynamic MAC,

SE -Statistics enabled, NM -Non configured MAC, P -Pinned MAC)

Routing instance : PBB_EVPN_I_COMP_CUSTOMER_G

Bridging domain : BD_500, ISID : 10500, VLAN : 500

MAC MAC Logical Remote

address flags interface BEB address

00:0c:29:49:aa:8c D rbeb.32768 00:01:01:01:01:01

00:0c:29:9a:09:c8 D rbeb.32768 00:01:01:01:01:01

00:0c:29:9a:1a:48 D rbeb.32769 00:03:03:03:03:03

00:0c:29:cb:1d:b0 D rbeb.32769 00:03:03:03:03:03

00:0c:29:db:19:5d D ge-0/0/2.500

00:0c:29:e4:3e:cd D ge-0/0/2.500

admin@PE_MXR03> show bridge mac-table

MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC

O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)

Routing instance : PBB_EVPN_B_COMP_CUSTOMER_G

Bridging domain : BD_500, VLAN : 500

MAC MAC Logical NH MAC

address flags interface Index property

00:01:01:01:01:01 DC 1048577

00:02:02:02:02:02 DC 1048575

01:1e:83:00:29:04 DC 1048578

MAC flags (S -static MAC, D -dynamic MAC,

SE -Statistics enabled, NM -Non configured MAC, P -Pinned MAC)

Routing instance : PBB_EVPN_I_COMP_CUSTOMER_G

Bridging domain : BD_500, ISID : 10500, VLAN : 500

MAC MAC Logical Remote

address flags interface BEB address

00:0c:29:49:aa:8c D rbeb.32768 00:01:01:01:01:01

00:0c:29:9a:09:c8 D rbeb.32768 00:01:01:01:01:01

00:0c:29:9a:1a:48 D ge-0/0/2.500

00:0c:29:cb:1d:b0 D ge-0/0/2.500

00:0c:29:db:19:5d D rbeb.32769 00:02:02:02:02:02

00:0c:29:e4:3e:cd D rbeb.32769 00:02:02:02:02:02

BGP Summary

Displays the iBGP peering to the other PEs.

admin@PE_MXR01> show bgp summary

Groups: 1 Peers: 2 Down peers: 0

Table Tot Paths Act Paths Suppressed History Damp State Pending

bgp.evpn.0

4 4 0 0 0 0

Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...

112.112.112.112 2345 78654 78647 0 0 5d 18:13:39 Establ

bgp.evpn.0: 2/2/2/0

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 2/2/2/0

__default_evpn__.evpn.0: 0/0/0/0

113.113.113.113 2345 78659 78652 0 0 5d 18:14:10 Establ

bgp.evpn.0: 2/2/2/0

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 2/2/2/0

__default_evpn__.evpn.0: 0/0/0/0

BGP Route Advertising

Displays the routes advertised to the other PEs. Each PE will advertise its B-MAC via a BGP Type 2 MAC route and a Type 3 Inclusive Multicast (IM) route.

admin@PE_MXR01> show route advertising-protocol bgp 112.112.112.112

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

Prefix Nexthop MED Lclpref AS path

2:111.111.111.111:50::10500::00:01:01:01:01:01/304 MAC/IP

* Self 100 I

3:111.111.111.111:50::10500::111.111.111.111/248 IM

* Self 100 I

admin@PE_MXR01> show route advertising-protocol bgp 113.113.113.113

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

Prefix Nexthop MED Lclpref AS path

2:111.111.111.111:50::10500::00:01:01:01:01:01/304 MAC/IP

* Self 100 I

3:111.111.111.111:50::10500::111.111.111.111/248 IM

* Self 100 I

BGP Route Receiving

Displays the routes received from the other PEs. In the ouput below, PE_MXR01 accepts Type 2 and 3 routes from PE_MXR02 and 03.

admin@PE_MXR01> show route receive-protocol bgp 112.112.112.112

inet.0: 46 destinations, 46 routes (46 active, 0 holddown, 0 hidden)

inet.3: 39 destinations, 39 routes (39 active, 0 holddown, 0 hidden)

MGMT.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)

mpls.0: 45 destinations, 45 routes (45 active, 0 holddown, 0 hidden)

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)

bgp.evpn.0: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)

Prefix Nexthop MED Lclpref AS path

2:112.112.112.112:50::10500::00:02:02:02:02:02/304 MAC/IP

* 112.112.112.112 100 I

3:112.112.112.112:50::10500::112.112.112.112/248 IM

* 112.112.112.112 100 I

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

Prefix Nexthop MED Lclpref AS path

2:112.112.112.112:50::10500::00:02:02:02:02:02/304 MAC/IP

* 112.112.112.112 100 I

3:112.112.112.112:50::10500::112.112.112.112/248 IM

* 112.112.112.112 100 I

admin@PE_MXR01> show route receive-protocol bgp 113.113.113.113

inet.0: 46 destinations, 46 routes (46 active, 0 holddown, 0 hidden)

inet.3: 39 destinations, 39 routes (39 active, 0 holddown, 0 hidden)

MGMT.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)

mpls.0: 45 destinations, 45 routes (45 active, 0 holddown, 0 hidden)

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)

bgp.evpn.0: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)

Prefix Nexthop MED Lclpref AS path

2:113.113.113.113:50::10500::00:03:03:03:03:03/304 MAC/IP

* 113.113.113.113 100 I

3:113.113.113.113:50::10500::113.113.113.113/248 IM

* 113.113.113.113 100 I

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

Prefix Nexthop MED Lclpref AS path

2:113.113.113.113:50::10500::00:03:03:03:03:03/304 MAC/IP

* 113.113.113.113 100 I

3:113.113.113.113:50::10500::113.113.113.113/248 IM

* 113.113.113.113 100 I

Route Table EVPN

Displays all EVPN routes in the B-COMP instance. The routing table outputs below are from all PEs for a comprehensive look at the entire PBB-EVPN network. Each PE contains 3 BGP Type 2 MACs and 3 Type 3 IM routes.

admin@PE_MXR01> show route table PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

2:111.111.111.111:50::10500::00:01:01:01:01:01/304 MAC/IP

*[EVPN/170] 3d 22:20:23

Indirect

2:112.112.112.112:50::10500::00:02:02:02:02:02/304 MAC/IP

*[BGP/170] 3d 22:16:12, localpref 100, from 112.112.112.112

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 329

2:113.113.113.113:50::10500::00:03:03:03:03:03/304 MAC/IP

*[BGP/170] 3d 22:13:00, localpref 100, from 113.113.113.113

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 330

3:111.111.111.111:50::10500::111.111.111.111/248 IM

*[EVPN/170] 3d 22:20:23

Indirect

3:112.112.112.112:50::10500::112.112.112.112/248 IM

*[BGP/170] 3d 22:16:12, localpref 100, from 112.112.112.112

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 329

3:113.113.113.113:50::10500::113.113.113.113/248 IM

*[BGP/170] 3d 22:13:00, localpref 100, from 113.113.113.113

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 330

admin@PE_MXR02> show route table PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

2:111.111.111.111:50::10500::00:01:01:01:01:01/304 MAC/IP

*[BGP/170] 3d 22:55:56, localpref 100, from 111.111.111.111

AS path: I, validation-state: unverified

> to 10.1.1.89 via ge-0/0/1.46, Push 328

2:112.112.112.112:50::10500::00:02:02:02:02:02/304 MAC/IP

*[EVPN/170] 3d 22:51:46

Indirect

2:113.113.113.113:50::10500::00:03:03:03:03:03/304 MAC/IP

*[BGP/170] 3d 22:48:33, localpref 100, from 113.113.113.113

AS path: I, validation-state: unverified

> to 10.1.1.89 via ge-0/0/1.46, Push 330

3:111.111.111.111:50::10500::111.111.111.111/248 IM

*[BGP/170] 3d 22:55:56, localpref 100, from 111.111.111.111

AS path: I, validation-state: unverified

> to 10.1.1.89 via ge-0/0/1.46, Push 328

3:112.112.112.112:50::10500::112.112.112.112/248 IM

*[EVPN/170] 3d 22:51:45

Indirect

3:113.113.113.113:50::10500::113.113.113.113/248 IM

*[BGP/170] 3d 22:48:33, localpref 100, from 113.113.113.113

AS path: I, validation-state: unverified

> to 10.1.1.89 via ge-0/0/1.46, Push 330

admin@PE_MXR03> show route table PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

2:111.111.111.111:50::10500::00:01:01:01:01:01/304 MAC/IP

*[BGP/170] 3d 22:37:14, localpref 100, from 111.111.111.111

AS path: I, validation-state: unverified

> to 10.1.1.97 via ge-0/0/1.48, Push 228

2:112.112.112.112:50::10500::00:02:02:02:02:02/304 MAC/IP

*[BGP/170] 3d 22:33:04, localpref 100, from 112.112.112.112

AS path: I, validation-state: unverified

> to 10.1.1.97 via ge-0/0/1.48, Push 229

2:113.113.113.113:50::10500::00:03:03:03:03:03/304 MAC/IP

*[EVPN/170] 3d 22:29:52

Indirect

3:111.111.111.111:50::10500::111.111.111.111/248 IM

*[BGP/170] 3d 22:37:14, localpref 100, from 111.111.111.111

AS path: I, validation-state: unverified

> to 10.1.1.97 via ge-0/0/1.48, Push 228

3:112.112.112.112:50::10500::112.112.112.112/248 IM

*[BGP/170] 3d 22:33:03, localpref 100, from 112.112.112.112

AS path: I, validation-state: unverified

> to 10.1.1.97 via ge-0/0/1.48, Push 229

3:113.113.113.113:50::10500::113.113.113.113/248 IM

*[EVPN/170] 3d 22:29:51

Indirect

Route Table EVPN (Type 2 NRLI)

Displays a deeper look into the PBB-EVPN’s BGP Type 2 NLRI (MAC Advertisement).

BGP Route Type [2]
Route Distinguisher [112.112.112.112:50]
I-SID [10500]
B-MAC Address [00:02:02:02:02:02]
MPLS Label [112050]
ESI [00:00:00:00:00:00:00:00:00:00]

admin@PE_MXR01> show route table PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0 extensive

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

.. snip..

2:112.112.112.112:50::10500::00:02:02:02:02:02/304 MAC/IP (1 entry, 1 announced)

*BGP Preference: 170/-101

Route Distinguisher: 112.112.112.112:50

Next hop type: Indirect, Next hop index: 0

Address: 0xb7a5fb0

Next-hop reference count: 4

Source: 112.112.112.112

Protocol next hop: 112.112.112.112

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

State: <Secondary Active Int Ext>

Local AS: 2345 Peer AS: 2345

Age: 3d 22:16:59 Metric2: 1

Validation State: unverified

Task: BGP_2345.112.112.112.112

Announcement bits (1): 0-PBB_EVPN_B_COMP_CUSTOMER_G-evpn

AS path: I

Communities: target:2345:50

Import Accepted

Route Label: 112050

ESI: 00:00:00:00:00:00:00:00:00:00

Localpref: 100

Router ID: 112.112.112.112

Primary Routing Table bgp.evpn.0

Indirect next hops: 1

Protocol next hop: 112.112.112.112 Metric: 1

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

Indirect path forwarding next hops: 1

Next hop type: Router

Next hop: 10.1.1.81 via ge-0/0/1.44

Session Id: 0x0

112.112.112.112/32 Originating RIB: inet.3

Metric: 1 Node path count: 1

Forwarding nexthops: 1

Nexthop: 10.1.1.81 via ge-0/0/1.44

Route Table EVPN (Type 3 NLRI)

Displays a deeper look into the EVPN BGP Type 3 NLRI (Inclusive Multicast).

BGP Route Type [3]
Route Distinguisher [112.112.112.112:50]
I-SID [10500]
MPLS Label [112051]
Tunnel Type [Ingress Replication]

admin@PE_MXR01> show route table PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0 extensive

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

..snip..

3:112.112.112.112:50::10500::112.112.112.112/248 IM (1 entry, 1 announced)

*BGP Preference: 170/-101

Route Distinguisher: 112.112.112.112:50

PMSI: Flags 0x0: Label 112051: Type INGRESS-REPLICATION 112.112.112.112

Next hop type: Indirect, Next hop index: 0

Address: 0xb7a5fb0

Next-hop reference count: 4

Source: 112.112.112.112

Protocol next hop: 112.112.112.112

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

State: <Secondary Active Int Ext>

Local AS: 2345 Peer AS: 2345

Age: 3d 22:16:59 Metric2: 1

Validation State: unverified

Task: BGP_2345.112.112.112.112

Announcement bits (1): 0-PBB_EVPN_B_COMP_CUSTOMER_G-evpn

AS path: I

Communities: target:2345:50

Import Accepted

Localpref: 100

Router ID: 112.112.112.112

Primary Routing Table bgp.evpn.0

Indirect next hops: 1

Protocol next hop: 112.112.112.112 Metric: 1

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

Indirect path forwarding next hops: 1

Next hop type: Router

Next hop: 10.1.1.81 via ge-0/0/1.44

Session Id: 0x0

112.112.112.112/32 Originating RIB: inet.3

Metric: 1 Node path count: 1

Forwarding nexthops: 1

Nexthop: 10.1.1.81 via ge-0/0/1.44

Traffic Observations

BUM Traffic Analysis

To observe BUM traffic handling in the PBB-EVPN network, a test scenario was setup to force CE_R27 to ARP for CE_R29's MAC address. While a couple of simultaneous captures were performed on the core links (the LSP between PE_MXR01 and PE_MXR02), a ping was initiated from CE_R27 (172.16.50.1) to CE_R29 (172.5.50.3). The ARP resolution process was then dissected from the packet captures.

CE_R27#ping 172.16.50.3

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.50.3, timeout is 2 seconds:

.!!!!

Success rate is 80 percent (4/5), round-trip min/avg/max = 4/26/43 ms

CE_R27#sh arp

Protocol Address Age (min) Hardware Addr Type Interface

Internet 172.16.50.1 - 000c.299a.09c8 ARPA GigabitEthernet1.500

Internet 172.16.50.3 0 000c.29e4.3ecd ARPA GigabitEthernet1.500

From the Wireshark captures, the first 3 frames were from the ARP resolution process and then the 4 ICMP requests/replies were seen as unicast traffic.

The first 2 odd looking frames were from Inclusive Multicast (IM) process. EVPN uses IM for BUM traffic handling and by looking closely at frame 37 and 38, it would reveal that these frames were sent to PE_MXR02 and PE_MXR03 respectively. Frame 39 was the properly formatted ARP response.

The Wireshark decodes for the 2 IM frames are shown below.

PE_MXR01 → PE_MXR02

PE_MXR01 → PE_MXR03

MPLS Labels (IM Frames)

The label stack was derived the Type 3 route lookups.

PE_MXR02’s Type 3 IM route is told to push a top label of 329 and bottom label of 112051.

admin@PE_MXR01> show route table PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

..snip..

3:112.112.112.112:50::10500::112.112.112.112/248 IM

*[BGP/170] 3d 04:36:36, localpref 100, from 112.112.112.112

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 329

admin@PE_MXR01> show route table PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0 extensive

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

..snip..

3:112.112.112.112:50::10500::112.112.112.112/248 IM (1 entry, 1 announced)

*BGP Preference: 170/-101

Route Distinguisher: 112.112.112.112:50

PMSI: Flags 0x0: Label 112051: Type INGRESS-REPLICATION 112.112.112.112

Next hop type: Indirect, Next hop index: 0

Address: 0xb7a5ad0

Next-hop reference count: 4

Source: 112.112.112.112

Protocol next hop: 112.112.112.112

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

State: <Secondary Active Int Ext>

Local AS: 2345 Peer AS: 2345

Age: 3d 4:36:59 Metric2: 1

Validation State: unverified

Task: BGP_2345.112.112.112.112

Announcement bits (1): 0-PBB_EVPN_B_COMP_CUSTOMER_G-evpn

AS path: I

Communities: target:2345:50

Import Accepted

Localpref: 100

Router ID: 112.112.112.112

Primary Routing Table bgp.evpn.0

Indirect next hops: 1

Protocol next hop: 112.112.112.112 Metric: 1

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

Indirect path forwarding next hops: 1

Next hop type: Router

Next hop: 10.1.1.81 via ge-0/0/1.44

Session Id: 0x0

112.112.112.112/32 Originating RIB: inet.3

Metric: 1 Node path count: 1

Forwarding nexthops: 1

Nexthop: 10.1.1.81 via ge-0/0/1.44

PE_MXR03’s Type 3 IM route is told to push a top label of 330 and bottom label of 113068.

admin@PE_MXR01> show route table PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

..snip..

3:113.113.113.113:50::10500::113.113.113.113/248 IM

*[BGP/170] 3d 04:36:40, localpref 100, from 113.113.113.113

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 330

admin@PE_MXR01> show route table PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0 extensive

PBB_EVPN_B_COMP_CUSTOMER_G.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

..snip..

3:113.113.113.113:50::10500::113.113.113.113/248 IM (1 entry, 1 announced)

*BGP Preference: 170/-101

Route Distinguisher: 113.113.113.113:50

PMSI: Flags 0x0: Label 113068: Type INGRESS-REPLICATION 113.113.113.113

Next hop type: Indirect, Next hop index: 0

Address: 0xb7a60d0

Next-hop reference count: 4

Source: 113.113.113.113

Protocol next hop: 113.113.113.113

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

State: <Secondary Active Int Ext>

Local AS: 2345 Peer AS: 2345

Age: 3d 4:37:03 Metric2: 1

Validation State: unverified

Task: BGP_2345.113.113.113.113

Announcement bits (1): 0-PBB_EVPN_B_COMP_CUSTOMER_G-evpn

AS path: I

Communities: target:2345:50

Import Accepted

Localpref: 100

Router ID: 113.113.113.113

Primary Routing Table bgp.evpn.0

Indirect next hops: 1

Protocol next hop: 113.113.113.113 Metric: 1

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

Indirect path forwarding next hops: 1

Next hop type: Router

Next hop: 10.1.1.81 via ge-0/0/1.44

Session Id: 0x0

113.113.113.113/32 Originating RIB: inet.3

Metric: 1 Node path count: 1

Forwarding nexthops: 1

Nexthop: 10.1.1.81 via ge-0/0/1.44

Source & Destination MAC Addresses (IM Frame)

When looking below the MPLS headers, the Ethernet header looks a little confusing, especially the source and destination MAC addresses (highlighted below). Based on PBB-EVPN’s BUM forwarding behavior, the source and destination MAC addresses should use known addresses, such as the Backbone MAC and Group Backbone MAC (seen from the “show bridge mac-table” & "show evpn database" commands). In the Wireshark's decode, these address do not make much sense.

Destination address Group B-MAC as seen from the MAC-Table.

admin@PE_MXR01> show bridge mac-table

MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC

O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)

Routing instance : PBB_EVPN_B_COMP_CUSTOMER_G

Bridging domain : BD_500, VLAN : 500

MAC MAC Logical NH MAC

address flags interface Index property

00:02:02:02:02:02 DC 1048577

00:03:03:03:03:03 DC 1048575

01:1e:83:00:29:04 DC 1048578

MAC flags (S -static MAC, D -dynamic MAC,

SE -Statistics enabled, NM -Non configured MAC, P -Pinned MAC)

Routing instance : PBB_EVPN_I_COMP_CUSTOMER_G

Bridging domain : BD_500, ISID : 10500, VLAN : 500

MAC MAC Logical Remote

address flags interface BEB address

00:0c:29:49:aa:8c D ge-0/0/2.500

00:0c:29:cb:1d:b0 D rbeb.32768 00:03:03:03:03:03

Source address Backbone MAC as seen from the EVPN database.

admin@PE_MXR01> show evpn database

Instance: PBB_EVPN_B_COMP_CUSTOMER_G

VLAN DomainId MAC address Active source Timestamp IP address

10500 00:01:01:01:01:01 Local Jun 14 20:54:06

10500 00:02:02:02:02:02 112.112.112.112 Jun 22 00:09:02

10500 00:03:03:03:03:03 113.113.113.113 Jun 22 00:08:58

Raw Data Decode (IM Frame)

In order to fully understand what was going on here, a raw HEX data decode was necessary. Using frame 37 as the example, each byte was meticulously examined, grouped and highlighted. This exercise revealed that this was indeed a PBB encapsulated ARP request with the proper B-MAC source address of 00:01:01:01:01:01 and a Group B-MAC destination address of 01:1e:83:00:29:04. The full decode and breakown is shown below.

Ethernet II (Core Link Header)

Destination MAC: 000c291eb42a (P_R03’s link towards PE_MXR01)
Source MAC: 000c29df993a (PE_MXR01’s link towards P_R03)

EtherType: 8847 (MPLS Unicast)

MPLS Header (Top)

MPLS Label: 00149 (329)
MPLS Bottom of Label Stack: 0 (No)
MPLS TTL: ff (255)

MPLS Header (Bottom)

MPLS Label: 1b5b3 (112051)
MPLS Bottom of Label Stack: 1 (Yes)
MPLS TTL: ff (255)

Ethernet II (PBB Header)

Destination MAC: 011e83002904 (Group B-MAC)
Source MAC: 000101010101 (PE_MXR01’s B-MAC)
EtherType: 88e7 (802.1ah Provider Backbone Bridge)

IEEE 802.1ah

I-Tag: 00
I-SID: 002904 (10500)
C-Destination MAC: ffffffffffff (Broadcast address)
C-Source MAC: 000c299a09c8 (CE_R27's MAC)
EtherType: 8100 (802.1q VLAN)

802.1Q Virtual LAN

Priority: 0 (Best Effort)
ID: 1f4 (500)
EtherType: 0806 (ARP)
Trailer: 000000000000000000000000000000000000

Address Resolution Protocol

Hardware Type: 0001 (Ethernet)
Protocol Type: 0800 (IPv4)
Hardware Size: 06 (6 Bytes)
Protocol Size: 04 (4 Bytes)
Opcode: 0001 (Request)
Sender MAC Address: 000c299a09c8 (CE_R27's MAC)
Sender IP Address: ac103201 (172.16.50.1 – CE_R27's IP)
Target MAC Address: 000000000000
Target IP Address: ac103203 (172.16.50.3 – CE_R29's IP)

ARP Response Packet

Once the destination PE (PE_MXR02) has processed the IM frame (frame 16) and forwards the ARP request to the end host, the ARP response is seen (frame 17).

The Wireshark decode shows the ARP response with the PBB header in red and ARP data in blue.

Unicast Traffic Analysis

After analyzing the inner-workings of BUM traffic in a PBB-EVPN network, unicast traffic is far easier understand. The example below will explain in detail the encapsulation and forwarding process for unicast traffic.

Once ARP resolution has successfully taken place, the end host CE_R27 (172.16.50.1) can now send traffic directly to CE_R29 (172.16.50.3) over the L2 network. Using Frame 43 as the example, the host’s ICMP IP packet is VLAN tagged with an ID of 500 along with its Source/Destination MACs.

The PE receives this traffic in the I-Comp instance then PBB encapsulates the frame (MAC in MAC). This process adds the I-SID with the new Backbone Source & Destination MAC addresses for forwarding in the B-Comp instance (backbone bridge). With MPLS as the data plane transport for the B-Comp, MPLS labels are attached for forwarding in the core network.

MPLS forwards the packet to the destination PE's B-Comp. Its corresponding I-Comp maps the B-MAC to the C-MAC, removes the PBB header and sends the orignal 802.1q frame to the destination host.

MPLS Headers

Top Label
Bottom Label

802.1ah PBB Header

Destination Backbone MAC
Source Backbone MAC
I-SID
Destination Client MAC
Source Client MAC

802.1q VLAN Header:

VLAN ID

Payload:

IP Header
ICMP Header & Data

References

BGP, MPLS, PBB, PBB-EVPN

https://www.juniper.net/documentation/en_US/junos/topics/concept/pbb-evpn-integration-for-dci-overview.html

https://www.juniper.net/documentation/en_US/junos/topics/example/example-pbb-sh-evpn-for-dci-configuring.html

https://sites.google.com/site/amitsciscozone/home/pbb/understanding-pbb

Other Blogs and Labs

https://tgregory.org/2016/11/15/evpn-vs-pbb-evpn-and-massive-scale/

the peering xchange : [a network engineer’s blog]

Pages

Thursday, July 5, 2018

MPLS L2VPN (PBB-EVPN on JunOS)

Report Abuse