MPLS L2VPN (EVPN on JunOS with All-Active Multi-Homing)

When EVPN introduced all-active multi-homing, it offered a new redundancy option not available with VPLS.  It was considered one of EVPN's significant features.  With the use of BGP as the control plane protocol, EVPN now enabled edge devices to utilize multiple active links and per flow load-balancing between the CE and PEs. 

To test this new feature, the topology has been modified from the original JunOS EVPN lab to accommodate a multi-homed host.  In this lab, we will look at the configurations but more importantly we will concentrate on its operation, such as route and label exchange in an effort to understand how load balancing occurs within the MPLS network.

Lab Environment

Note: The DSL hosts were swapped out with 3 routers as test hosts.

• (3) Cisco CSR1000v version 16.4.2

Diagram & Topology

Hosts only in VLAN 500 were swapped out with Cisco CSR1000vs.  This was done so one of the hosts could be ether-channeled (not using LACP).  This lab closely followed the configuration provided in this Juniper article.

• Infrastructure Information (changes only)
• Full mesh iBGP peering between all PEs.  The RR was removed for this lab (see Appendix section for more info).
• CUSTOMER G
• CE_R27 multi-homed to PE_MXR01 and PE_MXR03
• AC Interfaces: Gig1 to vSwitch 24 and Gig2 vSwitch 27
• PE01’s AC Interface: GE-0/0/2 to vSwitch 24
• PE03’s AC Interface: GE-0/0/3 to vSwitch 27
• All Active mode

Technology Overview

This section is an overview of key terms and concepts used throughout this post.

Ethernet Segment (ES) & Ethernet Segment Identifier (ESI)

The ES is simply referred to as the connection or link between the CE and PE.  Uniquely identifying these segments in the EVPN network is critical for a multi-homed deployment. Configuring an ID to these ESs is what is known as the ESI.  More information about ESI can be found in RFC 7432 Sec. 7.5.

Designated Forwarder (DF) & Backup Designated Forwarder (BDF)

In a multi-homed deployment, forwarding multi-destination traffic (i.e., BUM traffic) to all Ethernet segments can cause potential issues and is a huge waste of a PE's system resources.  By electing a DF, a single PE is authorized to forward this type of traffic for a given ESI, while the BDF waits in standby.  This method prevents the other PEs from having to perform redundant tasks, hence making BUM traffic delivery more efficient.  More information about the DF and BDF can be found in RFC 7432 Sec. 8.5.

Aliasing

Aliasing allows PEs to load balance traffic to multi-homed hosts.  More information about Aliasing can be found in RFC 7432 Sec. 8.4.

New EVPN BGP Route Types

EVPN multi-homing introduces new BGP route types.  There are two versions of Type 1s and a Type 4.  Each route types are discussed in more detail below.  More information about the route types can be found in RFC 7432 Sec. 7.

BGP Type 1 EAD per ESI Route

BGP Type 1 AD per EVI Route

BGP Type 4 Ethernet Segment Route

Configurations

Configurations below were from the devices that changed during the multi-homed lab.

CE_R27

version 16.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime show-timezone year
service timestamps log datetime localtime show-timezone year
service password-encryption
no platform punt-keepalive disable-kernel-core
platform console auto
!
hostname CE_R27
!
boot-start-marker
boot-end-marker
!
!
logging buffered 16000
no logging console
enable secret 5 <removed>
!
no aaa new-model
!
ip vrf MGMT
!
!
!
!
!
!
!
!
!

no ip domain lookup
ip domain name domain.local
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
!
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
!
!
!
license udi pid CSR1000V sn <removed>
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
username admin secret 5 <removed>
!
redundancy
!         
!
!
!
!
!
! 
!
!
!
!
!
!
!
!
!
!
!
!
! 
! 
! 
! 
! 
! 
!
!
interface Port-channel50
 description ETH CH TO PE_MXR01 & PE_MXR03
 ip address 172.16.50.1 255.255.255.0
 no negotiation auto
 no mop enabled
 no mop sysid
!
interface GigabitEthernet1
 description TO PE_MXR01
 no ip address
 negotiation auto
 no mop enabled
 no mop sysid
 channel-group 50
!
interface GigabitEthernet2
 description TO PE_MXR03
 no ip address
 negotiation auto
 no mop enabled
 no mop sysid
 channel-group 50
!
interface GigabitEthernet3
 no ip address
 shutdown
 negotiation auto
 no mop enabled
 no mop sysid
!
interface GigabitEthernet4
 description MANAGEMENT
 ip vrf forwarding MGMT
 ip address 10.1.1.247 255.255.255.0
 negotiation auto
 no mop enabled
 no mop sysid
!
!
virtual-service csr_mgmt
!
no ip forward-protocol nd
no ip http server
ip http authentication local
no ip http secure-server
!
ip bgp-community new-format
ip tftp source-interface GigabitEthernet4
ip route vrf MGMT 0.0.0.0 0.0.0.0 10.1.1.254 name MGMT_DEFAULT
!
!
!         
!
!
control-plane
!
 !
 !
 !
 !
!
!
!
!
!
line con 0
 exec-timeout 60 0
 logging synchronous
 stopbits 1
line vty 0 4
 exec-timeout 60 0
 logging synchronous
 login local
 transport input telnet ssh
!
ntp server pool.ntp.org
!
!
!
!
!
end

PE_MXR01

version 17.2R1.13;
system {
    host-name PE_MXR01;
    root-authentication {
        encrypted-password "<removed>"; ## SECRET-DATA
    }
    login {
        user admin {
            uid 2000;
            class super-user;
            authentication {
                encrypted-password "<removed>"; ## SECRET-DATA
            }
        }
    }
    services {
        ssh {
            root-login allow;
        }
    }
    syslog {
        user * {
            any emergency;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
    processes {
        dhcp-service {
            traceoptions {
                file dhcp_logfile size 10m;
                level all;
                flag all;
            }
        }
    }
}
chassis {
    fpc 0 {
        lite-mode;
    }
}
interfaces {
    ge-0/0/0 {
        unit 0 {
            description MGMT;
            family inet {
                address 10.1.1.204/24;
            }
        }
    }
    ge-0/0/1 {
        vlan-tagging;
        mtu 9234;
        unit 44 {
            description "TO P_R03";
            vlan-id 44;
            family inet {
                address 10.1.1.82/30;
            }
            family mpls;
        }
        unit 45 {
            description "TO P_R01";
            vlan-id 45;
            family inet {
                address 10.1.1.86/30;
            }
            family mpls;
        }
    }                                   
    ge-0/0/2 {
        description "TO CUSTOMER_G ELAN 500 VSWITCH_24";
        flexible-vlan-tagging;
        encapsulation flexible-ethernet-services;
        esi {
            00:11:22:33:44:55:66:77:88:99;
            all-active;
        }
        unit 500 {
            family bridge {
                interface-mode trunk;
                vlan-id-list [ 500 501 ];
            }
        }
    }
    fxp0 {
        description "RE MGMT";
        unit 0 {
            disable;
            family inet {
                dhcp {
                    vendor-id Juniper-vmx;
                }
            }
        }
    }
    lo0 {
        unit 111 {
            description RID;
            family inet {
                address 111.111.111.111/32;
            }
        }
    }
}
routing-options {
    router-id 111.111.111.111;
    autonomous-system 2345;
    forwarding-table {
        export EVPN_PPLB;
    }
}
protocols {
    mpls {
        label-range {
            dynamic-label-range 111000 111999;
        }
    }
    bgp {
        group PE {
            type internal;
            local-address 111.111.111.111;
            hold-time 21;
            family evpn {
                signaling;
            }
            neighbor 112.112.112.112;
            neighbor 113.113.113.113;
        }
    }
    ospf {
        area 0.0.0.0 {
            interface ge-0/0/1.44 {
                interface-type p2p;
                metric 1;
            }
            interface lo0.111 {
                passive;
            }                           
            interface ge-0/0/1.45 {
                interface-type p2p;
                metric 4;
            }
        }
    }
    ldp {
        interface ge-0/0/1.44;
        interface ge-0/0/1.45;
    }
}
policy-options {
    policy-statement EVPN_PPLB {
        from protocol evpn;
        then {
            load-balance per-packet;
        }
    }
}
routing-instances {
    EVPN_CUSTOMER_G_ELAN_500 {
        instance-type virtual-switch;
        interface ge-0/0/2.500;
        route-distinguisher 111.111.111.111:50;
        vrf-target target:2345:50;
        protocols {
            evpn {
                extended-vlan-list 500-501;
            }
        }
        bridge-domains {
            VL500 {
                domain-type bridge;
                vlan-id 500;
            }
            VL501 {
                domain-type bridge;
                vlan-id 501;
            }
        }
    }
    MGMT {
        instance-type virtual-router;
        interface ge-0/0/0.0;
        routing-options {
            static {
                route 0.0.0.0/0 next-hop 10.1.1.254;
            }
        }
    }
}

PE_MXR03

version 17.2R1.13;
system {
    host-name PE_MXR03;
    root-authentication {
        encrypted-password "<removed>"; ## SECRET-DATA
    }
    login {
        user admin {
            uid 2000;
            class super-user;
            authentication {
                encrypted-password "<removed>"; ## SECRET-DATA
            }
        }
    }
    services {
        ssh {
            root-login allow;
        }
    }
    syslog {
        user * {
            any emergency;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
    processes {
        dhcp-service {
            traceoptions {
                file dhcp_logfile size 10m;
                level all;
                flag all;
            }
        }
    }
}
chassis {
    fpc 0 {
        lite-mode;
    }
}
interfaces {
    ge-0/0/0 {
        unit 0 {
            description MGMT;
            family inet {
                address 10.1.1.206/24;
            }
        }
    }
    ge-0/0/1 {
        vlan-tagging;
        mtu 9234;
        unit 48 {
            description "TO P_R02";
            vlan-id 48;
            family inet {
                address 10.1.1.98/30;
            }
            family mpls;
        }
        unit 49 {
            description "TO P_R01";
            vlan-id 49;
            family inet {
                address 10.1.1.102/30;
            }
            family mpls;
        }
    }                                   
    ge-0/0/2 {
        description "TO CUSTOMER_G ELAN 500 VSWITCH_26";
        flexible-vlan-tagging;
        encapsulation flexible-ethernet-services;
        unit 500 {
            family bridge {
                interface-mode trunk;
                vlan-id-list [ 500 501 ];
            }
        }
    }
    ge-0/0/3 {
        description "TO CUSTOMER_G ELAN 500 VSWITCH_27";
        flexible-vlan-tagging;
        encapsulation flexible-ethernet-services;
        esi {
            00:11:22:33:44:55:66:77:88:99;
            all-active;
        }
        unit 500 {
            family bridge {
                interface-mode trunk;
                vlan-id-list [ 500 501 ];
            }
        }
    }
    fxp0 {
        description "RE MGMT";
        unit 0 {
            disable;
            family inet {
                dhcp {
                    vendor-id Juniper-vmx;
                }
            }
        }
    }
    lo0 {
        unit 113 {
            description RID;
            family inet {
                address 113.113.113.113/32;
            }
        }
    }
}
routing-options {
    router-id 113.113.113.113;
    autonomous-system 2345;
    forwarding-table {
        export EVPN_PPLB;
    }
}
protocols {
    mpls {
        label-range {
            dynamic-label-range 113000 113999;
        }
    }
    bgp {
        group PE {
            type internal;
            local-address 113.113.113.113;
            hold-time 21;
            family evpn {
                signaling;
            }
            neighbor 112.112.112.112;
            neighbor 111.111.111.111;
        }
    }
    ospf {
        area 0.0.0.0 {
            interface ge-0/0/1.48 {
                interface-type p2p;
                metric 1;
            }
            interface ge-0/0/1.49 {
                interface-type p2p;
                metric 5;
            }
            interface lo0.113 {
                passive;
            }
        }
    }
    ldp {
        interface ge-0/0/1.48;
        interface ge-0/0/1.49;
    }
}
policy-options {
    policy-statement EVPN_PPLB {
        from protocol evpn;
        then {
            load-balance per-packet;
        }
    }
}
routing-instances {
    EVPN_CUSTOMER_G_ELAN_500 {
        instance-type virtual-switch;
        interface ge-0/0/2.500;
        interface ge-0/0/3.500;
        route-distinguisher 113.113.113.113:50;
        vrf-target target:2345:50;
        protocols {
            evpn {
                extended-vlan-list 500-501;
            }
        }
        bridge-domains {
            VL500 {
                domain-type bridge;
                vlan-id 500;
            }
            VL501 {
                domain-type bridge;
                vlan-id 501;
            }
        }
    }
    MGMT {
        instance-type virtual-router;
        interface ge-0/0/0.0;
        routing-options {
            static {
                route 0.0.0.0/0 next-hop 10.1.1.254;
            }
        }
    }
}

EVPN Specific Configurations

Configurations for only PE_MXR01 and PE_MXR03 were referenced below.  PE_MXR02's configuration was not changed.

EVPN Instance

Interface GE-0/0/3 was configured on PE_MXR03’s EVPN instance as the second AC interface for CE_R27.  PE_MXR01 used its existing AC interface for the multi-homed host.

routing-instances {     EVPN_CUSTOMER_G_ELAN_500 {         instance-type virtual-switch;         interface ge-0/0/2.500;         interface ge-0/0/3.500;         route-distinguisher 113.113.113.113:50;         vrf-target target:2345:50;         protocols {             evpn {                 extended-vlan-list 500-501;             }         }         bridge-domains {             VL500 {                 domain-type bridge;                 vlan-id 500;             }             VL501 {                 domain-type bridge;                 vlan-id 501;             }         }     }

Attachment Circuit

A unique Ethernet Segment Identifier (ESI) was configured to a non-zero value on the AC interfaces and all-active was defined as the redundancy mode. 

PE_MXR01

interfaces {     ge-0/0/2 {         description "TO CUSTOMER_G ELAN 500 VSWITCH_24";         flexible-vlan-tagging;         encapsulation flexible-ethernet-services;         esi {             00:11:22:33:44:55:66:77:88:99;             all-active;         }         unit 500 {             family bridge {                 interface-mode trunk;                 vlan-id-list [ 500 501 ];             }         }     }

PE_MXR03

interfaces {     ge-0/0/3 {         description "TO CUSTOMER_G ELAN 500 VSWITCH_27";         flexible-vlan-tagging;         encapsulation flexible-ethernet-services;         esi {             00:11:22:33:44:55:66:77:88:99;             all-active;         }         unit 500 {             family bridge {                 interface-mode trunk;                 vlan-id-list [ 500 501 ];             }         }     }

MP-BGP

MP-BGP was changed to peer with all the PEs rather than the RR.  The RR was removed in this lab due to route exchange related issues (see Appendix for more information).

PE_MXR01

routing-options {     router-id 111.111.111.111;     autonomous-system 2345; } protocols {     bgp {         group PE {             type internal;             local-address 111.111.111.111;             hold-time 21;             family evpn {                 signaling;             }             neighbor 112.112.112.112;             neighbor 113.113.113.113;         }     }

PE_MXR03

routing-options {     router-id 113.113.113.113;     autonomous-system 2345; } protocols {     bgp {         group PE {             type internal;             local-address 113.113.113.113;             hold-time 21;             family evpn {                 signaling;             }             neighbor 112.112.112.112;             neighbor 111.111.111.111;         }     }

Testing Results

Pings were used from CE_R27 to verify connectivity and reachability to all hosts in the EVPN.

CE_R27 to CE_R28

CE_R27#ping 172.16.50.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 172.16.50.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 4/30/51 ms

CE_R27 to CE_R29

CE_R27#ping 172.16.50.3 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 172.16.50.3, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 2/6/19 ms

CE_R27’s ARP Table

CE_R27#sh arp Protocol  Address          Age (min)  Hardware Addr   Type   Interface Internet  172.16.50.1             -   001e.e5c8.0ff1  ARPA   Port-channel50 Internet  172.16.50.2           226   000c.2949.aa8c  ARPA   Port-channel50 Internet  172.16.50.3           199   000c.29e4.3ecd  ARPA   Port-channel50

Verification Tasks

The following commands were used to verify EVPN operation in an all-active configuration during normal operating conditions.

• Show EVPN Instance Extensive
• Show EVPN Database
• Show BGP Summary
• Show Route Advertising-Protocol BGP
• Show Route Table EVPN
• Show Route Table EVPN Extensive (Type 1 AD/ESI NLRI)
• Show Route Table EVPN Extensive (Type 1 AD/EVI NLRI)
• Show Route Table EVPN Extensive (Type 4 ES NLRI)

EVPN Instance (extensive)

The EVPN instance command was used to display operational information, such as:

• Local Interface and Configured ESI
• Redundancy Mode and Status
• Discovered ESI & Remote PE Mappings
• Elected Designated Forwarder & Backup Forwarder
• Advertised MPLS Labels

PE_MXR01’s EVPN instance output tells us the following information:

• A multi-homed host in all-active mode connects to this PE (non-zero ESI configured on local interface).
• The ESI learned from 113.113.113.113 is identical to what is locally configured.  This PE discovered that its local ES and the ES on PE 113.113.113.113 are connected to the same host.
• With both PE's ES configured in all-active mode, egress traffic from this PE can be load balanced via the local interface or the remote PE at 113.113.113.113 using its aliasing label.
• For BUM traffic forwarding, the DF is this PE and BDF is the remote PE at 113.113.113.113.

admin@PE_MXR01> show evpn instance extensive Instance: EVPN_CUSTOMER_G_ELAN_500   Route Distinguisher: 111.111.111.111:50   Per-instance MAC route label: 299776   MAC database status                     Local  Remote     MAC advertisements:                       0       0     MAC+IP advertisements:                    0       0     Default gateway MAC advertisements:       0       0   Number of local interfaces: 1 (1 up)     Interface name  ESI                            Mode             Status     AC-Role     ge-0/0/2.500    00:11:22:33:44:55:66:77:88:99  all-active       Up         Root   Number of IRB interfaces: 0 (0 up)   Number of bridge domains: 2     VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop     500                  1    1                Extended         Enabled   300656          Disabled     501                  1    1                Extended         Enabled   300672          Disabled   Number of neighbors: 2     Address               MAC    MAC+IP        AD        IM        ES Leaf-label     112.112.112.112         1         0         0         2         0     113.113.113.113         0         0         2         2         0   Number of ethernet segments: 1     ESI: 00:11:22:33:44:55:66:77:88:99       Status: Resolved by IFL ge-0/0/2.500       Local interface: ge-0/0/2.500, Status: Up/Forwarding       Number of remote PEs connected: 1         Remote PE        MAC label  Aliasing label  Mode         113.113.113.113  300512     300512          all-active         Designated forwarder: 111.111.111.111       Backup forwarder: 113.113.113.113       Last designated forwarder update: Jan 11 03:53:14       Advertised MAC label: 300640       Advertised aliasing label: 300640       Advertised split horizon label: 301184 Instance: __default_evpn__   Route Distinguisher: 111.111.111.111:0   Number of bridge domains: 0   Number of neighbors: 1     Address               MAC    MAC+IP        AD        IM        ES Leaf-label     113.113.113.113         0         0         0         0         1

PE_MXR02’s output tells us the following information:

• A single homed host connects to this PE (zero ESI configured on local interface).
• From the routes learned from the 2 remote PEs, the multi-homed host in this EVI is reachable via these 2 PEs, one via 111.111.111.111 and other via 113.113.113.113.  It can use the received aliasing labels to load balance the traffic.

admin@PE_MXR02> show evpn instance extensive Instance: EVPN_CUSTOMER_G_ELAN_500   Route Distinguisher: 112.112.112.112:50   Per-instance MAC route label: 299776   MAC database status                     Local  Remote     MAC advertisements:                       1       1     MAC+IP advertisements:                    0       0     Default gateway MAC advertisements:       0       0   Number of local interfaces: 1 (1 up)     Interface name  ESI                            Mode             Status     AC-Role     ge-0/0/2.500    00:00:00:00:00:00:00:00:00:00  single-homed     Up         Root   Number of IRB interfaces: 0 (0 up)   Number of bridge domains: 2     VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop     500                  1    1                Extended         Enabled   299840          Disabled     501                  1    1                Extended         Enabled   299856          Disabled   Number of neighbors: 2     Address               MAC    MAC+IP        AD        IM        ES Leaf-label     111.111.111.111         1         0         2         2         0     113.113.113.113         0         0         2         2         0   Number of ethernet segments: 1     ESI: 00:11:22:33:44:55:66:77:88:99       Status: Unresolved       Number of remote PEs connected: 2         Remote PE        MAC label  Aliasing label  Mode         111.111.111.111  300640     300640          all-active           113.113.113.113  300512     300512          all-active   Instance: __default_evpn__   Route Distinguisher: 112.112.112.112:0   Number of bridge domains: 0   Number of neighbors: 0

PE_MXR03’s output tells us the following information:

• A single homed host connects to this PE (zero ESI configured on local interface).
• A multi-homed host in all-active mode connects to this PE (non-zero ESI configured on local interface).
• The ESI learned from 111.111.111.111 is identical to what is locally configured.  This PE discovered that its local ES and the ES on PE 111.111.111.111 are connected to the same host.
• With both PE's ES configured in all-active mode, egress traffic from this PE can be load balanced via the local interface or the remote PE at 111.111.111.111 using its aliasing label.
• For BUM traffic forwarding, the DF is the remote PE at 111.111.111.111 and BDF is itself.

admin@PE_MXR03> show evpn instance extensive Instance: EVPN_CUSTOMER_G_ELAN_500   Route Distinguisher: 113.113.113.113:50   Per-instance MAC route label: 299776   MAC database status                     Local  Remote     MAC advertisements:                       0       0     MAC+IP advertisements:                    0       0     Default gateway MAC advertisements:       0       0   Number of local interfaces: 2 (2 up)     Interface name  ESI                            Mode             Status     AC-Role     ge-0/0/2.500    00:00:00:00:00:00:00:00:00:00  single-homed     Up         Root     ge-0/0/3.500    00:11:22:33:44:55:66:77:88:99  all-active       Up         Root   Number of IRB interfaces: 0 (0 up)   Number of bridge domains: 2     VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop     500                  2    2                Extended         Enabled   300528          Disabled     501                  2    2                Extended         Enabled   300544          Disabled   Number of neighbors: 2     Address               MAC    MAC+IP        AD        IM        ES Leaf-label     111.111.111.111         0         0         2         2         0     112.112.112.112         0         0         0         2         0   Number of ethernet segments: 1     ESI: 00:11:22:33:44:55:66:77:88:99       Status: Resolved by IFL ge-0/0/3.500       Local interface: ge-0/0/3.500, Status: Up/Forwarding       Number of remote PEs connected: 1         Remote PE        MAC label  Aliasing label  Mode         111.111.111.111  300640     300640          all-active         Designated forwarder: 111.111.111.111       Backup forwarder: 113.113.113.113       Last designated forwarder update: Jan 11 03:53:17       Advertised MAC label: 300512       Advertised aliasing label: 300512       Advertised split horizon label: 301200 Instance: __default_evpn__   Route Distinguisher: 113.113.113.113:0   Number of bridge domains: 0   Number of neighbors: 1     Address               MAC    MAC+IP        AD        IM        ES Leaf-label     111.111.111.111         0         0         0         0         1

EVPN Database

The EVPN database output displayed a different active source for multi-homed hosts.  Since traffic can now take the path of multiple PEs, an ESI lookup was now required.

admin@PE_MXR01> show evpn database    Instance: EVPN_CUSTOMER_G_ELAN_500 VLAN  DomainId  MAC address        Active source                  Timestamp        IP address 500             00:0c:29:49:aa:8c  112.112.112.112                Jan 14 07:22:32 500             00:0c:29:e4:3e:cd  113.113.113.113                Jan 14 07:22:48 500             00:1e:e5:c8:0f:f1  00:11:22:33:44:55:66:77:88:99  Jan 14 07:22:31

admin@PE_MXR02> show evpn database    Instance: EVPN_CUSTOMER_G_ELAN_500 VLAN  DomainId  MAC address        Active source                  Timestamp        IP address 500             00:0c:29:49:aa:8c  ge-0/0/2.500                   Jan 14 07:22:31 500             00:0c:29:e4:3e:cd  113.113.113.113                Jan 14 08:18:17 500             00:1e:e5:c8:0f:f1  00:11:22:33:44:55:66:77:88:99  Jan 14 07:22:32

admin@PE_MXR03> show evpn database Instance: EVPN_CUSTOMER_G_ELAN_500 VLAN  DomainId  MAC address        Active source                  Timestamp        IP address 500             00:0c:29:49:aa:8c  112.112.112.112                Jan 14 07:22:32 500             00:0c:29:e4:3e:cd  ge-0/0/2.500                   Jan 14 07:22:47 500             00:1e:e5:c8:0f:f1  00:11:22:33:44:55:66:77:88:99  Jan 14 07:22:32

Show BGP Summary

MP-BGP peering was configured between the PEs due to a route exchange issue with the RR. (See appendix for more information.)

admin@PE_MXR01> show bgp summary Groups: 1 Peers: 2 Down peers: 0 Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending bgp.evpn.0                                  9          9          0          0          0          0 Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 112.112.112.112        2345      43252      43279       0       1  3d 3:55:51 Establ   bgp.evpn.0: 3/3/3/0   EVPN_CUSTOMER_G_ELAN_500.evpn.0: 3/3/3/0   __default_evpn__.evpn.0: 0/0/0/0 113.113.113.113        2345      43241      43268       0       1  3d 3:54:46 Establ   bgp.evpn.0: 6/6/6/0   EVPN_CUSTOMER_G_ELAN_500.evpn.0: 5/5/5/0   __default_evpn__.evpn.0: 1/1/1/0

admin@PE_MXR02> show bgp summary Groups: 1 Peers: 2 Down peers: 0 Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending bgp.evpn.0                                 10         10          0          0          0          0 Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 111.111.111.111        2345      43284      43255       0       1  3d 3:56:24 Establ   bgp.evpn.0: 5/5/5/0   EVPN_CUSTOMER_G_ELAN_500.evpn.0: 5/5/5/0   __default_evpn__.evpn.0: 0/0/0/0 113.113.113.113        2345      43248      43247       0       1  3d 3:55:26 Establ   bgp.evpn.0: 5/5/5/0   EVPN_CUSTOMER_G_ELAN_500.evpn.0: 5/5/5/0   __default_evpn__.evpn.0: 0/0/0/0

admin@PE_MXR03> show bgp summary Groups: 1 Peers: 2 Down peers: 0 Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending bgp.evpn.0                                  9          9          0          0          0          0 Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 111.111.111.111        2345      43280      43251       0       1  3d 3:55:50 Establ   bgp.evpn.0: 6/6/6/0   EVPN_CUSTOMER_G_ELAN_500.evpn.0: 5/5/5/0   __default_evpn__.evpn.0: 1/1/1/0 112.112.112.112        2345      43252      43251       0       1  3d 3:55:57 Establ   bgp.evpn.0: 3/3/3/0   EVPN_CUSTOMER_G_ELAN_500.evpn.0: 3/3/3/0   __default_evpn__.evpn.0: 0/0/0/0

BGP Route Advertising

PEs with multi-homed hosts will advertise additional BGP Type 1 EAD per ESI, Type 1 EAD per EVI and Type 4 ES routes.  These route are used to signal and discover other PEs with multi-homed hosts.  They will advertise information such as their ESI value, MPLS labels and redundancy mode.

In the output below, PE_MXR01 advertised its Type 1 and Type 4 routes, signaling to the other PEs it has a multi-homed host.

admin@PE_MXR01> show route advertising-protocol bgp 112.112.112.112 EVPN_CUSTOMER_G_ELAN_500.evpn.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:111.111.111.111:50::112233445566778899::0/192 AD/EVI            *                         Self                         100        I   2:111.111.111.111:50::500::00:1e:e5:c8:0f:f1/304 MAC/IP            *                         Self                         100        I   3:111.111.111.111:50::500::111.111.111.111/248 IM                *                         Self                         100        I   3:111.111.111.111:50::501::111.111.111.111/248 IM                *                         Self                         100        I __default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:111.111.111.111:0::112233445566778899::FFFF:FFFF/192 AD/ESI            *                         Self                         100        I   4:111.111.111.111:0::112233445566778899:111.111.111.111/296 ES                *                         Self                         100        I

admin@PE_MXR01> show route advertising-protocol bgp 113.113.113.113    EVPN_CUSTOMER_G_ELAN_500.evpn.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:111.111.111.111:50::112233445566778899::0/192 AD/EVI            *                         Self                         100        I   2:111.111.111.111:50::500::00:1e:e5:c8:0f:f1/304 MAC/IP            *                         Self                         100        I   3:111.111.111.111:50::500::111.111.111.111/248 IM                *                         Self                         100        I   3:111.111.111.111:50::501::111.111.111.111/248 IM                 *                         Self                         100        I __default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:111.111.111.111:0::112233445566778899::FFFF:FFFF/192 AD/ESI            *                         Self                         100        I   4:111.111.111.111:0::112233445566778899:111.111.111.111/296 ES                *                         Self                         100        I

PE_MXR02 didn't need to advertise any Type 1 or 4 routes since it only has a single homed host.

admin@PE_MXR02> show route advertising-protocol bgp 111.111.111.111 EVPN_CUSTOMER_G_ELAN_500.evpn.0: 13 destinations, 13 routes (13 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP            *                         Self                         100        I   3:112.112.112.112:50::500::112.112.112.112/248 IM                *                         Self                         100        I   3:112.112.112.112:50::501::112.112.112.112/248 IM                *                         Self                         100        I

admin@PE_MXR02> show route advertising-protocol bgp 113.113.113.113    EVPN_CUSTOMER_G_ELAN_500.evpn.0: 13 destinations, 13 routes (13 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP            *                         Self                         100        I   3:112.112.112.112:50::500::112.112.112.112/248 IM                *                         Self                         100        I   3:112.112.112.112:50::501::112.112.112.112/248 IM                *                         Self                         100        I

PE_MXR03 advertised its Type 1 and Type 4 routes, signaling to the other PEs it has a multi-homed host.

admin@PE_MXR03> show route advertising-protocol bgp 111.111.111.111 EVPN_CUSTOMER_G_ELAN_500.evpn.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:113.113.113.113:50::112233445566778899::0/192 AD/EVI            *                         Self                         100        I   2:113.113.113.113:50::500::00:0c:29:e4:3e:cd/304 MAC/IP            *                         Self                         100        I   3:113.113.113.113:50::500::113.113.113.113/248 IM                *                         Self                         100        I   3:113.113.113.113:50::501::113.113.113.113/248 IM                *                         Self                         100        I __default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI            *                         Self                         100        I   4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES                *                         Self                         100        I

admin@PE_MXR03> show route advertising-protocol bgp 112.112.112.112    EVPN_CUSTOMER_G_ELAN_500.evpn.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:113.113.113.113:50::112233445566778899::0/192 AD/EVI             *                         Self                         100        I   2:113.113.113.113:50::500::00:0c:29:e4:3e:cd/304 MAC/IP            *                         Self                         100        I   3:113.113.113.113:50::500::113.113.113.113/248 IM                *                         Self                         100        I   3:113.113.113.113:50::501::113.113.113.113/248 IM                *                         Self                         100        I __default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI            *                         Self                         100        I   4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES                *                         Self                         100        I

Route Table EVPN

The BGP EVPN table is the primary route table for the EVI and displays all imported routes.  Every PE will import the Type 1, 2 and 3 routes.  However, importing of the Type 4 route depends on PEs who have matching ESI values (via its ES-Import extended community).

PE_MXR01’s imported all route types, including the Type 4.  Based on this route table, this PE discovered a redundant ES via 113.113.113.113.

admin@PE_MXR01> show route table bgp.evpn.0 bgp.evpn.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI                           *[BGP/170] 3d 04:27:49, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 330 1:113.113.113.113:50::112233445566778899::0/192 AD/EVI                            *[BGP/170] 3d 04:27:50, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 330 2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP                           *[BGP/170] 00:58:32, localpref 100, from 112.112.112.112                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 328 2:113.113.113.113:50::500::00:0c:29:e4:3e:cd/304 MAC/IP                           *[BGP/170] 00:02:47, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 330 3:112.112.112.112:50::500::112.112.112.112/248 IM                               *[BGP/170] 3d 04:28:58, localpref 100, from 112.112.112.112                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 328 3:112.112.112.112:50::501::112.112.112.112/248 IM                               *[BGP/170] 3d 04:28:58, localpref 100, from 112.112.112.112                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 328 3:113.113.113.113:50::500::113.113.113.113/248 IM                               *[BGP/170] 3d 04:27:53, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 330 3:113.113.113.113:50::501::113.113.113.113/248 IM                               *[BGP/170] 3d 04:27:53, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 330 4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES                               *[BGP/170] 3d 04:27:50, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 330

PE_MXR02 imported all route types except the type 4 route.  This PE can reach the multi-homed ES via 2 remote PEs.

admin@PE_MXR02> show route table bgp.evpn.0 bgp.evpn.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 1:111.111.111.111:0::112233445566778899::FFFF:FFFF/192 AD/ESI                           *[BGP/170] 3d 04:29:05, localpref 100, from 111.111.111.111                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 329 1:111.111.111.111:50::112233445566778899::0/192 AD/EVI                           *[BGP/170] 3d 04:29:06, localpref 100, from 111.111.111.111                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 329 1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI                           *[BGP/170] 3d 04:28:07, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 330 1:113.113.113.113:50::112233445566778899::0/192 AD/EVI                           *[BGP/170] 3d 04:28:08, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 330 2:111.111.111.111:50::500::00:1e:e5:c8:0f:f1/304 MAC/IP                           *[BGP/170] 00:58:50, localpref 100, from 111.111.111.111                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 329 2:113.113.113.113:50::500::00:0c:29:e4:3e:cd/304 MAC/IP                           *[BGP/170] 00:03:05, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 330 3:111.111.111.111:50::500::111.111.111.111/248 IM                               *[BGP/170] 3d 04:29:16, localpref 100, from 111.111.111.111                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 329 3:111.111.111.111:50::501::111.111.111.111/248 IM                               *[BGP/170] 3d 04:29:16, localpref 100, from 111.111.111.111                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 329 3:113.113.113.113:50::500::113.113.113.113/248 IM                               *[BGP/170] 3d 04:28:18, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 330 3:113.113.113.113:50::501::113.113.113.113/248 IM                               *[BGP/170] 3d 04:28:18, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 330

PE_MXR03’s imported all route types, including the Type 4.  Based on this route table, this PE discovered a redundant ES via 111.111.111.111.

admin@PE_MXR03> show route table bgp.evpn.0 bgp.evpn.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 1:111.111.111.111:0::112233445566778899::FFFF:FFFF/192 AD/ESI                           *[BGP/170] 3d 04:28:31, localpref 100, from 111.111.111.111                       AS path: I, validation-state: unverified                     > to 10.1.1.97 via ge-0/0/1.48, Push 229 1:111.111.111.111:50::112233445566778899::0/192 AD/EVI                           *[BGP/170] 3d 04:28:31, localpref 100, from 111.111.111.111                       AS path: I, validation-state: unverified                     > to 10.1.1.97 via ge-0/0/1.48, Push 229 2:111.111.111.111:50::500::00:1e:e5:c8:0f:f1/304 MAC/IP                           *[BGP/170] 00:59:10, localpref 100, from 111.111.111.111                       AS path: I, validation-state: unverified                     > to 10.1.1.97 via ge-0/0/1.48, Push 229 2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP                           *[BGP/170] 00:59:10, localpref 100, from 112.112.112.112                       AS path: I, validation-state: unverified                     > to 10.1.1.97 via ge-0/0/1.48, Push 228 3:111.111.111.111:50::500::111.111.111.111/248 IM                               *[BGP/170] 3d 04:28:31, localpref 100, from 111.111.111.111                       AS path: I, validation-state: unverified                     > to 10.1.1.97 via ge-0/0/1.48, Push 229 3:111.111.111.111:50::501::111.111.111.111/248 IM                               *[BGP/170] 3d 04:28:31, localpref 100, from 111.111.111.111                       AS path: I, validation-state: unverified                     > to 10.1.1.97 via ge-0/0/1.48, Push 229 3:112.112.112.112:50::500::112.112.112.112/248 IM                               *[BGP/170] 3d 04:28:37, localpref 100, from 112.112.112.112                       AS path: I, validation-state: unverified                     > to 10.1.1.97 via ge-0/0/1.48, Push 228 3:112.112.112.112:50::501::112.112.112.112/248 IM                               *[BGP/170] 3d 04:28:37, localpref 100, from 112.112.112.112                       AS path: I, validation-state: unverified                     > to 10.1.1.97 via ge-0/0/1.48, Push 228 4:111.111.111.111:0::112233445566778899:111.111.111.111/296 ES                               *[BGP/170] 3d 04:28:31, localpref 100, from 111.111.111.111                       AS path: I, validation-state: unverified                     > to 10.1.1.97 via ge-0/0/1.48, Push 229

Route Table EVPN (Type 1 EAD per ESI NLRI)

Here’s a deeper look into the BGP Type 1 Ethernet Auto Discovery per Ethernet Segment NLRI (Fast Convergence).  More information about the Type 1 EAD per ESI route can be found in RFC 7432, Sec. 8.2.

• Route Distinguisher
• ESI
• Ethernet Tag
• MPLS Label 
• Note: Label set to 0. Not seen in extensive output, however was seen in the Wireshark capture below.
• ESI-Label Extended Community
• Set to 0 for All-Active
• Split-Horizon MPLS Label

Route output from PE_MXR01

1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI (1 entry, 0 announced)         *BGP    Preference: 170/-101                 Route Distinguisher: 113.113.113.113:0                 Next hop type: Indirect, Next hop index: 0                 Address: 0xb7a3a30                 Next-hop reference count: 12                 Source: 113.113.113.113                 Protocol next hop: 113.113.113.113                 Indirect next hop: 0x2 no-forward INH Session ID: 0x0                 State: <Active Int Ext>                 Local AS:  2345 Peer AS:  2345                 Age: 3d 4:34:49         Metric2: 1                 Validation State: unverified                 Task: BGP_2345.113.113.113.113                 AS path: I                 Communities: target:2345:50 esi-label:0x0:all-active (label 301200)                 Import Accepted                 Localpref: 100                 Router ID: 113.113.113.113                 Secondary Tables: EVPN_CUSTOMER_G_ELAN_500.evpn.0                 Indirect next hops: 1                         Protocol next hop: 113.113.113.113 Metric: 1                         Indirect next hop: 0x2 no-forward INH Session ID: 0x0                         Indirect path forwarding next hops: 1                                 Next hop type: Router                                 Next hop: 10.1.1.81 via ge-0/0/1.44                                 Session Id: 0x0                         113.113.113.113/32 Originating RIB: inet.3                           Metric: 1                       Node path count: 1                           Forwarding nexthops: 1                                 Nexthop: 10.1.1.81 via ge-0/0/1.44

Wireshark Capture NLRI for Type1 EAD per ESI

Route Table EVPN (Type 1 EAD per EVI NLRI)

Here’s a deeper look into the BGP Type 1 Ethernet Auto Discovery per EVPN Instance NLRI (Aliasing).  More information about the Type 1 EAD per EVI route can be found in RFC 7432, Sec. 8.4.1.

• Route Distinguisher
• ESI
• Ethernet Tag (None or 0)
• MPLS Aliasing Label

Route output from PE_MXR01

1:113.113.113.113:50::112233445566778899::0/192 AD/EVI (1 entry, 0 announced)         *BGP    Preference: 170/-101                 Route Distinguisher: 113.113.113.113:50                 Next hop type: Indirect, Next hop index: 0                 Address: 0xb7a3a30                 Next-hop reference count: 12                 Source: 113.113.113.113                 Protocol next hop: 113.113.113.113                 Indirect next hop: 0x2 no-forward INH Session ID: 0x0                 State: <Active Int Ext>                 Local AS:  2345 Peer AS:  2345                 Age: 3d 4:34:50         Metric2: 1                 Validation State: unverified                 Task: BGP_2345.113.113.113.113                 AS path: I                 Communities: target:2345:50                 Import Accepted                 Route Label: 300512                 Localpref: 100                 Router ID: 113.113.113.113                 Secondary Tables: EVPN_CUSTOMER_G_ELAN_500.evpn.0                 Indirect next hops: 1                         Protocol next hop: 113.113.113.113 Metric: 1                         Indirect next hop: 0x2 no-forward INH Session ID: 0x0                         Indirect path forwarding next hops: 1                                 Next hop type: Router                                 Next hop: 10.1.1.81 via ge-0/0/1.44                                 Session Id: 0x0                         113.113.113.113/32 Originating RIB: inet.3                           Metric: 1                       Node path count: 1                           Forwarding nexthops: 1                                 Nexthop: 10.1.1.81 via ge-0/0/1.44

Wireshark Capture NLRI for Type1 EAD per EVI

Route Table EVPN (Type 4 ES Route NLRI)

Here’s a look at the BGP Type 4 Ethernet Segment Route NLRI (Ethernet Segment Route).  More information about the Type 4 Ethernet Segment route can be found in RFC 7432, Sec. 8.5.

• Route Distinguisher
• ESI
• IP Address Length
• Note: Set to 32. Not seen in route output, but seen in the Wireshark capture below.
• Originating Router’s IP Address
• ES-Import Extended Community

Route output from PE_MXR01

4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES (1 entry, 0 announced)         *BGP    Preference: 170/-101                 Route Distinguisher: 113.113.113.113:0                 Next hop type: Indirect, Next hop index: 0                 Address: 0xb7a3a30                 Next-hop reference count: 12                 Source: 113.113.113.113                 Protocol next hop: 113.113.113.113                 Indirect next hop: 0x2 no-forward INH Session ID: 0x0                 State: <Active Int Ext>                 Local AS:  2345 Peer AS:  2345                 Age: 3d 4:34:50         Metric2: 1                 Validation State: unverified                 Task: BGP_2345.113.113.113.113                 AS path: I                 Communities: es-import-target:11-22-33-44-55-66                 Import Accepted                 Localpref: 100                 Router ID: 113.113.113.113                 Secondary Tables: __default_evpn__.evpn.0                 Indirect next hops: 1                         Protocol next hop: 113.113.113.113 Metric: 1                         Indirect next hop: 0x2 no-forward INH Session ID: 0x0                         Indirect path forwarding next hops: 1                                 Next hop type: Router                                 Next hop: 10.1.1.81 via ge-0/0/1.44                                 Session Id: 0x0                         113.113.113.113/32 Originating RIB: inet.3                           Metric: 1                       Node path count: 1                           Forwarding nexthops: 1                                 Nexthop: 10.1.1.81 via ge-0/0/1.44

Wireshark Capture NLRI for Type4 ES Route

Verification Tasks (During Failure)

Failure Testing - Simulated Failure of an Ethernet Segment on PE_MXR01

CE_R27’s G1 interface was shutdown to simulate an Ethernet segment failure to PE_MXR01.

CE_R27#conf t Enter configuration commands, one per line.  End with CNTL/Z. CE_R27(config)#int g1 CE_R27(config-if)#shut CE_R27(config-if)#end

CE_R27#sh int desc Interface                      Status         Protocol Description Gi1                            admin down     down     TO PE_MXR01 Gi2                            up             up       TO PE_MXR03 Gi3                            admin down     down     Gi4                            up             up       MANAGEMENT Po50                           up             up       ETH CH TO PE_MXR01 & PE_MXR03

PE_MXR01’s AC interface towards CE_R27’s G1 interface was shutdown to ensure a failure at both ends.

admin@PE_MXR01> configure Entering configuration mode [edit] admin@PE_MXR01# set interfaces ge-0/0/2 disable [edit] admin@PE_MXR01# commit commit complete

admin@PE_MXR01# run show interfaces terse ge-0/0/2  Interface               Admin Link Proto    Local                 Remote ge-0/0/2                down  down ge-0/0/2.500            up    down bridge  ge-0/0/2.32767          up    down multiservice

The list below outlined commands used to verify EVPN operation in an all-active configuration during a failure condition.

• Show EVPN Instance Extensive
• Show Route Advertising-Protocol BGP
• Show Route Table EVPN

EVPN Instance (During Failure)

PE_MXR01’s EVPN Instance output displayed its local ES as down.  The DF was switched and the ESI was still available via remote PE at 113.113.113.113.

admin@PE_MXR01> show evpn instance extensive Instance: EVPN_CUSTOMER_G_ELAN_500   Route Distinguisher: 111.111.111.111:50   Per-instance MAC route label: 299776   MAC database status                     Local  Remote     MAC advertisements:                       0       2     MAC+IP advertisements:                    0       0     Default gateway MAC advertisements:       0       0   Number of local interfaces: 1 (0 up)     Interface name  ESI                            Mode             Status     AC-Role     ge-0/0/2.500    00:11:22:33:44:55:66:77:88:99  all-active       Down       Root   Number of IRB interfaces: 0 (0 up)   Number of bridge domains: 2     VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop     500                  1    0                Extended         Enabled                   Disabled     501                  1    0                Extended         Enabled                   Disabled   Number of neighbors: 2     Address               MAC    MAC+IP        AD        IM        ES Leaf-label     112.112.112.112         1         0         0         2         0     113.113.113.113         1         0         2         2         0   Number of ethernet segments: 1     ESI: 00:11:22:33:44:55:66:77:88:99       Status: Resolved by NH 1048578       Local interface: ge-0/0/2.500, Status: Down       Number of remote PEs connected: 1         Remote PE        MAC label  Aliasing label  Mode         113.113.113.113  300512     300512          all-active         Designated forwarder: 113.113.113.113       Last designated forwarder update: Jan 17 07:47:23       Advertised MAC label: 300640       Advertised aliasing label: 300640       Advertised split horizon label: 301184 Instance: __default_evpn__   Route Distinguisher: 111.111.111.111:0   Number of bridge domains: 0   Number of neighbors: 1     Address               MAC    MAC+IP        AD        IM        ES Leaf-label     113.113.113.113         0         0         0         0         1

From PE_MXR02's perspective, the ES was available via remote PE at 113.113.113.113.

admin@PE_MXR02> show evpn instance extensive Instance: EVPN_CUSTOMER_G_ELAN_500   Route Distinguisher: 112.112.112.112:50   Per-instance MAC route label: 299776   MAC database status                     Local  Remote     MAC advertisements:                       1       1     MAC+IP advertisements:                    0       0     Default gateway MAC advertisements:       0       0   Number of local interfaces: 1 (1 up)     Interface name  ESI                            Mode             Status     AC-Role     ge-0/0/2.500    00:00:00:00:00:00:00:00:00:00  single-homed     Up         Root   Number of IRB interfaces: 0 (0 up)   Number of bridge domains: 2     VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop     500                  1    1                Extended         Enabled   299840          Disabled     501                  1    1                Extended         Enabled   299856          Disabled   Number of neighbors: 1     Address               MAC    MAC+IP        AD        IM        ES Leaf-label     113.113.113.113         1         0         2         2         0   Number of ethernet segments: 1     ESI: 00:11:22:33:44:55:66:77:88:99       Status: Resolved by NH 1048579       Number of remote PEs connected: 1         Remote PE        MAC label  Aliasing label  Mode         113.113.113.113  300512     300512          all-active   Instance: __default_evpn__   Route Distinguisher: 112.112.112.112:0   Number of bridge domains: 0   Number of neighbors: 0

PE_MXR03's EVPN Instance output displayed the ES was still reachable via the local interface of GE-0/0/3.500.  The DF was switched to this PE.

admin@PE_MXR03> show evpn instance extensive Instance: EVPN_CUSTOMER_G_ELAN_500   Route Distinguisher: 113.113.113.113:50   Per-instance MAC route label: 299776   MAC database status                     Local  Remote     MAC advertisements:                       1       1     MAC+IP advertisements:                    0       0     Default gateway MAC advertisements:       0       0   Number of local interfaces: 2 (2 up)     Interface name  ESI                            Mode             Status     AC-Role     ge-0/0/2.500    00:00:00:00:00:00:00:00:00:00  single-homed     Up         Root     ge-0/0/3.500    00:11:22:33:44:55:66:77:88:99  all-active       Up         Root   Number of IRB interfaces: 0 (0 up)   Number of bridge domains: 2     VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop     500                  2    2                Extended         Enabled   300528          Disabled     501                  2    2                Extended         Enabled   300544          Disabled   Number of neighbors: 1     Address               MAC    MAC+IP        AD        IM        ES Leaf-label     112.112.112.112         1         0         0         2         0   Number of ethernet segments: 1     ESI: 00:11:22:33:44:55:66:77:88:99       Status: Resolved by IFL ge-0/0/3.500       Local interface: ge-0/0/3.500, Status: Up/Forwarding       Designated forwarder: 113.113.113.113       Last designated forwarder update: Jan 17 07:47:24       Advertised MAC label: 300512       Advertised aliasing label: 300512       Advertised split horizon label: 301200 Instance: __default_evpn__   Route Distinguisher: 113.113.113.113:0   Number of bridge domains: 0   Number of neighbors: 0

BGP Route Advertising (During Failure)

PE_MXR01 withdrew all routes during the failure of its ES.  This was done to signal the other PEs to update their forwarding adjacency to the backup PE.

admin@PE_MXR01> show route advertising-protocol bgp 112.112.112.112 admin@PE_MXR01>

admin@PE_MXR01> show route advertising-protocol bgp 113.113.113.113 admin@PE_MXR01>

PE_MXR02 didn't contain any related ES and therefore its advertisements did not change during a failure.

admin@PE_MXR02> show route advertising-protocol bgp 111.111.111.111 EVPN_CUSTOMER_G_ELAN_500.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP            *                         Self                         100        I   3:112.112.112.112:50::500::112.112.112.112/248 IM                *                         Self                         100        I   3:112.112.112.112:50::501::112.112.112.112/248 IM                *                         Self                         100        I

admin@PE_MXR02> show route advertising-protocol bgp 113.113.113.113    EVPN_CUSTOMER_G_ELAN_500.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP            *                         Self                         100        I   3:112.112.112.112:50::500::112.112.112.112/248 IM                *                         Self                         100        I   3:112.112.112.112:50::501::112.112.112.112/248 IM                *                         Self                         100        I

PE_MXR03 continued to advertise Type 1 and 4 routes since its ES was still up.

admin@PE_MXR03> show route advertising-protocol bgp 111.111.111.111 EVPN_CUSTOMER_G_ELAN_500.evpn.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:113.113.113.113:50::112233445566778899::0/192 AD/EVI            *                         Self                         100        I   2:113.113.113.113:50::500::00:1e:e5:c8:0f:f1/304 MAC/IP            *                         Self                         100        I   3:113.113.113.113:50::500::113.113.113.113/248 IM                *                         Self                         100        I   3:113.113.113.113:50::501::113.113.113.113/248 IM                *                         Self                         100        I __default_evpn__.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI            *                         Self                         100        I   4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES                *                         Self                         100        I

admin@PE_MXR03> show route advertising-protocol bgp 112.112.112.112   EVPN_CUSTOMER_G_ELAN_500.evpn.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:113.113.113.113:50::112233445566778899::0/192 AD/EVI            *                         Self                         100        I   2:113.113.113.113:50::500::00:1e:e5:c8:0f:f1/304 MAC/IP            *                         Self                         100        I   3:113.113.113.113:50::500::113.113.113.113/248 IM                *                         Self                         100        I   3:113.113.113.113:50::501::113.113.113.113/248 IM                *                         Self                         100        I __default_evpn__.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI            *                         Self                         100        I   4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES                *                         Self                         100        I

Route Table EVPN (During Failure)

PE_MXR01 only saw the Type 1 and 4 routes from 113.113.113.113.  With its local ES down, this indicated that the ES was only reachable via 113.113.113.113.

admin@PE_MXR01> show route table bgp.evpn.0 bgp.evpn.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI                           *[BGP/170] 6d 04:57:19, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 330 1:113.113.113.113:50::112233445566778899::0/192 AD/EVI                           *[BGP/170] 6d 04:57:20, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 330 2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP                           *[BGP/170] 01:07:09, localpref 100, from 112.112.112.112                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 328 2:113.113.113.113:50::500::00:1e:e5:c8:0f:f1/304 MAC/IP                           *[BGP/170] 01:03:06, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 330 3:112.112.112.112:50::500::112.112.112.112/248 IM                               *[BGP/170] 6d 04:58:28, localpref 100, from 112.112.112.112                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 328 3:112.112.112.112:50::501::112.112.112.112/248 IM                               *[BGP/170] 6d 04:58:28, localpref 100, from 112.112.112.112                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 328 3:113.113.113.113:50::500::113.113.113.113/248 IM                               *[BGP/170] 6d 04:57:23, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 330 3:113.113.113.113:50::501::113.113.113.113/248 IM                               *[BGP/170] 6d 04:57:23, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 330 4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES                                *[BGP/170] 6d 04:57:20, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 330

The same was true for PE_MXR02.  Since it imported Type 1 routes only from 113.113.113.113, the ES was only reachable via that address.

admin@PE_MXR02> show route table bgp.evpn.0 bgp.evpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI                           *[BGP/170] 6d 04:57:33, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 330 1:113.113.113.113:50::112233445566778899::0/192 AD/EVI                           *[BGP/170] 6d 04:57:34, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 330 2:113.113.113.113:50::500::00:1e:e5:c8:0f:f1/304 MAC/IP                           *[BGP/170] 01:03:20, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 330 3:113.113.113.113:50::500::113.113.113.113/248 IM                               *[BGP/170] 6d 04:57:44, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 330 3:113.113.113.113:50::501::113.113.113.113/248 IM                               *[BGP/170] 6d 04:57:44, localpref 100, from 113.113.113.113                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 330

PE_MXR03 saw no routes from 111.111.111.111.  With its local ES as up, this was the active PE.

admin@PE_MXR03> show route table bgp.evpn.0 bgp.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP                           *[BGP/170] 01:07:32, localpref 100, from 112.112.112.112                       AS path: I, validation-state: unverified                     > to 10.1.1.97 via ge-0/0/1.48, Push 228 3:112.112.112.112:50::500::112.112.112.112/248 IM                               *[BGP/170] 6d 04:57:52, localpref 100, from 112.112.112.112                       AS path: I, validation-state: unverified                     > to 10.1.1.97 via ge-0/0/1.48, Push 228 3:112.112.112.112:50::501::112.112.112.112/248 IM                               *[BGP/170] 6d 04:57:52, localpref 100, from 112.112.112.112                       AS path: I, validation-state: unverified                     > to 10.1.1.97 via ge-0/0/1.48, Push 228

Appendix

BGP Type 4 Route Reflection Issue

Using Cisco IOS XE Version 03.16.05.S as the BGP Route Reflector (RR) may have caused an issue where the Type 4 routes were not reflected correctly to all PEs.  Since these Type 4 routes were not imported/exported correctly, the DF and BDF elections were not taking place.  I can’t confirm if this is a misconfiguration or software bug, however this was the closest matching bug I saw on CCO.

Below were my results based on this unexpected behavior of the RR.

PE_MXR01’s advertised routes to the RR.  Type 1 and 4 routes were being advertised.

admin@PE_MXR01> show route advertising-protocol bgp 11.11.11.11              EVPN_CUSTOMER_G_ELAN_500.evpn.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:111.111.111.111:50::112233445566778899::0/192 AD/EVI            *                         Self                         100        I   2:111.111.111.111:50::500::00:1e:e5:c8:0f:f1/304 MAC/IP            *                         Self                         100        I   3:111.111.111.111:50::500::111.111.111.111/248 IM                *                         Self                         100        I   3:111.111.111.111:50::501::111.111.111.111/248 IM                *                         Self                         100        I __default_evpn__.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:111.111.111.111:0::112233445566778899::FFFF:FFFF/192 AD/ESI            *                         Self                         100        I   4:111.111.111.111:0::112233445566778899:111.111.111.111/296 ES                *                         Self                         100        I

PE_MXR01’s received routes from the RR.  Type 1 routes were received while the Type 4 route was not.

admin@PE_MXR01> show route receive-protocol bgp 11.11.11.11    inet.0: 46 destinations, 46 routes (46 active, 0 holddown, 0 hidden) inet.3: 39 destinations, 39 routes (39 active, 0 holddown, 0 hidden) MGMT.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden) mpls.0: 47 destinations, 48 routes (47 active, 0 holddown, 0 hidden) inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) bgp.evpn.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI            *                         113.113.113.113              100        I   1:113.113.113.113:50::112233445566778899::0/192 AD/EVI            *                         113.113.113.113              100        I   2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP            *                         112.112.112.112              100        I   3:112.112.112.112:50::500::112.112.112.112/248 IM                *                         112.112.112.112              100        I   3:112.112.112.112:50::501::112.112.112.112/248 IM                *                         112.112.112.112              100        I   3:113.113.113.113:50::500::113.113.113.113/248 IM                 *                         113.113.113.113              100        I   3:113.113.113.113:50::501::113.113.113.113/248 IM                *                         113.113.113.113              100        I EVPN_CUSTOMER_G_ELAN_500.evpn.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI            *                         113.113.113.113              100        I   1:113.113.113.113:50::112233445566778899::0/192 AD/EVI            *                         113.113.113.113              100        I   2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP            *                         112.112.112.112              100        I   3:112.112.112.112:50::500::112.112.112.112/248 IM                *                         112.112.112.112              100        I   3:112.112.112.112:50::501::112.112.112.112/248 IM                *                         112.112.112.112              100        I   3:113.113.113.113:50::500::113.113.113.113/248 IM                *                         113.113.113.113              100        I   3:113.113.113.113:50::501::113.113.113.113/248 IM                *                         113.113.113.113              100        I __default_evpn__.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)

PE_MXR03’s advertised routes to the RR.  Type 1 and 4 routes were being advertised.

admin@PE_MXR03> show route advertising-protocol bgp 11.11.11.11 EVPN_CUSTOMER_G_ELAN_500.evpn.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:113.113.113.113:50::112233445566778899::0/192 AD/EVI            *                         Self                         100        I   3:113.113.113.113:50::500::113.113.113.113/248 IM                *                         Self                         100        I   3:113.113.113.113:50::501::113.113.113.113/248 IM                *                         Self                         100        I __default_evpn__.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI            *                         Self                         100        I   4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES                *                         Self                         100        I

PE_MXR03’s received routes from the RR.  Type 1 routes were received while the Type 4 route was not.

admin@PE_MXR03> show route receive-protocol bgp 11.11.11.11                                         inet.0: 46 destinations, 46 routes (46 active, 0 holddown, 0 hidden) inet.3: 39 destinations, 39 routes (39 active, 0 holddown, 0 hidden) MGMT.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden) mpls.0: 44 destinations, 44 routes (44 active, 0 holddown, 0 hidden) inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) bgp.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:111.111.111.111:0::112233445566778899::FFFF:FFFF/192 AD/ESI            *                         111.111.111.111              100        I   1:111.111.111.111:50::112233445566778899::0/192 AD/EVI            *                         111.111.111.111              100        I   2:111.111.111.111:50::500::00:1e:e5:c8:0f:f1/304 MAC/IP            *                         111.111.111.111              100        I   2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP            *                         112.112.112.112              100        I   3:111.111.111.111:50::500::111.111.111.111/248 IM                *                         111.111.111.111              100        I   3:111.111.111.111:50::501::111.111.111.111/248 IM                *                         111.111.111.111              100        I   3:112.112.112.112:50::500::112.112.112.112/248 IM                *                         112.112.112.112              100        I   3:112.112.112.112:50::501::112.112.112.112/248 IM                *                         112.112.112.112              100        I EVPN_CUSTOMER_G_ELAN_500.evpn.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)   Prefix                  Nexthop              MED     Lclpref    AS path   1:111.111.111.111:0::112233445566778899::FFFF:FFFF/192 AD/ESI            *                         111.111.111.111              100        I   1:111.111.111.111:50::112233445566778899::0/192 AD/EVI            *                         111.111.111.111              100        I   2:111.111.111.111:50::500::00:1e:e5:c8:0f:f1/304 MAC/IP            *                         111.111.111.111              100        I   2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP            *                         112.112.112.112              100        I   3:111.111.111.111:50::500::111.111.111.111/248 IM                *                         111.111.111.111              100        I   3:111.111.111.111:50::501::111.111.111.111/248 IM                *                         111.111.111.111              100        I   3:112.112.112.112:50::500::112.112.112.112/248 IM                *                         112.112.112.112              100        I   3:112.112.112.112:50::501::112.112.112.112/248 IM                *                         112.112.112.112              100        I __default_evpn__.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)

The RR’s BGP table for the EVPN address family contained no Type 4 routes.  These routes were not imported from the PEs.

RR_R11#sh ip bgp l2vpn evpn          BGP table version is 21, local router ID is 11.11.11.11 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,               r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,               x best-external, a additional-path, c RIB-compressed, Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found      Network          Next Hop            Metric LocPrf Weight Path Route Distinguisher: 111.111.111.111:0  *>i [1][111.111.111.111:0][00112233445566778899][4294967295][0]/25                        111.111.111.111               100      0 i Route Distinguisher: 111.111.111.111:50  *>i [1][111.111.111.111:50][00112233445566778899][0][300640]/25                        111.111.111.111               100      0 i Route Distinguisher: 113.113.113.113:0  *>i [1][113.113.113.113:0][00112233445566778899][4294967295][0]/25                        113.113.113.113               100      0 i Route Distinguisher: 113.113.113.113:50  *>i [1][113.113.113.113:50][00112233445566778899][0][300512]/25                        113.113.113.113               100      0 i Route Distinguisher: 111.111.111.111:50  *>i [2][111.111.111.111:50][00112233445566778899][500][48][001EE5C80FF1][0][*][300640]/33                        111.111.111.111               100      0 i Route Distinguisher: 112.112.112.112:50  *>i [2][112.112.112.112:50][00000000000000000000][500][48][000C2949AA8C][0][*][299776]/33                        112.112.112.112               100      0 i Route Distinguisher: 111.111.111.111:50  *>i [3][111.111.111.111:50][500][32][578748416.30d0.06a8.227f]/17                        111.111.111.111               100      0 i  *>i [3][111.111.111.111:50][501][32][578748416.30d0.06a8.227f]/17                        111.111.111.111               100      0 i Route Distinguisher: 112.112.112.112:50  *>i [3][112.112.112.112:50][500][32][578748416.30d0.06a8.227f]/17                        112.112.112.112               100      0 i  *>i [3][112.112.112.112:50][501][32][578748416.30d0.06a8.227f]/17                        112.112.112.112               100      0 i Route Distinguisher: 113.113.113.113:50  *>i [3][113.113.113.113:50][500][32][578748416.30d0.06a8.227f]/17                        113.113.113.113               100      0 i  *>i [3][113.113.113.113:50][501][32][578748416.30d0.06a8.227f]/17                        113.113.113.113               100      0 i

The RR received all route types from PE_MXR01 except the Type 4.

RR_R11#sh ip bgp l2vpn evpn neighbors 111.111.111.111 received-routes BGP table version is 21, local router ID is 11.11.11.11 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,               r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,               x best-external, a additional-path, c RIB-compressed, Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found      Network          Next Hop            Metric LocPrf Weight Path Route Distinguisher: 111.111.111.111:0  *>i [1][111.111.111.111:0][00112233445566778899][4294967295][0]/25                        111.111.111.111               100      0 i Route Distinguisher: 111.111.111.111:50  *>i [1][111.111.111.111:50][00112233445566778899][0][300640]/25                        111.111.111.111               100      0 i  *>i [2][111.111.111.111:50][00112233445566778899][500][48][001EE5C80FF1][0][*][300640]/33                        111.111.111.111               100      0 i  *>i [3][111.111.111.111:50][500][32][UNKNOWN]/17                        111.111.111.111               100      0 i  *>i [3][111.111.111.111:50][501][32][UNKNOWN]/17                        111.111.111.111               100      0 i Total number of prefixes 5

The RR received all route types from PE_MXR03 except the Type 4.

RR_R11#sh ip bgp l2vpn evpn neighbors 113.113.113.113 received-routes BGP table version is 21, local router ID is 11.11.11.11 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,               r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,               x best-external, a additional-path, c RIB-compressed, Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found      Network          Next Hop            Metric LocPrf Weight Path Route Distinguisher: 113.113.113.113:0  *>i [1][113.113.113.113:0][00112233445566778899][4294967295][0]/25                        113.113.113.113               100      0 i Route Distinguisher: 113.113.113.113:50  *>i [1][113.113.113.113:50][00112233445566778899][0][300512]/25                        113.113.113.113               100      0 i  *>i [3][113.113.113.113:50][500][32][UNKNOWN]/17                        113.113.113.113               100      0 i  *>i [3][113.113.113.113:50][501][32][UNKNOWN]/17                        113.113.113.113               100      0 i Total number of prefixes 4

Without the Type 4 route, DF election on PE_MXR01 did not occur and was set to itself.

admin@PE_MXR01> show evpn instance EVPN_CUSTOMER_G_ELAN_500 extensive Instance: EVPN_CUSTOMER_G_ELAN_500   Route Distinguisher: 111.111.111.111:50   Per-instance MAC route label: 299776   MAC database status                     Local  Remote     MAC advertisements:                       1       1     MAC+IP advertisements:                    0       0     Default gateway MAC advertisements:       0       0   Number of local interfaces: 1 (1 up)     Interface name  ESI                            Mode             Status     AC-Role     ge-0/0/2.500    00:11:22:33:44:55:66:77:88:99  all-active       Up         Root   Number of IRB interfaces: 0 (0 up)   Number of bridge domains: 2     VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop     500                  1    1                Extended         Enabled   300480          Disabled     501                  1    1                Extended         Enabled   300496          Disabled   Number of neighbors: 2     Address               MAC    MAC+IP        AD        IM        ES Leaf-label     112.112.112.112         1         0         0         2         0     113.113.113.113         0         0         2         2         0   Number of ethernet segments: 1     ESI: 00:11:22:33:44:55:66:77:88:99       Status: Resolved by IFL ge-0/0/2.500       Local interface: ge-0/0/2.500, Status: Up/Forwarding       Number of remote PEs connected: 1         Remote PE        MAC label  Aliasing label  Mode         113.113.113.113  300640     300640          all-active         Designated forwarder: 111.111.111.111                       Last designated forwarder update: Dec 31 10:49:25       Advertised MAC label: 300448       Advertised aliasing label: 300448       Advertised split horizon label: 300464

PE_MXR03’s DF was also set to itself.  DF election did not occur on this PE.

admin@PE_MXR03> show evpn instance EVPN_CUSTOMER_G_ELAN_500 extensive Instance: EVPN_CUSTOMER_G_ELAN_500   Route Distinguisher: 113.113.113.113:50   Per-instance MAC route label: 299776   MAC database status                     Local  Remote     MAC advertisements:                       0       2     MAC+IP advertisements:                    0       0     Default gateway MAC advertisements:       0       0   Number of local interfaces: 2 (2 up)     Interface name  ESI                            Mode             Status     AC-Role     ge-0/0/2.500    00:00:00:00:00:00:00:00:00:00  single-homed     Up         Root     ge-0/0/3.500    00:11:22:33:44:55:66:77:88:99  all-active       Up         Root   Number of IRB interfaces: 0 (0 up)   Number of bridge domains: 2     VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop     500                  2    2                Extended         Enabled   299840          Disabled     501                  2    2                Extended         Enabled   299856          Disabled   Number of neighbors: 2     Address               MAC    MAC+IP        AD        IM        ES Leaf-label     111.111.111.111         1         0         2         2         0     112.112.112.112         1         0         0         2         0   Number of ethernet segments: 1     ESI: 00:11:22:33:44:55:66:77:88:99       Status: Resolved by IFL ge-0/0/3.500       Local interface: ge-0/0/3.500, Status: Up/Forwarding       Number of remote PEs connected: 1         Remote PE        MAC label  Aliasing label  Mode         111.111.111.111  300448     300448          all-active         Designated forwarder: 113.113.113.113                Last designated forwarder update: Dec 31 10:51:45       Advertised MAC label: 300640       Advertised aliasing label: 300640       Advertised split horizon label: 300656

References

EVPN All-Active Multi-Homing

https://onestore.nokia.com/asset/178187/Nokia_EVPN_Conference_Presentation_EN.pdf

https://www.juniper.net/documentation/en_US/junos/topics/concept/evpn-bgp-multihoming-overview.html

https://www.juniper.net/documentation/en_US/junos/topics/example/example-evpn-a-a-basic.html

https://forums.juniper.net/t5/Data-Center-Technologists/MC-LAG-is-dead-Long-live-EVPN-Multi-homing/ba-p/298924

Other Blogs and Labs

https://tgregory.org/tag/evpn-multihoming/

https://kb.wisc.edu/uwsysnet/page.php?id=56133

https://tgregory.org/2016/06/16/evpn-single-active-redundancy/