Saturday, March 17, 2018

MPLS L2VPN (EVPN on JunOS with Single-Active Multi-Homing)

Exploring EVPN’s multi-homing capabilities wouldn’t be complete without a detailed look into Single-Active mode.  To finish off my testing, I modified my previous All-Active lab to Single-Active to observe the differences between the two redundancy modes.

I was also motivated to understand MPLS label usage, as I experienced some inconsistent behavior with label usage during the All-Active lab.  My results and documentation can be found here.  Hopefully this lab will reinforce my understanding of multi-homing operation in EVPN.


Lab Environment

Note: No additional devices were added or changed.


Diagram & Topology

CE_R27’s G1 and G2 interfaces were reconfigured from an Ether-Channel to an L2 bridge domain with a BDI acting as the IP enabled interface (highlighted).





Configurations Changes

Minor configuration changes were only made to PE_MXR01, PE_MXR03 and CE_R27.  No changes to PE_MXR02 were made.


Attachment Circuit

Redundancy mode was changed from all-active to single-active.

PE_MXR01

interfaces {
    ge-0/0/2 {
        description "TO CUSTOMER_G ELAN 500 VSWITCH_24";
        flexible-vlan-tagging;
        encapsulation flexible-ethernet-services;
        esi {
            00:11:22:33:44:55:66:77:88:99;
            single-active;
        }
        unit 500 {
            family bridge {
                interface-mode trunk;
                vlan-id-list [ 500 501 ];
            }
        }
    }


PE_MXR03

interfaces {
    ge-0/0/3 {
        description "TO CUSTOMER_G ELAN 500 VSWITCH_27";
        flexible-vlan-tagging;
        encapsulation flexible-ethernet-services;
        esi {
            00:11:22:33:44:55:66:77:88:99;
            single-active;
        }
        unit 500 {
            family bridge {
                interface-mode trunk;
                vlan-id-list [ 500 501 ];
            }
        }
    }


CE_R27

The CE’s Ethernet interfaces facing the PEs were re-configured to be in a L2 bridge domain.

!
bridge-domain 50
 mac aging-time 30
!
interface GigabitEthernet1
 description TO PE_MXR01
 no ip address
 load-interval 30
 negotiation auto
 no mop enabled
 no mop sysid
 service instance 50 ethernet
  encapsulation untagged
  bridge-domain 50
 !
!
interface GigabitEthernet2
 description TO PE_MXR03
 no ip address
 load-interval 30
 negotiation auto
 no mop enabled
 no mop sysid
 service instance 50 ethernet
  encapsulation untagged
  bridge-domain 50
 !
!
interface BDI50
 description BD TO PE_MXR01 & PE_MXR03
 ip address 172.16.50.1 255.255.255.0
 load-interval 30
 no mop enabled
 no mop sysid
!



Verification Tasks

The list below outlines commands used to verify EVPN operation in a single-active configuration during normal operating conditions.
  • Show EVPN Instance Extensive
  • Show Route Advertising-Protocol BGP
  • Show Route Table EVPN


EVPN Instance

The outputs below displays the EVPN operational information from all PEs during normal conditions.  In single-active redundancy mode, DF and BDF election determines which PE is the primary and backup PE for the ES (as explained in RFC 7432 Sec. 14.1.1).

PE_MXR01 is the DF and is the primary PE.  The remote PE 113.113.113.113 (PE_MXR03) is the backup.  This PE will advertise its aliasing label of 306800 (via the Type 1 EAD per EVI route) to signal other PEs to only forward traffic to this PE for the ES.  

admin@PE_MXR01> show evpn instance extensive   
Instance: EVPN_CUSTOMER_G_ELAN_500
  Route Distinguisher: 111.111.111.111:50
  Per-instance MAC route label: 299776
  MAC database status                     Local  Remote
    MAC advertisements:                       1       1
    MAC+IP advertisements:                    0       0
    Default gateway MAC advertisements:       0       0
  Number of local interfaces: 1 (1 up)
    Interface name  ESI                            Mode             Status     AC-Role
    ge-0/0/2.500    00:11:22:33:44:55:66:77:88:99  single-active    Up         Root
  Number of IRB interfaces: 0 (0 up)
  Number of bridge domains: 2
    VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop
    500                  1    1                Extended         Enabled   307248          Disabled
    501                  1    1                Extended         Enabled   307264          Disabled
  Number of neighbors: 2
    Address               MAC    MAC+IP        AD        IM        ES Leaf-label
    112.112.112.112         1         0         0         2         0
    113.113.113.113         0         0         1         2         0
  Number of ethernet segments: 1
    ESI: 00:11:22:33:44:55:66:77:88:99
      Status: Resolved by IFL ge-0/0/2.500
      Local interface: ge-0/0/2.500, Status: Up/Forwarding
      Number of remote PEs connected: 1
        Remote PE        MAC label  Aliasing label  Mode
        113.113.113.113  304176     0               single-active
      Designated forwarder: 111.111.111.111
      Backup forwarder: 113.113.113.113
      Last designated forwarder update: Feb 01 19:06:20
      Advertised MAC label: 306800
      Advertised aliasing label: 306800
      Advertised split horizon label: 0

Instance: __default_evpn__
  Route Distinguisher: 111.111.111.111:0
  Number of bridge domains: 0
  Number of neighbors: 1
    Address               MAC    MAC+IP        AD        IM        ES Leaf-label
    113.113.113.113         0         0         0         0         1


PE_MXR02 shows its received the aliasing label from 111.111.111.111 but not from 113.113.113.113.  This would indicate that 111.111.111.111 is the primary PE and any traffic to the ES should be forwarded there. 

admin@PE_MXR02> show evpn instance extensive   
Instance: EVPN_CUSTOMER_G_ELAN_500
  Route Distinguisher: 112.112.112.112:50
  Per-instance MAC route label: 299776
  MAC database status                     Local  Remote
    MAC advertisements:                       1       2
    MAC+IP advertisements:                    0       0
    Default gateway MAC advertisements:       0       0
  Number of local interfaces: 1 (1 up)
    Interface name  ESI                            Mode             Status     AC-Role
    ge-0/0/2.500    00:00:00:00:00:00:00:00:00:00  single-homed     Up         Root
  Number of IRB interfaces: 0 (0 up)
  Number of bridge domains: 2
    VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop
    500                  1    1                Extended         Enabled   299840          Disabled
    501                  1    1                Extended         Enabled   299856          Disabled
  Number of neighbors: 2
    Address               MAC    MAC+IP        AD        IM        ES Leaf-label
    111.111.111.111         1         0         2         2         0
    113.113.113.113         1         0         1         2         0
  Number of ethernet segments: 1
    ESI: 00:11:22:33:44:55:66:77:88:99
      Status: Resolved by NH 1048576
      Number of remote PEs connected: 2
        Remote PE        MAC label  Aliasing label  Mode
        113.113.113.113  304176     0               single-active
        111.111.111.111  306800     306800          single-active

Instance: __default_evpn__
  Route Distinguisher: 112.112.112.112:0
  Number of bridge domains: 0
  Number of neighbors: 0


PE_MXR03 is the backup PE for the ES based on DF/BDF election and is in the blocking state.  Egress traffic from this PE to the ES will take the path towards 111.111.111.111 via its received aliasing label.

admin@PE_MXR03> show evpn instance extensive
Instance: EVPN_CUSTOMER_G_ELAN_500
  Route Distinguisher: 113.113.113.113:50
  Per-instance MAC route label: 299776
  MAC database status                     Local  Remote
    MAC advertisements:                       0       3
    MAC+IP advertisements:                    0       0
    Default gateway MAC advertisements:       0       0
  Number of local interfaces: 2 (2 up)
    Interface name  ESI                            Mode             Status     AC-Role
    ge-0/0/2.500    00:00:00:00:00:00:00:00:00:00  single-homed     Up         Root
    ge-0/0/3.500    00:11:22:33:44:55:66:77:88:99  single-active    Up         Root
  Number of IRB interfaces: 0 (0 up)
  Number of bridge domains: 2
    VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop
    500                  2    2                Extended         Enabled   300528          Disabled
    501                  2    2                Extended         Enabled   300544          Disabled
  Number of neighbors: 2
    Address               MAC    MAC+IP        AD        IM        ES Leaf-label
    111.111.111.111         1         0         2         2         0
    112.112.112.112         2         0         0         2         0
  Number of ethernet segments: 1
    ESI: 00:11:22:33:44:55:66:77:88:99
      Status: Resolved by NH 1048578
      Local interface: ge-0/0/3.500, Status: Up/Blocking
      Number of remote PEs connected: 1
        Remote PE        MAC label  Aliasing label  Mode
        111.111.111.111  306800     306800          single-active
      Designated forwarder: 111.111.111.111
      Backup forwarder: 113.113.113.113
      Last designated forwarder update: Feb 01 01:21:38
      Advertised MAC label: 304176
      Advertised aliasing label: 304176
      Advertised split horizon label: 0

Instance: __default_evpn__
  Route Distinguisher: 113.113.113.113:0
  Number of bridge domains: 0
  Number of neighbors: 1
    Address               MAC    MAC+IP        AD        IM        ES Leaf-label
    111.111.111.111         0         0         0         0         1


Route Advertising

With the redundancy mode changed to single-active, Type 1 EAD per EVI route export behave slightly differently than in an all-active configuration.  This route type is responsible for advertising the aliasing label used to reach a multi-homed ES.  Since one PE is forwarding at any given time, the designated backup PE will withhold its Type 1 EAD per EVI route advertisement.  This method is used to signal other PEs to only forward to the primary PE.  The Type 4 ES route exchange must still take place to elect the DF and BDF.

PE_MXR01, being the primary PE, will advertise its Type 1 EVI route with the aliasing label.  The Type 4 route is still advertised for DF/BDF purposes.

admin@PE_MXR01> show route advertising-protocol bgp 112.112.112.112 extensive   

EVPN_CUSTOMER_G_ELAN_500.evpn.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
* 1:111.111.111.111:50::112233445566778899::0/192 AD/EVI (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 111.111.111.111:50
     Route Label: 306800
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50

* 2:111.111.111.111:50::500::00:1e:e5:c8:0f:bf/304 MAC/IP (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 111.111.111.111:50
     Route Label: 306800
     ESI: 00:11:22:33:44:55:66:77:88:99
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50

* 3:111.111.111.111:50::500::111.111.111.111/248 IM (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 111.111.111.111:50
     Route Label: 307248
     PMSI: Flags 0x0: Label 307248: Type INGRESS-REPLICATION 111.111.111.111
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50
     PMSI: Flags 0x0: Label 307248: Type INGRESS-REPLICATION 111.111.111.111

* 3:111.111.111.111:50::501::111.111.111.111/248 IM (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 111.111.111.111:50
     Route Label: 307264
     PMSI: Flags 0x0: Label 307264: Type INGRESS-REPLICATION 111.111.111.111
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50
     PMSI: Flags 0x0: Label 307264: Type INGRESS-REPLICATION 111.111.111.111

__default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)

* 1:111.111.111.111:0::112233445566778899::FFFF:FFFF/192 AD/ESI (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 111.111.111.111:0
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50 esi-label:0x1:single-active (label 0)

* 4:111.111.111.111:0::112233445566778899:111.111.111.111/296 ES (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 111.111.111.111:0
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: es-import-target:11-22-33-44-55-66


admin@PE_MXR01> show route advertising-protocol bgp 113.113.113.113 extensive   

EVPN_CUSTOMER_G_ELAN_500.evpn.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
* 1:111.111.111.111:50::112233445566778899::0/192 AD/EVI (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 111.111.111.111:50
     Route Label: 306800
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50

* 2:111.111.111.111:50::500::00:1e:e5:c8:0f:bf/304 MAC/IP (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 111.111.111.111:50
     Route Label: 306800
     ESI: 00:11:22:33:44:55:66:77:88:99
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50

* 3:111.111.111.111:50::500::111.111.111.111/248 IM (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 111.111.111.111:50
     Route Label: 307248
     PMSI: Flags 0x0: Label 307248: Type INGRESS-REPLICATION 111.111.111.111
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50
     PMSI: Flags 0x0: Label 307248: Type INGRESS-REPLICATION 111.111.111.111

* 3:111.111.111.111:50::501::111.111.111.111/248 IM (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 111.111.111.111:50
     Route Label: 307264
     PMSI: Flags 0x0: Label 307264: Type INGRESS-REPLICATION 111.111.111.111
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50
     PMSI: Flags 0x0: Label 307264: Type INGRESS-REPLICATION 111.111.111.111

__default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)

* 1:111.111.111.111:0::112233445566778899::FFFF:FFFF/192 AD/ESI (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 111.111.111.111:0
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50 esi-label:0x1:single-active (label 0)

* 4:111.111.111.111:0::112233445566778899:111.111.111.111/296 ES (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 111.111.111.111:0
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: es-import-target:11-22-33-44-55-66


PE_MXR03, being the backup PE, will not advertise its Type 1 EVI and not seen here.  The Type 4 route is still advertised for DF/BDF purposes.

admin@PE_MXR03> show route advertising-protocol bgp 111.111.111.111 extensive   

EVPN_CUSTOMER_G_ELAN_500.evpn.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
* 3:113.113.113.113:50::500::113.113.113.113/248 IM (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 300528
     PMSI: Flags 0x0: Label 300528: Type INGRESS-REPLICATION 113.113.113.113
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50
     PMSI: Flags 0x0: Label 300528: Type INGRESS-REPLICATION 113.113.113.113

* 3:113.113.113.113:50::501::113.113.113.113/248 IM (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 300544
     PMSI: Flags 0x0: Label 300544: Type INGRESS-REPLICATION 113.113.113.113
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50
     PMSI: Flags 0x0: Label 300544: Type INGRESS-REPLICATION 113.113.113.113

__default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)

* 1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:0
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50 esi-label:0x1:single-active (label 0)

* 4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:0
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: es-import-target:11-22-33-44-55-66


admin@PE_MXR03> show route advertising-protocol bgp 112.112.112.112 extensive   

EVPN_CUSTOMER_G_ELAN_500.evpn.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
* 3:113.113.113.113:50::500::113.113.113.113/248 IM (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 300528
     PMSI: Flags 0x0: Label 300528: Type INGRESS-REPLICATION 113.113.113.113
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50
     PMSI: Flags 0x0: Label 300528: Type INGRESS-REPLICATION 113.113.113.113

* 3:113.113.113.113:50::501::113.113.113.113/248 IM (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 300544
     PMSI: Flags 0x0: Label 300544: Type INGRESS-REPLICATION 113.113.113.113
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50
     PMSI: Flags 0x0: Label 300544: Type INGRESS-REPLICATION 113.113.113.113

__default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)

* 1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:0
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50 esi-label:0x1:single-active (label 0)

* 4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:0
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: es-import-target:11-22-33-44-55-66



Route Table EVPN

The EVPN route table will show the full route table for all PEs during normal operation. 

PE_MXR01, being the primary PE for the ES, will not receive any Type 1 EVI routes from the other PEs.  The Type 4 route is still imported from 113.113.113.113 for DF/BDF purposes.

admin@PE_MXR01> show route table bgp.evpn.0

bgp.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI       
                   *[BGP/170] 00:19:10, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 330
2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP       
                   *[BGP/170] 00:17:37, localpref 100, from 112.112.112.112
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 328
3:112.112.112.112:50::500::112.112.112.112/248 IM           
                   *[BGP/170] 1d 00:52:39, localpref 100, from 112.112.112.112
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 328
3:112.112.112.112:50::501::112.112.112.112/248 IM           
                   *[BGP/170] 1d 00:52:39, localpref 100, from 112.112.112.112
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 328
3:113.113.113.113:50::500::113.113.113.113/248 IM           
                   *[BGP/170] 1d 00:52:39, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 330
3:113.113.113.113:50::501::113.113.113.113/248 IM           
                   *[BGP/170] 1d 00:52:39, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 330
4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES           
                   *[BGP/170] 00:19:10, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 330


PE_MXR02 imports the Type 1 EVI route from the primary PE (111.111.111.111).  Traffic to the ES egressing this PE will forward only to 111.111.111.111.  Since this PE contains no multi-homed hosts, no type 4 routes were imported.

admin@PE_MXR02> show route table bgp.evpn.0

bgp.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1:111.111.111.111:0::112233445566778899::FFFF:FFFF/192 AD/ESI       
                   *[BGP/170] 00:24:58, localpref 100, from 111.111.111.111
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 329
1:111.111.111.111:50::112233445566778899::0/192 AD/EVI       
                   *[BGP/170] 00:24:56, localpref 100, from 111.111.111.111
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 329
1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI       
                   *[BGP/170] 00:24:55, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 330
2:111.111.111.111:50::500::00:1e:e5:c8:0f:bf/304 MAC/IP       
                   *[BGP/170] 00:24:21, localpref 100, from 111.111.111.111
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 329
3:111.111.111.111:50::500::111.111.111.111/248 IM           
                   *[BGP/170] 1d 00:58:24, localpref 100, from 111.111.111.111
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 329
3:111.111.111.111:50::501::111.111.111.111/248 IM           
                   *[BGP/170] 1d 00:58:24, localpref 100, from 111.111.111.111
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 329
3:113.113.113.113:50::500::113.113.113.113/248 IM           
                   *[BGP/170] 1d 01:15:47, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 330
3:113.113.113.113:50::501::113.113.113.113/248 IM           
                   *[BGP/170] 1d 01:15:47, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 330


PE_MXR03, being the backup PE, receives the Type 1 EVI route from 111.111.111.111.  The Type 4 route is still imported from 111.111.111.111 for DF/BDF purposes.

admin@PE_MXR03> show route table bgp.evpn.0                           

bgp.evpn.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1:111.111.111.111:0::112233445566778899::FFFF:FFFF/192 AD/ESI       
                   *[BGP/170] 1d 04:27:03, localpref 100, from 111.111.111.111
                      AS path: I, validation-state: unverified
                    > to 10.1.1.97 via ge-0/0/1.48, Push 229
1:111.111.111.111:50::112233445566778899::0/192 AD/EVI       
                   *[BGP/170] 1d 04:27:01, localpref 100, from 111.111.111.111
                      AS path: I, validation-state: unverified
                    > to 10.1.1.97 via ge-0/0/1.48, Push 229
2:111.111.111.111:50::500::00:1e:e5:c8:0f:bf/304 MAC/IP       
                   *[BGP/170] 05:08:49, localpref 100, from 111.111.111.111
                      AS path: I, validation-state: unverified
                    > to 10.1.1.97 via ge-0/0/1.48, Push 229
2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP       
                   *[BGP/170] 05:08:49, localpref 100, from 112.112.112.112
                      AS path: I, validation-state: unverified
                    > to 10.1.1.97 via ge-0/0/1.48, Push 228
3:111.111.111.111:50::500::111.111.111.111/248 IM           
                   *[BGP/170] 1d 04:27:02, localpref 100, from 111.111.111.111
                      AS path: I, validation-state: unverified
                    > to 10.1.1.97 via ge-0/0/1.48, Push 229
3:111.111.111.111:50::501::111.111.111.111/248 IM           
                   *[BGP/170] 1d 04:27:02, localpref 100, from 111.111.111.111
                      AS path: I, validation-state: unverified
                    > to 10.1.1.97 via ge-0/0/1.48, Push 229
3:112.112.112.112:50::500::112.112.112.112/248 IM           
                   *[BGP/170] 2d 23:02:39, localpref 100, from 112.112.112.112
                      AS path: I, validation-state: unverified
                    > to 10.1.1.97 via ge-0/0/1.48, Push 228
3:112.112.112.112:50::501::112.112.112.112/248 IM           
                   *[BGP/170] 2d 23:02:39, localpref 100, from 112.112.112.112
                      AS path: I, validation-state: unverified
                    > to 10.1.1.97 via ge-0/0/1.48, Push 228
4:111.111.111.111:0::112233445566778899:111.111.111.111/296 ES           
                   *[BGP/170] 1d 04:27:04, localpref 100, from 111.111.111.111
                      AS path: I, validation-state: unverified
                    > to 10.1.1.97 via ge-0/0/1.48, Push 229



Verification Tasks (During Failure)

The CE’s AC interfaces towards PE_MXR01 were shutdown to simulate an ES failure.


Simulated Failure of an Ethernet Segment on PE_MXR01

CE_R27’s G1 interface was shutdown to simulate an Ethernet segment failure to PE_MXR01.

CE_R27#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
CE_R27(config)#int g1
CE_R27(config-if)#shut
CE_R27(config-if)#end

CE_R27#sh int desc
Interface                      Status         Protocol Description
Gi1                            admin down     down     TO PE_MXR01
Gi2                            up             up       TO PE_MXR03
Gi3                            admin down     down    
Gi4                            up             up       MANAGEMENT
BD50                           up             up       BD TO PE_MXR01 & PE_MXR03

PE_MXR01’s AC interface towards CE_R27’s G1 interface was shutdown to ensure a failure at both ends.

admin@PE_MXR01> configure
Entering configuration mode

[edit]
admin@PE_MXR01# set interfaces ge-0/0/2 disable

[edit]
admin@PE_MXR01# commit
commit complete

admin@PE_MXR01# run show interfaces terse ge-0/0/2 
Interface               Admin Link Proto    Local                 Remote
ge-0/0/2                down  down
ge-0/0/2.500            up    down bridge 
ge-0/0/2.32767          up    down multiservice


The list below outlines commands used to verify EVPN operation in a single-active configuration during a failure condition.
  • Show EVPN Instance Extensive
  • Show Route Advertising-Protocol BGP
  • Show Route Table EVPN

EVPN Instance (During Failure)

The outputs below will display the EVPN operational information from all PEs during an ES failure.

During PE_MXR01’s local ES failure, the DF switches to PE_MXR03.  The redundant ES is now reachable via the backup PE.

admin@PE_MXR01> show evpn instance extensive
Instance: EVPN_CUSTOMER_G_ELAN_500
  Route Distinguisher: 111.111.111.111:50
  Per-instance MAC route label: 299776
  MAC database status                     Local  Remote
    MAC advertisements:                       0       3
    MAC+IP advertisements:                    0       0
    Default gateway MAC advertisements:       0       0
  Number of local interfaces: 1 (0 up)
    Interface name  ESI                            Mode             Status     AC-Role
    ge-0/0/2.500    00:11:22:33:44:55:66:77:88:99  single-active    Down       Root
  Number of IRB interfaces: 0 (0 up)
  Number of bridge domains: 2
    VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop
    500                  1    0                Extended         Enabled                   Disabled
    501                  1    0                Extended         Enabled                   Disabled
  Number of neighbors: 2
    Address               MAC    MAC+IP        AD        IM        ES Leaf-label
    112.112.112.112         2         0         0         2         0
    113.113.113.113         1         0         2         2         0
  Number of ethernet segments: 1
    ESI: 00:11:22:33:44:55:66:77:88:99
      Status: Resolved by NH 1048578
      Local interface: ge-0/0/2.500, Status: Down
      Number of remote PEs connected: 1
        Remote PE        MAC label  Aliasing label  Mode
        113.113.113.113  304176     304176          single-active
      Designated forwarder: 113.113.113.113
      Last designated forwarder update: Feb 01 01:54:29
      Advertised MAC label: 306800
      Advertised aliasing label: 306800
      Advertised split horizon label: 0

Instance: __default_evpn__
  Route Distinguisher: 111.111.111.111:0
  Number of bridge domains: 0
  Number of neighbors: 1
    Address               MAC    MAC+IP        AD        IM        ES Leaf-label
    113.113.113.113         0         0         0         0         1


From PE_MXR02, the ES is now reachable via the backup PE 113.113.113.113 (PE_MXR03).

admin@PE_MXR02> show evpn instance extensive
Instance: EVPN_CUSTOMER_G_ELAN_500
  Route Distinguisher: 112.112.112.112:50
  Per-instance MAC route label: 299776
  MAC database status                     Local  Remote
    MAC advertisements:                       1       1
    MAC+IP advertisements:                    0       0
    Default gateway MAC advertisements:       0       0
  Number of local interfaces: 1 (1 up)
    Interface name  ESI                            Mode             Status     AC-Role
    ge-0/0/2.500    00:00:00:00:00:00:00:00:00:00  single-homed     Up         Root
  Number of IRB interfaces: 0 (0 up)
  Number of bridge domains: 2
    VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop
    500                  1    1                Extended         Enabled   299840          Disabled
    501                  1    1                Extended         Enabled   299856          Disabled
  Number of neighbors: 1
    Address               MAC    MAC+IP        AD        IM        ES Leaf-label
    113.113.113.113         1         0         2         2         0
  Number of ethernet segments: 1
    ESI: 00:11:22:33:44:55:66:77:88:99
      Status: Resolved by NH 1048576
      Number of remote PEs connected: 1
        Remote PE        MAC label  Aliasing label  Mode
        113.113.113.113  304176     304176          single-active

Instance: __default_evpn__
  Route Distinguisher: 112.112.112.112:0
  Number of bridge domains: 0
  Number of neighbors: 0


This PE is now the forwarding PE and the ES is reachable via its local interface.

admin@PE_MXR03> show evpn instance extensive
Instance: EVPN_CUSTOMER_G_ELAN_500
  Route Distinguisher: 113.113.113.113:50
  Per-instance MAC route label: 299776
  MAC database status                     Local  Remote
    MAC advertisements:                       1       1
    MAC+IP advertisements:                    0       0
    Default gateway MAC advertisements:       0       0
  Number of local interfaces: 2 (2 up)
    Interface name  ESI                            Mode             Status     AC-Role
    ge-0/0/2.500    00:00:00:00:00:00:00:00:00:00  single-homed     Up         Root
    ge-0/0/3.500    00:11:22:33:44:55:66:77:88:99  single-active    Up         Root
  Number of IRB interfaces: 0 (0 up)
  Number of bridge domains: 2
    VLAN  Domain ID   Intfs / up    IRB intf   Mode             MAC sync  IM route label  SG sync  IM core nexthop
    500                  2    2                Extended         Enabled   300528          Disabled
    501                  2    2                Extended         Enabled   300544          Disabled
  Number of neighbors: 1
    Address               MAC    MAC+IP        AD        IM        ES Leaf-label
    112.112.112.112         1         0         0         2         0
  Number of ethernet segments: 1
    ESI: 00:11:22:33:44:55:66:77:88:99
      Status: Resolved by IFL ge-0/0/3.500
      Local interface: ge-0/0/3.500, Status: Up/Forwarding
      Designated forwarder: 113.113.113.113
      Last designated forwarder update: Feb 01 01:54:29
      Advertised MAC label: 304176
      Advertised aliasing label: 304176
      Advertised split horizon label: 0

Instance: __default_evpn__
  Route Distinguisher: 113.113.113.113:0
  Number of bridge domains: 0
  Number of neighbors: 0



Route Advertising

With PE_MXR01’s ES down, it withdraws all its routes to signal other PEs of the failure.

admin@PE_MXR01> show route advertising-protocol bgp 112.112.112.112   

admin@PE_MXR01>

admin@PE_MXR01> show route advertising-protocol bgp 113.113.113.113   

admin@PE_MXR01>


PE_MXR03 now advertises its Type 1 EAD per EVI route (with aliasing label) to advertise its reachability to this ES.

admin@PE_MXR03> show route advertising-protocol bgp 111.111.111.111 extensive

EVPN_CUSTOMER_G_ELAN_500.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
* 1:113.113.113.113:50::112233445566778899::0/192 AD/EVI (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 304176
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50

* 2:113.113.113.113:50::500::00:0c:29:e4:3e:cd/304 MAC/IP (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 299776
     ESI: 00:00:00:00:00:00:00:00:00:00
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50

* 2:113.113.113.113:50::500::00:1e:e5:c8:0f:bf/304 MAC/IP (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 304176
     ESI: 00:11:22:33:44:55:66:77:88:99
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50

* 3:113.113.113.113:50::500::113.113.113.113/248 IM (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 300528
     PMSI: Flags 0x0: Label 300528: Type INGRESS-REPLICATION 113.113.113.113
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50
     PMSI: Flags 0x0: Label 300528: Type INGRESS-REPLICATION 113.113.113.113

* 3:113.113.113.113:50::501::113.113.113.113/248 IM (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 300544
     PMSI: Flags 0x0: Label 300544: Type INGRESS-REPLICATION 113.113.113.113
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50
     PMSI: Flags 0x0: Label 300544: Type INGRESS-REPLICATION 113.113.113.113

__default_evpn__.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)

* 1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:0
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50 esi-label:0x1:single-active (label 0)

* 4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:0
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: es-import-target:11-22-33-44-55-66


admin@PE_MXR03> show route advertising-protocol bgp 112.112.112.112 extensive   

EVPN_CUSTOMER_G_ELAN_500.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
* 1:113.113.113.113:50::112233445566778899::0/192 AD/EVI (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 304176
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50

* 2:113.113.113.113:50::500::00:0c:29:e4:3e:cd/304 MAC/IP (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 299776
     ESI: 00:00:00:00:00:00:00:00:00:00
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50

* 2:113.113.113.113:50::500::00:1e:e5:c8:0f:bf/304 MAC/IP (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 304176
     ESI: 00:11:22:33:44:55:66:77:88:99
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50

* 3:113.113.113.113:50::500::113.113.113.113/248 IM (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 300528
     PMSI: Flags 0x0: Label 300528: Type INGRESS-REPLICATION 113.113.113.113
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50
     PMSI: Flags 0x0: Label 300528: Type INGRESS-REPLICATION 113.113.113.113

* 3:113.113.113.113:50::501::113.113.113.113/248 IM (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:50
     Route Label: 300544
     PMSI: Flags 0x0: Label 300544: Type INGRESS-REPLICATION 113.113.113.113
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50
     PMSI: Flags 0x0: Label 300544: Type INGRESS-REPLICATION 113.113.113.113

__default_evpn__.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)

* 1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:0
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: target:2345:50 esi-label:0x1:single-active (label 0)

* 4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES (1 entry, 1 announced)
 BGP group PE type Internal
     Route Distinguisher: 113.113.113.113:0
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [2345] I
     Communities: es-import-target:11-22-33-44-55-66


Route Table EVPN (During Failure)

The outputs below will observe the route exchange from all PEs during an ES failure.

With PE_MXR01’s ES down, it imports the Type 1 EAD per EVI (aliasing) route from the backup PE 113.113.113.113 (PE_MXR03).  Traffic egressing this PE will forward to 113.113.113.113.

admin@PE_MXR01> show route table bgp.evpn.0

bgp.evpn.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI       
                   *[BGP/170] 00:44:23, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 330
1:113.113.113.113:50::112233445566778899::0/192 AD/EVI       
                   *[BGP/170] 00:11:27, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 330
2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP       
                   *[BGP/170] 00:42:50, localpref 100, from 112.112.112.112
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 328
2:113.113.113.113:50::500::00:1e:e5:c8:0f:bf/304 MAC/IP        
                   *[BGP/170] 00:11:24, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 330
3:112.112.112.112:50::500::112.112.112.112/248 IM            
                   *[BGP/170] 1d 01:17:52, localpref 100, from 112.112.112.112
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 328
3:112.112.112.112:50::501::112.112.112.112/248 IM           
                   *[BGP/170] 1d 01:17:52, localpref 100, from 112.112.112.112
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 328
3:113.113.113.113:50::500::113.113.113.113/248 IM           
                   *[BGP/170] 1d 01:17:52, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 330
3:113.113.113.113:50::501::113.113.113.113/248 IM           
                   *[BGP/170] 1d 01:17:52, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 330
4:113.113.113.113:0::112233445566778899:113.113.113.113/296 ES           
                   *[BGP/170] 00:44:23, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 330


PE_MXR02 also receives the Type 1 EAD per EVI route from the backup PE 113.113.113.113 (PE_MXR03) and will now have an aliasing label to forward traffic to it.

admin@PE_MXR02> show route table bgp.evpn.0

bgp.evpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1:113.113.113.113:0::112233445566778899::FFFF:FFFF/192 AD/ESI       
                   *[BGP/170] 00:47:31, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 330
1:113.113.113.113:50::112233445566778899::0/192 AD/EVI       
                   *[BGP/170] 00:14:36, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 330
2:113.113.113.113:50::500::00:1e:e5:c8:0f:bf/304 MAC/IP       
                   *[BGP/170] 00:14:33, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 330
3:113.113.113.113:50::500::113.113.113.113/248 IM           
                   *[BGP/170] 1d 01:38:23, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 330
3:113.113.113.113:50::501::113.113.113.113/248 IM           
                   *[BGP/170] 1d 01:38:23, localpref 100, from 113.113.113.113
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 330


With PE_MXR01’s ES down, no routes are received from 111.111.111.111.

admin@PE_MXR03> show route table bgp.evpn.0   

bgp.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP       
                   *[BGP/170] 00:50:19, localpref 100, from 112.112.112.112
                      AS path: I, validation-state: unverified
                    > to 10.1.1.97 via ge-0/0/1.48, Push 228
3:112.112.112.112:50::500::112.112.112.112/248 IM           
                   *[BGP/170] 1d 01:42:43, localpref 100, from 112.112.112.112
                      AS path: I, validation-state: unverified
                    > to 10.1.1.97 via ge-0/0/1.48, Push 228
3:112.112.112.112:50::501::112.112.112.112/248 IM           
                   *[BGP/170] 1d 01:42:43, localpref 100, from 112.112.112.112
                      AS path: I, validation-state: unverified
                    > to 10.1.1.97 via ge-0/0/1.48, Push 22



References

EVPN Single-Active Multi-Homing



Other Blogs and Labs