EVPN Single-Active Multi-Homing – Load Balancing Traffic Analysis

Here are my traffic analysis notes for the Single-Active lab.  After doing a deep dive on the All-Active traffic, I felt this final step was necessary to get a detailed understanding of both multi-homing redundancy modes.  The test plan was identical to the All-Active traffic analysis and the Wireshark traces were performed on the same backbone links.

Test Plan

• Simulate traffic flow from CE_R27 to CE_R29
• Observe interface statistics to determine traffic routing
• Observe BGP routing and label exchange
• Perform targeted packet captures on PE links
• Examine packet captures for label usage

Test Traffic

Continuous ping from host CE_R27 to CE_R29.

CE_R27#ping 172.16.50.2 rep 2147483647 Type escape sequence to abort. Sending 2147483647, 100-byte ICMP Echos to 172.16.50.2, timeout is 2 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! .. snip ..

Traffic Observation

CE_R27’s Traffic Distribution

CE_R27’s G1 and G2 Bridge-Domain interface.  The input/output rate were equal and normal.

CE_R27#sh int bdi50 | in rate   Queueing strategy: fifo   30 second input rate 38000 bits/sec, 42 packets/sec   30 second output rate 38000 bits/sec, 42 packets/sec

CE_R27’s G1 Ethernet interface to PE_MXR01.  The primary forwarding interface’s input/output rates were in-line with the BDI interface.

CE_R27#sh int g1 | in rate     Queueing strategy: fifo   30 second input rate 38000 bits/sec, 42 packets/sec   30 second output rate 38000 bits/sec, 42 packets/sec

CE_R27’s G2 Ethernet interface to PE_MXR03.  The standby interface’s has no traffic.

CE_R27#sh int g2 | in rate   Queueing strategy: fifo   30 second input rate 0 bits/sec, 0 packets/sec   30 second output rate 0 bits/sec, 0 packets/sec

PE_MXR01's Traffic Distribution

PE_MXR01’s AC interface to CE_R27 (Gig1).  The active AC interface’s input/output rates were in-line with the CE.

admin@PE_MXR01> show interfaces ge-0/0/2.500 detail |grep pps         Input  packets:                19049                   42 pps      Output packets:                19046                   42 pps

PE_MXR01 MPLS interface to P_R03.  The link towards the MPLS core saw all traffic to P3.

admin@PE_MXR01> show interfaces ge-0/0/1.44 detail |grep pps          Input  packets:              6172520                   42 pps      Output packets:              7677321                   42 pps

PE_MXR01 MPLS interface to P_R01.  The link towards the MPLS core saw no traffic to P1.

admin@PE_MXR01> show interfaces ge-0/0/1.45 detail |grep pps         Input  packets:                    3                    0 pps      Output packets:                    1                    0 pps

PE_MXR02's Traffic Distribution

PE_MXR02’s AC interface to CE_R28.  The destination AC interface’s input/output rates were in-line with the CE.

admin@PE_MXR02> show interfaces ge-0/0/2.500 detail |grep pps                                                                          Input  packets:              6473475                   42 pps      Output packets:              6914532                   42 pps

PE_MXR02’s MPLS interface to P_R03.  The link towards the MPLS core saw all traffic to P3.

admin@PE_MXR02> show interfaces ge-0/0/1.46 detail |grep pps         Input  packets:              6914474                   42 pps      Output packets:             12623504                   42 pps

PE_MXR02’s MPLS interface to P_R04.  The link towards the MPLS core saw no traffic to P4.

admin@PE_MXR02> show interfaces ge-0/0/1.47 detail |grep pps         Input  packets:                    2                    0 pps      Output packets:                    0                    0 pps

PE_MXR03's Traffic Distribution

PE_MXR03’s AC interface to CE_R27 (Gig2).  The redundant AC interface, configured in standby mode, saw no traffic.

admin@PE_MXR03> show interfaces ge-0/0/3.500 detail |grep pps       Input  packets:                    0                    0 pps      Output packets:                    0                    0 pps

PE_MXR03’s MPLS interface to P_R02.  No traffic was forwarded to this PE under normal operating conditions.

admin@PE_MXR03> show interfaces ge-0/0/1.48 detail |grep pps          Input  packets:              7514322                    0 pps      Output packets:               299204                    0 pps

PE_MXR03’s MPLS interface to P_R01.  No traffic was forwarded to this PE under normal operating conditions.

admin@PE_MXR03> show interfaces ge-0/0/1.49 detail |grep pps         Input  packets:                    4                    0 pps      Output packets:                   27                    0 pps

Wireshark Traffic Analysis

Capture 1 – Link between PE_MXR01 and P_R03

Capture 1, Frame 2 was the ICMP request from CE_R27 to CE_R28 as seen between PE_MXR01 and P_R03.  This request was sent to CE_R28 with destination MAC address 00:0c:29:49:aa:8c.  

Based on the Type 2 MAC route lookup, the zero ESI indicated that the destination host was single-homed and a Type 1 aliasing route lookup wasn’t necessary.  As seen from the Wireshark capture, its label usage (bottom label 299776, top label 328) was consistent with the route lookup.

admin@PE_MXR01> show route table bgp.evpn.0 extensive               bgp.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden) .. snip .. 2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP (1 entry, 0 announced)         *BGP    Preference: 170/-101                 Route Distinguisher: 112.112.112.112:50                 Next hop type: Indirect, Next hop index: 0                 Address: 0xb7a3a90                 Next-hop reference count: 8                 Source: 112.112.112.112                 Protocol next hop: 112.112.112.112                 Indirect next hop: 0x2 no-forward INH Session ID: 0x0                 State: <Active Int Ext>                 Local AS:  2345 Peer AS:  2345                 Age: 17:57      Metric2: 1                 Validation State: unverified                 Task: BGP_2345.112.112.112.112                 AS path: I                 Communities: target:2345:50                 Import Accepted                 Route Label: 299776                 ESI: 00:00:00:00:00:00:00:00:00:00                 Localpref: 100                 Router ID: 112.112.112.112                 Secondary Tables: EVPN_CUSTOMER_G_ELAN_500.evpn.0                 Indirect next hops: 1                         Protocol next hop: 112.112.112.112 Metric: 1                         Indirect next hop: 0x2 no-forward INH Session ID: 0x0                         Indirect path forwarding next hops: 1                                 Next hop type: Router                                 Next hop: 10.1.1.81 via ge-0/0/1.44                                 Session Id: 0x0                         112.112.112.112/32 Originating RIB: inet.3                           Metric: 1                       Node path count: 1                           Forwarding nexthops: 1                                 Nexthop: 10.1.1.81 via ge-0/0/1.44

admin@PE_MXR01> show route table bgp.evpn.0 bgp.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both .. snip .. 2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP                           *[BGP/170] 00:17:37, localpref 100, from 112.112.112.112                       AS path: I, validation-state: unverified                     > to 10.1.1.81 via ge-0/0/1.44, Push 328

Capture 2 – Link between P_R03 and PE_MXR02

Capture 2, Frame 1 was the ICMP request from CE_R27 to CE_R28 as seen from P_R03 and PE_MXR02.  This request was sent to CE_R28's MAC address of 00:0c:29:49:aa:8c.  

As P_R03 forwarded the packet to PE_MXR02, it popped the top label of 328.  The VPN label of 299776 was looked up, associated to the EVPN instance (via Ingress-MAC since the Type 2 route’s label was used) and packet delivered to CE_R28.  At this point everything looked normal.

admin@PE_MXR02> show route table mpls.0 label 299776 mpls.0: 52 destinations, 52 routes (52 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 299776             *[EVPN/7] 00:24:26, routing-instance EVPN_CUSTOMER_G_ELAN_500, route-type Ingress-MAC, vlan-id 500                       to table EVPN_CUSTOMER_G_ELAN_500.evpn-mac.0

Capture 2, Frame 2 was the ICMP reply from CE_R28 to CE_R27 as seen from PE_MXR02 to P_R03.  According to Wireshark’s analysis (arrows), this was Frame 1’s corresponding reply.  

PE_MXR02 initially performed a Type 2 MAC route lookup for the destination MAC address and discovered this host was multi-homed (from non-zero ESI).  A Type 1 EAD per EVI route lookup was subsequently performed and its aliasing label was then used to construct the label stack of 306800 and 329.

In single-active mode, a Type 1 EAD per EVI route was only advertised from the primary forwarding PE.  In this case, this route was only received from 111.111.111.111 (PE_MXR01), therefore PE_MXR02 would forward any traffic to 111.111.111.111.

admin@PE_MXR02> show route table bgp.evpn.0 extensive    bgp.evpn.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden) .. snip .. 1:111.111.111.111:50::112233445566778899::0/192 AD/EVI (1 entry, 0 announced)         *BGP    Preference: 170/-101                 Route Distinguisher: 111.111.111.111:50                 Next hop type: Indirect, Next hop index: 0                 Address: 0xb7a5950                 Next-hop reference count: 10                 Source: 111.111.111.111                 Protocol next hop: 111.111.111.111                 Indirect next hop: 0x2 no-forward INH Session ID: 0x0                 State: <Active Int Ext>                 Local AS:  2345 Peer AS:  2345                 Age: 25:22      Metric2: 1                 Validation State: unverified                 Task: BGP_2345.111.111.111.111                 AS path: I                 Communities: target:2345:50                 Import Accepted                 Route Label: 306800                 Localpref: 100                 Router ID: 111.111.111.111                 Secondary Tables: EVPN_CUSTOMER_G_ELAN_500.evpn.0                 Indirect next hops: 1                         Protocol next hop: 111.111.111.111 Metric: 1                         Indirect next hop: 0x2 no-forward INH Session ID: 0x0                         Indirect path forwarding next hops: 1                                 Next hop type: Router                                 Next hop: 10.1.1.89 via ge-0/0/1.46                                 Session Id: 0x0                         111.111.111.111/32 Originating RIB: inet.3                           Metric: 1                       Node path count: 1                           Forwarding nexthops: 1                                 Nexthop: 10.1.1.89 via ge-0/0/1.46 2:111.111.111.111:50::500::00:1e:e5:c8:0f:bf/304 MAC/IP (1 entry, 0 announced)         *BGP    Preference: 170/-101                 Route Distinguisher: 111.111.111.111:50                 Next hop type: Indirect, Next hop index: 0                 Address: 0xb7a5950                 Next-hop reference count: 10                 Source: 111.111.111.111                 Protocol next hop: 111.111.111.111                 Indirect next hop: 0x2 no-forward INH Session ID: 0x0                 State: <Active Int Ext>                 Local AS:  2345 Peer AS:  2345                 Age: 24:47      Metric2: 1                 Validation State: unverified                 Task: BGP_2345.111.111.111.111                 AS path: I                 Communities: target:2345:50                 Import Accepted                 Route Label: 306800                 ESI: 00:11:22:33:44:55:66:77:88:99                 Localpref: 100                 Router ID: 111.111.111.111                 Secondary Tables: EVPN_CUSTOMER_G_ELAN_500.evpn.0                 Indirect next hops: 1                         Protocol next hop: 111.111.111.111 Metric: 1                         Indirect next hop: 0x2 no-forward INH Session ID: 0x0                         Indirect path forwarding next hops: 1                                 Next hop type: Router                                 Next hop: 10.1.1.89 via ge-0/0/1.46                                 Session Id: 0x0                         111.111.111.111/32 Originating RIB: inet.3                           Metric: 1                       Node path count: 1                           Forwarding nexthops: 1                                 Nexthop: 10.1.1.89 via ge-0/0/1.46

admin@PE_MXR02> show route table bgp.evpn.0 bgp.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both .. snip .. 1:111.111.111.111:50::112233445566778899::0/192 AD/EVI                           *[BGP/170] 00:24:56, localpref 100, from 111.111.111.111                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 329 2:111.111.111.111:50::500::00:1e:e5:c8:0f:bf/304 MAC/IP                           *[BGP/170] 00:24:21, localpref 100, from 111.111.111.111                       AS path: I, validation-state: unverified                     > to 10.1.1.89 via ge-0/0/1.46, Push 329

Capture 1 – Link between P_R03 and PE_MXR01

Capture 1, Frame 4 was the ICMP reply from CE_R28 to CE_R27 as seen from P_R03 and PE_MXR01.  According to Wireshark’s analysis (arrows), this was Frame 2’s corresponding reply.  

As P_R03 forwarded the packet to PE_MXR01, it popped the top label of 329.  The VPN label of 306800 was looked up, associated to the EVPN instance (via Ingress-Aliasing since Type 1 route’s label was used) and packet delivered to CE_R27.  Again, traffic forwarding in this direction looked normal.

admin@PE_MXR01> show route table mpls.0 label 306800 mpls.0: 44 destinations, 44 routes (44 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 306800             *[EVPN/7] 00:19:49, routing-instance EVPN_CUSTOMER_G_ELAN_500, route-type Ingress-Aliasing                       to table EVPN_CUSTOMER_G_ELAN_500.evpn-mac.0                     [EVPN/7] 00:19:15, routing-instance EVPN_CUSTOMER_G_ELAN_500, route-type Ingress-MAC, vlan-id 500                       to table EVPN_CUSTOMER_G_ELAN_500.evpn-mac.0

Conclusion

From my observation and analysis in this lab, traffic forwarding in single-active redundancy mode operated as expected per my understanding of EVPN multi-homing.  I didn’t experience any odd behaviors in regards to routing exchange and label usage.