Sunday, April 8, 2018

EVPN Single-Active Multi-Homing – Load Balancing Traffic Analysis

Here are my traffic analysis notes for the Single-Active lab.  After doing a deep dive on the All-Active traffic, I felt this final step was necessary to get a detailed understanding of both multi-homing redundancy modes.  The test plan was identical to the All-Active traffic analysis and the Wireshark traces were performed on the same backbone links.


Test Plan
  • Simulate traffic flow from CE_R27 to CE_R29
  • Observe interface statistics to determine traffic routing
  • Observe BGP routing and label exchange
  • Perform targeted packet captures on PE links
  • Examine packet captures for label usage



Test Traffic

Continuous ping from host CE_R27 to CE_R29.

CE_R27#ping 172.16.50.2 rep 2147483647
Type escape sequence to abort.
Sending 2147483647, 100-byte ICMP Echos to 172.16.50.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
.. snip ..



Traffic Observation

CE_R27’s Traffic Distribution

CE_R27’s G1 and G2 Bridge-Domain interface.  The input/output rate were equal and normal.

CE_R27#sh int bdi50 | in rate
  Queueing strategy: fifo
  30 second input rate 38000 bits/sec, 42 packets/sec
  30 second output rate 38000 bits/sec, 42 packets/sec

CE_R27’s G1 Ethernet interface to PE_MXR01.  The primary forwarding interface’s input/output rates were in-line with the BDI interface.

CE_R27#sh int g1 | in rate  
  Queueing strategy: fifo
  30 second input rate 38000 bits/sec, 42 packets/sec
  30 second output rate 38000 bits/sec, 42 packets/sec

CE_R27’s G2 Ethernet interface to PE_MXR03.  The standby interface’s has no traffic.

CE_R27#sh int g2 | in rate
  Queueing strategy: fifo
  30 second input rate 0 bits/sec, 0 packets/sec
  30 second output rate 0 bits/sec, 0 packets/sec



PE_MXR01's Traffic Distribution

PE_MXR01’s AC interface to CE_R27 (Gig1).  The active AC interface’s input/output rates were in-line with the CE.

admin@PE_MXR01> show interfaces ge-0/0/2.500 detail |grep pps   
     Input  packets:                19049                   42 pps
     Output packets:                19046                   42 pps

PE_MXR01 MPLS interface to P_R03.  The link towards the MPLS core saw all traffic to P3.

admin@PE_MXR01> show interfaces ge-0/0/1.44 detail |grep pps    
     Input  packets:              6172520                   42 pps
     Output packets:              7677321                   42 pps

PE_MXR01 MPLS interface to P_R01.  The link towards the MPLS core saw no traffic to P1.

admin@PE_MXR01> show interfaces ge-0/0/1.45 detail |grep pps   
     Input  packets:                    3                    0 pps
     Output packets:                    1                    0 pps



PE_MXR02's Traffic Distribution

PE_MXR02’s AC interface to CE_R28.  The destination AC interface’s input/output rates were in-line with the CE.

admin@PE_MXR02> show interfaces ge-0/0/2.500 detail |grep pps                                                                    
     Input  packets:              6473475                   42 pps
     Output packets:              6914532                   42 pps

PE_MXR02’s MPLS interface to P_R03.  The link towards the MPLS core saw all traffic to P3.

admin@PE_MXR02> show interfaces ge-0/0/1.46 detail |grep pps   
     Input  packets:              6914474                   42 pps
     Output packets:             12623504                   42 pps

PE_MXR02’s MPLS interface to P_R04.  The link towards the MPLS core saw no traffic to P4.

admin@PE_MXR02> show interfaces ge-0/0/1.47 detail |grep pps   
     Input  packets:                    2                    0 pps
     Output packets:                    0                    0 pps



PE_MXR03's Traffic Distribution

PE_MXR03’s AC interface to CE_R27 (Gig2).  The redundant AC interface, configured in standby mode, saw no traffic.

admin@PE_MXR03> show interfaces ge-0/0/3.500 detail |grep pps 
     Input  packets:                    0                    0 pps
     Output packets:                    0                    0 pps

PE_MXR03’s MPLS interface to P_R02.  No traffic was forwarded to this PE under normal operating conditions.

admin@PE_MXR03> show interfaces ge-0/0/1.48 detail |grep pps    
     Input  packets:              7514322                    0 pps
     Output packets:               299204                    0 pps

PE_MXR03’s MPLS interface to P_R01.  No traffic was forwarded to this PE under normal operating conditions.

admin@PE_MXR03> show interfaces ge-0/0/1.49 detail |grep pps   
     Input  packets:                    4                    0 pps
     Output packets:                   27                    0 pps



Wireshark Traffic Analysis

Capture 1 – Link between PE_MXR01 and P_R03

Capture 1, Frame 2 was the ICMP request from CE_R27 to CE_R28 as seen between PE_MXR01 and P_R03.  This request was sent to CE_R28 with destination MAC address 00:0c:29:49:aa:8c.  




Based on the Type 2 MAC route lookup, the zero ESI indicated that the destination host was single-homed and a Type 1 aliasing route lookup wasn’t necessary.  As seen from the Wireshark capture, its label usage (bottom label 299776, top label 328) was consistent with the route lookup.

admin@PE_MXR01> show route table bgp.evpn.0 extensive              

bgp.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)

.. snip ..

2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 112.112.112.112:50
                Next hop type: Indirect, Next hop index: 0
                Address: 0xb7a3a90
                Next-hop reference count: 8
                Source: 112.112.112.112
                Protocol next hop: 112.112.112.112
                Indirect next hop: 0x2 no-forward INH Session ID: 0x0
                State: <Active Int Ext>
                Local AS:  2345 Peer AS:  2345
                Age: 17:57      Metric2: 1
                Validation State: unverified
                Task: BGP_2345.112.112.112.112
                AS path: I
                Communities: target:2345:50
                Import Accepted
                Route Label: 299776
                ESI: 00:00:00:00:00:00:00:00:00:00
                Localpref: 100
                Router ID: 112.112.112.112
                Secondary Tables: EVPN_CUSTOMER_G_ELAN_500.evpn.0
                Indirect next hops: 1
                        Protocol next hop: 112.112.112.112 Metric: 1
                        Indirect next hop: 0x2 no-forward INH Session ID: 0x0
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 10.1.1.81 via ge-0/0/1.44
                                Session Id: 0x0
                        112.112.112.112/32 Originating RIB: inet.3
                          Metric: 1                       Node path count: 1
                          Forwarding nexthops: 1
                                Nexthop: 10.1.1.81 via ge-0/0/1.44


admin@PE_MXR01> show route table bgp.evpn.0

bgp.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

.. snip ..

2:112.112.112.112:50::500::00:0c:29:49:aa:8c/304 MAC/IP       
                   *[BGP/170] 00:17:37, localpref 100, from 112.112.112.112
                      AS path: I, validation-state: unverified
                    > to 10.1.1.81 via ge-0/0/1.44, Push 328



Capture 2 – Link between P_R03 and PE_MXR02

Capture 2, Frame 1 was the ICMP request from CE_R27 to CE_R28 as seen from P_R03 and PE_MXR02.  This request was sent to CE_R28's MAC address of 00:0c:29:49:aa:8c.  




As P_R03 forwarded the packet to PE_MXR02, it popped the top label of 328.  The VPN label of 299776 was looked up, associated to the EVPN instance (via Ingress-MAC since the Type 2 route’s label was used) and packet delivered to CE_R28.  At this point everything looked normal.

admin@PE_MXR02> show route table mpls.0 label 299776

mpls.0: 52 destinations, 52 routes (52 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299776             *[EVPN/7] 00:24:26, routing-instance EVPN_CUSTOMER_G_ELAN_500, route-type Ingress-MAC, vlan-id 500
                      to table EVPN_CUSTOMER_G_ELAN_500.evpn-mac.0



Capture 2, Frame 2 was the ICMP reply from CE_R28 to CE_R27 as seen from PE_MXR02 to P_R03.  According to Wireshark’s analysis (arrows), this was Frame 1’s corresponding reply.  




PE_MXR02 initially performed a Type 2 MAC route lookup for the destination MAC address and discovered this host was multi-homed (from non-zero ESI).  A Type 1 EAD per EVI route lookup was subsequently performed and its aliasing label was then used to construct the label stack of 306800 and 329.

In single-active mode, a Type 1 EAD per EVI route was only advertised from the primary forwarding PE.  In this case, this route was only received from 111.111.111.111 (PE_MXR01), therefore PE_MXR02 would forward any traffic to 111.111.111.111.

admin@PE_MXR02> show route table bgp.evpn.0 extensive   

bgp.evpn.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)

.. snip ..

1:111.111.111.111:50::112233445566778899::0/192 AD/EVI (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 111.111.111.111:50
                Next hop type: Indirect, Next hop index: 0
                Address: 0xb7a5950
                Next-hop reference count: 10
                Source: 111.111.111.111
                Protocol next hop: 111.111.111.111
                Indirect next hop: 0x2 no-forward INH Session ID: 0x0
                State: <Active Int Ext>
                Local AS:  2345 Peer AS:  2345
                Age: 25:22      Metric2: 1
                Validation State: unverified
                Task: BGP_2345.111.111.111.111
                AS path: I
                Communities: target:2345:50
                Import Accepted
                Route Label: 306800
                Localpref: 100
                Router ID: 111.111.111.111
                Secondary Tables: EVPN_CUSTOMER_G_ELAN_500.evpn.0
                Indirect next hops: 1
                        Protocol next hop: 111.111.111.111 Metric: 1
                        Indirect next hop: 0x2 no-forward INH Session ID: 0x0
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 10.1.1.89 via ge-0/0/1.46
                                Session Id: 0x0
                        111.111.111.111/32 Originating RIB: inet.3
                          Metric: 1                       Node path count: 1
                          Forwarding nexthops: 1
                                Nexthop: 10.1.1.89 via ge-0/0/1.46


2:111.111.111.111:50::500::00:1e:e5:c8:0f:bf/304 MAC/IP (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 111.111.111.111:50
                Next hop type: Indirect, Next hop index: 0
                Address: 0xb7a5950
                Next-hop reference count: 10
                Source: 111.111.111.111
                Protocol next hop: 111.111.111.111
                Indirect next hop: 0x2 no-forward INH Session ID: 0x0
                State: <Active Int Ext>
                Local AS:  2345 Peer AS:  2345
                Age: 24:47      Metric2: 1
                Validation State: unverified
                Task: BGP_2345.111.111.111.111
                AS path: I
                Communities: target:2345:50
                Import Accepted
                Route Label: 306800
                ESI: 00:11:22:33:44:55:66:77:88:99
                Localpref: 100
                Router ID: 111.111.111.111
                Secondary Tables: EVPN_CUSTOMER_G_ELAN_500.evpn.0
                Indirect next hops: 1
                        Protocol next hop: 111.111.111.111 Metric: 1
                        Indirect next hop: 0x2 no-forward INH Session ID: 0x0
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 10.1.1.89 via ge-0/0/1.46
                                Session Id: 0x0
                        111.111.111.111/32 Originating RIB: inet.3
                          Metric: 1                       Node path count: 1
                          Forwarding nexthops: 1
                                Nexthop: 10.1.1.89 via ge-0/0/1.46


admin@PE_MXR02> show route table bgp.evpn.0

bgp.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

.. snip ..

1:111.111.111.111:50::112233445566778899::0/192 AD/EVI       
                   *[BGP/170] 00:24:56, localpref 100, from 111.111.111.111
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 329

2:111.111.111.111:50::500::00:1e:e5:c8:0f:bf/304 MAC/IP       
                   *[BGP/170] 00:24:21, localpref 100, from 111.111.111.111
                      AS path: I, validation-state: unverified
                    > to 10.1.1.89 via ge-0/0/1.46, Push 329



Capture 1 – Link between P_R03 and PE_MXR01

Capture 1, Frame 4 was the ICMP reply from CE_R28 to CE_R27 as seen from P_R03 and PE_MXR01.  According to Wireshark’s analysis (arrows), this was Frame 2’s corresponding reply.  




As P_R03 forwarded the packet to PE_MXR01, it popped the top label of 329.  The VPN label of 306800 was looked up, associated to the EVPN instance (via Ingress-Aliasing since Type 1 route’s label was used) and packet delivered to CE_R27.  Again, traffic forwarding in this direction looked normal.

admin@PE_MXR01> show route table mpls.0 label 306800

mpls.0: 44 destinations, 44 routes (44 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

306800             *[EVPN/7] 00:19:49, routing-instance EVPN_CUSTOMER_G_ELAN_500, route-type Ingress-Aliasing
                      to table EVPN_CUSTOMER_G_ELAN_500.evpn-mac.0
                    [EVPN/7] 00:19:15, routing-instance EVPN_CUSTOMER_G_ELAN_500, route-type Ingress-MAC, vlan-id 500
                      to table EVPN_CUSTOMER_G_ELAN_500.evpn-mac.0



Conclusion

From my observation and analysis in this lab, traffic forwarding in single-active redundancy mode operated as expected per my understanding of EVPN multi-homing.  I didn’t experience any odd behaviors in regards to routing exchange and label usage.