the peering xchange : [a network engineer’s blog]: 2017

This lab was configured in an effort to learn the inner workings of BGP MPLS-Based Ethernet VPN (RFC 7432). After I experienced some disappointing issues with the PBB-EVPN lab build out, I felt compelled to lab up another EVPN solution. Since Cisco didn't offer RFC 7432 based EVPN in IOS, I looked into Juniper routers. Juniper did support EVPN in their code base and offered their virtualized version of the MX routers under a free trial basis so I took this opportunity to learn this technology as well as get more exposure to JunOS.

Lab Environment

Note: These resources were appended to the existing IP/MPLS core network running 4 Cisco IOS XE Version 03.16.05.S.

(3) Juniper vMX: Version 17.2R1.13
(6) Damn Small Linux (DSL) 4.x as test hosts

Diagram & Topology

Three additional PEs (PE_MXR01, PE_MXR02 and PE_MXR03) were added to the existing IP/MPLS core network and two DSL VMs were attached to each of those PEs as end hosts (using a virtual switch). The customer L2VPN was then created between the three PEs over the MPLS backbone.

Infrastructure Information

PE_MXR01, PE_MXR02 and PE_MXR03 iBGP to Route Reflector (R11)
CUSTOMER G

EVPN/EVI Name: CUSTOMER_G_ELAN_500
Customer VLANs: 500 & 501
CE Network IP Addresses:

VLAN 500: 172.16.50.0/24
VLAN 501: 172.16.51.0/24

CE AC Interface: vSwitch 24, 25 & 26
PE AC Interface: GE-0/0/2
Singled Homed

Technology Overview

This section is an overview of key terms and concepts used throughout this post.

EVPN Control & Data Plane

EVPN’s unified control plane allows it to work with multiple data plane technologies, such as MPLS or VXLAN. This lab will concentrate on the base implementation of RFC 7432, BGP MPLS based EVPN.

EVPN Terms and Concepts

Frequently used terms are highlighted below, such as CE, PE, EVI and Ethernet Segment. It is important to have a basic understanding of these EVPN components moving forward.

MP-BGP

BGP as the control plane protocol establishes all information exchange between the PEs. As a proven scalable protocol, it can learn and accommodate thousands of routes. The new address-family for EVPN defines a new NLRI which advertises reachability information.

EVPN Route Types

As part of the RFC 7432 specification, new BGP routes types and communities were introduced. Single-homed EVPN deployments would only involve Route Types 2 and 3. Multi-homed deployments, will involve additional Types 1 and 4 routes. Route types will be further discussed in detail throughout this post.

EVPN Use Cases

The primary use case for EVPN is for Data Center Interconnects (DCI). With large virtualized datacenters being commonplace in today’s landscape, L2/L3 VPN services with flexible and scalable features such as active-active multi-homing, fast convergence/MAC mobility and ARP flooding protection become extremely important.

Configurations

Configurations below were from the three new PEs added to the MPLS core.

PE_MXR01

version 17.2R1.13;
system {
host-name PE_MXR01;
root-authentication {
encrypted-password "<removed>"; ## SECRET-DATA
}
login {
user admin {
uid 2000;
class super-user;
authentication {
encrypted-password "<removed>"; ## SECRET-DATA
}
}
}
services {
ssh {
root-login allow;
}
}
syslog {
user * {
any emergency;
}
file messages {
any notice;
authorization info;
}
file interactive-commands {
interactive-commands any;
}
}
processes {
dhcp-service {
traceoptions {
file dhcp_logfile size 10m;
level all;
flag all;
}
}
}
}
chassis {
fpc 0 {
lite-mode;
}
}
interfaces {
ge-0/0/0 {
unit 0 {
description MGMT;
family inet {
address 10.1.1.204/24;
}
}
}
ge-0/0/1 {
vlan-tagging;
mtu 9234;
unit 44 {
description "TO P_R03";
vlan-id 44;
family inet {
address 10.1.1.82/30;
}
family mpls;
}
unit 45 {
description "TO P_R01";
vlan-id 45;
family inet {
address 10.1.1.86/30;
}
family mpls;
}
}
ge-0/0/2 {
description "TO CUSTOMER_G ELAN 500 VSWITCH";
flexible-vlan-tagging;
encapsulation flexible-ethernet-services;
unit 500 {
family bridge {
interface-mode trunk;
vlan-id-list [ 500 501 ];
}
}
}
fxp0 {
description "RE MGMT";
unit 0 {
disable;
family inet {
dhcp {
vendor-id Juniper-vmx;
}
}
}
}
lo0 {
unit 111 {
description RID;
family inet {
address 111.111.111.111/32;
}
}
}
}
routing-options {
router-id 111.111.111.111;
autonomous-system 2345;
}
protocols {
mpls {
label-range {
dynamic-label-range 111000 111999;
}
}
bgp {
group RR {
type internal;
local-address 111.111.111.111;
hold-time 21;
family evpn {
signaling;
}
neighbor 11.11.11.11;
}
}
ospf {
area 0.0.0.0 {
interface ge-0/0/1.44 {
interface-type p2p;
metric 1;
}
interface lo0.111 {
passive;
}
interface ge-0/0/1.45 {
interface-type p2p;
metric 4;
}
}
}
ldp {
interface ge-0/0/1.44;
interface ge-0/0/1.45;
}
}
routing-instances {
EVPN_CUSTOMER_G_ELAN_500 {
instance-type virtual-switch;
interface ge-0/0/2.500;
route-distinguisher 111.111.111.111:50;
vrf-target target:2345:50;
protocols {
evpn {
extended-vlan-list 500-501;
}
}
bridge-domains {
VL500 {
domain-type bridge;
vlan-id 500;
}
VL501 {
domain-type bridge;
vlan-id 501;
}
}
}
MGMT {
instance-type virtual-router;
interface ge-0/0/0.0;
routing-options {
static {
route 0.0.0.0/0 next-hop 10.1.1.254;
}
}
}
}

PE_MXR02

version 17.2R1.13;
system {
host-name PE_MXR02;
root-authentication {
encrypted-password "<removed>"; ## SECRET-DATA
}
login {
user admin {
uid 2000;
class super-user;
authentication {
encrypted-password "<removed>"; ## SECRET-DATA
}
}
}
services {
ssh {
root-login allow;
}
}
syslog {
user * {
any emergency;
}
file messages {
any notice;
authorization info;
}
file interactive-commands {
interactive-commands any;
}
}
processes {
dhcp-service {
traceoptions {
file dhcp_logfile size 10m;
level all;
flag all;
}
}
}
}
chassis {
fpc 0 {
lite-mode;
}
}
interfaces {
ge-0/0/0 {
unit 0 {
description MGMT;
family inet {
address 10.1.1.205/24;
}
}
}
ge-0/0/1 {
vlan-tagging;
mtu 9234;
unit 46 {
description "TO P_R03";
vlan-id 46;
family inet {
address 10.1.1.90/30;
}
family mpls;
}
unit 47 {
description "TO P_R04";
vlan-id 47;
family inet {
address 10.1.1.94/30;
}
family mpls;
}
}
ge-0/0/2 {
description "TO CUSTOMER_G ELAN 500 VSWITCH";
flexible-vlan-tagging;
encapsulation flexible-ethernet-services;
unit 500 {
family bridge {
interface-mode trunk;
vlan-id-list [ 500 501 ];
}
}
}
fxp0 {
description "RE MGMT";
unit 0 {
disable;
family inet {
dhcp {
vendor-id Juniper-vmx;
}
}
}
}
lo0 {
unit 112 {
description RID;
family inet {
address 112.112.112.112/32;
}
}
}
}
routing-options {
router-id 112.112.112.112;
autonomous-system 2345;
}
protocols {
mpls {
label-range {
dynamic-label-range 112000 112999;
}
}
bgp {
group RR {
type internal;
local-address 112.112.112.112;
hold-time 21;
family evpn {
signaling;
}
neighbor 11.11.11.11;
}
}
ospf {
area 0.0.0.0 {
interface ge-0/0/1.46 {
interface-type p2p;
metric 1;
}
interface ge-0/0/1.47 {
interface-type p2p;
metric 5;
}
interface lo0.112 {
passive;
}
}
}
ldp {
interface ge-0/0/1.46;
interface ge-0/0/1.47;
}
}
routing-instances {
EVPN_CUSTOMER_G_ELAN_500 {
instance-type virtual-switch;
interface ge-0/0/2.500;
route-distinguisher 112.112.112.112:50;
vrf-target target:2345:50;
protocols {
evpn {
extended-vlan-list 500-501;
}
}
bridge-domains {
VL500 {
domain-type bridge;
vlan-id 500;
}
VL501 {
domain-type bridge;
vlan-id 501;
}
}
}
MGMT {
instance-type virtual-router;
interface ge-0/0/0.0;
routing-options {
static {
route 0.0.0.0/0 next-hop 10.1.1.254;
}
}
}
}

PE_MXR03

version 17.2R1.13;
system {
host-name PE_MXR03;
root-authentication {
encrypted-password "<removed>"; ## SECRET-DATA
}
login {
user admin {
uid 2000;
class super-user;
authentication {
encrypted-password "<removed>"; ## SECRET-DATA
}
}
}
services {
ssh {
root-login allow;
}
}
syslog {
user * {
any emergency;
}
file messages {
any notice;
authorization info;
}
file interactive-commands {
interactive-commands any;
}
}
processes {
dhcp-service {
traceoptions {
file dhcp_logfile size 10m;
level all;
flag all;
}
}
}
}
chassis {
fpc 0 {
lite-mode;
}
}
interfaces {
ge-0/0/0 {
unit 0 {
description MGMT;
family inet {
address 10.1.1.206/24;
}
}
}
ge-0/0/1 {
vlan-tagging;
mtu 9234;
unit 48 {
description "TO P_R02";
vlan-id 48;
family inet {
address 10.1.1.98/30;
}
family mpls;
}
unit 49 {
description "TO P_R01";
vlan-id 49;
family inet {
address 10.1.1.102/30;
}
family mpls;
}
}
ge-0/0/2 {
description "TO CUSTOMER_G ELAN 500 VSWITCH";
flexible-vlan-tagging;
encapsulation flexible-ethernet-services;
unit 500 {
family bridge {
interface-mode trunk;
vlan-id-list [ 500 501 ];
}
}
}
fxp0 {
description "RE MGMT";
unit 0 {
disable;
family inet {
dhcp {
vendor-id Juniper-vmx;
}
}
}
}
lo0 {
unit 113 {
description RID;
family inet {
address 113.113.113.113/32;
}
}
}
}
routing-options {
router-id 113.113.113.113;
autonomous-system 2345;
}
protocols {
mpls {
label-range {
dynamic-label-range 113000 113999;
}
}
bgp {
group RR {
type internal;
local-address 113.113.113.113;
hold-time 21;
family evpn {
signaling;
}
neighbor 11.11.11.11;
}
}
ospf {
area 0.0.0.0 {
interface ge-0/0/1.48 {
interface-type p2p;
metric 1;
}
interface ge-0/0/1.49 {
interface-type p2p;
metric 5;
}
interface lo0.113 {
passive;
}
}
}
ldp {
interface ge-0/0/1.48;
interface ge-0/0/1.49;
}
}
routing-instances {
EVPN_CUSTOMER_G_ELAN_500 {
instance-type virtual-switch;
interface ge-0/0/2.500;
route-distinguisher 113.113.113.113:50;
vrf-target target:2345:50;
protocols {
evpn {
extended-vlan-list 500-501;
}
}
bridge-domains {
VL500 {
domain-type bridge;
vlan-id 500;
}
VL501 {
domain-type bridge;
vlan-id 501;
}
}
}
MGMT {
instance-type virtual-router;
interface ge-0/0/0.0;
routing-options {
static {
route 0.0.0.0/0 next-hop 10.1.1.254;
}
}
}
}

EVPN Specific Configurations

Configuration below were taken only from PE_MXR01 for brevity.

EVPN Instance

This configuration defines the EVI, such as its name, AC interface, RD, RT and VLAN ID’s.

routing-instances {

EVPN_CUSTOMER_G_ELAN_500 {

instance-type virtual-switch;

interface ge-0/0/2.500;

route-distinguisher 111.111.111.111:50;

vrf-target target:2345:50;

protocols {

evpn {

extended-vlan-list 500-501;

}

bridge-domains {

VL500 {

domain-type bridge;

vlan-id 500;

}

VL501 {

domain-type bridge;

vlan-id 501;

}

Attachment Circuit

The attachment circuit configuration classifies the customer's traffic into the EVPN by VLAN ID.

interfaces {

ge-0/0/2 {

description "TO CUSTOMER_G ELAN 500 VSWITCH";

flexible-vlan-tagging;

encapsulation flexible-ethernet-services;

unit 500 {

family bridge {

interface-mode trunk;

vlan-id-list [ 500 501 ];

}

MP-BGP

MP-BGP configuration establishes route peering to the other PEs via a Route Reflector (RR) under AFI 25 (L2VPN) & SAFI 70 (EVPN).

PE_MXR01 BGP configuration to RR.

routing-options {

router-id 111.111.111.111;

autonomous-system 2345;

}

protocols {

bgp {

group RR {

type internal;

local-address 111.111.111.111;

hold-time 21;

family evpn {

signaling;

}

neighbor 11.11.11.11;

}

Route Reflector BGP configuration to the PEs.

router bgp 2345

bgp log-neighbor-changes

no bgp default ipv4-unicast

neighbor PE_MXR01 peer-group

neighbor PE_MXR01 remote-as 2345

neighbor PE_MXR01 description TO PE MXR01

neighbor PE_MXR01 update-source Loopback0

neighbor PE_MXR01 timers 7 21

neighbor PE_MXR02 peer-group

neighbor PE_MXR02 remote-as 2345

neighbor PE_MXR02 description TO PE MXR02

neighbor PE_MXR02 update-source Loopback0

neighbor PE_MXR02 timers 7 21

neighbor PE_MXR03 peer-group

neighbor PE_MXR03 remote-as 2345

neighbor PE_MXR03 description TO PE MXR03

neighbor PE_MXR03 update-source Loopback0

neighbor PE_MXR03 timers 7 21

neighbor 111.111.111.111 peer-group PE_MXR01

neighbor 112.112.112.112 peer-group PE_MXR02

neighbor 113.113.113.113 peer-group PE_MXR03

address-family ipv4

exit-address-family

address-family l2vpn evpn

neighbor PE_MXR01 send-community both

neighbor PE_MXR01 route-reflector-client

neighbor PE_MXR02 send-community both

neighbor PE_MXR02 route-reflector-client

neighbor PE_MXR03 send-community both

neighbor PE_MXR03 route-reflector-client

neighbor 111.111.111.111 activate

neighbor 112.112.112.112 activate

neighbor 113.113.113.113 activate

exit-address-family

Testing Results

H01 to H02

H01 to H03

H01's ARP Table

H04 to H05

H04 to H06

H04's ARP Table

Verification Tasks

The following commands were used to verify the operation of the MPLS and EVPN infrastructure. Outputs were only from PE_MXR01 and RR for brevity.

Show OSPF Neighbors to Core
Show LDP Neighbors to Core
Show EVPN Instance Extensive
Show EVPN Database
Show EVPN Database Extensive
Show Bridge MAC Table
Show BGP Summary
Show Route Advertising-Protocol BGP (To RR)
Show Route Receive-Protocol BGP (To RR)
Show IP BGP L2VPN EVPN Summary (From RR)
Show IP BGP L2VPN EVPN (From RR)
Show Route Table EVPN
Show Route Table EVPN Extensive (Type 2 NLRI)
Show Route Table EVPN Extensive (Type 3 NLRI)

OSPF Neighbors

The OSPF neighbors command verified IGP peering between PE_MXR01 to MPLS core network (P1 and P3).

admin@PE_MXR01> show ospf neighbor

Address Interface State ID Pri Dead

10.1.1.81 ge-0/0/1.44 Full 3.3.3.3 1 31

10.1.1.85 ge-0/0/1.45 Full 1.1.1.1 1 31

LDP Neighbors

LDP neighbors command verified LDP peering between PE_MXR01 to MPLS core network (P1 and P3).

admin@PE_MXR01> show ldp neighbor

Address Interface Label space ID Hold time

10.1.1.81 ge-0/0/1.44 3.3.3.3:0 12

10.1.1.85 ge-0/0/1.45 1.1.1.1:0 12

EVPN Instance (extensive)

The EVPN instance output displays operational information, such as:

EVPN Instance Name
MPLS VPN Label
MAC Database Status
AC Interface, operational mode and status
Bridge Domains and associated VLANs
EVPN Neighbors and learned MAC addresses

admin@PE_MXR01> show evpn instance extensive

Instance: EVPN_CUSTOMER_G_ELAN_500

Route Distinguisher: 111.111.111.111:50

Per-instance MAC route label: 300608

MAC database status Local Remote

MAC advertisements: 2 4

MAC+IP advertisements: 0 0

Default gateway MAC advertisements: 0 0

Number of local interfaces: 1 (1 up)

Interface name ESI Mode Status AC-Role

ge-0/0/2.500 00:00:00:00:00:00:00:00:00:00 single-homed Up Root

Number of IRB interfaces: 0 (0 up)

Number of bridge domains: 2

VLAN Domain ID Intfs / up IRB intf Mode MAC sync IM route label SG sync IM core nexthop

500 1 1 Extended Enabled 300672 Disabled

501 1 1 Extended Enabled 300688 Disabled

Number of neighbors: 2

Address MAC MAC+IP AD IM ES Leaf-label

112.112.112.112 2 0 0 2 0

113.113.113.113 2 0 0 2 0

Number of ethernet segments: 0

Instance: __default_evpn__

Route Distinguisher: 111.111.111.111:0

Number of bridge domains: 0

Number of neighbors: 0

EVPN Database

The EVPN database displays a simple table of MAC addresses to PE mappings.

From PE_MXR01's output, local MACs were mapped to their AC interface while remote MACs mapped to their respective PEs.

admin@PE_MXR01> show evpn database

Instance: EVPN_CUSTOMER_G_ELAN_500

VLAN DomainId MAC address Active source Timestamp IP address

500 00:0c:29:50:20:24 112.112.112.112 Dec 21 19:15:41

500 00:0c:29:c4:09:26 113.113.113.113 Dec 21 19:15:50

500 00:0c:29:f9:a1:ff ge-0/0/2.500 Dec 21 19:15:40

501 00:0c:29:77:ea:21 ge-0/0/2.500 Dec 21 19:16:59

501 00:0c:29:98:c1:3c 113.113.113.113 Dec 21 19:17:10

501 00:0c:29:a8:ac:de 112.112.112.112 Dec 21 19:17:00

EVPN Database (extensive)

The extensive output displays a verbose version of the EVPN database.

The MAC label in this output identified the MPLS VPN label that was allocated on each PE on a per instance basis. In other words, label 300656 was the EVPN instance label for PE_MXR02 and label 300352 for PE_MXR03.

admin@PE_MXR01> show evpn database extensive

Instance: EVPN_CUSTOMER_G_ELAN_500

VLAN ID: 500, MAC address:: 00:0c:29:50:20:24

Nexthop ID: 1048576

Source: 112.112.112.112, Rank: 1, Status: Active

MAC label: 300656

Timestamp: Dec 21 19:15:41 (0x5a3c085d)

State: <Remote-To-Local-Adv-Done>

VLAN ID: 500, MAC address:: 00:0c:29:c4:09:26

Nexthop ID: 1048579

Source: 113.113.113.113, Rank: 1, Status: Active

MAC label: 300352

Timestamp: Dec 21 19:15:50 (0x5a3c0866)

State: <Remote-To-Local-Adv-Done>

VLAN ID: 500, MAC address:: 00:0c:29:f9:a1:ff

Source: ge-0/0/2.500, Rank: 1, Status: Active

Timestamp: Dec 21 19:15:40 (0x5a3c085c)

State: <Local-MAC-Only Local-To-Remote-Adv-Allowed>

VLAN ID: 501, MAC address:: 00:0c:29:77:ea:21

Source: ge-0/0/2.500, Rank: 1, Status: Active

Timestamp: Dec 21 19:16:59 (0x5a3c08ab)

State: <Local-MAC-Only Local-To-Remote-Adv-Allowed>

VLAN ID: 501, MAC address:: 00:0c:29:98:c1:3c

Nexthop ID: 1048579

Source: 113.113.113.113, Rank: 1, Status: Active

MAC label: 300352

Timestamp: Dec 21 19:17:10 (0x5a3c08b6)

State: <Remote-To-Local-Adv-Done>

VLAN ID: 501, MAC address:: 00:0c:29:a8:ac:de

Nexthop ID: 1048576

Source: 112.112.112.112, Rank: 1, Status: Active

MAC label: 300656

Timestamp: Dec 21 19:17:00 (0x5a3c08ac)

State: <Remote-To-Local-Adv-Done>

Bridge MAC Table

This bridge mac table displays the MAC addresses learned from the router's data plane side.

The MAC flags displayed which addresses were learned locally vs. what was learned over the control plane.

admin@PE_MXR01> show bridge mac-table

MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC

O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)

Routing instance : EVPN_CUSTOMER_G_ELAN_500

Bridging domain : VL500, VLAN : 500

MAC MAC Logical NH MAC

address flags interface Index property

00:0c:29:50:20:24 DC 1048576

00:0c:29:c4:09:26 DC 1048579

00:0c:29:f9:a1:ff D ge-0/0/2.500

MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC

O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)

Routing instance : EVPN_CUSTOMER_G_ELAN_500

Bridging domain : VL501, VLAN : 501

MAC MAC Logical NH MAC

address flags interface Index property

00:0c:29:77:ea:21 D ge-0/0/2.500

00:0c:29:98:c1:3c DC 1048579

00:0c:29:a8:ac:de DC 1048576

BGP Summary

BGP summary displays peering statistic information.

All PEs had a single peer configured to the RR.

admin@PE_MXR01> show bgp summary

Groups: 1 Peers: 1 Down peers: 0

Table Tot Paths Act Paths Suppressed History Damp State Pending

bgp.evpn.0

8 8 0 0 0 0

Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...

11.11.11.11 2345 225406 242013 0 0 2w3d 15:41:31 Establ

bgp.evpn.0: 8/8/8/0

EVPN_CUSTOMER_G_ELAN_500.evpn.0: 8/8/8/0

__default_evpn__.evpn.0: 0/0/0/0

BGP Route Advertising

This command displays routes advertised to the RR. EVPN single-homed deployments will only advertise BGP Type 2 and 3 routes.

In this output, PE_MXR01 advertised two Type 2 routes for the locally attached hosts and two Type 3 inclusive multicast routes, one for each VLAN. Type 2 routes advertise MAC address reachability and Type 3 IM routes advertise reachability for multi-destination traffic, such as BUM traffic.

admin@PE_MXR01> show route advertising-protocol bgp 11.11.11.11

EVPN_CUSTOMER_G_ELAN_500.evpn.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)

Prefix Nexthop MED Lclpref AS path

2:111.111.111.111:50::500::00:0c:29:f9:a1:ff/304 MAC/IP

* Self 100 I

2:111.111.111.111:50::501::00:0c:29:77:ea:21/304 MAC/IP

* Self 100 I

3:111.111.111.111:50::500::111.111.111.111/248 IM

* Self 100 I

3:111.111.111.111:50::501::111.111.111.111/248 IM

* Self 100 I

BGP Route Receiving

This command displays the routes received from the RR.

PE_MXR01 has learned all Type 2 MAC and Type 3 IM routes in the EVI.

admin@PE_MXR01> show route receive-protocol bgp 11.11.11.11

inet.0: 49 destinations, 49 routes (49 active, 0 holddown, 0 hidden)

inet.3: 42 destinations, 42 routes (42 active, 0 holddown, 0 hidden)

MGMT.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)

mpls.0: 49 destinations, 50 routes (49 active, 0 holddown, 0 hidden)

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)

bgp.evpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)

Prefix Nexthop MED Lclpref AS path

2:112.112.112.112:50::500::00:0c:29:50:20:24/304 MAC/IP

* 112.112.112.112 100 I

2:112.112.112.112:50::501::00:0c:29:a8:ac:de/304 MAC/IP

* 112.112.112.112 100 I

2:113.113.113.113:50::500::00:0c:29:c4:09:26/304 MAC/IP

* 113.113.113.113 100 I

2:113.113.113.113:50::501::00:0c:29:98:c1:3c/304 MAC/IP

* 113.113.113.113 100 I

3:112.112.112.112:50::500::112.112.112.112/248 IM

* 112.112.112.112 100 I

3:112.112.112.112:50::501::112.112.112.112/248 IM

* 112.112.112.112 100 I

3:113.113.113.113:50::500::113.113.113.113/248 IM

* 113.113.113.113 100 I

3:113.113.113.113:50::501::113.113.113.113/248 IM

* 113.113.113.113 100 I

EVPN_CUSTOMER_G_ELAN_500.evpn.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)

Prefix Nexthop MED Lclpref AS path

2:112.112.112.112:50::500::00:0c:29:50:20:24/304 MAC/IP

* 112.112.112.112 100 I

2:112.112.112.112:50::501::00:0c:29:a8:ac:de/304 MAC/IP

* 112.112.112.112 100 I

2:113.113.113.113:50::500::00:0c:29:c4:09:26/304 MAC/IP

* 113.113.113.113 100 I

2:113.113.113.113:50::501::00:0c:29:98:c1:3c/304 MAC/IP

* 113.113.113.113 100 I

3:112.112.112.112:50::500::112.112.112.112/248 IM

* 112.112.112.112 100 I

3:112.112.112.112:50::501::112.112.112.112/248 IM

* 112.112.112.112 100 I

3:113.113.113.113:50::500::113.113.113.113/248 IM

* 113.113.113.113 100 I

3:113.113.113.113:50::501::113.113.113.113/248 IM

* 113.113.113.113 100 I

BGP L2VPN EVPN Summary (From RR)

This command displays BGP neighbors from the RR's perspective.

The RR has peering established to all PEs. Each PE learned 4 routes each, two Type 2 and two Type 3 routes (one Type 2 and 3 route per host per VLAN).

RR_R11#sh ip bgp l2vpn evpn summary

BGP router identifier 11.11.11.11, local AS number 2345

BGP table version is 270, main routing table version 270

12 network entries using 4128 bytes of memory

12 path entries using 1968 bytes of memory

7/7 BGP path/bestpath attribute entries using 1848 bytes of memory

1 BGP extended community entries using 24 bytes of memory

0 BGP route-map cache entries using 0 bytes of memory

0 BGP filter-list cache entries using 0 bytes of memory

BGP using 7968 total bytes of memory

BGP activity 113/101 prefixes, 128/116 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

111.111.111.111 4 2345 242062 225450 270 0 0 2w3d 4

112.112.112.112 4 2345 231070 215162 270 0 0 2w2d 4

113.113.113.113 4 2345 217118 202241 270 0 0 2w1d 4

BGP L2VPN EVPN (From RR)

This command displays the EVPN BGP route table from the RR's perspective.

RR_R11#sh ip bgp l2vpn evpn

BGP table version is 270, local router ID is 11.11.11.11

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,

x best-external, a additional-path, c RIB-compressed,

Origin codes: i - IGP, e - EGP, ? - incomplete

RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path

Route Distinguisher: 111.111.111.111:50

*>i [2][111.111.111.111:50][00000000000000000000][500][48][000C29F9A1FF][0][*][300608]/33

111.111.111.111 100 0 i

*>i [2][111.111.111.111:50][00000000000000000000][501][48][000C2977EA21][0][*][300608]/33

111.111.111.111 100 0 i

Route Distinguisher: 112.112.112.112:50

*>i [2][112.112.112.112:50][00000000000000000000][500][48][000C29502024][0][*][300656]/33

112.112.112.112 100 0 i

*>i [2][112.112.112.112:50][00000000000000000000][501][48][000C29A8ACDE][0][*][300656]/33

112.112.112.112 100 0 i

Route Distinguisher: 113.113.113.113:50

*>i [2][113.113.113.113:50][00000000000000000000][500][48][000C29C40926][0][*][300352]/33

113.113.113.113 100 0 i

*>i [2][113.113.113.113:50][00000000000000000000][501][48][000C2998C13C][0][*][300352]/33

113.113.113.113 100 0 i

Route Distinguisher: 111.111.111.111:50

*>i [3][111.111.111.111:50][500][32][662634496.30f0.b684.277f]/17

111.111.111.111 100 0 i

*>i [3][111.111.111.111:50][501][32][662634496.30f0.b684.277f]/17

111.111.111.111 100 0 i

Route Distinguisher: 112.112.112.112:50

*>i [3][112.112.112.112:50][500][32][662634496.30f0.b684.277f]/17

112.112.112.112 100 0 i

*>i [3][112.112.112.112:50][501][32][662634496.30f0.b684.277f]/17

112.112.112.112 100 0 i

Route Distinguisher: 113.113.113.113:50

*>i [3][113.113.113.113:50][500][32][662634496.30f0.b684.277f]/17

113.113.113.113 100 0 i

*>i [3][113.113.113.113:50][501][32][662634496.30f0.b684.277f]/17

113.113.113.113 100 0 i

Route Table EVPN

This command displays all routes in the EVPN table, both local and learned.

admin@PE_MXR01> show route table EVPN_CUSTOMER_G_ELAN_500.evpn.0

EVPN_CUSTOMER_G_ELAN_500.evpn.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

2:111.111.111.111:50::500::00:0c:29:f9:a1:ff/304 MAC/IP

*[EVPN/170] 00:38:52

Indirect

2:111.111.111.111:50::501::00:0c:29:77:ea:21/304 MAC/IP

*[EVPN/170] 00:37:33

Indirect

2:112.112.112.112:50::500::00:0c:29:50:20:24/304 MAC/IP

*[BGP/170] 00:38:52, localpref 100, from 11.11.11.11

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 326

2:112.112.112.112:50::501::00:0c:29:a8:ac:de/304 MAC/IP

*[BGP/170] 00:37:33, localpref 100, from 11.11.11.11

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 326

2:113.113.113.113:50::500::00:0c:29:c4:09:26/304 MAC/IP

*[BGP/170] 00:38:42, localpref 100, from 11.11.11.11

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 339

2:113.113.113.113:50::501::00:0c:29:98:c1:3c/304 MAC/IP

*[BGP/170] 00:37:23, localpref 100, from 11.11.11.11

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 339

3:111.111.111.111:50::500::111.111.111.111/248 IM

*[EVPN/170] 2w1d 20:51:22

Indirect

3:111.111.111.111:50::501::111.111.111.111/248 IM

*[EVPN/170] 2w1d 20:51:22

Indirect

3:112.112.112.112:50::500::112.112.112.112/248 IM

*[BGP/170] 2w1d 20:46:31, localpref 100, from 11.11.11.11

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 326

3:112.112.112.112:50::501::112.112.112.112/248 IM

*[BGP/170] 2w1d 20:46:31, localpref 100, from 11.11.11.11

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 326

3:113.113.113.113:50::500::113.113.113.113/248 IM

*[BGP/170] 2w1d 20:28:21, localpref 100, from 11.11.11.11

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 339

3:113.113.113.113:50::501::113.113.113.113/248 IM

*[BGP/170] 2w1d 20:28:21, localpref 100, from 11.11.11.11

AS path: I, validation-state: unverified

> to 10.1.1.81 via ge-0/0/1.44, Push 339

Route Table EVPN (Type 2 MAC NLRI)

Here’s a deeper look into the EVPN BGP Type 2 MAC NLRI (MAC Advertisement). More information about the Type 2 MAC Advertisement route can be found in RFC 7432, Sec. 7.2.

BGP Route Type
Route Distinguisher
VLAN ID
C-MAC Address
MPLS Label
Ethernet Segment Identifier

admin@PE_MXR01> show route table EVPN_CUSTOMER_G_ELAN_500.evpn.0 extensive

EVPN_CUSTOMER_G_ELAN_500.evpn.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)

..snip..

2:112.112.112.112:50::500::00:0c:29:50:20:24/304 MAC/IP (1 entry, 1 announced)

*BGP Preference: 170/-101

Route Distinguisher: 112.112.112.112:50

Next hop type: Indirect, Next hop index: 0

Address: 0xb7a5950

Next-hop reference count: 8

Source: 11.11.11.11

Protocol next hop: 112.112.112.112

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

State: <Secondary Active Int Ext>

Local AS: 2345 Peer AS: 2345

Age: 7:13:58 Metric2: 1

Validation State: unverified

Task: BGP_2345.11.11.11.11

Announcement bits (1): 0-EVPN_CUSTOMER_G_ELAN_500-evpn

AS path: I (Originator)

Cluster list: 11.11.11.11

Originator ID: 112.112.112.112

Communities: target:2345:50

Import Accepted

Route Label: 300656

ESI: 00:00:00:00:00:00:00:00:00:00

Localpref: 100

Router ID: 11.11.11.11

Primary Routing Table bgp.evpn.0

Indirect next hops: 1

Protocol next hop: 112.112.112.112 Metric: 1

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

Indirect path forwarding next hops: 1

Next hop type: Router

Next hop: 10.1.1.81 via ge-0/0/1.44

Session Id: 0x0

112.112.112.112/32 Originating RIB: inet.3

Metric: 1 Node path count: 1

Forwarding nexthops: 1

Nexthop: 10.1.1.81 via ge-0/0/1.44

Route Table EVPN (Type 3 IM NLRI)

Here’s a deeper look into the EVPN BGP Type 3 IM NLRI (Inclusive-Multicast). More information about the Type 3 Inclusive Multicast route can be found in RFC 7432, Sec. 7.3.

BGP Route Type
Route Distinguisher
VLAN ID
IP Address of originating router
MPLS Label
Tunnel Type

admin@PE_MXR01> show route table EVPN_CUSTOMER_G_ELAN_500.evpn.0 extensive

EVPN_CUSTOMER_G_ELAN_500.evpn.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)

..snip..

3:112.112.112.112:50::500::112.112.112.112/248 IM (1 entry, 1 announced)

*BGP Preference: 170/-101

Route Distinguisher: 112.112.112.112:50

PMSI: Flags 0x0: Label 300784: Type INGRESS-REPLICATION 112.112.112.112

Next hop type: Indirect, Next hop index: 0

Address: 0xb7a5950

Next-hop reference count: 8

Source: 11.11.11.11

Protocol next hop: 112.112.112.112

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

State: <Secondary Active Int Ext>

Local AS: 2345 Peer AS: 2345

Age: 2w2d 3:20:03 Metric2: 1

Validation State: unverified

Task: BGP_2345.11.11.11.11

Announcement bits (1): 0-EVPN_CUSTOMER_G_ELAN_500-evpn

AS path: I (Originator)

Cluster list: 11.11.11.11

Originator ID: 112.112.112.112

Communities: target:2345:50

Import Accepted

Localpref: 100

Router ID: 11.11.11.11

Primary Routing Table bgp.evpn.0

Indirect next hops: 1

Protocol next hop: 112.112.112.112 Metric: 1

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

Indirect path forwarding next hops: 1

Next hop type: Router

Next hop: 10.1.1.81 via ge-0/0/1.44

Session Id: 0x0

112.112.112.112/32 Originating RIB: inet.3

Metric: 1 Node path count: 1

Forwarding nexthops: 1

Nexthop: 10.1.1.81 via ge-0/0/1.44

MPLS Forwarding Trace

Here's a step by step look at a frame from Host01 being forwarded throughout the EVPN network to Host03.

Host01 sent a frame to Host03. PE_MXR01 received that frame and performed an EVPN database and route lookup of Host03’s MAC address of 00:0c:29:c4:09:26. The BGP Type 2 MAC route received a VPN label of 300352. This label was imposed as the inner label, which will later identify the VPN at the destination. This destination MAC's next-hop IP was set to 113.113.113.113.

admin@PE_MXR01> show route table EVPN_CUSTOMER_G_ELAN_500.evpn.0 evpn-mac-address 00:0c:29:c4:09:26 extensive

EVPN_CUSTOMER_G_ELAN_500.evpn.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)

2:113.113.113.113:50::500::00:0c:29:c4:09:26/304 MAC/IP (1 entry, 1 announced)

*BGP Preference: 170/-101

Route Distinguisher: 113.113.113.113:50

Next hop type: Indirect, Next hop index: 0

Address: 0xb7a6970

Next-hop reference count: 8

Source: 11.11.11.11

Protocol next hop: 113.113.113.113

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

State: <Secondary Active Int Ext>

Local AS: 2345 Peer AS: 2345

Age: 11:13:36 Metric2: 1

Validation State: unverified

Task: BGP_2345.11.11.11.11

Announcement bits (1): 0-EVPN_CUSTOMER_G_ELAN_500-evpn

AS path: I (Originator)

Cluster list: 11.11.11.11

Originator ID: 113.113.113.113

Communities: target:2345:50

Import Accepted

Route Label: 300352

ESI: 00:00:00:00:00:00:00:00:00:00

Localpref: 100

Router ID: 11.11.11.11

Primary Routing Table bgp.evpn.0

Indirect next hops: 1

Protocol next hop: 113.113.113.113 Metric: 1

Indirect next hop: 0x2 no-forward INH Session ID: 0x0

Indirect path forwarding next hops: 1

Next hop type: Router

Next hop: 10.1.1.81 via ge-0/0/1.44

Session Id: 0x0

113.113.113.113/32 Originating RIB: inet.3

Metric: 1 Node path count: 1

Forwarding nexthops: 1

Nexthop: 10.1.1.81 via ge-0/0/1.44

admin@PE_MXR01> show evpn database mac-address 00:0c:29:c4:09:26 extensive

Instance: EVPN_CUSTOMER_G_ELAN_500

VLAN ID: 500, MAC address:: 00:0c:29:c4:09:26

Nexthop ID: 1048579

Source: 113.113.113.113, Rank: 1, Status: Active

MAC label: 300352

Timestamp: Dec 21 19:15:50 (0x5a3c0866)

State: <Remote-To-Local-Adv-Done>

PE_MXR01 performed a route forwarding lookup for prefix 113.113.113.113. Then it pushed an outer forwarding label of 339 and forwarded the packet to its next hop (P3).

admin@PE_MXR01> show route table inet.3 113.113.113.113

inet.3: 42 destinations, 42 routes (42 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

113.113.113.113/32 *[LDP/9] 2w5d 03:53:06, metric 1

> to 10.1.1.81 via ge-0/0/1.44, Push 339

P3 received the packet, then performed a MPLS forwarding lookup for label 339. It swapped the label to 239 and forwarded the packet to its next hop (P2).

P_R03#sh mpls forwarding-table labels 339 detail

Local Outgoing Prefix Bytes Label Outgoing Next Hop

Label Label or Tunnel Id Switched interface

339 239 113.113.113.113/32 \

10620072 Gi1.14 10.0.0.17

MAC/Encaps=18/22, MRU=9216, Label Stack{239}

000C294A72B4000C291EB42A8100000E8847 000EF000

No output feature configured

P2 received the packet, then performed a MPLS forwarding lookup for label 239. As the PHP PE, it popped the label off, then forwarded the packet to its next hop (PE_MXR03).

P_R02#sh mpls forwarding-table labels 239 detail

Local Outgoing Prefix Bytes Label Outgoing Next Hop

Label Label or Tunnel Id Switched interface

239 Pop Label 113.113.113.113/32 \

38943282 Gi1.48 10.1.1.98

MAC/Encaps=18/18, MRU=9220, Label Stack{}

000C29866C6E000C294A72B4810000308847

No output feature configured

PE_MXR03 received the packet and performed a final MPLS look up for label of 300352. This label identified the EVPN instance as "EVPN_CUSTOMER_G_ELAN_500".

admin@PE_MXR03> show route table mpls.0 label 300352

mpls.0: 49 destinations, 50 routes (49 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

300352 *[EVPN/7] 11:32:50, routing-instance EVPN_CUSTOMER_G_ELAN_500, route-type Ingress-MAC, vlan-id 500

to table EVPN_CUSTOMER_G_ELAN_500.evpn-mac.0

[EVPN/7] 11:31:30, routing-instance EVPN_CUSTOMER_G_ELAN_500, route-type Ingress-MAC, vlan-id 501

to table EVPN_CUSTOMER_G_ELAN_500.evpn-mac.0

The EVPN database and bridge domain lookup for the routing instance "EVPN_CUSTOMER_G_ELAN_500" mapped MAC address 00:0c:29:c4:09:26 to the local AC interface. The frame was forwarded out ge-0/0/2.500.

admin@PE_MXR03> show evpn database mac-address 00:0c:29:c4:09:26 extensive

Instance: EVPN_CUSTOMER_G_ELAN_500

VLAN ID: 500, MAC address:: 00:0c:29:c4:09:26

Source: ge-0/0/2.500, Rank: 1, Status: Active

Timestamp: Dec 21 19:15:50 (0x5a3c0866)

State: <Local-MAC-Only Local-To-Remote-Adv-Allowed>

admin@PE_MXR03> show bridge mac-table 00:0c:29:c4:09:26 extensive

MAC address: 00:0c:29:c4:09:26

Routing instance: EVPN_CUSTOMER_G_ELAN_500

Bridging domain: VL500, VLAN : 500

Learning interface: ge-0/0/2.500

Base learning interface: ge-0/0/2.500