Cisco IOS Interoperability Issues with ZBFW, NVI NAT and FVRF’s

Recently I had a requirement to update our firewall feature set on our Cisco ISR G2 routers from classic firewall (CBAC) to Zone Based Firewall (ZBFW).  As I was preparing for this and tested this in my lab, I ran into a strange issue and wanted to share my experience.

The problem can only be explained as an interoperability issue between Zone Based Firewall, while using Front door VRF’s (FVRF) and NAT virtual interface (NVI).  From what I can tell, this issue seems to be occurring on only Cisco ISR G2 platforms (800 – 3900) that runs IOS classic.  The newer generation ISR 4000 series that run IOS XE do not experience this problem.  As of this writing, I still run into this issue as of IOS version 15.5(3)M2.

Cisco seems to know about this issue too. There is an existing bug ID that closely resemble this behavior (see link below).  However, when I opened a TAC case and went through the typical rigmarole, I really didn’t get anywhere.  The case even got escalated to their security team and their only advice was to simply implement one of the workarounds.

https://quickview.cloudapps.cisco.com/quickview/bug/CSCsh12490

Workaround:

Remove Zone-Based Policy Firewall and apply Classic Firewall

-or-

Remove NAT-Virtual Interface configuration and apply Inside/Outside NAT.

The main issue for me was that both workarounds couldn’t apply in my situation.  The first workaround was not feasible since I has a hard requirement to run ZBFW.  The second option was also a no go since NAT Virtual Interface was the only NAT method that seemed to interoperate with FVRF’s.  For some reason, the older style Inside/Outside NAT didn’t seem to work with FVRF’s.  I’ve tried all configuration permutations of NVI, and trust me it just didn’t work.  So as you can tell, I was in between a rock and a hard place and had to figure out a solution.

The issue with ZBFW was strange.  It definitely wasn’t behaving normally.  When the two zones were setup (Inside and Outside) and their corresponding policies and zone-pairs were applied (e.g., inside to outside and outside to inside), the inspection failed as soon as I applied a zone pair that inspected traffic from outside to self.   It only failed when I applied that zone pair.  Other zone pairs in the direction of “self to outside” or “outside to inside” did not affect the test traffic.  If I made special provisions to exclude or “pass” the test traffic (say ICMP) in the outside to self policy, it would work.  So that proved ZBFW inspected traffic it shouldn’t have.  Somewhere in the router, the return traffic wasn’t matching the same policy as it passed through initially.  It thought that traffic was meant for the router itself, hence the outside to self policy dropped it.  

Lab Diagram

Test Protocol

Ping and Telnet from LAN_SWITCH’s VL200 with IP 10.1.1.253 to INET_SWITCH IP of 4.2.2.2.

Configuration

Router

version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname ROUTER
!
boot-start-marker
boot-end-marker
!
!
logging buffered 16000
enable secret 5 <removed>
!
no aaa new-model
ethernet lmi ce
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
ip vrf INET_PUBLIC
!
ip vrf MGMT
!
!
!
!
ip domain lookup source-interface GigabitEthernet0/3
ip domain name domain.local
ip name-server vrf MGMT 8.8.8.8
ip name-server 8.8.8.8
ip cef
no ipv6 cef
!
parameter-map type inspect global
 log dropped-packets enable
 max-incomplete low 18000
 max-incomplete high 20000
parameter-map type inspect tcp
 audit-trail on
 alert on
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
license udi pid C3900-SPE250/K9 sn <removed>
!
!         
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
!
class-map type inspect match-any CMAP_FW_PASS_OUTSIDE_TO_SELF
 match access-group name ACL_DHCP_IN
class-map type inspect match-any CMAP_FW_PASS_SELF_TO_OUTSIDE
 match access-group name ACL_DHCP_OUT
class-map type inspect match-any CMAP_FW_INSPECT_INSIDE_TO_OUTSIDE
 match access-group name ACL_LAN_INSIDE_TO_OUTSIDE
class-map type inspect match-any CMAP_FW_INSPECT_OUTSIDE_TO_SELF
 match access-group name ACL_SSH_IN
 match access-group name ACL_ICMP_IN
class-map type inspect match-any CMAP_FW_INSPECT_SELF_TO_OUTSIDE
 match access-group name ACL_NTP_OUT
 match access-group name ACL_ICMP_OUT
 match access-group name ACL_HTTP_OUT
 match access-group name ACL_DNS_OUT
class-map type inspect match-any CMAP_FW_INSPECT_OUTSIDE_TO_INSIDE
 match access-group name ACL_LAN_OUTSIDE_TO_INSIDE
!
policy-map type inspect PMAP_FW_INSIDE_TO_OUTSIDE
 class type inspect CMAP_FW_INSPECT_INSIDE_TO_OUTSIDE
  inspect 
 class class-default
  drop log
policy-map type inspect PMAP_FW_SELF_TO_OUTSIDE
 class type inspect CMAP_FW_INSPECT_SELF_TO_OUTSIDE
  inspect 
 class type inspect CMAP_FW_PASS_SELF_TO_OUTSIDE
  pass
 class class-default
  drop log
policy-map type inspect PMAP_FW_OUTSIDE_TO_INSIDE
 class type inspect CMAP_FW_INSPECT_OUTSIDE_TO_INSIDE
  inspect 
 class class-default
  drop log
policy-map type inspect PMAP_FW_OUTSIDE_TO_SELF
 class type inspect CMAP_FW_INSPECT_OUTSIDE_TO_SELF
  inspect 
 class type inspect CMAP_FW_PASS_OUTSIDE_TO_SELF
  pass
 class class-default
  drop log
!
zone security INSIDE
zone security OUTSIDE
zone-pair security ZPAIR_FW_INSIDE_TO_OUTSIDE source INSIDE destination OUTSIDE
 service-policy type inspect PMAP_FW_INSIDE_TO_OUTSIDE
zone-pair security ZPAIR_FW_SELF_TO_OUTSIDE source self destination OUTSIDE
 service-policy type inspect PMAP_FW_SELF_TO_OUTSIDE
zone-pair security ZPAIR_FW_OUTSIDE_TO_INSIDE source OUTSIDE destination INSIDE
 service-policy type inspect PMAP_FW_OUTSIDE_TO_INSIDE
zone-pair security ZPAIR_FW_OUTSIDE_TO_SELF source OUTSIDE destination self
 service-policy type inspect PMAP_FW_OUTSIDE_TO_SELF
! 
!
!
interface GigabitEthernet0/0
 description INTERNET
 ip vrf forwarding INET_PUBLIC
 ip address 1.1.1.1 255.255.255.0
 ip nat outside
 ip virtual-reassembly in
 zone-member security OUTSIDE
 ip policy route-map RMAP_INTERNET_RETURN
 duplex auto
 speed auto
!
interface GigabitEthernet0/1
 description LAN
 ip address 10.1.1.254 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 zone-member security INSIDE
 duplex auto
 speed auto
!
interface GigabitEthernet0/2
 no ip address
 shutdown
 duplex auto
 speed auto
!
interface GigabitEthernet0/3
 ip vrf forwarding MGMT
 ip address 192.168.1.200 255.255.255.0
 duplex auto
 speed auto
 no cdp enable
!
!
!
!
no ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip dns view vrf MGMT default
ip nat inside source route-map RMAP_NAT_POLICY interface GigabitEthernet0/0 overload
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0 1.1.1.254 name DEFAULT
ip route vrf INET_PUBLIC 0.0.0.0 0.0.0.0 GigabitEthernet0/0 1.1.1.254 name INET_DEFAULT
ip route vrf MGMT 0.0.0.0 0.0.0.0 192.168.1.254 name MGMT_DEFAULT
!
ip access-list extended ACL_DHCP_IN
 permit udp any eq bootpc any eq bootps
 permit udp any eq bootps any eq bootpc
ip access-list extended ACL_DHCP_OUT
 permit udp any eq bootps any eq bootpc
 permit udp any eq bootpc any eq bootps
ip access-list extended ACL_DNS_OUT
 permit udp any any eq domain
ip access-list extended ACL_HTTP_OUT
 permit tcp any any eq www
ip access-list extended ACL_ICMP_IN
 permit icmp any any echo
 permit icmp any any echo-reply
 permit icmp any any ttl-exceeded
 permit icmp any any port-unreachable
ip access-list extended ACL_ICMP_OUT
 permit icmp any any
ip access-list extended ACL_INTERNET_RETURN
 permit ip any 10.0.0.0 0.255.255.255
ip access-list extended ACL_LAN_INSIDE_TO_OUTSIDE
 permit ip 10.0.0.0 0.255.255.255 any
ip access-list extended ACL_LAN_OUTSIDE_TO_INSIDE
 deny   ip any any
ip access-list extended ACL_NAT
 permit ip 10.0.0.0 0.255.255.255 any
ip access-list extended ACL_NTP_OUT
 permit udp any any eq ntp
ip access-list extended ACL_SSH_OUT
 permit tcp any any eq 22
!
!
route-map RMAP_INTERNET_RETURN permit 10
 match ip address ACL_INTERNET_RETURN
 set global
!
route-map RMAP_NAT_POLICY permit 10
 description ROUTE-MAP FOR NAT
 match ip address ACL_NAT
!
!
!
control-plane
!
!
!
line con 0
 logging synchronous
line aux 0
line vty 0 4
 exec-timeout 60 0
 logging synchronous
 login local
 transport input telnet ssh
!
scheduler allocate 20000 1000
!
end

Test Results

Logs shows ICMP drop for unknown reasons and telnet drop due to policy reasons.   The policy drop doesn’t make sense because traffic should have only been inspected by the ZPAIR_FW_INSIDE_TO_OUTSIDE policy.

May 30 17:37:14.956: %FW-6-DROP_PKT: Dropping icmp session 4.2.2.2:0 1.1.1.1:0 on zone-pair ZPAIR_FW_OUTSIDE_TO_SELF class CMAP_FW_INSPECT_OUTSIDE_TO_SELF due to  Internal Error with ip ident 3918 May 30 17:39:37.870: %FW-6-DROP_PKT: Dropping tcp session 4.2.2.2:23 1.1.1.1:22660 on zone-pair ZPAIR_FW_OUTSIDE_TO_SELF class class-default due to  DROP action found in policy-map with ip ident 34337

ZBFW Sessions (Broken)

ROUTER#show policy-map type inspect zone-pair sessions policy exists on zp ZPAIR_FW_INSIDE_TO_OUTSIDE   Zone-pair: ZPAIR_FW_INSIDE_TO_OUTSIDE   Service-policy inspect : PMAP_FW_INSIDE_TO_OUTSIDE     Class-map: CMAP_FW_INSPECT_INSIDE_TO_OUTSIDE (match-any)        Match: access-group name ACL_LAN_INSIDE_TO_OUTSIDE         13 packets, 816 bytes         30 second rate 0 bps    Inspect       Number of Half-open Sessions = 2       Half-open Sessions         Session 264AE8A8 (10.1.1.253:28673)=>(4.2.2.2:23) tcp SIS_OPENING/TCP_SYNSENT           Created 00:00:12, Last heard 00:00:12           Bytes sent (initiator:responder) [0:0]         Session 264AE540 (10.1.1.253:8)=>(4.2.2.2:0) icmp SIS_OPENING           Created 00:00:05, Last heard 00:00:01           ECHO request           Bytes sent (initiator:responder) [216:0]     Class-map: class-default (match-any)        Match: any       Drop         208980 packets, 8818290 bytes ..snip.. policy exists on zp ZPAIR_FW_OUTSIDE_TO_SELF   Zone-pair: ZPAIR_FW_OUTSIDE_TO_SELF   Service-policy inspect : PMAP_FW_OUTSIDE_TO_SELF     Class-map: CMAP_FW_INSPECT_OUTSIDE_TO_SELF (match-any)        Match: access-group name ACL_SSH_IN         0 packets, 0 bytes         30 second rate 0 bps       Match: access-group name ACL_ICMP_IN         0 packets, 0 bytes         30 second rate 0 bps    Inspect     Class-map: CMAP_FW_PASS_OUTSIDE_TO_SELF (match-any)        Match: access-group name ACL_DHCP_IN         0 packets, 0 bytes         30 second rate 0 bps       Pass         430665 packets, 49375220 bytes     Class-map: class-default (match-any)        Match: any       Drop         2 packets, 72 bytes

During the course of my testing, I wanted to try the second workaround to see if reconfiguring the NAT to use Inside/Outside would make a difference.  As I said before, the classic NAT feature didn’t seem to interoperate with FVRF’s so that could have been a potential culprit to all this.  When I changed over the NAT and ran tests again, I noticed ZBFW wasn’t log the drops as it was before when NVI was configured.  That led me to believe the NAT had something to do with this.

After many more hours researching and testing, I was able to address the NAT issue.  For some weird reason, the router wasn’t able to send the NAT’ed traffic back to the global route table.  Again, I tried all permutations of the NAT command, even though the NAT command was VRF aware.  The solution that worked was to create a policy route that set the post NAT traffic back to the global route table.

//Create ACL and Route-Map for Policy Routing ip access-list extended ACL_INTERNET_RETURN  permit ip any 10.0.0.0 0.255.255.255 route-map RMAP_INTERNET_RETURN permit 10  match ip address ACL_INTERNET_RETURN  set global //Apply Policy Route to Outside Interface interface GigabitEthernet0/0  ip policy route-map RMAP_INTERNET_RETURN

Once the NAT issue was resolved, I reapplied ZBFW and that seemed to work as well.  I guess the issue wasn’t ZBWF at all but rather an internal routing issue between a VRF and global table.  Cisco never told me during the course of the TAC case, nor was it ever documented so I wanted to share this solution.  Hopefully others can benefit from this.

Debug of Policy Routing

Jun  7 21:49:56.910: IP: route map RMAP_INTERNET_RETURN, item 10, permit Jun  7 21:49:56.910: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253, len 40, set global Jun  7 21:49:56.910: IP: set global to outidb 0x23F82540 Jun  7 21:49:56.910: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253 (GigabitEthernet0/1), len 40, policy routed Jun  7 21:49:56.910: IP: GigabitEthernet0/0 to GigabitEthernet0/1 10.1.1.253 Jun  7 21:49:56.912: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253, len 40, FIB policy match Jun  7 21:49:56.912: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253, len 40, PBR Counted Jun  7 21:49:56.912: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253, len 40, FIB policy routed set global

ZBFW Sessions (Working)

ROUTER#show policy-map type inspect zone-pair sessions policy exists on zp ZPAIR_FW_INSIDE_TO_OUTSIDE   Zone-pair: ZPAIR_FW_INSIDE_TO_OUTSIDE   Service-policy inspect : PMAP_FW_INSIDE_TO_OUTSIDE     Class-map: CMAP_FW_INSPECT_INSIDE_TO_OUTSIDE (match-any)        Match: access-group name ACL_LAN_INSIDE_TO_OUTSIDE         9 packets, 608 bytes         30 second rate 0 bps    Inspect       Number of Established Sessions = 2       Established Sessions         Session 264AE8A8 (10.1.1.253:59905)=>(4.2.2.2:23) tcp SIS_OPEN/TCP_ESTAB           Created 00:00:17, Last heard 00:00:15           Bytes sent (initiator:responder) [79:140]         Session 264AEF78 (10.1.1.253:8)=>(4.2.2.2:0) icmp SIS_OPEN           Created 00:00:05, Last heard 00:00:00           ECHO request           Bytes sent (initiator:responder) [102744:102672]     Class-map: class-default (match-any)        Match: any       Drop         208926 packets, 8816202 bytes ..snip.. policy exists on zp ZPAIR_FW_OUTSIDE_TO_SELF   Zone-pair: ZPAIR_FW_OUTSIDE_TO_SELF   Service-policy inspect : PMAP_FW_OUTSIDE_TO_SELF     Class-map: CMAP_FW_INSPECT_OUTSIDE_TO_SELF (match-any)        Match: access-group name ACL_SSH_IN         0 packets, 0 bytes         30 second rate 0 bps       Match: access-group name ACL_ICMP_IN         0 packets, 0 bytes         30 second rate 0 bps    Inspect     Class-map: CMAP_FW_PASS_OUTSIDE_TO_SELF (match-any)        Match: access-group name ACL_DHCP_IN         0 packets, 0 bytes         30 second rate 0 bps       Pass         430549 packets, 49362148 bytes     Class-map: class-default (match-any)        Match: any       Drop         2 packets, 72 bytes

MPLS L2 VPN (VPWS - AToM)

After learning a great deal about L2VPNs during the VPLS lab, I figured taking on Virtual Private Wire Service (VPWS) wouldn't be that much more difficult.  Here’s a quick look at my lab setup for VPWS, also known as Any Transport over MPLS (AToM).

Lab Environment

Note: These resources were appended to the existing MPLS L2VPN lab.

• (2) CSR1000v: Version 03.16.05.S
• Used default vCPU setting of 1
• Used default RAM at 4Gb

Diagram & Topology

I added 2 additional CEs (CE12 and CE19), one to PE13 and PE15.  I used a separate interface connecting the PE to the CE so the attachment circuit (AC) doesn't interfere with other labs.

• Infrastructure Information
• LDP Signaling
• (2) Customer Edge (CE Routers)
• CUSTOMER C
• IP Address: 192.168.0.0/24
• L2VPN XConnect Name: VPWS
• VC ID: 20
• VLAN: 2000

Technology Overview

Here were some good informational slides I found on the technology.  It covers common terms and acronyms that are used throughout the post.  Please see references section for links to these documents.

The two slides below provide a good representation on why VPWS is also known as AToM.  VPWS can transport any protocol, such as Ethernet, ATM or FR, over a MPLS backbone.

Similar to VPLS, VPWS still uses the pseudowire so it fundamentally operates the same.

VPWS continues to use a two label MPLS stack. The inner label identifies the VC and outer label for forwarding.

This lab still uses LDP for signaling, however auto-discovery was unnecessary.  MP-BGP is not used  here.  The pseudowire is manually created between PEs.

Configurations

PE_R13

version 15.5
service timestamps debug datetime localtime show-timezone year
service timestamps log datetime localtime show-timezone year
service password-encryption
no platform punt-keepalive disable-kernel-core
platform console virtual
!
hostname PE_R13
!
boot-start-marker
boot-end-marker
!
!
logging buffered 16000
no logging console
enable secret 5 <removed>
!
no aaa new-model
!
ip vrf MGMT
!
!
!
!
!
!
!
!
!
!
!
no ip domain lookup
ip domain name domain.local
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
mpls label range 1300 1399
mpls label protocol ldp
multilink bundle-name authenticated
l2vpn
 router-id 13.13.13.13
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
license udi pid CSR1000V sn 9T86RBMQW36
!         
spanning-tree extend system-id
!
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
!
cdp run
!
! 
!
!
!
!
!
!
!
!
!
!
!
!
! 
! 
! 
! 
! 
! 
!
!
interface Loopback0
 description RID
 ip address 13.13.13.13 255.255.255.255
 ip ospf 100 area 0
!
interface GigabitEthernet1
 mtu 9216
 no ip address
 negotiation auto
!
interface GigabitEthernet1.24
 description TO P_R01
 encapsulation dot1Q 24
 ip address 10.1.1.18 255.255.255.252
 ip ospf network point-to-point
 ip ospf 100 area 0
 ip ospf cost 10
 mpls ip
!
interface GigabitEthernet1.25
 description TO P_R02
 encapsulation dot1Q 25
 ip address 10.1.1.22 255.255.255.252
 ip ospf network point-to-point
 ip ospf 100 area 0
 mpls ip
!
interface GigabitEthernet2
 mtu 9216
 no ip address
 negotiation auto
 cdp enable
 shutdown
 !        
!
interface GigabitEthernet3
 description TO CE_R12 - VPWS
 mtu 9216
 no ip address
 negotiation auto
 no keepalive
 service instance 20 ethernet
  encapsulation dot1q 2000-2100
 !
!
interface GigabitEthernet4
 description MANAGEMENT
 ip vrf forwarding MGMT
 ip address 10.1.1.233 255.255.255.0
 negotiation auto
!
router ospf 100
 router-id 13.13.13.13
 passive-interface default
 no passive-interface GigabitEthernet1.24
 no passive-interface GigabitEthernet1.25
!
router bgp 2345
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor RR peer-group
 neighbor RR remote-as 2345
 neighbor RR description ROUTE REFLECTOR - R11
 neighbor RR update-source Loopback0
 neighbor RR timers 7 21
 neighbor 11.11.11.11 peer-group RR
 !
 address-family ipv4
 exit-address-family
!
!
virtual-service csr_mgmt
!
no ip forward-protocol nd
!
ip bgp-community new-format
no ip http server
no ip http secure-server
ip tftp source-interface GigabitEthernet4
ip route vrf MGMT 0.0.0.0 0.0.0.0 10.1.1.254 name MGMT_DEFAULT
!
!
mpls ldp router-id Loopback0
!
l2vpn xconnect context VPWS
 description VPWS-ATOM FOR CUSTOMER_C
 remote link failure notification
 member 15.15.15.15 20 encapsulation mpls
 member GigabitEthernet3 service-instance 20 
!
!
control-plane
!
 !
 !
 !
 !
!
!
!
!
!
line con 0
 logging synchronous
 stopbits 1
line vty 0 4
 exec-timeout 60 0
 logging synchronous
 login local
 transport input telnet ssh
!
ntp server pool.ntp.org
!
end

PE_R15

version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
platform console virtual
!
hostname PE_R15
!
boot-start-marker
boot-end-marker
!
!
enable secret 5 <removed>
!
no aaa new-model
!
ip vrf MGMT
!
!
!
!
!
!
!
!
!
!
!
no ip domain lookup
ip domain name domain.local
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
mpls label range 1500 1599
mpls label protocol ldp
multilink bundle-name authenticated
l2vpn
 router-id 15.15.15.15
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
license udi pid CSR1000V sn 9NP1VCY8F2T
!
spanning-tree extend system-id
!
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
!
cdp run
!
! 
!
!
!
!
!
!
!
!
!
!
!
!
! 
! 
! 
! 
! 
! 
!
!
interface Loopback0
 description RID
 ip address 15.15.15.15 255.255.255.255
 ip ospf 100 area 0
!
interface GigabitEthernet1
 mtu 9216
 no ip address
 negotiation auto
!
interface GigabitEthernet1.28
 description TO P_R03
 encapsulation dot1Q 28
 ip address 10.1.1.34 255.255.255.252
 ip ospf network point-to-point
 ip ospf 100 area 0
 mpls ip
!
interface GigabitEthernet1.29
 description TO P_R04
 encapsulation dot1Q 29
 ip address 10.1.1.38 255.255.255.252
 ip ospf network point-to-point
 ip ospf 100 area 0
 mpls ip
!
interface GigabitEthernet2
 mtu 9216
 no ip address
 negotiation auto
 cdp enable
 shutdown
 !
!
interface GigabitEthernet3
 description TO CE_R19 - VPWS
 mtu 9216 
 no ip address
 negotiation auto
 no keepalive
 cdp enable
 service instance 20 ethernet
  encapsulation dot1q 2000-2100
 !
!
interface GigabitEthernet4
 description MANAGEMENT
 ip vrf forwarding MGMT
 ip address 10.1.1.235 255.255.255.0
 negotiation auto
!
router ospf 100
 router-id 15.15.15.15
 passive-interface default
 no passive-interface GigabitEthernet1.28
 no passive-interface GigabitEthernet1.29
!
router bgp 2345
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor RR peer-group
 neighbor RR remote-as 2345
 neighbor RR description ROUTE REFLECTOR - R11
 neighbor RR update-source Loopback0
 neighbor RR timers 7 21
 neighbor 11.11.11.11 peer-group RR
 !
 address-family ipv4
 exit-address-family
!
!
virtual-service csr_mgmt
!
no ip forward-protocol nd
!
ip bgp-community new-format
no ip http server
no ip http secure-server
ip tftp source-interface GigabitEthernet1
ip route vrf MGMT 0.0.0.0 0.0.0.0 10.1.1.254 name MGMT_DEFAULT
!
!
mpls ldp router-id Loopback0
!
l2vpn xconnect context VPWS
 description VPWS-ATOM FOR CUSTOMER_C
 remote link failure notification
 member 13.13.13.13 20 encapsulation mpls
 member GigabitEthernet3 service-instance 20 
!
!
control-plane
!
 !
 !
 !
 !
!
!
!
!
!
line con 0
 logging synchronous
 stopbits 1
line vty 0 4
 exec-timeout 60 0
 logging synchronous
 login local
 transport input telnet ssh
!
ntp server pool.ntp.org
!
end

CE_R12

version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
platform console virtual
!
hostname CE_R12
!
boot-start-marker
boot-end-marker
!
!
enable secret 5 <removed>
!
no aaa new-model
!
ip vrf MGMT
!
!
!
!
!
!
!
!
!
!
!
no ip domain lookup
ip domain name domain.local
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
!
!
license udi pid CSR1000V sn 99EJNS7SIZJ
!
spanning-tree extend system-id
!
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
cdp run
!
!         
!
!
!
!
!
!
!
!
!
!
!
!
! 
! 
! 
! 
! 
! 
!
!
interface GigabitEthernet1
 mtu 9216
 no ip address
 negotiation auto
 cdp enable
!
interface GigabitEthernet1.2000
 description TO PE_R13 - VPWS
 encapsulation dot1Q 2000
 ip address 192.168.0.1 255.255.255.0
 cdp enable
!
interface GigabitEthernet2
 no ip address
 shutdown
 negotiation auto
 cdp enable
!
interface GigabitEthernet3
 no ip address
 shutdown
 negotiation auto
!
interface GigabitEthernet4
 description MANAGEMENT
 ip vrf forwarding MGMT
 ip address 10.1.1.232 255.255.255.0
 negotiation auto
!
!
virtual-service csr_mgmt
!
no ip forward-protocol nd
!
no ip http server
no ip http secure-server
ip route vrf MGMT 0.0.0.0 0.0.0.0 10.1.1.254 name MGMT_DEFAULT
ip ssh time-out 60
!
!
!
!
control-plane
!
 !
 !
 !
 !
!
!
!
!
!
line con 0
 logging synchronous
 stopbits 1
line vty 0 4
 exec-timeout 60 0
 logging synchronous
 login local
 transport input telnet ssh
!
ntp server pool.ntp.org
!
end

CE_R19

version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
platform console virtual
!
hostname CE_R19
!
boot-start-marker
boot-end-marker
!
!
enable secret 5 <removed>
!
no aaa new-model
!
ip vrf MGMT
!
!
!
!
!
!
!
!
!
!
!
no ip domain lookup
ip domain name domain.local
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
!
!
license udi pid CSR1000V sn 9YGZ15PZSP9
!
spanning-tree extend system-id
!
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
cdp run
!
!         
!
!
!
!
!
!
!
!
!
!
!
!
! 
! 
! 
! 
! 
! 
!
!
interface GigabitEthernet1
 mtu 9216
 no ip address
 negotiation auto
 cdp enable
!
interface GigabitEthernet1.2000
 description TO PE_R15 - VPWS
 encapsulation dot1Q 2000
 ip address 192.168.0.2 255.255.255.0
 cdp enable
!
interface GigabitEthernet2
 no ip address
 shutdown
 negotiation auto
 cdp enable
!
interface GigabitEthernet3
 no ip address
 shutdown
 negotiation auto
!
interface GigabitEthernet4
 description MANAGEMENT
 ip vrf forwarding MGMT
 ip address 10.1.1.239 255.255.255.0
 negotiation auto
!
!
virtual-service csr_mgmt
!
no ip forward-protocol nd
!
no ip http server
no ip http secure-server
ip tftp source-interface GigabitEthernet4
ip route vrf MGMT 0.0.0.0 0.0.0.0 10.1.1.254 name MGMT_DEFAULT
!
!
!
!
control-plane
!
 !
 !
 !
 !
!
!
!
!
!
line con 0
 logging synchronous
 stopbits 1
line vty 0 4
 exec-timeout 60 0
 logging synchronous
 login local
 transport input telnet ssh
!
ntp server pool.ntp.org
!
end

VPWS Lab Specific Configurations

We will look at the configuration for only PE13 since the other PE will have similar configurations.

L2VPN XConnect

The xconnect command builds a static peering/pseudowire to the destination PE, binds the attachment circuit and defines the encapsulation.  The pseudowire interface is automatically created with this command.

l2vpn xconnect context VPWS  description VPWS-ATOM FOR CUSTOMER_C  remote link failure notification  member 15.15.15.15 20 encapsulation mpls  member GigabitEthernet3 service-instance 20

Attachment Circuit

The attachment circuit that connects the CE uses an Ethernet Flow Point (EFP)/Bridge-Domain.  The Ethernet service instance ID and VLANs are defined on the PE.  In the example below, the PE matches VLANs 2000 – 2100 for traffic to be classified into service instance 20.

interface GigabitEthernet3  description TO CE_R12 - VPWS  mtu 9216  no ip address  negotiation auto  no keepalive  service instance 20 ethernet   encapsulation dot1q 2000-2100

CE Interface

The CE interface must use VLANs in the range of 2000 - 2100 as transport to be classified correctly on the PE.  Here VLAN 2000 was configured.

interface GigabitEthernet1  mtu 9216  no ip address  negotiation auto ! interface GigabitEthernet1.2000  description TO PE_R13 - VPWS  encapsulation dot1Q 2000  ip address 192.168.0.1 255.255.255.0

Testing Results

Test results were simple pings from site A to B to verify full L3 reachability.  ARP tables lookup verified L2 reachability.

SiteA to SiteB Pings

CE_R12#ping 192.168.0.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.0.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 9/9/10 ms

SiteA’s ARP Table

CE_R12#sh arp Protocol  Address          Age (min)  Hardware Addr   Type   Interface Internet  192.168.0.1             -   000c.29c8.372e  ARPA   GigabitEthernet1.2000 Internet  192.168.0.2           218   000c.29ba.b90c  ARPA   GigabitEthernet1.2000

SiteB’s ARP Table

CE_R19#sh arp Protocol  Address          Age (min)  Hardware Addr   Type   Interface Internet  192.168.0.1           219   000c.29c8.372e  ARPA   GigabitEthernet1.2000 Internet  192.168.0.2             -   000c.29ba.b90c  ARPA   GigabitEthernet1.2000

Verification Tasks

From the customer perspective the topology looks like a point to point cable between devices.  However, there are many underlying infrastructure components at play.  In this section, we will look into the VPWS specific operation.

The list below outlines the VPWS verification tasks.

• L2VPN AToM VC
• L2VPN XConnect details
• MPLS L2Transport VC details
• L2VPN Psuedowire details

L2VPN AToM VC

This command displays a summary of the virtual circuit and its state.

PE_R13#sh l2vpn atom vc                                        Service Interface Peer ID         VC ID      Type   Name                     Status --------- --------------- ---------- ------ ------------------------ ----------   pw100002  15.15.15.15     20         p2p    VPWS                     UP

L2VPN XConnect

This command shows the pseudo-wire interface peer/encapsulation/state, VC labels and AC binding/state.

PE_R13#sh l2vpn service xconnect all detail Legend: St=State    XC St=State in the L2VPN Service      Prio=Priority         UP=Up       DN=Down            AD=Admin Down      IA=Inactive         SB=Standby  HS=Hot Standby     RV=Recovering      NH=No Hardware         m=manually selected   Interface          Group       Encapsulation                   Prio  St  XC St   ---------          -----       -------------                   ----  --  ----- VPWS name: VPWS, State: UP   pw100002                       15.15.15.15:20(MPLS)            0     UP  UP                                    Local VC label 1328                                              Remote VC label 1528                                                                             Gi3                            Gi3:20(Ethernet)                0     UP  UP                                    Interworking: none

MPLS L2Transport VC

The MPLS L2Tranport command will show LDP peering and label stack information.

PE_R13#sh mpls l2transport vc vcid 20 detail Local interface: Gi3 up, line protocol up, Ethernet:20 up   Destination address: 15.15.15.15, VC ID: 20, VC status: up     Output interface: Gi1.25, imposed label stack {221 1500}     Preferred path: not configured      Default path: active     Next hop: 10.1.1.21   Create time: 15:31:30, last status change time: 15:31:16     Last label FSM state change time: 15:31:16   Signaling protocol: LDP, peer 15.15.15.15:0 up     Targeted Hello: 13.13.13.13(LDP Id) -> 15.15.15.15, LDP is UP     Graceful restart: not configured and not enabled     Non stop routing: not configured and not enabled     Status TLV support (local/remote)   : enabled/supported       LDP route watch                   : enabled       Label/status state machine        : established, LruRru       Last local dataplane   status rcvd: No fault       Last BFD dataplane     status rcvd: Not sent       Last BFD peer monitor  status rcvd: No fault       Last local AC  circuit status rcvd: No fault       Last local AC  circuit status sent: No fault       Last local PW i/f circ status rcvd: No fault       Last local LDP TLV     status sent: No fault       Last remote LDP TLV    status rcvd: No fault       Last remote LDP ADJ    status rcvd: No fault     MPLS VC labels: local 1300, remote 1500     Group ID: local 8, remote 8     MTU: local 9216, remote 9216     Remote interface description: TO CE_R19 - VPWS   Sequencing: receive disabled, send disabled   Control Word: On (configured: autosense)   SSO Descriptor: 15.15.15.15/20, local label: 1300   Dataplane:     SSM segment/switch IDs: 28700/4121 (used), PWID: 1   VC statistics:     transit packet totals: receive 9, send 9     transit byte totals:   receive 846, send 1116     transit packet drops:  receive 0, seq error 0, send 0

L2VPN Psuedowire

A detailed look at the pseudowire (PW) will show a bit more information, such as PW type and its AC interworking (Like2Like), which basically means it’s Ethernet to Ethernet.

PE_R13#sh l2vpn atom vc pseudowire 100002 detail pseudowire100002 is up, VC status is up PW type: Ethernet   Create time: 05:09:29, last status change time: 05:09:29     Last label FSM state change time: 05:09:29   Destination address: 15.15.15.15 VC ID: 20     Output interface: Gi1.25, imposed label stack {221 1528}     Preferred path: not configured      Default path: active     Next hop: 10.1.1.21   Member of xconnect service VPWS     Associated member Gi3 is up, status is up     Interworking type is Like2Like     Service id: 0x3d000001   Signaling protocol: LDP, peer 15.15.15.15:0 up     Targeted Hello: 13.13.13.13(LDP Id) -> 15.15.15.15, LDP is UP     Graceful restart: not configured and not enabled     Non stop routing: not configured and not enabled     PWid FEC (128), VC ID: 20     Status TLV support (local/remote)         : enabled/supported       LDP route watch                         : enabled       Label/status state machine              : established, LruRru       Local dataplane status received         : No fault       BFD dataplane status received           : Not sent       BFD peer monitor status received        : No fault       Status received from access circuit     : No fault       Status sent to access circuit           : No fault       Status received from pseudowire i/f     : No fault       Status sent to network peer             : No fault       Status received from network peer       : No fault       Adjacency status of remote peer         : No fault   Sequencing: receive disabled, send disabled   Bindings     Parameter    Local                          Remote     ------------ ------------------------------ ------------------------------     Label        1328                           1528     Group ID     8                              8     Interface    TO CE_R12 - VPWS               TO CE_R19 - VPWS                  MTU          9216                           9216     Control word on (configured: autosense)     on     PW type      Ethernet                       Ethernet     VCCV CV type 0x02                           0x02                    LSPV [2]                       LSPV [2]                        VCCV CC type 0x07                           0x07                    CW [1], RA [2], TTL [3]       CW [1], RA [2], TTL [3]     Status TLV   enabled                        supported   SSO Descriptor: 15.15.15.15/20, local label: 1328   Dataplane:     SSM segment/switch IDs: 8215/4118 (used), PWID: 1   Rx Counters     1 input transit packets, 64 bytes     0 drops, 0 seq err   Tx Counters     1 output transit packets, 94 bytes     0 drops

Packet Walk

This is an example packet walk for traffic sent from CE12 (SiteA) to CE19 (SiteB) for Customer D.  I'll explain step by step as traffic traverses the core network.

1. CE12 sends traffic to CE19 and arrives on PE13’s Gig3 interface.  Interface Gig3 is the attachment circuit for VC 20.



PE_R13#sh mpls l2transport vc Local intf     Local circuit              Dest address    VC ID      Status -------------  -------------------------- --------------- ---------- ---------- Gi3            Ethernet:20                15.15.15.15     20         UP




2. A look at VCID 20 in more details tell us the peer is to 15.15.15.15 (PE15). PE13 imposes (pushes) inner VC label of 1500 and outer LSP label 221 and sends to next hop 10.1.1.21 (P2).



PE_R13#sh mpls l2transport vc vcid 20 detail Local interface: Gi3 up, line protocol up, Ethernet:20 up   Destination address: 15.15.15.15, VC ID: 20, VC status: up     Output interface: Gi1.25, imposed label stack {221 1500}     Preferred path: not configured      Default path: active     Next hop: 10.1.1.21   Create time: 15:31:30, last status change time: 15:31:16     Last label FSM state change time: 15:31:16   Signaling protocol: LDP, peer 15.15.15.15:0 up     Targeted Hello: 13.13.13.13(LDP Id) -> 15.15.15.15, LDP is UP     Graceful restart: not configured and not enabled     Non stop routing: not configured and not enabled     Status TLV support (local/remote)   : enabled/supported       LDP route watch                   : enabled       Label/status state machine        : established, LruRru       Last local dataplane   status rcvd: No fault       Last BFD dataplane     status rcvd: Not sent       Last BFD peer monitor  status rcvd: No fault       Last local AC  circuit status rcvd: No fault       Last local AC  circuit status sent: No fault       Last local PW i/f circ status rcvd: No fault       Last local LDP TLV     status sent: No fault       Last remote LDP TLV    status rcvd: No fault       Last remote LDP ADJ    status rcvd: No fault     MPLS VC labels: local 1300, remote 1500     Group ID: local 8, remote 8     MTU: local 9216, remote 9216     Remote interface description: TO CE_R19 - VPWS   Sequencing: receive disabled, send disabled   Control Word: On (configured: autosense)   SSO Descriptor: 15.15.15.15/20, local label: 1300   Dataplane:     SSM segment/switch IDs: 28700/4121 (used), PWID: 1   VC statistics:     transit packet totals: receive 9, send 9     transit byte totals:   receive 846, send 1116     transit packet drops:  receive 0, seq error 0, send 0






PE_R13#sh mpls forwarding-table 15.15.15.15 Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    Label      Label      or Tunnel Id     Switched      interface              1323       221        15.15.15.15/32   0             Gi1.25     10.1.1.21




3. P2 receives packet, does a forwarding lookup, swaps label to 416 and send to next hop (P4).



P_R02#sh mpls forwarding-table labels 221 Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    Label      Label      or Tunnel Id     Switched      interface              221        416        15.15.15.15/32   105517286     Gi1.13     10.0.0.14




4. P4 receives the packet, does a forwarding lookup, pops label (uses PHP) and send to next hop (PE15).



P_R04#sh mpls forwarding-table labels 416 Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    Label      Label      or Tunnel Id     Switched      interface              416        Pop Label  15.15.15.15/32   104502664     Gi1.29     10.1.1.38




5. PE15 forward the frame out its AC, interface Gig3. 



PE_R15#sh mpls l2transport vc Local intf     Local circuit              Dest address    VC ID      Status -------------  -------------------------- --------------- ---------- ---------- Gi3            Ethernet:20                13.13.13.13     20         UP

References

https://nsrc.org/workshops/2015/apricot2015/raw-attachment/wiki/Track3MPLS/10_Deploying%20MPLS%20L2VPN.pdf

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/mp_l2_vpns/configuration/xe-3s/mp-l2-vpns-xe-3s-book/l2vpn-prot-based.html#GUID-2E9D5F84-6E3B-4A44-9A87-76E1A1619674