Tunnel Path MTU Discovery in a DMVPN Network: Use with Caution

Everyone knows one of the main issues in managing a DMVPN network is dealing with fragmentation.  Typically, when calculating the tunnel MTU and MSS, we are under the assumption that we are working with a network transport with normal MTU of 1500.  However I’ve been seeing more cases of Internet services being delivered to customers with lower than normal MTU.

Recently, I discovered a couple of sites in Europe where an ISP delivered a DSL service with a backend MTU of 1444.  This MTU was not disclosed to us and the MTU between the router and service provider edge device was set at 1500, giving the appearance of a normal working MTU.  This site performed normally for a good amount of time until the “tunnel path-mtu discovery” command (or tunnel PMTUD) was enabled on the tunnel interface (we discovered later it was added accidentally during a maintenance period). This is when I was alerted about a debilitating performance issue affecting that site.

At first I thought that this tunnel PMTUD feature couldn't have possibly cause such an issue. However after researching and testing it, it made more sense on why things broke.  I wanted to be able to share the experience in this post so others wouldn't get burned by it too.

As a small disclaimer, the following is based on my own experience, so I’m not saying this is a bad feature or to never use it.  But my advice is to use it with caution and enable it after fully understanding how the feature can affect your network.

Lab Environment

• Headend Router: Cisco 3945 (ISRG2) with IOS version 15.5(3)M2
• Branch Router: Cisco 2951 (ISRG2) with IOS version 15.5(3)M2
• Internet R01: Cisco 1801 with IOS version 12.4(15)T4
• Internet S01: Cisco 3560 with IOS version 12.2(55)SE5

Diagram & Topology

Technology Overview

When looking at what the Tunnel PMTUD feature is doing, it basically does two main things.

• Copy DF bit from the original IP packet to new GRE header
• Router listens for ICMP Unreachables with Fragmentation Needed and Don't Fragment Set (Type 3 Code 4)

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/interface/command/ir-cr-book/ir-t2.html#wp1581717163

On the other hand, when Tunnel PMTUD is not configured, the default behavior of GRE is to not copy the DF bit from the original IP header.  This basically allows fragmentation to occur when it encounters a path that has a MTU lower than 1500.

https://supportforums.cisco.com/document/50111/dynamic-multipoint-vpn-dmvpn-design-and-positioning-questions-and-answers-live#Q._What_is_the_best_way_to_solve_the_packet_fragmentation_issue_Many_articles_exist_with_different_options_like_-clear_df-bit-._What_is_recommended_for_DMVPN

Testing and Demonstration

So with these two bits of information in mind, let’s run through a few different scenarios.  This will show Tunnel PMTUD operation on a greater detail.  All scenarios will run through a network with a lower than normal MTU of 1300 in its path (see lab diagram).

1. Traffic sent from Headend LAN Switch to Branch over the DMVPN tunnel with Tunnel PMTUD disabled.
2. Traffic sent from Headend LAN Switch to Branch over the DMVPN tunnel with Tunnel PMTUD enabled and working.
3. Traffic sent from Headend LAN Switch to Branch over the DMVPN tunnel with Tunnel PMTUD enabled and not working (ICMP unreachable blocked).

Scenario 1 (Tunnel PMTUD Disabled)

In this scenario, we do not have any tunnel PMTUD configured.  The tunnel MTU is set at 1400 and the test traffic uses at 1400 byte packet sent from Headend to Branch.

• Ping with size of 1400 and DF bit set from LAN Switch to Branch network.  Packet makes it to destination without issue.

LAN_SWITCH#ping 10.100.100.254 size 1400 df Type escape sequence to abort. Sending 5, 1400-byte ICMP Echos to 10.100.100.254, timeout is 2 seconds: Packet sent with the DF bit set !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/9 ms

• Headend router’s tunnel MTU is 1400 and shows no fragmentation.

HEADEND#sh ip traffic | i Frag|frag   Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble          0 fragmented, 0 fragments, 0 couldn't fragment

• The Headend adds encryption and new GRE header with no DF bit.  Based on the Cisco IPSec Overhead Calculator tool, the packet size is now 1480 bytes.

• Packet reaches router INTERNET_R01 and fragments to pass the MTU path of 1300.  One fragment has a length of 1300 and other with 200. Note: Output below is showing only 1 of 5 packets for brevity.

INTERNET_R01# * Jul 27 16:12:17.770: IP: tableid=0, s=1.1.1.1 (Vlan100), d=3.3.3.1 (FastEthernet0), routed via FIB * Jul 27 16:12:17.770: IP: s=1.1.1.1 (Vlan100), d=3.3.3.1 (FastEthernet0), g=3.3.3.1, len 1480, forward, proto=50 * Jul 27 16:12:17.770: IP: s=1.1.1.1 (Vlan100), d=3.3.3.1 (FastEthernet0), len 1300, sending fragment * Jul 27 16:12:17.770:     IP Fragment, Ident = 40355, fragment offset = 0, proto=50 * Jul 27 16:12:17.770: IP: s=1.1.1.1 (Vlan100), d=3.3.3.1 (FastEthernet0), len 200, sending last fragment * Jul 27 16:12:17.770:     IP Fragment, Ident = 40355, fragment offset = 1280

• IP traffic statistic shows 5 packets has been fragmented.

INTERNET_R01#sh ip traffic | in Frag|frag   Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble          5 fragmented, 10 fragments, 0 couldn't fragment

• Branch router receives packets.

BRANCH# Jul 27 16:20:11.634: ICMP: echo reply sent, src 10.100.100.254, dst 10.1.1.253, topology BASE, dscp 0 topoid 0 Jul 27 16:20:11.638: ICMP: echo reply sent, src 10.100.100.254, dst 10.1.1.253, topology BASE, dscp 0 topoid 0 Jul 27 16:20:11.642: ICMP: echo reply sent, src 10.100.100.254, dst 10.1.1.253, topology BASE, dscp 0 topoid 0 Jul 27 16:20:11.650: ICMP: echo reply sent, src 10.100.100.254, dst 10.1.1.253, topology BASE, dscp 0 topoid 0 Jul 27 16:20:11.654: ICMP: echo reply sent, src 10.100.100.254, dst 10.1.1.253, topology BASE, dscp 0 topoid 0

• Branch router reassembles fragmented packets.

BRANCH#sh ip traffic | i Frag|frag   Frags: 5 reassembled, 0 timeouts, 0 couldn't reassemble          0 fragmented, 0 fragments, 0 couldn't fragment

Scenario 2 (Tunnel PMTUD Enabled)

In this scenario, tunnel PMTUD is configured.  The tunnel MTU is set at 1400 and the test traffic uses at 1400 byte packet sent from Headend to Branch.

• Ping with size of 1400 and DF bit set from LAN Switch to Branch network.  This fails.

LAN_SWITCH#p 10.100.100.254 si 1400 df Type escape sequence to abort. Sending 5, 1400-byte ICMP Echos to 10.100.100.254, timeout is 2 seconds: Packet sent with the DF bit set .M.M. Success rate is 0 percent (0/5)

• With Tunnel PMTUD enabled, the router is doing the following:
• Router sends an ICMP unreachable to itself to adjust MTU.
• Tunnel PMTUD process calculates the new MTU based on the configured policy and current MTU of 1400 and drops it down to 1334.
• Router sends an ICMP unreachable to the source telling it the new MTU is 1334.

HEADEND# Jul 27 06:09:22.507: ICMP: dst (3.3.3.1) frag. needed and DF set unreachable sent to 1.1.1.1 Jul 27 06:09:22.507: ICMP: dst (1.1.1.1) frag. needed and DF set unreachable rcv from 1.1.1.1 mtu:1362 Jul 27 06:09:22.507: Tunnel1: dest 3.3.3.1, received frag needed (mtu 1362), adjusting soft state MTU from 0 to 1334 Jul 27 06:09:22.507: Tunnel1: tunnel endpoint for transport dest 3.3.3.1, change MTU from 0 to 1334 Jul 27 06:09:24.510: ICMP: dst (10.100.100.254) frag. needed and DF set unreachable sent to 10.1.1.253mtu:1334 Jul 27 06:09:26.514: ICMP: dst (10.100.100.254) frag. needed and DF set unreachable sent to 10.1.1.253mtu:1334

• When trying to figure out why the tunnel PMTUD process is using a value of 1334, the IPSec Overhead Calculator was very useful tool.  We can now see why a payload size of 1334, after encryption and GRE, would yield a packet size is 1400.

• This begs the question on why PMTUD need to alter the MTU at this point?  It seems inefficient to do so because we are clearly sending data that will fit the tunnel MTU of 1400.  I can understand it if we sent a 1500 byte packet.

• Headend’s Tunnel interface sets new MTU of 1334 via the PMTUD feature.

HEADEND#sh int tun1 | in Path           Path MTU Discovery, ager 10 mins, min MTU 92   Path destination 3.3.3.1: MTU 1334, expires 00:09:23

• Headend’s IP statistics shows that it dropped the packet because it couldn’t fragment it.

HEADEND#sh ip traffic | i Frag|frag   Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble          0 fragmented, 0 fragments, 2 couldn't fragment

• The LAN Switch re-attempts to send traffic to the Branch site with the new MTU of 1334.  It still fails because we have a MTU of 1300 somewhere in the path.

LAN_SWITCH#ping 10.100.100.254 size 1334 df Type escape sequence to abort. Sending 5, 1334-byte ICMP Echos to 10.100.100.254, timeout is 2 seconds: Packet sent with the DF bit set .M.M. Success rate is 0 percent (0/5)

• Internet router received a 1400 byte packet and sends an ICMP unreachable to source because it can’t pass the interface with MTU of 1300.

INTERNET_R01# * Jul 27 22:59:31.764: IP: tableid=0, s=1.1.1.1 (Vlan100), d=3.3.3.1 (FastEthernet0), routed via FIB * Jul 27 22:59:31.764: IP: s=1.1.1.1 (Vlan100), d=3.3.3.1 (FastEthernet0), g=3.3.3.1, len 1400, forward, proto=50 * Jul 27 22:59:31.764: ICMP: dst (3.3.3.1) frag. needed and DF set unreachable sent to 1.1.1.1

• Internet router drops packet because it couldn’t fragment.

INTERNET_R01#sh ip traffic | i Frag|frag   Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble          0 fragmented, 0 fragments, 1 couldn't fragment

• This time the Headend’s Tunnel PMTU process is doing the following:
• Router receives an ICMP unreachable from the Internet router with the MTU of 1300.
• Router send an ICMP unreachable to itself to adjust the MTU.
• Tunnel PMTUD process calculates the new MTU based on MTU of 1250 and drops it down to 1222.  I’m unsure how it got the 1250 value.
• Router sends a new ICMP unreachable to the source telling it to drop the MTU to 1222.

HEADEND# Jul 27 23:24:01.752: ICMP: dst (1.1.1.1) frag. needed and DF set unreachable rcv from 1.0.0.1 mtu:1300 Jul 27 23:24:03.752: ICMP: dst (3.3.3.1) frag. needed and DF set unreachable sent to 1.1.1.1 Jul 27 23:24:03.752: ICMP: dst (1.1.1.1) frag. needed and DF set unreachable rcv from 1.1.1.1 mtu:1250 Jul 27 23:24:03.752: Tunnel1: dest 3.3.3.1, received frag needed (mtu 1250), adjusting soft state MTU from 0 to 1222 Jul 27 23:24:03.752: Tunnel1: tunnel endpoint for transport dest 3.3.3.1, change MTU from 0 to 1222 Jul 27 23:24:05.757: ICMP: dst (10.100.100.254) frag. needed and DF set unreachable sent to 10.1.1.253mtu:1222 Jul 27 23:24:07.761: ICMP: dst (10.100.100.254) frag. needed and DF set unreachable sent to 10.1.1.253mtu:1222

• Headend’s Tunnel interface sets new MTU of 1222 via the PMTUD feature.

HEADEND#sh int tun1 | In Pa          Path MTU Discovery, ager 10 mins, min MTU 92   Path destination 3.3.3.1: MTU 1222, expires 00:01:06

• When we use the IPSec Overhead Calculator with a payload size of 1222, after encryption and GRE, the packet size is 1288.  Now this will fit over the 1300 MTU link.

Note: The Tunnel PMTUD process must know the exact overhead calculations to be able to set the correct MTU.  As an example, I set the payload size 1 byte higher and the total is now bigger than 1300.

• The LAN Switch re-attempts to send traffic to the Branch site with the new MTU of 1222.  This time it succeeds.

LAN_SWITCH#p 10.100.100.254 size 1222 df Type escape sequence to abort. Sending 5, 1222-byte ICMP Echos to 10.100.100.254, timeout is 2 seconds: Packet sent with the DF bit set !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms

•  Internet router with MTU 1300 receives the post encrypted packet of 1288 and forwards it (showing only 1 of 5 packets).

INTERNET_R01# * Jul 27 23:59:53.800: IP: tableid=0, s=1.1.1.1 (Vlan100), d=3.3.3.1 (FastEthernet0), routed via FIB * Jul 27 23:59:53.800: IP: s=1.1.1.1 (Vlan100), d=3.3.3.1 (FastEthernet0), g=3.3.3.1, len 1288, forward

•  Branch router receives packet and replies.

BRANCH# Jul 27 23:55:37.862: ICMP: echo reply sent, src 10.100.100.254, dst 10.1.1.253, topology BASE, dscp 0 topoid 0 Jul 27 23:55:37.866: ICMP: echo reply sent, src 10.100.100.254, dst 10.1.1.253, topology BASE, dscp 0 topoid 0 Jul 27 23:55:37.870: ICMP: echo reply sent, src 10.100.100.254, dst 10.1.1.253, topology BASE, dscp 0 topoid 0 Jul 27 23:55:37.874: ICMP: echo reply sent, src 10.100.100.254, dst 10.1.1.253, topology BASE, dscp 0 topoid 0 Jul 27 23:55:37.882: ICMP: echo reply sent, src 10.100.100.254, dst 10.1.1.253, topology BASE, dscp 0 topoid 0

Scenario 3 (Tunnel PMTUD Enabled, ICMP Unreachable Blocked)

In this scenario, tunnel PMTUD is configured but ICMP unreachables are purposely blocked in the simulated Internet infrastructure.  The tunnel MTU is set at 1400 and the test traffic uses at 1400 byte packet sent from Headend to Branch.

To simulate a situation where the ICMP unreachables are blocked or lost within a network, an ACL is added to prevent any ICMP unreachables from reaching the Headend router from the Internet router.

ip access-list extended BLOCK_UNREACHABLES  deny   icmp any any unreachable log  permit ip any any

INET_SWITCH#conf t Enter configuration commands, one per line.  End with CNTL/Z. INET_SWITCH(config)#int f0/5                                 INET_SWITCH(config-if)#ip access-group BLOCK_UNREACHABLES in INET_SWITCH(config-if)#end

• With the block in place, let see what happens now.  We ping with size of 1400 and DF bit set from LAN Switch to Branch network.  It fails.

LAN_SWITCH#ping 10.100.100.254 size 1400 df Type escape sequence to abort. Sending 5, 1400-byte ICMP Echos to 10.100.100.254, timeout is 2 seconds: Packet sent with the DF bit set .M.M. Success rate is 0 percent (0/5)

• With Tunnel PMTUD still enabled, the router is doing the following:
• Router sends an ICMP unreachable to itself to adjust MTU.
• Tunnel PMTUD process calculates the new MTU based on the configured policy and current MTU of 1400 and drops it down to 1334.
• Router sends an ICMP unreachable to the source telling it the new MTU is 1334.

HEADEND# Jul 27 18:28:21.312: ICMP: dst (3.3.3.1) frag. needed and DF set unreachable sent to 1.1.1.1 Jul 27 18:28:21.312: ICMP: dst (1.1.1.1) frag. needed and DF set unreachable rcv from 1.1.1.1 mtu:1362 Jul 27 18:28:21.312: Tunnel1: dest 3.3.3.1, received frag needed (mtu 1362), adjusting soft state MTU from 0 to 1334 Jul 27 18:28:21.312: Tunnel1: tunnel endpoint for transport dest 3.3.3.1, change MTU from 0 to 1334 Jul 27 18:28:23.317: ICMP: dst (10.100.100.254) frag. needed and DF set unreachable sent to 10.1.1.253mtu:1334 Jul 27 18:28:25.322: ICMP: dst (10.100.100.254) frag. needed and DF set unreachable sent to 10.1.1.253mtu:1334

• Headend’s Tunnel interface sets new MTU of 1334 via the PMTUD feature.

HEADEND#sh int tun1 | in Path         Path MTU Discovery, ager 10 mins, min MTU 92   Path destination 3.3.3.1: MTU 1334, expires 00:07:13

• The LAN Switch re-attempts to send traffic to the Branch site with the new MTU of 1334.  It still fails because we have a MTU of 1300 somewhere in the path.

LAN_SWITCH#ping 10.100.100.254 si 1334 df Type escape sequence to abort. Sending 5, 1334-byte ICMP Echos to 10.100.100.254, timeout is 2 seconds: Packet sent with the DF bit set ..... Success rate is 0 percent (0/5)

• The Internet router receives a 1400 byte packet.  A MTU 1300 path exists so an ICMP unreachable is sent to the Headend router (1.1.1.1).

INTERNET_R01# * Jul 27 18:21:02.661: IP: tableid=0, s=1.1.1.1 (Vlan100), d=3.3.3.1 (FastEthernet0), routed via FIB * Jul 27 18:21:02.661: IP: s=1.1.1.1 (Vlan100), d=3.3.3.1 (FastEthernet0), g=3.3.3.1, len 1400, forward, proto=50 * Jul 27 18:21:02.661: ICMP: dst (3.3.3.1) frag. needed and DF set unreachable sent to 1.1.1.1

•  The Internet Switch blocks the ICMP unreachable per the ACL.

INET_SWITCH# * Jul 27 20:20:36.813: %SEC-6-IPACCESSLOGDP: list BLOCK_UNREACHABLES denied icmp 1.0.0.1 -> 1.1.1.1 (3/4), 1 packet

At this point the Headend router doesn’t see the ICMP unreachable message so it does nothing to react.  It doesn’t know to drop the MTU down to 1300 and continues to use the MTU value of 1334 as previously set by the Tunnel PMTU feature.  Without any further mechanisms to correct the MTU, the source will continue to send packets with a size of 1334 and will be dropped, which will create a serious network performance problem (due to TCP retransmits etc.)  This scenario demonstrates firsthand that receiving the ICMP is key for this feature to work correctly.

Conclusion

Besides the fact that this tunnel PMTUD feature seems to be a bit chatty and inefficient at times, the main caveat I would like to mention here is that for this to work successfully in the practical sense, we have to rely on our Internet service providers to generate and forward ICMP unreachables.  In my opinion, that’s a tall order.  We frequently hear of ISPs blocking ICMP’s for security reasons and with most common deployments of DMVPN (or any IPSec VPN network) being created over the Internet, I think its a bad idea to use it in the first place.

The best practice, in my opinion, is to lower tunnel MTU and re-adjust MSS for sites that have an Internet service with a lower than normal MTU.  It’s a quick and easy fix.  Otherwise, if there a specific need for this feature, you just have to be aware of your network environment and ensure the ICMP unreachable prerequisite is met.

Cisco IOS Interoperability Issues with ZBFW, NVI NAT and FVRF’s

Recently I had a requirement to update our firewall feature set on our Cisco ISR G2 routers from classic firewall (CBAC) to Zone Based Firewall (ZBFW).  As I was preparing for this and tested this in my lab, I ran into a strange issue and wanted to share my experience.

The problem can only be explained as an interoperability issue between Zone Based Firewall, while using Front door VRF’s (FVRF) and NAT virtual interface (NVI).  From what I can tell, this issue seems to be occurring on only Cisco ISR G2 platforms (800 – 3900) that runs IOS classic.  The newer generation ISR 4000 series that run IOS XE do not experience this problem.  As of this writing, I still run into this issue as of IOS version 15.5(3)M2.

Cisco seems to know about this issue too. There is an existing bug ID that closely resemble this behavior (see link below).  However, when I opened a TAC case and went through the typical rigmarole, I really didn’t get anywhere.  The case even got escalated to their security team and their only advice was to simply implement one of the workarounds.

https://quickview.cloudapps.cisco.com/quickview/bug/CSCsh12490

Workaround:

Remove Zone-Based Policy Firewall and apply Classic Firewall

-or-

Remove NAT-Virtual Interface configuration and apply Inside/Outside NAT.

The main issue for me was that both workarounds couldn’t apply in my situation.  The first workaround was not feasible since I has a hard requirement to run ZBFW.  The second option was also a no go since NAT Virtual Interface was the only NAT method that seemed to interoperate with FVRF’s.  For some reason, the older style Inside/Outside NAT didn’t seem to work with FVRF’s.  I’ve tried all configuration permutations of NVI, and trust me it just didn’t work.  So as you can tell, I was in between a rock and a hard place and had to figure out a solution.

The issue with ZBFW was strange.  It definitely wasn’t behaving normally.  When the two zones were setup (Inside and Outside) and their corresponding policies and zone-pairs were applied (e.g., inside to outside and outside to inside), the inspection failed as soon as I applied a zone pair that inspected traffic from outside to self.   It only failed when I applied that zone pair.  Other zone pairs in the direction of “self to outside” or “outside to inside” did not affect the test traffic.  If I made special provisions to exclude or “pass” the test traffic (say ICMP) in the outside to self policy, it would work.  So that proved ZBFW inspected traffic it shouldn’t have.  Somewhere in the router, the return traffic wasn’t matching the same policy as it passed through initially.  It thought that traffic was meant for the router itself, hence the outside to self policy dropped it.  

Lab Diagram

Test Protocol

Ping and Telnet from LAN_SWITCH’s VL200 with IP 10.1.1.253 to INET_SWITCH IP of 4.2.2.2.

Configuration

Router

version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname ROUTER
!
boot-start-marker
boot-end-marker
!
!
logging buffered 16000
enable secret 5 <removed>
!
no aaa new-model
ethernet lmi ce
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
ip vrf INET_PUBLIC
!
ip vrf MGMT
!
!
!
!
ip domain lookup source-interface GigabitEthernet0/3
ip domain name domain.local
ip name-server vrf MGMT 8.8.8.8
ip name-server 8.8.8.8
ip cef
no ipv6 cef
!
parameter-map type inspect global
 log dropped-packets enable
 max-incomplete low 18000
 max-incomplete high 20000
parameter-map type inspect tcp
 audit-trail on
 alert on
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
license udi pid C3900-SPE250/K9 sn <removed>
!
!         
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
!
class-map type inspect match-any CMAP_FW_PASS_OUTSIDE_TO_SELF
 match access-group name ACL_DHCP_IN
class-map type inspect match-any CMAP_FW_PASS_SELF_TO_OUTSIDE
 match access-group name ACL_DHCP_OUT
class-map type inspect match-any CMAP_FW_INSPECT_INSIDE_TO_OUTSIDE
 match access-group name ACL_LAN_INSIDE_TO_OUTSIDE
class-map type inspect match-any CMAP_FW_INSPECT_OUTSIDE_TO_SELF
 match access-group name ACL_SSH_IN
 match access-group name ACL_ICMP_IN
class-map type inspect match-any CMAP_FW_INSPECT_SELF_TO_OUTSIDE
 match access-group name ACL_NTP_OUT
 match access-group name ACL_ICMP_OUT
 match access-group name ACL_HTTP_OUT
 match access-group name ACL_DNS_OUT
class-map type inspect match-any CMAP_FW_INSPECT_OUTSIDE_TO_INSIDE
 match access-group name ACL_LAN_OUTSIDE_TO_INSIDE
!
policy-map type inspect PMAP_FW_INSIDE_TO_OUTSIDE
 class type inspect CMAP_FW_INSPECT_INSIDE_TO_OUTSIDE
  inspect 
 class class-default
  drop log
policy-map type inspect PMAP_FW_SELF_TO_OUTSIDE
 class type inspect CMAP_FW_INSPECT_SELF_TO_OUTSIDE
  inspect 
 class type inspect CMAP_FW_PASS_SELF_TO_OUTSIDE
  pass
 class class-default
  drop log
policy-map type inspect PMAP_FW_OUTSIDE_TO_INSIDE
 class type inspect CMAP_FW_INSPECT_OUTSIDE_TO_INSIDE
  inspect 
 class class-default
  drop log
policy-map type inspect PMAP_FW_OUTSIDE_TO_SELF
 class type inspect CMAP_FW_INSPECT_OUTSIDE_TO_SELF
  inspect 
 class type inspect CMAP_FW_PASS_OUTSIDE_TO_SELF
  pass
 class class-default
  drop log
!
zone security INSIDE
zone security OUTSIDE
zone-pair security ZPAIR_FW_INSIDE_TO_OUTSIDE source INSIDE destination OUTSIDE
 service-policy type inspect PMAP_FW_INSIDE_TO_OUTSIDE
zone-pair security ZPAIR_FW_SELF_TO_OUTSIDE source self destination OUTSIDE
 service-policy type inspect PMAP_FW_SELF_TO_OUTSIDE
zone-pair security ZPAIR_FW_OUTSIDE_TO_INSIDE source OUTSIDE destination INSIDE
 service-policy type inspect PMAP_FW_OUTSIDE_TO_INSIDE
zone-pair security ZPAIR_FW_OUTSIDE_TO_SELF source OUTSIDE destination self
 service-policy type inspect PMAP_FW_OUTSIDE_TO_SELF
! 
!
!
interface GigabitEthernet0/0
 description INTERNET
 ip vrf forwarding INET_PUBLIC
 ip address 1.1.1.1 255.255.255.0
 ip nat outside
 ip virtual-reassembly in
 zone-member security OUTSIDE
 ip policy route-map RMAP_INTERNET_RETURN
 duplex auto
 speed auto
!
interface GigabitEthernet0/1
 description LAN
 ip address 10.1.1.254 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 zone-member security INSIDE
 duplex auto
 speed auto
!
interface GigabitEthernet0/2
 no ip address
 shutdown
 duplex auto
 speed auto
!
interface GigabitEthernet0/3
 ip vrf forwarding MGMT
 ip address 192.168.1.200 255.255.255.0
 duplex auto
 speed auto
 no cdp enable
!
!
!
!
no ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip dns view vrf MGMT default
ip nat inside source route-map RMAP_NAT_POLICY interface GigabitEthernet0/0 overload
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0 1.1.1.254 name DEFAULT
ip route vrf INET_PUBLIC 0.0.0.0 0.0.0.0 GigabitEthernet0/0 1.1.1.254 name INET_DEFAULT
ip route vrf MGMT 0.0.0.0 0.0.0.0 192.168.1.254 name MGMT_DEFAULT
!
ip access-list extended ACL_DHCP_IN
 permit udp any eq bootpc any eq bootps
 permit udp any eq bootps any eq bootpc
ip access-list extended ACL_DHCP_OUT
 permit udp any eq bootps any eq bootpc
 permit udp any eq bootpc any eq bootps
ip access-list extended ACL_DNS_OUT
 permit udp any any eq domain
ip access-list extended ACL_HTTP_OUT
 permit tcp any any eq www
ip access-list extended ACL_ICMP_IN
 permit icmp any any echo
 permit icmp any any echo-reply
 permit icmp any any ttl-exceeded
 permit icmp any any port-unreachable
ip access-list extended ACL_ICMP_OUT
 permit icmp any any
ip access-list extended ACL_INTERNET_RETURN
 permit ip any 10.0.0.0 0.255.255.255
ip access-list extended ACL_LAN_INSIDE_TO_OUTSIDE
 permit ip 10.0.0.0 0.255.255.255 any
ip access-list extended ACL_LAN_OUTSIDE_TO_INSIDE
 deny   ip any any
ip access-list extended ACL_NAT
 permit ip 10.0.0.0 0.255.255.255 any
ip access-list extended ACL_NTP_OUT
 permit udp any any eq ntp
ip access-list extended ACL_SSH_OUT
 permit tcp any any eq 22
!
!
route-map RMAP_INTERNET_RETURN permit 10
 match ip address ACL_INTERNET_RETURN
 set global
!
route-map RMAP_NAT_POLICY permit 10
 description ROUTE-MAP FOR NAT
 match ip address ACL_NAT
!
!
!
control-plane
!
!
!
line con 0
 logging synchronous
line aux 0
line vty 0 4
 exec-timeout 60 0
 logging synchronous
 login local
 transport input telnet ssh
!
scheduler allocate 20000 1000
!
end

Test Results

Logs shows ICMP drop for unknown reasons and telnet drop due to policy reasons.   The policy drop doesn’t make sense because traffic should have only been inspected by the ZPAIR_FW_INSIDE_TO_OUTSIDE policy.

May 30 17:37:14.956: %FW-6-DROP_PKT: Dropping icmp session 4.2.2.2:0 1.1.1.1:0 on zone-pair ZPAIR_FW_OUTSIDE_TO_SELF class CMAP_FW_INSPECT_OUTSIDE_TO_SELF due to  Internal Error with ip ident 3918 May 30 17:39:37.870: %FW-6-DROP_PKT: Dropping tcp session 4.2.2.2:23 1.1.1.1:22660 on zone-pair ZPAIR_FW_OUTSIDE_TO_SELF class class-default due to  DROP action found in policy-map with ip ident 34337

ZBFW Sessions (Broken)

ROUTER#show policy-map type inspect zone-pair sessions policy exists on zp ZPAIR_FW_INSIDE_TO_OUTSIDE   Zone-pair: ZPAIR_FW_INSIDE_TO_OUTSIDE   Service-policy inspect : PMAP_FW_INSIDE_TO_OUTSIDE     Class-map: CMAP_FW_INSPECT_INSIDE_TO_OUTSIDE (match-any)        Match: access-group name ACL_LAN_INSIDE_TO_OUTSIDE         13 packets, 816 bytes         30 second rate 0 bps    Inspect       Number of Half-open Sessions = 2       Half-open Sessions         Session 264AE8A8 (10.1.1.253:28673)=>(4.2.2.2:23) tcp SIS_OPENING/TCP_SYNSENT           Created 00:00:12, Last heard 00:00:12           Bytes sent (initiator:responder) [0:0]         Session 264AE540 (10.1.1.253:8)=>(4.2.2.2:0) icmp SIS_OPENING           Created 00:00:05, Last heard 00:00:01           ECHO request           Bytes sent (initiator:responder) [216:0]     Class-map: class-default (match-any)        Match: any       Drop         208980 packets, 8818290 bytes ..snip.. policy exists on zp ZPAIR_FW_OUTSIDE_TO_SELF   Zone-pair: ZPAIR_FW_OUTSIDE_TO_SELF   Service-policy inspect : PMAP_FW_OUTSIDE_TO_SELF     Class-map: CMAP_FW_INSPECT_OUTSIDE_TO_SELF (match-any)        Match: access-group name ACL_SSH_IN         0 packets, 0 bytes         30 second rate 0 bps       Match: access-group name ACL_ICMP_IN         0 packets, 0 bytes         30 second rate 0 bps    Inspect     Class-map: CMAP_FW_PASS_OUTSIDE_TO_SELF (match-any)        Match: access-group name ACL_DHCP_IN         0 packets, 0 bytes         30 second rate 0 bps       Pass         430665 packets, 49375220 bytes     Class-map: class-default (match-any)        Match: any       Drop         2 packets, 72 bytes

During the course of my testing, I wanted to try the second workaround to see if reconfiguring the NAT to use Inside/Outside would make a difference.  As I said before, the classic NAT feature didn’t seem to interoperate with FVRF’s so that could have been a potential culprit to all this.  When I changed over the NAT and ran tests again, I noticed ZBFW wasn’t log the drops as it was before when NVI was configured.  That led me to believe the NAT had something to do with this.

After many more hours researching and testing, I was able to address the NAT issue.  For some weird reason, the router wasn’t able to send the NAT’ed traffic back to the global route table.  Again, I tried all permutations of the NAT command, even though the NAT command was VRF aware.  The solution that worked was to create a policy route that set the post NAT traffic back to the global route table.

//Create ACL and Route-Map for Policy Routing ip access-list extended ACL_INTERNET_RETURN  permit ip any 10.0.0.0 0.255.255.255 route-map RMAP_INTERNET_RETURN permit 10  match ip address ACL_INTERNET_RETURN  set global //Apply Policy Route to Outside Interface interface GigabitEthernet0/0  ip policy route-map RMAP_INTERNET_RETURN

Once the NAT issue was resolved, I reapplied ZBFW and that seemed to work as well.  I guess the issue wasn’t ZBWF at all but rather an internal routing issue between a VRF and global table.  Cisco never told me during the course of the TAC case, nor was it ever documented so I wanted to share this solution.  Hopefully others can benefit from this.

Debug of Policy Routing

Jun  7 21:49:56.910: IP: route map RMAP_INTERNET_RETURN, item 10, permit Jun  7 21:49:56.910: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253, len 40, set global Jun  7 21:49:56.910: IP: set global to outidb 0x23F82540 Jun  7 21:49:56.910: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253 (GigabitEthernet0/1), len 40, policy routed Jun  7 21:49:56.910: IP: GigabitEthernet0/0 to GigabitEthernet0/1 10.1.1.253 Jun  7 21:49:56.912: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253, len 40, FIB policy match Jun  7 21:49:56.912: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253, len 40, PBR Counted Jun  7 21:49:56.912: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253, len 40, FIB policy routed set global

ZBFW Sessions (Working)

ROUTER#show policy-map type inspect zone-pair sessions policy exists on zp ZPAIR_FW_INSIDE_TO_OUTSIDE   Zone-pair: ZPAIR_FW_INSIDE_TO_OUTSIDE   Service-policy inspect : PMAP_FW_INSIDE_TO_OUTSIDE     Class-map: CMAP_FW_INSPECT_INSIDE_TO_OUTSIDE (match-any)        Match: access-group name ACL_LAN_INSIDE_TO_OUTSIDE         9 packets, 608 bytes         30 second rate 0 bps    Inspect       Number of Established Sessions = 2       Established Sessions         Session 264AE8A8 (10.1.1.253:59905)=>(4.2.2.2:23) tcp SIS_OPEN/TCP_ESTAB           Created 00:00:17, Last heard 00:00:15           Bytes sent (initiator:responder) [79:140]         Session 264AEF78 (10.1.1.253:8)=>(4.2.2.2:0) icmp SIS_OPEN           Created 00:00:05, Last heard 00:00:00           ECHO request           Bytes sent (initiator:responder) [102744:102672]     Class-map: class-default (match-any)        Match: any       Drop         208926 packets, 8816202 bytes ..snip.. policy exists on zp ZPAIR_FW_OUTSIDE_TO_SELF   Zone-pair: ZPAIR_FW_OUTSIDE_TO_SELF   Service-policy inspect : PMAP_FW_OUTSIDE_TO_SELF     Class-map: CMAP_FW_INSPECT_OUTSIDE_TO_SELF (match-any)        Match: access-group name ACL_SSH_IN         0 packets, 0 bytes         30 second rate 0 bps       Match: access-group name ACL_ICMP_IN         0 packets, 0 bytes         30 second rate 0 bps    Inspect     Class-map: CMAP_FW_PASS_OUTSIDE_TO_SELF (match-any)        Match: access-group name ACL_DHCP_IN         0 packets, 0 bytes         30 second rate 0 bps       Pass         430549 packets, 49362148 bytes     Class-map: class-default (match-any)        Match: any       Drop         2 packets, 72 bytes