Cisco IOS Interoperability Issues with ZBFW, NVI NAT and FVRF’s

Recently I had a requirement to update our firewall feature set on our Cisco ISR G2 routers from classic firewall (CBAC) to Zone Based Firewall (ZBFW).  As I was preparing for this and tested this in my lab, I ran into a strange issue and wanted to share my experience.

The problem can only be explained as an interoperability issue between Zone Based Firewall, while using Front door VRF’s (FVRF) and NAT virtual interface (NVI).  From what I can tell, this issue seems to be occurring on only Cisco ISR G2 platforms (800 – 3900) that runs IOS classic.  The newer generation ISR 4000 series that run IOS XE do not experience this problem.  As of this writing, I still run into this issue as of IOS version 15.5(3)M2.

Cisco seems to know about this issue too. There is an existing bug ID that closely resemble this behavior (see link below).  However, when I opened a TAC case and went through the typical rigmarole, I really didn’t get anywhere.  The case even got escalated to their security team and their only advice was to simply implement one of the workarounds.

https://quickview.cloudapps.cisco.com/quickview/bug/CSCsh12490

Workaround:

Remove Zone-Based Policy Firewall and apply Classic Firewall

-or-

Remove NAT-Virtual Interface configuration and apply Inside/Outside NAT.

The main issue for me was that both workarounds couldn’t apply in my situation.  The first workaround was not feasible since I has a hard requirement to run ZBFW.  The second option was also a no go since NAT Virtual Interface was the only NAT method that seemed to interoperate with FVRF’s.  For some reason, the older style Inside/Outside NAT didn’t seem to work with FVRF’s.  I’ve tried all configuration permutations of NVI, and trust me it just didn’t work.  So as you can tell, I was in between a rock and a hard place and had to figure out a solution.

The issue with ZBFW was strange.  It definitely wasn’t behaving normally.  When the two zones were setup (Inside and Outside) and their corresponding policies and zone-pairs were applied (e.g., inside to outside and outside to inside), the inspection failed as soon as I applied a zone pair that inspected traffic from outside to self.   It only failed when I applied that zone pair.  Other zone pairs in the direction of “self to outside” or “outside to inside” did not affect the test traffic.  If I made special provisions to exclude or “pass” the test traffic (say ICMP) in the outside to self policy, it would work.  So that proved ZBFW inspected traffic it shouldn’t have.  Somewhere in the router, the return traffic wasn’t matching the same policy as it passed through initially.  It thought that traffic was meant for the router itself, hence the outside to self policy dropped it.  

Lab Diagram

Test Protocol

Ping and Telnet from LAN_SWITCH’s VL200 with IP 10.1.1.253 to INET_SWITCH IP of 4.2.2.2.

Configuration

Router

version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname ROUTER
!
boot-start-marker
boot-end-marker
!
!
logging buffered 16000
enable secret 5 <removed>
!
no aaa new-model
ethernet lmi ce
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
ip vrf INET_PUBLIC
!
ip vrf MGMT
!
!
!
!
ip domain lookup source-interface GigabitEthernet0/3
ip domain name domain.local
ip name-server vrf MGMT 8.8.8.8
ip name-server 8.8.8.8
ip cef
no ipv6 cef
!
parameter-map type inspect global
 log dropped-packets enable
 max-incomplete low 18000
 max-incomplete high 20000
parameter-map type inspect tcp
 audit-trail on
 alert on
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
license udi pid C3900-SPE250/K9 sn <removed>
!
!         
username admin secret 5 <removed>
!
redundancy
!
!
!
!
!
!
class-map type inspect match-any CMAP_FW_PASS_OUTSIDE_TO_SELF
 match access-group name ACL_DHCP_IN
class-map type inspect match-any CMAP_FW_PASS_SELF_TO_OUTSIDE
 match access-group name ACL_DHCP_OUT
class-map type inspect match-any CMAP_FW_INSPECT_INSIDE_TO_OUTSIDE
 match access-group name ACL_LAN_INSIDE_TO_OUTSIDE
class-map type inspect match-any CMAP_FW_INSPECT_OUTSIDE_TO_SELF
 match access-group name ACL_SSH_IN
 match access-group name ACL_ICMP_IN
class-map type inspect match-any CMAP_FW_INSPECT_SELF_TO_OUTSIDE
 match access-group name ACL_NTP_OUT
 match access-group name ACL_ICMP_OUT
 match access-group name ACL_HTTP_OUT
 match access-group name ACL_DNS_OUT
class-map type inspect match-any CMAP_FW_INSPECT_OUTSIDE_TO_INSIDE
 match access-group name ACL_LAN_OUTSIDE_TO_INSIDE
!
policy-map type inspect PMAP_FW_INSIDE_TO_OUTSIDE
 class type inspect CMAP_FW_INSPECT_INSIDE_TO_OUTSIDE
  inspect 
 class class-default
  drop log
policy-map type inspect PMAP_FW_SELF_TO_OUTSIDE
 class type inspect CMAP_FW_INSPECT_SELF_TO_OUTSIDE
  inspect 
 class type inspect CMAP_FW_PASS_SELF_TO_OUTSIDE
  pass
 class class-default
  drop log
policy-map type inspect PMAP_FW_OUTSIDE_TO_INSIDE
 class type inspect CMAP_FW_INSPECT_OUTSIDE_TO_INSIDE
  inspect 
 class class-default
  drop log
policy-map type inspect PMAP_FW_OUTSIDE_TO_SELF
 class type inspect CMAP_FW_INSPECT_OUTSIDE_TO_SELF
  inspect 
 class type inspect CMAP_FW_PASS_OUTSIDE_TO_SELF
  pass
 class class-default
  drop log
!
zone security INSIDE
zone security OUTSIDE
zone-pair security ZPAIR_FW_INSIDE_TO_OUTSIDE source INSIDE destination OUTSIDE
 service-policy type inspect PMAP_FW_INSIDE_TO_OUTSIDE
zone-pair security ZPAIR_FW_SELF_TO_OUTSIDE source self destination OUTSIDE
 service-policy type inspect PMAP_FW_SELF_TO_OUTSIDE
zone-pair security ZPAIR_FW_OUTSIDE_TO_INSIDE source OUTSIDE destination INSIDE
 service-policy type inspect PMAP_FW_OUTSIDE_TO_INSIDE
zone-pair security ZPAIR_FW_OUTSIDE_TO_SELF source OUTSIDE destination self
 service-policy type inspect PMAP_FW_OUTSIDE_TO_SELF
! 
!
!
interface GigabitEthernet0/0
 description INTERNET
 ip vrf forwarding INET_PUBLIC
 ip address 1.1.1.1 255.255.255.0
 ip nat outside
 ip virtual-reassembly in
 zone-member security OUTSIDE
 ip policy route-map RMAP_INTERNET_RETURN
 duplex auto
 speed auto
!
interface GigabitEthernet0/1
 description LAN
 ip address 10.1.1.254 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 zone-member security INSIDE
 duplex auto
 speed auto
!
interface GigabitEthernet0/2
 no ip address
 shutdown
 duplex auto
 speed auto
!
interface GigabitEthernet0/3
 ip vrf forwarding MGMT
 ip address 192.168.1.200 255.255.255.0
 duplex auto
 speed auto
 no cdp enable
!
!
!
!
no ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip dns view vrf MGMT default
ip nat inside source route-map RMAP_NAT_POLICY interface GigabitEthernet0/0 overload
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0 1.1.1.254 name DEFAULT
ip route vrf INET_PUBLIC 0.0.0.0 0.0.0.0 GigabitEthernet0/0 1.1.1.254 name INET_DEFAULT
ip route vrf MGMT 0.0.0.0 0.0.0.0 192.168.1.254 name MGMT_DEFAULT
!
ip access-list extended ACL_DHCP_IN
 permit udp any eq bootpc any eq bootps
 permit udp any eq bootps any eq bootpc
ip access-list extended ACL_DHCP_OUT
 permit udp any eq bootps any eq bootpc
 permit udp any eq bootpc any eq bootps
ip access-list extended ACL_DNS_OUT
 permit udp any any eq domain
ip access-list extended ACL_HTTP_OUT
 permit tcp any any eq www
ip access-list extended ACL_ICMP_IN
 permit icmp any any echo
 permit icmp any any echo-reply
 permit icmp any any ttl-exceeded
 permit icmp any any port-unreachable
ip access-list extended ACL_ICMP_OUT
 permit icmp any any
ip access-list extended ACL_INTERNET_RETURN
 permit ip any 10.0.0.0 0.255.255.255
ip access-list extended ACL_LAN_INSIDE_TO_OUTSIDE
 permit ip 10.0.0.0 0.255.255.255 any
ip access-list extended ACL_LAN_OUTSIDE_TO_INSIDE
 deny   ip any any
ip access-list extended ACL_NAT
 permit ip 10.0.0.0 0.255.255.255 any
ip access-list extended ACL_NTP_OUT
 permit udp any any eq ntp
ip access-list extended ACL_SSH_OUT
 permit tcp any any eq 22
!
!
route-map RMAP_INTERNET_RETURN permit 10
 match ip address ACL_INTERNET_RETURN
 set global
!
route-map RMAP_NAT_POLICY permit 10
 description ROUTE-MAP FOR NAT
 match ip address ACL_NAT
!
!
!
control-plane
!
!
!
line con 0
 logging synchronous
line aux 0
line vty 0 4
 exec-timeout 60 0
 logging synchronous
 login local
 transport input telnet ssh
!
scheduler allocate 20000 1000
!
end

Test Results

Logs shows ICMP drop for unknown reasons and telnet drop due to policy reasons.   The policy drop doesn’t make sense because traffic should have only been inspected by the ZPAIR_FW_INSIDE_TO_OUTSIDE policy.

May 30 17:37:14.956: %FW-6-DROP_PKT: Dropping icmp session 4.2.2.2:0 1.1.1.1:0 on zone-pair ZPAIR_FW_OUTSIDE_TO_SELF class CMAP_FW_INSPECT_OUTSIDE_TO_SELF due to  Internal Error with ip ident 3918 May 30 17:39:37.870: %FW-6-DROP_PKT: Dropping tcp session 4.2.2.2:23 1.1.1.1:22660 on zone-pair ZPAIR_FW_OUTSIDE_TO_SELF class class-default due to  DROP action found in policy-map with ip ident 34337

ZBFW Sessions (Broken)

ROUTER#show policy-map type inspect zone-pair sessions policy exists on zp ZPAIR_FW_INSIDE_TO_OUTSIDE   Zone-pair: ZPAIR_FW_INSIDE_TO_OUTSIDE   Service-policy inspect : PMAP_FW_INSIDE_TO_OUTSIDE     Class-map: CMAP_FW_INSPECT_INSIDE_TO_OUTSIDE (match-any)        Match: access-group name ACL_LAN_INSIDE_TO_OUTSIDE         13 packets, 816 bytes         30 second rate 0 bps    Inspect       Number of Half-open Sessions = 2       Half-open Sessions         Session 264AE8A8 (10.1.1.253:28673)=>(4.2.2.2:23) tcp SIS_OPENING/TCP_SYNSENT           Created 00:00:12, Last heard 00:00:12           Bytes sent (initiator:responder) [0:0]         Session 264AE540 (10.1.1.253:8)=>(4.2.2.2:0) icmp SIS_OPENING           Created 00:00:05, Last heard 00:00:01           ECHO request           Bytes sent (initiator:responder) [216:0]     Class-map: class-default (match-any)        Match: any       Drop         208980 packets, 8818290 bytes ..snip.. policy exists on zp ZPAIR_FW_OUTSIDE_TO_SELF   Zone-pair: ZPAIR_FW_OUTSIDE_TO_SELF   Service-policy inspect : PMAP_FW_OUTSIDE_TO_SELF     Class-map: CMAP_FW_INSPECT_OUTSIDE_TO_SELF (match-any)        Match: access-group name ACL_SSH_IN         0 packets, 0 bytes         30 second rate 0 bps       Match: access-group name ACL_ICMP_IN         0 packets, 0 bytes         30 second rate 0 bps    Inspect     Class-map: CMAP_FW_PASS_OUTSIDE_TO_SELF (match-any)        Match: access-group name ACL_DHCP_IN         0 packets, 0 bytes         30 second rate 0 bps       Pass         430665 packets, 49375220 bytes     Class-map: class-default (match-any)        Match: any       Drop         2 packets, 72 bytes

During the course of my testing, I wanted to try the second workaround to see if reconfiguring the NAT to use Inside/Outside would make a difference.  As I said before, the classic NAT feature didn’t seem to interoperate with FVRF’s so that could have been a potential culprit to all this.  When I changed over the NAT and ran tests again, I noticed ZBFW wasn’t log the drops as it was before when NVI was configured.  That led me to believe the NAT had something to do with this.

After many more hours researching and testing, I was able to address the NAT issue.  For some weird reason, the router wasn’t able to send the NAT’ed traffic back to the global route table.  Again, I tried all permutations of the NAT command, even though the NAT command was VRF aware.  The solution that worked was to create a policy route that set the post NAT traffic back to the global route table.

//Create ACL and Route-Map for Policy Routing ip access-list extended ACL_INTERNET_RETURN  permit ip any 10.0.0.0 0.255.255.255 route-map RMAP_INTERNET_RETURN permit 10  match ip address ACL_INTERNET_RETURN  set global //Apply Policy Route to Outside Interface interface GigabitEthernet0/0  ip policy route-map RMAP_INTERNET_RETURN

Once the NAT issue was resolved, I reapplied ZBFW and that seemed to work as well.  I guess the issue wasn’t ZBWF at all but rather an internal routing issue between a VRF and global table.  Cisco never told me during the course of the TAC case, nor was it ever documented so I wanted to share this solution.  Hopefully others can benefit from this.

Debug of Policy Routing

Jun  7 21:49:56.910: IP: route map RMAP_INTERNET_RETURN, item 10, permit Jun  7 21:49:56.910: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253, len 40, set global Jun  7 21:49:56.910: IP: set global to outidb 0x23F82540 Jun  7 21:49:56.910: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253 (GigabitEthernet0/1), len 40, policy routed Jun  7 21:49:56.910: IP: GigabitEthernet0/0 to GigabitEthernet0/1 10.1.1.253 Jun  7 21:49:56.912: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253, len 40, FIB policy match Jun  7 21:49:56.912: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253, len 40, PBR Counted Jun  7 21:49:56.912: IP: s=4.2.2.2 (GigabitEthernet0/0), d=10.1.1.253, len 40, FIB policy routed set global

ZBFW Sessions (Working)

ROUTER#show policy-map type inspect zone-pair sessions policy exists on zp ZPAIR_FW_INSIDE_TO_OUTSIDE   Zone-pair: ZPAIR_FW_INSIDE_TO_OUTSIDE   Service-policy inspect : PMAP_FW_INSIDE_TO_OUTSIDE     Class-map: CMAP_FW_INSPECT_INSIDE_TO_OUTSIDE (match-any)        Match: access-group name ACL_LAN_INSIDE_TO_OUTSIDE         9 packets, 608 bytes         30 second rate 0 bps    Inspect       Number of Established Sessions = 2       Established Sessions         Session 264AE8A8 (10.1.1.253:59905)=>(4.2.2.2:23) tcp SIS_OPEN/TCP_ESTAB           Created 00:00:17, Last heard 00:00:15           Bytes sent (initiator:responder) [79:140]         Session 264AEF78 (10.1.1.253:8)=>(4.2.2.2:0) icmp SIS_OPEN           Created 00:00:05, Last heard 00:00:00           ECHO request           Bytes sent (initiator:responder) [102744:102672]     Class-map: class-default (match-any)        Match: any       Drop         208926 packets, 8816202 bytes ..snip.. policy exists on zp ZPAIR_FW_OUTSIDE_TO_SELF   Zone-pair: ZPAIR_FW_OUTSIDE_TO_SELF   Service-policy inspect : PMAP_FW_OUTSIDE_TO_SELF     Class-map: CMAP_FW_INSPECT_OUTSIDE_TO_SELF (match-any)        Match: access-group name ACL_SSH_IN         0 packets, 0 bytes         30 second rate 0 bps       Match: access-group name ACL_ICMP_IN         0 packets, 0 bytes         30 second rate 0 bps    Inspect     Class-map: CMAP_FW_PASS_OUTSIDE_TO_SELF (match-any)        Match: access-group name ACL_DHCP_IN         0 packets, 0 bytes         30 second rate 0 bps       Pass         430549 packets, 49362148 bytes     Class-map: class-default (match-any)        Match: any       Drop         2 packets, 72 bytes