I have a guest post today that I am happy to present to you. The following content was created by a fellow F5 DevCentral MVP member named Chetan Bhatt who works for NBC Universal as a Senior Network Engineer.

In an article that TheF5Guy.com posted back in September 22, 2010 I explained a method about creating a F5-Cisco VLAN to VLAN Bypass for Cisco IOS gear.  With the introduction to Cisco Nexus and vPC (Virtual Port Channel) technology the configurations to make the VLAN-to-VLAN bypass would need to be updated.  (Previous article can be found here)

So now we have the following similar scenario with the added twist of Nexus and vPC.

I have a pair of F5 ADC in an Internet DMZ, where nodes behind the load balancer need to access NAS system(s) on a VLAN located on a separate VLAN that is not behind the load balancer. The problem is that in my current design I have to route through the F5 Load balancer to access the NAS system(s).  Unfortunately the amount of bandwidth it takes supersedes the F5 ADC’s total throughput.  I would like to bypass this without adding extra network cards or recreating a new VLAN and would like preserve the IP addresses as much as possible.  Also the F5 ADC is sitting on a network design that participates in vPC within Cisco Nexus Datacenter gear.

Based on this description above you extrapolate a high-level logical network design as shown in Figure 1 ( I have removed vPC design for now as you read on you will see it introduced into the article):

In the figure 1, we VIP VLAN which is a routable VLAN. Node VLAN is a non-routable VLAN, which is strictly Layer 2.  Since the VLAN is non-routable no external devices except the F5 can access the Nodes directly.  Finally we have Server VLAN Z which is where the NAS system is connected to.  In order to have communication between Server VLAN Z and Node VLAN, the traffic must route through the F5 via VIP VLAN. This is done by a static route pointing to .11 on VIP VLAN which is the F5 floating address on VIP VLAN to reach node VLAN address block. In figure 1 you also have all servers in Node VLAN pointing to .1 as their default gateway which is the floating address of the F5. The F5’s default gateway is .1 on VIP VLAN. Now that we have described the current behavior of Figure 1, we can start looking at making some changes.

So how do we change the network to accommodate the result that is being looked for? It is actually much easier then you might think.

The first item you want to remove is the static route on the switch pointing to point to .11 on VIP VLAN to access NODE VLAN. You will not need this since the end result is to allow SERVER VLAN and NODE VLAN to communicate directly via the Cisco Nexus Switch router.

Next you will need to change NODE VLAN from a non-routable network to a routable network. Thus, NODE VLAN will have a gateway of .1 on the switch router. The F5 will then change its own floating address to say .11 and subsequently change the self-addresses to .12 and .13.  All the servers in NODE VLAN will continue to use .1  as the default gateway.

 

Thus the network will now look more like Figure 2:

At this point, you are thinking how is the traffic going to return to F5 load balancer when it’s traffic via VIP. The easy way is to apply SNAT Automap. Which works, but then you run into another problem where you lose the client IP address. Normally this might be work, but will make tracking clients more difficult especially around traffic that is not HTTP based.

The short answer to this is utilizing a Cisco’s Policy Based Route.  How does that work?

On a Cisco switch router, you can do the following configuration (NX OS Syntax):


ip access-list from_node_vlan_deny
10 permit ip y.y.y.0/24 z.z.z.0/24
ip access-list from_node_vlan_allow
10 permit ip y.y.y.0/24 any
route map to_node_vlan deny 10
match ip address from_node_vlan_deny
route map to_node_vlan permit 10
match ip address from_node_vlan_allow
set ip next-hop y.y.y.11
interface VIP_VLAN
ip policy route-map to_node_vlan

NOTE: You must have feature pbr enabled.

If you are a student of Cisco IOS you might notice that IP access-list does not contain deny statements.  This is because PBR statements in the Nexus OS was designed to ignore the deny statements within IP access-lists.  I haven’t received an official reason of why this happened, but the best case was that they wanted  to make the ultimate PERMIT/DENY decision at the route map level.   The good news is that this new behavior only exists when applied to the pBR. Meaning Deny statements within an IP access-list will not be ignored when applying as a standard ACL for security access.   Also you can use the same access-list for security access and route-maps so just keep in mind that that DENY statements will be ignored by the route-maps ONLY.

Looking at the configuration example above the behavior is that if the NODE VLAN traffic is destined to the SERVER VLAN, skip the route-map statement and use the internal routing table of the switch. Thus allowing NODE VLAN to communicate directly to SERVER VLAN and vice versa. Subsequently, if traffic from NODE VLAN is attempting to talk to the internet then it will match the IP access-list “from_node_vlan_allow” within route map “to_node_vlan permit 10”.  It will then apply the next command which is a next hop of y.y.y.11 (Floating address of the F5) within NODE VLAN.

If we left everything alone, this story would be complete.   Unfortunately the network example I used is also using vPC, which adds another layer of complexity which needs to be accounted.   Figure 3 shows us what a vPC topology would look like with an F5:

You see F5 had decided to optimize the Ethernet Frames.   To optimize F5 typically ignores the arp reply given by the HSRP primary and instead forwards Ethernet frames to which ever MAC address it receives frames from the result is a faster response time.   NAS storage vendors also do this and it’s wide spread.  Unfortunately this is not a nonstandard behavior.   If you are well versed enough on the F5 you would immediately think to turn off the auto Last hop feature would counteract this behavior.  Unfortunately, this does not work in Cisco Nexus OS world.  Cisco recognized that many vendors had this same issue so they introduced the command “peer-gateway” command. This command in affect disabled the optimization.

So basically you would introduce the command in the following configuration example, in our diagram it would be on Nexus 7010 MDF A and MDF B

vpc domain 1

role priority 10

peer-keepalive destination 10.1.1.2 source 10.1.1.1 vrf VPC-KeepAlive

peer-gateway

Of course this is still not end of the story because peer-gateway has a caveat as stated in the Nexus OS Layer 2 guide

Packets arriving at the peer-gateway vPC device will have their TTL decremented, so packets carrying TTL = 1 may be dropped in transit due to TTL expire. This needs to be taken into account when the peer-gateway feature is enabled and particular network protocols sourcing packets with TTL = 1 operate on a vPC VLAN.

This means that the traffic will be treated like a layer 3 hop which means we need to make small adjustment in our access list

From:

ip access-list from_node_vlan_deny
10 permit ip y.y.y.0/24 z.z.z.0/24
ip access-list from_node_vlan_allow
10 permit ip y.y.y.0/24 any
route map to_node_vlan deny 10
match ip address from_node_vlan_deny
route map to_node_vlan permit 10
match ip address from_node_vlan_allow
set ip next-hop y.y.y.11
interface VIP_VLAN
ip policy route-map to_node_vlan

To:

ip access-list from_node_vlan_deny
5 permit ip y.y.y.0/24 y.y.y.0/24
10 permit ip y.y.y.0/24 z.z.z.0/24
ip access-list from_node_vlan_allow
10 permit ip y.y.y.0/24 any
route map to_node_vlan deny 10
match ip address from_node_vlan_deny
route map to_node_vlan permit 10
match ip address from_node_vlan_allow
set ip next-hop y.y.y.11
interface VIP_VLAN
ip policy route-map to_node_vlan

If you have been following closely on the difference you might be wondering why should you have a permit for traffic between NODE VLAN to NODE VLAN?  After all the access-list looks at Layer 3, not Layer 2 traffic.   As I mentioned above “Packets arriving at the peer-gateway vPC device will have their TTL decremented…”  Which means that Layer 2 traffic under vPC Peer Gateway will treat any traffic within that VLAN as a layer 3 hop and it will be processed within the access-list.

Conclusion

If you are running a F5 ADC which routes through F5 Nexus devices, then you don’t need peer-gateway , but you will if you  if you are directly attached to a Nexus Device that is configured to use vPC.

I have yet to face any issues with this configuration so it might be a good idea to add Peer-gateway into your vpc configuration as a default.

Share

No Comment.

Add Your Comment