Often overlooked by those seeking BIG-IP answers to web related problems is a very powerful feature called “Stream Profiles”.  So what exactly is a stream profile?  Well I am glad you asked!

In short a stream profile is a profile that can be used to replace strings of your choosing in server side response data.  They are generally pretty lightweight as far as CPU ticks go and are pretty easy to write.  When I have used them in the past, I have kept most of mine simple, doing what I call string for string replacements such as replacing the word “old” with the word “new”.  However, the stream profile can leverage basic regex syntax to for your more creative solutions if you ever have need.

Now when do stream profiles come in handy?  Well I can give you a real world example.  I was troubleshooting an issue with the login page of a web application the other day and realized that the submit button for the application was hard coded to POST to an HTTP address but I was attempting to use the application over HTTPS.

Being no stranger to iRules and laughing to myself how easy this one would be to solve, I simple created a VIP to listen on HTTP and threw my trusty HTTP_TO_HTTPS iRule on it.  Then I went back and checked the application.

I typed in the URL, using HTTP this time to check the redirect was taking place now and of course was forwarded over to HTTPS via the iRule.  Success!  Or so I thought…. I plugged in the test username and password, hit SUBMIT and received the page that said I had submitted the wrong username and password.  Thinking I fat fingered it, I went back, plugged in my credentials again (this time doing the super slow typing trying while saying my password out loud, yes you know what I am talking about) and hit submit again.  And was thwarted again.

I pulled up my trusty HTTP Watch program and went through the series of events once again.  The redirect was working for HTTP over to HTTPS, but something seemed to be going wrong where the web application was using the POST method.  The POST data was still intact after the redirect (here is how to pull that off), but something else was messing with the code.  Hmmm….  Could it be related to http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html Section 10.3.3 which states “If the 302 status code is received in response to a request other than GET or HEAD, the user agent MUST NOT automatically redirect the request unless it can be confirmed by the user, since this might change the conditions under which the request was issued.”

Well that certainly might cause a problem with the code we were testing!  Then drilling down a bit further into our test application we realized that the URL that the “Submit” button was performing the POST to was actually being pulled dynamically by the application from a database server entry.  Being unable to modify that database entry because of a variety of reasons we decided to leverage the BIG-IP’s Stream profile abilities.

So you see, it was a bit of a complex problem in our case, yet the solution was “BIG-IP Easy”.  I logged into the LTM, click Profiles, Other, Stream and then the Create button.

Give it a name, select “stream” as the parent profile, leave the source blank and then input your target information.  This is the part that allows you to substitute one outbound word for another.  For example we want to replace and old URL with a new URL.  The old URL is http://myold.url.com:80 and the new URL is https://mynew.url.com.

In the “Target” box you would type:

@http://myold.url.com:80@https://mynew.url.com@

Then save the profile and apply it to the VIP that is in need of the fix.  That is it!  Now the data in the content stream going back to the client it will be re-written according to your selection.  You of course can use different delimiters than the @ sign if you like and you can even add another string for the profile to replace if you like.  All you have to do in that case is add a space after the last delimiter, add another delimiter and then the next string/replacement string combo.

If you are liking what you are hearing so far but want to use different delimiters, leverage regex and/or do all of this in an iRule rather than a profile, I highly suggest you check out a Tech Tip on DevCentral written by Deb Allen on September 11th, 2007.  Here is shortcut to that article.

In an article that TheF5Guy.com posted back in September 22, 2010 I explained a method about creating a F5-Cisco VLAN to VLAN Bypass for Cisco IOS gear.  With the introduction to Cisco Nexus and vPC (Virtual Port Channel) technology the configurations to make the VLAN-to-VLAN bypass would need to be updated.  (Previous article can be found here)

So now we have the following similar scenario with the added twist of Nexus and vPC.

I have a pair of F5 ADC in an Internet DMZ, where nodes behind the load balancer need to access NAS system(s) on a VLAN located on a separate VLAN that is not behind the load balancer. The problem is that in my current design I have to route through the F5 Load balancer to access the NAS system(s).  Unfortunately the amount of bandwidth it takes supersedes the F5 ADC’s total throughput.  I would like to bypass this without adding extra network cards or recreating a new VLAN and would like preserve the IP addresses as much as possible.  Also the F5 ADC is sitting on a network design that participates in vPC within Cisco Nexus Datacenter gear.

Based on this description above you extrapolate a high-level logical network design as shown in Figure 1 ( I have removed vPC design for now as you read on you will see it introduced into the article):

In the figure 1, we VIP VLAN which is a routable VLAN. Node VLAN is a non-routable VLAN, which is strictly Layer 2.  Since the VLAN is non-routable no external devices except the F5 can access the Nodes directly.  Finally we have Server VLAN Z which is where the NAS system is connected to.  In order to have communication between Server VLAN Z and Node VLAN, the traffic must route through the F5 via VIP VLAN. This is done by a static route pointing to .11 on VIP VLAN which is the F5 floating address on VIP VLAN to reach node VLAN address block. In figure 1 you also have all servers in Node VLAN pointing to .1 as their default gateway which is the floating address of the F5. The F5’s default gateway is .1 on VIP VLAN. Now that we have described the current behavior of Figure 1, we can start looking at making some changes.

So how do we change the network to accommodate the result that is being looked for? It is actually much easier then you might think.

The first item you want to remove is the static route on the switch pointing to point to .11 on VIP VLAN to access NODE VLAN. You will not need this since the end result is to allow SERVER VLAN and NODE VLAN to communicate directly via the Cisco Nexus Switch router.

Next you will need to change NODE VLAN from a non-routable network to a routable network. Thus, NODE VLAN will have a gateway of .1 on the switch router. The F5 will then change its own floating address to say .11 and subsequently change the self-addresses to .12 and .13.  All the servers in NODE VLAN will continue to use .1  as the default gateway.

Thus the network will now look more like Figure 2:

At this point, you are thinking how is the traffic going to return to F5 load balancer when it’s traffic via VIP. The easy way is to apply SNAT Automap. Which works, but then you run into another problem where you lose the client IP address. Normally this might be work, but will make tracking clients more difficult especially around traffic that is not HTTP based.

The short answer to this is utilizing a Cisco’s Policy Based Route.  How does that work?

On a Cisco switch router, you can do the following configuration (NX OS Syntax):


ip access-list from_node_vlan_deny
10 permit ip y.y.y.0/24 z.z.z.0/24
ip access-list from_node_vlan_allow
10 permit ip y.y.y.0/24 any
route map to_node_vlan deny 10
match ip address from_node_vlan_deny
route map to_node_vlan permit 10
match ip address from_node_vlan_allow
set ip next-hop y.y.y.11
interface VIP_VLAN
ip policy route-map to_node_vlan


NOTE: You must have feature pbr enabled.

If you are a student of Cisco IOS you might notice that IP access-list does not contain deny statements.  This is because PBR statements in the Nexus OS was designed to ignore the deny statements within IP access-lists.  I haven’t received an official reason of why this happened, but the best case was that they wanted  to make the ultimate PERMIT/DENY decision at the route map level.   The good news is that this new behavior only exists when applied to the pBR. Meaning Deny statements within an IP access-list will not be ignored when applying as a standard ACL for security access.   Also you can use the same access-list for security access and route-maps so just keep in mind that that DENY statements will be ignored by the route-maps ONLY.

Looking at the configuration example above the behavior is that if the NODE VLAN traffic is destined to the SERVER VLAN, skip the route-map statement and use the internal routing table of the switch. Thus allowing NODE VLAN to communicate directly to SERVER VLAN and vice versa. Subsequently, if traffic from NODE VLAN is attempting to talk to the internet then it will match the IP access-list “from_node_vlan_allow” within route map “to_node_vlan permit 10”.  It will then apply the next command which is a next hop of y.y.y.11 (Floating address of the F5) within NODE VLAN.

If we left everything alone, this story would be complete.   Unfortunately the network example I used is also using vPC, which adds another layer of complexity which needs to be accounted.   Figure 3 shows us what a vPC topology would look like with an F5:

You see F5 had decided to optimize the Ethernet Frames.   To optimize F5 typically ignores the arp reply given by the HSRP primary and instead forwards Ethernet frames to which ever MAC address it receives frames from the result is a faster response time.   NAS storage vendors also do this and it’s wide spread.  Unfortunately this is not a nonstandard behavior.   If you are well versed enough on the F5 you would immediately think to turn off the auto Last hop feature would counteract this behavior.  Unfortunately, this does not work in Cisco Nexus OS world.  Cisco recognized that many vendors had this same issue so they introduced the command “peer-gateway” command. This command in affect disabled the optimization.

So basically you would introduce the command in the following configuration example, in our diagram it would be on Nexus 7010 MDF A and MDF B

vpc domain 1

role priority 10

peer-keepalive destination 10.1.1.2 source 10.1.1.1 vrf VPC-KeepAlive

peer-gateway

Of course this is still not end of the story because peer-gateway has a caveat as stated in the Nexus OS Layer 2 guide

Packets arriving at the peer-gateway vPC device will have their TTL decremented, so packets carrying TTL = 1 may be dropped in transit due to TTL expire. This needs to be taken into account when the peer-gateway feature is enabled and particular network protocols sourcing packets with TTL = 1 operate on a vPC VLAN.

This means that the traffic will be treated like a layer 3 hop which means we need to make small adjustment in our access list

From:

ip access-list from_node_vlan_deny
10 permit ip y.y.y.0/24 z.z.z.0/24
ip access-list from_node_vlan_allow
10 permit ip y.y.y.0/24 any
route map to_node_vlan deny 10
match ip address from_node_vlan_deny
route map to_node_vlan permit 10
match ip address from_node_vlan_allow
set ip next-hop y.y.y.11
interface VIP_VLAN
ip policy route-map to_node_vlan


To:

ip access-list from_node_vlan_deny
5 permit ip y.y.y.0/24 y.y.y.0/24
10 permit ip y.y.y.0/24 z.z.z.0/24
ip access-list from_node_vlan_allow
10 permit ip y.y.y.0/24 any
route map to_node_vlan deny 10
match ip address from_node_vlan_deny
route map to_node_vlan permit 10
match ip address from_node_vlan_allow
set ip next-hop y.y.y.11
interface VIP_VLAN
ip policy route-map to_node_vlan

If you have been following closely on the difference you might be wondering why should you have a permit for traffic between NODE VLAN to NODE VLAN?  After all the access-list looks at Layer 3, not Layer 2 traffic.   As I mentioned above “Packets arriving at the peer-gateway vPC device will have their TTL decremented…”  Which means that Layer 2 traffic under vPC Peer Gateway will treat any traffic within that VLAN as a layer 3 hop and it will be processed within the access-list.

Conclusion

If you are running a F5 ADC which routes through F5 Nexus devices, then you don’t need peer-gateway , but you will if you  if you are directly attached to a Nexus Device that is configured to use vPC.

I have yet to face any issues with this configuration so it might be a good idea to add Peer-gateway into your vpc configuration as a default.

I thought it would be fun/interesting/(useful?) for the Umbraco and F5 Networks community to create a series of posts based on my experiences in using the F5 BIG-IP to deliver this application in a fast, secure and highly available manner.

The first post that I want to throw out there for folks in both communities is related to security and iRules.  There are always “Best Practice” things that you want to do with every web application and Umbraco is no different.  I have two issues that I want to cover.

One of the first things that you will want to do is turn off access to the built-in debug feature included with Umbraco.  According to the official Umbraco documentation found here: http://our.umbraco.org/wiki/how-tos/hide-debugging-features-for-production-systems this feature cannot be turned off inside of Umbraco.  The documentation then goes on to contradict itself  and mentions that you CAN turn off debugging.  It is a bit confusing I know, but I guess we have to work with the information that we have right?

In that same document it also mentions that debugging can be blocked from within Umbraco using the built in URL rewriting feature, but if you are going to be doing some URL manipulation… well, I think you know where I am going with this!

The basic iRule below will keep hackers from being able to see what is going on behind the scenes on you production Umbraco servers which accomplishes our Best Practice goals.

when HTTP_REQUEST {
if { ([string tolower [HTTP::uri]] contains "umbdebug")} {
HTTP::redirect "https://mycompany.com/default.aspx"
}
elseif { ([string tolower [HTTP::uri]] contains "umbraco")} {
HTTP::redirect "https://mycompany.com/default.aspx"
}
}

The first part of this simply scans your incoming HTTP Request URI’s looking for “umbdebug” and when found it redirects the request back out to the homepage or whatever location you choose to send them.

The second part of the iRule I have added because it will prevent people from accessing the Umbraco Administration console.  This is not only a good idea for security but is also another Umbraco Best Practice.  It is important because it prevents your content developers from accessing that area via the load balanced URL.

If you are using DFS as your storage method on the backend of Umbraco and you attempt to use the load balanced URL to upload documents their experience will not be a pleasant one.  Documents will hang while they are uploading them and may even lock-up their web browser.  They will need to access one (and only one) server directly for site administration.

Like the first part of the iRule, it scans incoming HTTP Request URI’s but this looks for “umbraco” in the URI path and if it is found redirects the user to the location of your choosing.  You could also just drop the packets or something along that line, but I find dumping people out to the root of the site is adequate enough in most cases.

For those of you who do not have a lot of experience with HTML parameters you probably have heard to them referred to as fields in your web application.  For example, many web applications have username and password fields and these are essentially parameter fields.  There are sometimes hidden parameters and dynamic parameters that are not associated with a field on the page, but today I want to discuss the basic ones.  I have chosen the TARGET parameter because it is deprecated and it can be used in phishing attacks as a form of “Open Redirect” attack on your web sites.  The user parameter was chosen because it is a pretty common parameter/field name and it just seemed to make sense to include it in the discussion.

An open redirect type of attack will often consist of an attacker creating a URL that will redirect a victim to a site that they control.  This URL is then used in a phishing attack where a user is presented with a valid link in an email and companywebsite.com redirects the user to companywebsite-justgotowned.com… which is the site the attacker controls!  That’s just one type of open redirect attack though, another type focus’s on using the TARGET parameter to redirect a user behind the scenes to a malicious web site.

Needless to say, that’s not good.  What is good though is that protecting against the malicious use of parameters is very EASY to do with BIG-IP ASM.  The first thing that you will want to do, provided you already have an application security policy in place, is to create a Parameter.  Navigate to Application Security, Parameter, Parameters List, select the application policy that you want to modify and click the GO button.

Then click Create.  Give your parameter an explicit name (I used TARGET in my example), select Global Parameter, Data Type should be Alpha-Numeric and check the “Regular Expression” box.  Now you will need to come up with a regular expression that fits your environment.  In my example I am going to define two things.  First I will use the hostname of the web site that is valid and then after the pipe I will define a value for a URL that is still being called in our own code via the TARGET method.  Since it is a relative URL I have to include it because the regex for just the hostname will not cover it.  Below is a screenshot for reference:

The regex looks like this:

.*mycompany.com.*|.*myurlpath.*

Something very important to remember when creating these regular expressions is that whenever you create a parameter value and check the Regular Expression box it is automatically setup as a POSITIVE regular expression.  Therefore whatever is in this box defines what is legal for this parameter/field.  In the example above if a TARGET value is submitted to the web application it must contain “mycompany.com” or “myurlpath” or it will be shot down by the ASM.  This will prevent someone from setting a target of somewhere other than your web site.  This will stop a blatant open redirect attack but certainly not all.  Then click the create button.

Now you will need to tell your web application policy to be on the lookout for violations of this type.  Navigate to Application Security, Policy, Blocking, Settings.  Then scroll down the list until you see “Parameter value does not comply with regular expression”, check the Learn, Alarm and Block check boxes.  Save and then Apply the policy.  That’s it!

When ever a violation happens you will now see this in the manual traffic learning section:

Now to tackle the “user” parameter.  I am going to take a different angle on this one because like I mentioned before, once you understand the principal behind it you will see it can be used in a million different ways to protect your web application.

After looking over a few security logs you might notice that some hackers attempt to utilize the “user” parameter/field in your web application and they will try to throw all kinds of things in there.  One common element I have seen is that they will try to inject a username@yourdomain.com into the field.  Since that is not a valid character for the application I am looking to protect, I am going to block this kind of attack configuring the ASM to block based off of an invalid metacharacter value being placed in the parameter value.

Following the instructions above for creating a new Parameter, except this time instead of using a regular expression, click the Value Meta Characters tab.  Select “@ (0×40)” from the list on the right hand side of the page and then set the value to be disallowed using the drop down box under the set state heading.  Put a check mark in the check characters on this parameter value check box.  Now to configure your web application policy to listen, alarm and block on these kinds of attacks.  Navigate to Application Security, Policy, Blocking, Settings.  Then scroll down the list until you see “Illegal meta character in parameter value”.  Check the appropriate boxes, save and then apply.

Now whenever a would be hacker attempts to inject an invalid character into that field (the @ character in this case, but like I said you can use countless others) they will be smacked down by the ASM.

It’s a piece of cake really once you do it a time or two.  If you get hung up on the regular expression part have no fear!  The kind folks over at F5 Networks have thought ahead and have included a regular expression validator inside of the ASM module.  Just navigate to Application Security, Options, Tools and RegExp Validator.  You can use that tool to compile your regular expression if need be.

Remember when thinking about security related things it is best to take the defense in-depth approach.  Little things added here and there to your web application security policy that do no harm but can mitigate attacks can be very effective.

NRPT – Name Resolution Policy Table.  If you have messed around with Direct Access much at all you have to had come across this term at some point.  It basically tells your Direct Access clients how to behave when it comes to DNS queries.  Think host file on steroids…

I recently discovered that the NRPT pushed out via Group Policy can EASILY be corrupted if the script that applies the GPO’s fails during it’s activation.  How did I figure this out?  Well I had about 62 NRPT entries to push out, so I queued them all up, hit the apply button and walked away for lunch.  Thinking happily to myself that I would grab some lunch, come back, my updates will have been pushed out and I can jump back onto a little F5 BIG-IP project I am working on.  Imagine the look on my face when I arrived back from lunch and all of my “Test” subjects (aka co-workers) were mentioning that they could no longer access any LAN resources!  I sheepishly hunkered down into my cube and furiously began working on a fix.

Well Microsoft promised this couldn’t happen as of UAG/DA update 1, but I am running UAG/DA update 2 and I can assure you, it can still happen.  The fix is easy enough though as long as you have a computer that is running Direct Access and it has not pulled down a corrupt NRPT table.  The problem generally happens when a computer checks in with the Domain Controllers and does a GP refresh.  This happens periodically and it is hard to tell when a machine might check in.  If you are in the middle of pushing out a new NRPT or it halted in the middle of an update when the client checks in, poof!  Corrupt NRPT.

The fastest way to tell if you have a corrupt NRPT is to open a command line and type:

netsh name show effective policy

If you get back the dreaded message of: “Name resolution policy table has been corrupted. DNS resolution will fail until it is fixed. Contact your network administrator.”  Then welcome to can’t do anything on the LAN land.

So how do you fix it?  On the computer that has a valid NRPT table go and export the following registry key, save it to a thumb drive and sneakernet it over to victims PC.  The key you want to export is “HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\DnsClient\DnsPolicyConfig”.  Then on the victims PC open up the same spot in the registry and remove the subkeys UNDER the DnsPolicyConfig key.  Don’t change anything in that particular key, just delete the ones underneath it.  They will usually all have a name similar to UAGDA Rule 1, UAGDA Rule 2… you get the idea.

Once you have all of those deleted out, import the good registry key which contains the NRPT and then reboot the PC.  And that’s it!

I have used the WAM (WebAccelerator Module) to accelerate a few SharePoint 2007 sites in the past and have been able to achieve a 45%-55% reduction in the number of hits on our web front end servers.  To me that is a pretty dramatic reduction to say the least.  Those servers have since been upgraded to SharePoint 2010 so I will hopefully be doing another blog post in a few weeks where I will show you how we use WAM to accelerate SharePoint 2010 web applications.  In this post I am going to cover using the default WAM IIS template to accelerate our main web site and show you the results.

Previously I was using TMOS Version 9.x so starting off couldn’t be more simple in Version 10.2.  One very nice thing that I want to point out with this version is that when you click on the WebAccelerator section in the GUI it no longer opens up in a separate window.  That used to really annoy me and I was glad to see it is more cohesive in this version.  After clicking into there, click the “Applications” menu option and then click “Create”.  Type in a name, select the central policy template that you want to use (MS IIS in my case), type in your requested host name and click save.

You then create a Class Profile by clicking “Class Profiles” and “Create”.  Assign a name to it and leave the default values as they are.  That way if you decide to change  or modify something in that profile in the future you can easily do so and it will not effect any of your other profiles.  Then go back into the Local Traffic portion of the GUI, select the Virtual Server that you want to add the policy to, click “Resources” and then click the “Manage” button under HTTP Class Profiles, select the newly created acceleration profile in the list, click the << button to add it to the list and then click the finished button.

That’s it ladies and gentlemen!  You now have a accelerated web site.  How easy is that?!  I can’t imagine it being any easier than that, of course those folks at F5 Networks are always improving things.

So what kind of results can you expect from such a simple setup?  Well lets take a look.  From the graphs below you can see that the BIG-IP WAM has a response time of about 21ms for content requests.  This is the length of time it takes the WebAccelerator system to respond to a request from the client.

The second picture below shows you that the unit responded to 48,000 requests and the unit was able to successfully accelerate around 37,000 requests via Smart Cache.  That is a lot of happy users and represents 37,000 requests that our web servers did not have to respond to!  The errors that show up in the report are mostly my fault because I have not cleaned up my traffic reports like George Watkins explains how to do over on DevCentral http://devcentral.f5.com/weblogs/watkins/archive/2010/08/18/clean-up-those-webaccelerator-performance-reports.aspx.  Thanks again George for that excellent post!  Once I have had a chance to clean those reports up I will try to post a prettier picture.

Then last but certainly not least, is a picture that shows you my CPU utilization on that particular unit over the last 24 hours.  This picture was taken roughly 9 hours after implementing the web acceleration profile.  As you can see there has been only a slight increase (maybe 1%) in my CPU utilization.

Looking at those facts it is safe to say this was a very successful deployment of a WAM profile on a production web site that has generated some very positive results.  Faster responses for the end users and less load on the back web servers, it is a win-win solution in my book.

From time to time, I usually receive a request that goes something like this.

“I have a pair of F5 ADC in an Internet DMZ, where the servers behind the load balancer need to access NAS system(s) on a VLAN located in the same network on another VLAN that is not behind the load balancer.

The problem is that in my current design I have to route through the F5 Load balancer to access the NAS system(s).  Unfortunately the amount of bandwidth it takes supersedes the F5 ADC’s total throughput.  I would like to by pass this without adding extra network cards or recreating a new VLAN and would like preserve the IP addresses as much as possible.”

For the purposes of the blog we will call the person requesting this Keyser Söze

Based on this description above you extrapolate a high-level logical network design as shown in Figure 1.

Figure 1

In the figure 1, we VLAN 10 which is a routable VLAN. VLAN 12 is an empty VLAN, which is strictly Layer 2, and no other traffic allowed to it from the router itself.  Finally we have VLAN13 which is where the NAS servers is connected to .  In order to access VLAN12 you need to route through the F5 that is also connected on VLAN10. This is done by a static route pointing to .11 on VLAN10 which is the F5 floating address on VLAN 10 to reach VLAN12 address block. In figure 1 you also have all servers in VLAN12 pointing to .1 as their default gateway which is the floating address of the F5. The F5’s default gateway is .1 on VLAN10. Now that we have described the current behavior of Figure 1, we can start looking at making some changes.

So how do we change the network to accommodate the result that Kyser is looking for? It is actually much easier then you might think.

For the purposes of this explanation, let us assume the switches are connected on Cisco Switch routers

The first item you want to remove is the the static route on the switch pointing to point to .11 on VLAN10 to access VLAN12. You will not need this since the end result is to allow VLAN 12 and VLAN 11 to communicate directly via the Cisco Switch router.

Next you will need to change VLAN11 from a non-routable network to a routable network. Thus, VLAN 11 will have a gateway of .1 on the switch router. The F5 will then change its own floating address to say .11 and subsequently change the self-addresses. All the servers will continue to use .1 on VLAN11 as their default gateway.

Thus the network will now look more like Figure 2

Figure 2

At this point, you are thinking well if that is the case then how do we get traffic back to the F5 for Load balancing traffic. Well the easy way is to apply SNAT Automap across all the Virtual addresses. Which works, but then you run into another problem where you lose the client IP address. Normally this might be work, BUT if you are tracking clients for statistical purposes, this is not going to work.

The short answer to this is utilizing a Cisco’s Policy Based Route. How does that work?

On a Cisco switch you can do the following configuration (IOS Syntax):

ip access-list extended from_vlan11
Deny y.y.y.0 0.0.0.255 z.z.z.0 0.0.0.255
Permit y.y.y.0 0.0.0.255 any
route map to_lb_vlan11
Match ip address from_vlan11
ip default next-hop y.y.y.11
interface Vlan11
ip policy route-map to_lb_vlan11

What these statements mean is that any traffic from VLAN11 is destined to addresses on VLAN12, skip the route-map statement and use the internal routing table of the switch. Thus allowing VLAN11 to communicate directly to VLAN12 and vice versa. Subsequently, if traffic from VLAN11 is attempting to talk to the internet then it will match the permit statement in the IP access list “from_vlan11” then apply the route map statement and thus your next hope is .11, which is hosted on VLAN11.

That pretty much sums up how to use the switches throughput for VLAN to VLAN traffic and the F5 ADC continues to do what it does best while Kyser can go home happy.

Thanks,

CB

Setting Up BIG-IP and Live Meeting Portal Server

Prerequisites:

Please consult the Live Meeting Portal Server documentation and ensure that your servers meet all the perquisites before installation. All the examples in this guide are setup so that you will end up with a website at this URL: https://livemeeting.mycompany.com/lmportal. Please feel free to substitute your company’s name for “mycompany”.

IIS Setup:
1. Download the latest version of Office Live Meeting Service Portal. As of 4/20/2010 that can be found here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=429bb528-fd1b-45b7-af2b-cbbf4a8e65ff&displaylang=en

2. Create a basic website in IIS and name it Live Meeting. This empty shell of a website will be used by the Live Meeting installer and will basically be taken over by it after you run through the installation.

3. Create a folder named “Livemeeting” in the directory of your choice. In this example we will use ”E:\web\content\”

4. Double click the lmportal.exe to begin the installation and choose custom when the option appears. Then select the directory you created above so the files will be placed in your normal custom web content location.

5. Remote Desktop (RDP) to the web server and open IIS. DO NOT USE THE IIS CONSOLE ON YOUR LOCAL MACHINE as you will not have access to everything that you need.

6. The screenshots below will help guide you through the configuration of the web site in IIS. Things that do need to be changed:
a. Add 443 to the SSL port and select the unique IP address for the site to use. We will be terminating SSL on the F5 BIG-IP and then re-encrypting before sending it back on to the server.

b. Allow Scripts and Executables under execute permissions. Verify application pool is set to Live Meeting Intranet Portal AppPool.

c. Verify that ASP.NET is set to version 1.1.4.322.

d. Under Directory Security, click Edit and make sure there is a check mark on the “Enable anonymous access” and “Integrated Windows authentication” box.

e. Go to the application pool, right click and go to properties. Click the Health tab and uncheck “Enable Rapid-Fail protection”. Not including a screenshot of this one.

7. Navigate to “E:\web\content\Livemeeting\Portal” on the server. Then find the file named “Portal.config”, right-click it and click the Security tab. Click Add and then add the “Network Service” user account and give it full control. You have to do this or you cannot modify the configuration settings from the GUI.

8. Do the same thing listed in step 7 for the “PortalExport” folder located in the directory you should currently be in: “E:\web\content\Livemeeting\Portal”

9. Now you have to import the SSL certificate that you are going to use into IIS website that you just set up. You will need to obtain the .crt file for the SSL certificate and the .key file for that certificate. We terminate our SSL on the BIG-IP so these can both be obtained from there. I will skip the steps regarding purchasing an SSL certificate for a site if you do not already have one. It kind of falls outside the scope of this guide.

10. Use a search engine and search for OpenSSL. You should find their homepage at: http://www.openssl.org/

11. Download OpenSSL and install it on your Local machine. I don’t recommend installing it on the server for a wide variety of reasons. I installed my copy of OpenSSL into “C:\OpenSSL”.

12. Take the .key file and the .crt file and put them into OpenSSL’s “bin” directory. It’s just a folder inside of your OpenSSL folder called bin.

13. Open a command line and change directory over to C:\OpenSSL\bin. The example I am going to provide is for a fictitious company named “MyCompany” that is using a wildcard ssl certificate on a few of their websites.

14. Then type in the following command:

This all needs to be on one line. Spaces are ok, but no carriage returns or anything like that. This command is modeled after this example for future reference:

openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt -certfile CACert.crt

certificate.pfx = the name of the new pfx file you want to create
privateKey.key = the private key you got off of the F5 BIG-IP
certificate.crt = the crt file that you got off the F5 BIG-IP
CACert.crt = the crt file that you got off the F5 BIG-IP

15. After you type the command and hit enter, you will be prompted for a password. You can use any password that you like but you will need to remember it because IIS asks you for the same password when you go to import it.

16. OpenSSL will compile a new .pfx file for you in the C:/OpenSSL/bin directory. Take that SSL certificate and copy it over to your web server.

17. RDP over to the server and open IIS. Again here is the disclaimer, DO NOT USE THE IIS CONSOLE ON YOUR LOCAL MACHINE. Right-click on the Live Meeting web site that you created and click on the Directory Security tab. Under “Secure Communications”, click the “Server Certificate…” button.

18. Click Next and then click the “Import a certificate from a .pfx file” radio button and click next. Browse to the .pfx file that you uploaded to the web server. Click next and enter your password information that you used when you created the certificate. Then finish clicking through the wizard. Then restart IIS on the server and delete the certificate off of your local machine. This completes the IIS setup. Now move on to the Live Meeting Portal setup.
Live Meeting Portal Setup

19. Navigate to the URL:

https://livemeeting.mycompany.com/LMPortal/settings.aspx

Where livemeeting.mycompany.com is the name of the website you setup. The screen will look like the one shown on the next page. This is the Settings-Portal Configuration page. You will want to use the following settings which are also pictured in the screenshot on the next page.

Conference Center URL = https://www.livemeeting.com/cc/mycompany
Conference Center Administrator
User Id =
Password =
Email address for escalation =
Enabled Portal Services = Check the Account Create, Account Login, Account Update and Web Method Calls
Ticket Timeout = 300 Seconds
Directory Service Parameters = AccountNamePolicy=LogonUsername

20. Then click Save. If you receive an error at this point, refer back to step #7.

21. Click on the Roles link on the left side of the page. This will take you to the Roles-Portal Configuration page. Under “Live Meeting Administrators” add the users who will be the Live Meeting Administrators. Use domain\name format. IE: mydomain\username
22. Then under the “Live Meeting Organizers” settings I recommend adding the “Domain Users” from the varies domains on your network. So if you have three domains on you network named ABC, 123 and XYZ you would list ABC\Domain Users, 123\Domain Users and XYZ\Domain Users.

23. Then click the “Export Configurations Settings” link on the left hand side of the page. This is not really labeled right because what it actually does is back up your configuration. If you mess something up in the running configuration, simply click on the “Import Configuration Settings” to restore the last configuration that you exported.

24. Then click on the “Events” link on the left side of the page. Change the log file directory to a directory that you want to have all the logs written into. In this example I chose the E: drive of the server I was working on. Whether you create a new one or use an existing one you must make sure that the “Network Service” account has permissions on that folder to Read, Write and Modify. Otherwise you will receive a nasty .NET error when you go to save the changes you just made. Click Save.

Live Meeting Portal Server BIG-IP LTM Setup

The BIG-IP LTM set up for this can be very easy to configure. You will need to create nodes for each of your web servers, assign them to a pool named “Live_Meeting_Pool” and then create a Virtual Server for the application. I named my virtual server “Live Meeting” in the example pictured below. You may need to customize it to match your environment, but the basic settings are:

Service Port: 443
Type: Standard
Protocol: TCP
Protocol Profile (Client): tcp
HTTP Profile: http
SSL Profile (Client): wildcard
SSL Profile (Server): serverssl

I also assigned the Live_Meeting_Pool to the Virtual Server, set the Default Persistence Profile to “Cookie” and Fallback Persistence Profile to “source_addr”.

Both of the previously mentioned deployment guides discuss editing files on the Citrix farms Web Interface servers so that it looks for the client IP address in the X-Forwarded-For HTTP header.  Otherwise, every connection will appear to be originating from the BIG-IP LTM and not from its true IP.  After reading both guides and looking at my current environment I was dismayed to find that the files and locations mentioned were no longer valid.  I then turned to my top three resources on the web in the search for an answer: AskF5, DevCentral and Google.

I struck out on the first two (which seldom happens) but my Google search did turn up some interesting results on the Citrix Forums.  I finally found some code posted by Sam Jacobs back in August 2009 that modifies the way the Citrix farm looks up the client IP address.  His method allows for the use of the X-Forwarded-For header.

The first file that you will want to find and edit is the Include.java file.  You will want to locate and change this file on every Web Interface XenApp server in the farm.  Speaking from experience, save a copy of the original file to a safe location such as your desktop or flash drive.  DO NOT copy the file and rename the original to Include.old and leave it on the server.  It may sound crazy, but doing that will not work.  I’m not a programmer, so I cannot tell you why that will not work, but I can tell you I know for a fact it will not.  That being said, here is the file path for the Include.java file:

“\Inetpub\wwwroot\Citrix\XenApp\app_code\PagesJava\com\citrix\wi\pageutils\Include.java”

Now that you have found the file, open it up with a text editor (I use Textpad) and find the Java routine named “getClientAddress”.  Replace the code for that routine with the code listed below.

public static String getClientAddress(WIContext wiContext) {
String ageClientAddress = AGEUtilities.getAGEClientIPAddress(wiContext);
String userIPAddress = wiContext.getWebAbstraction().getRequestHeader("X-FORWARDED-FOR");
if (userIPAddress == null) {
userIPAddress = wiContext.getWebAbstraction().getUserHostAddress();
}
return (ageClientAddress != null ? ageClientAddress : userIPAddress);
}

Save the file and wash/rinse/repeat this step on every Web Interface server in the farm.  The next thing that you will want to do is to modify the login page so that it displays the client IP address being obtained from the X-Forwarded-For header.  The file you will want to edit is called “loginView.ascx” and can be found in the following file path on your Web Interface Servers:

”\inetpub\wwwroot\Citrix\XenApp\app_data\include\loginView.ascx”

The code you will want to add is:

Client IP: <%= com.citrix.wi.pageutils.Include.getClientAddress(wiContext) %>

I added the code directly below the LoginPageControl viewControl line and it works well for me.  Save the file and repeat this step on every Web Interface server in the farm and reboot each Web Interface Server after you are done.  Then it is time for the moment of truth… fire up your browser of choice and navigate to the Citrix login page.  If you have successfully set everything up and have finished following the rest of the deployment guide you should see a screen similar to the one below:

If you receive an error message or the screen doesn’t load, then you might want to go back and check your settings again.  Then that’s it!  I am aiming to develop some custom monitors for the Web Interface Server and for the XML Broker Servers over the next few weeks.  Once I have those done I will put them out in the Devcentral forums for the community enjoy.

I am very happy to mention that the kind folks over at F5 Networks allowed me to submit this as a Tech Tip article which you can find on their site at:

http://devcentral.f5.com/Default.aspx?tabid=63&articleType=ArticleView&articleId=1082335

No problem, I identified all of the expired certificates, checked the box beside them and hit the delete button at the bottom of the page.  After verifying everything was still happy and the support tickets didn’t start flooding my inbox I decided to run a config sync and push the config changes over to the standby box.

The config sync ran without a problem and the gui showed Config Sync: OK.  I then proceeded to check my changes on the standby unit, just for verification purposes.  And that ladies and gentlemen, is when the fun began….

As I was verifying the changes I noticed something I thought was rather strange.  The old SSL certificates that I deleted on the Active unit, were still there in the Standby units SSL Certificate store!  My first thought, oops, my Trusted Device Certificates must be out of whack.  I then proceeded to delete the trusted device certs and ran the “big_ip add” command from the CLI on each unit.  I checked my trusted device certificates and like magic there they were.  I ran another Config Sync thinking that probably fixed the problem, but wait… no such luck.

The Config Sync ran and didn’t kick out any errors, but the old SSL certificates were still in there in all their expired glory.  Frustrated and humbled once again, I decided to run a quick test by deleting a VS on the Active Unit to see if it would be removed once I ran a Config Sync.  I blew away the VIP I use for testing and ran the Config Sync again.  The VS was deleted off of the Standby Unit.  Not knowing off the top of my head what to do next, I then proceeded to open a ticket with my good friends over at F5 Networks.  I didn’t have a lot of faith in my running configuration at the time so I went ahead and opened the ticket as a level 2 ticket (site at risk).

I quickly received a phone call from a Network Support Engineer named Kevin “CB” Midkiff.  We went through the standard procedure of qkview files and few other tests.  After going over the problem Mr. Midkiff proceeded to explain to me that while the SSL Certificates store is indeed carried over when you run a Config Sync IT DOES NOT DELETE SSL Certificates on the unit that you push the config to.  In my case it was the Standby Unit.  The Config Sync function only appends SSL Certificates.

Moral to the story?  If you are double checking your configurations and happen to see some lingering SSL certificates don’t worry, just select them and let the delete button work its magic on them.  Also as an FYI, “CB” was great to work with and very knowledgeable.  Thanks again for your help Mr. Midkiff.

The unit that was upgraded has three modules running on it, the GTM, LTM and WA modules.  The issue is caused by the WebAccelerator module logging to many messages out to the PVAC log, which can lead to excessive disk I/O and may cause the log file to grow so large it crashes the WebAccelerator module.  It is now a Known Issue and is being tracked in CR127854.  So if you have upgraded to TMOS 9.4.8 and you are running the WebAcceleration module you might want to keep an eye out for this!

If you believe you have a unit experiencing this issue I would advise you to contact F5 Technical Support and open a case with them.  An Engineering Hotfix can be provided to you that addresses this issue.  In the meantime, if you are able to stop using the WebAccelerator class profiles, then I would suggest not using those until you have downloaded and applied the hotfix.  Below is the text from AskF5.com regarding the issue.

Known Issue
Updated: 9/17/09 10:11 AM

Now this was an afront to my logic!  My brain was so used to thinking “If this, then this”, that it really was hard for me to wrap my brain around how I was going to pull this off.  So of course, I did what any sane F5′er does when he is looking for an answer to a puzzle he cannot solve.  I turned to Devcentral and the community forums.  I dug around for a while and eventually I found an old 4.0 iRule where an individual had used the “not” Logical Operator.

So I gave myself a big slap on the forehead and muttered a Homer Simpson’ish “DOH!!”.  I later went on to discover that the “not” Logical Operator is well documented on DevCentral here.  Below is the simple iRule that has saved our company thousands of dollars, saved the help desk many man hours of labor, prevented users from going insane because of broken links and keeps things simple.  It is amazing how an iRule so simple, can have such a dramatic impact.  So, the next time you are writing an iRule, just think of all the things you could “NOT” be doing!


when HTTP_REQUEST {
if { not ([string tolower [HTTP::host]] contains ".mycompany.com")}{
HTTP::redirect "https://[HTTP::host].mycompany.com[HTTP::uri]"
}
}


1.  It had to support Ubuntu 64-bit Version 8.10.

2.  I wanted an official support channel that I could lean on if I ran into problems.

3.  It had to be inexpensive.

I found a number of $300+ solutions, but I don’t consider that cheap… To me, cheap is $50 or less.  Then as luck would have it I found a great solution for a mere $29.99.  The program is called “Image For Linux” and it is available at:

http://www.terabyteunlimited.com/image-for-linux.htm

I followed the setup directions provided with the download and within an hour I had successfully cloned my server over to a VM.  The only bump in the road was easy to overcome and I think the problem may have been caused by me choosing an incorrect setting before the clone process.  Because of that, I had to reinstall Grub on the VM which was no big deal.  I just loaded the Ubuntu disc, fired up the VM and had it boot from the Live-CD.  Then I went to the command line and typed (be sure to hit the enter key after each command):

sudo grub
find /boot/grub/stage1
root (hd0,4)
setup (hd0)
quit

The first line loads grub.  The second finds your boot partition and displays that information back out on the command line for you to read.  The output varies based on your install of course.  In my case hd0,4 is the only thing the second command displayed on screen after running so my choice was pretty easy.  Using that input, lines three and four tell grub where to install and the fifth line of course kicked me back out of grub.  I then removed the Live-CD, rebooted the VM and that was it!  The whole process was very painless and the software proved to be very capable.  If you have a need to clone or P2V a Ubuntu Linux machine then I recommend this software.