Often overlooked by those seeking BIG-IP answers to web related problems is a very powerful feature called “Stream Profiles”.  So what exactly is a stream profile?  Well I am glad you asked!

In short a stream profile is a profile that can be used to replace strings of your choosing in server side response data.  They are generally pretty lightweight as far as CPU ticks go and are pretty easy to write.  When I have used them in the past, I have kept most of mine simple, doing what I call string for string replacements such as replacing the word “old” with the word “new”.  However, the stream profile can leverage basic regex syntax to for your more creative solutions if you ever have need.

Now when do stream profiles come in handy?  Well I can give you a real world example.  I was troubleshooting an issue with the login page of a web application the other day and realized that the submit button for the application was hard coded to POST to an HTTP address but I was attempting to use the application over HTTPS.

Being no stranger to iRules and laughing to myself how easy this one would be to solve, I simple created a VIP to listen on HTTP and threw my trusty HTTP_TO_HTTPS iRule on it.  Then I went back and checked the application.

I typed in the URL, using HTTP this time to check the redirect was taking place now and of course was forwarded over to HTTPS via the iRule.  Success!  Or so I thought…. I plugged in the test username and password, hit SUBMIT and received the page that said I had submitted the wrong username and password.  Thinking I fat fingered it, I went back, plugged in my credentials again (this time doing the super slow typing trying while saying my password out loud, yes you know what I am talking about) and hit submit again.  And was thwarted again.

I pulled up my trusty HTTP Watch program and went through the series of events once again.  The redirect was working for HTTP over to HTTPS, but something seemed to be going wrong where the web application was using the POST method.  The POST data was still intact after the redirect (here is how to pull that off), but something else was messing with the code.  Hmmm….  Could it be related to http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html Section 10.3.3 which states “If the 302 status code is received in response to a request other than GET or HEAD, the user agent MUST NOT automatically redirect the request unless it can be confirmed by the user, since this might change the conditions under which the request was issued.”

Well that certainly might cause a problem with the code we were testing!  Then drilling down a bit further into our test application we realized that the URL that the “Submit” button was performing the POST to was actually being pulled dynamically by the application from a database server entry.  Being unable to modify that database entry because of a variety of reasons we decided to leverage the BIG-IP’s Stream profile abilities.

So you see, it was a bit of a complex problem in our case, yet the solution was “BIG-IP Easy”.  I logged into the LTM, click Profiles, Other, Stream and then the Create button.

Give it a name, select “stream” as the parent profile, leave the source blank and then input your target information.  This is the part that allows you to substitute one outbound word for another.  For example we want to replace and old URL with a new URL.  The old URL is http://myold.url.com:80 and the new URL is https://mynew.url.com.

In the “Target” box you would type:

@http://myold.url.com:80@https://mynew.url.com@

Then save the profile and apply it to the VIP that is in need of the fix.  That is it!  Now the data in the content stream going back to the client it will be re-written according to your selection.  You of course can use different delimiters than the @ sign if you like and you can even add another string for the profile to replace if you like.  All you have to do in that case is add a space after the last delimiter, add another delimiter and then the next string/replacement string combo.

If you are liking what you are hearing so far but want to use different delimiters, leverage regex and/or do all of this in an iRule rather than a profile, I highly suggest you check out a Tech Tip on DevCentral written by Deb Allen on September 11th, 2007.  Here is shortcut to that article.

Let me first start off with saying how much fun I had.  Getting to attend an event that is hosted like this is more fun than going to Six Flags for me!  Anyway, as I was saying…

This year our gracious host set us up in a very plush and idea inducing location about a mile from the loop in Chicago.  I have to admit that I was starting to get worried as we were driving to the meeting location.  The neighborhood looked a little sketchy in places, plus I didn’t have “Kim” with me because of the insane Chicago anti-gun laws.  To top it off I think our cab driver was under the impression that he was trying out for the Indianapolis 500, but once we got there the building was just awesome.  I am going to include a few pics below but you can also go out to the Thinkubator web site and check it out yourself here.

Needless to say we enjoyed the nice furniture, the iCade, the Xbox 360, the good food and there may have been an “Adult Beverage” or two.  They had several cool freebies for us MVP’s to!  A nice pen, a shirt and a very nice TSA compliant messenger bag made by Timbuk2.  I even tried my hand at a Kinect game, but got schooled by Joe and George, but I’m pretty sure I owned Jeff though… hahaha!  Yea, I think I just threw an iron gauntlet on a glass table!  Still, I think my hands were built more for slinging guns and working on F5 gear, not playing kinect games.

Now as fun as all of that is, the real fun came from the conversations that we had.  We had several presentations given to us by core F5 Networks people and each and every one was fantastic.  They held nothing back and spared no detail, I loved it.  To top it off even the MVP’s were even provided the opportunity to get up in front of everyone and speak about what was on our minds.  I am happy to say that is exactly what they did, sharing their success stories, identifying issues and they gave real world examples of how they leveraged F5 gear to solve some major problems.  It was really something.

The next day we were provided the opportunity to attend an Agility Event of our choosing and I chose the Application Security Manager and APM Labs.  Both labs were entertaining and very informative.  We were set up with accounts on BIG-IP’s to our own instances, given guidance/instruction on new features and then actually went through and configured things ourselves so we could have some real hands on experience with TMOS version 11.

I would have to say the coolest things that I saw in the ASM lab were the new policy builder, the AJAX protection capabilities and some awesome GUI enhancements that really help clarify things.

The APM stuff was completely new to me so I don’t have anything really to compare it to.  I did like the interface for it, but I think there is a bit of a learning curve required to configure that module.  Rather than run at the mouth about things I don’t have much experience with, I will redirect you over to the F5 Networks web site where you can find more information.

I want to take a second to say thank you to everyone at F5 Networks for all the hard work that you poured into making this event so great.  It was really great, the atmosphere was relaxing and I thought that really lead to a lot more openness and sharing of ideas than you get at a more corporate location.  I also wanted to thank the other MVP’s for being so friendly and willing to share your vast knowledge.  I hope that you all continue to give to the F5 Networks/DevCentral community whatever you can whenever you can, as I know that we will all be better for it.  I certainly feel better for knowing all of you and having shared this experience with you.

I look forward to contributing content myself and look forward to hopefully seeing you all at the next event!

In an article that TheF5Guy.com posted back in September 22, 2010 I explained a method about creating a F5-Cisco VLAN to VLAN Bypass for Cisco IOS gear.  With the introduction to Cisco Nexus and vPC (Virtual Port Channel) technology the configurations to make the VLAN-to-VLAN bypass would need to be updated.  (Previous article can be found here)

So now we have the following similar scenario with the added twist of Nexus and vPC.

I have a pair of F5 ADC in an Internet DMZ, where nodes behind the load balancer need to access NAS system(s) on a VLAN located on a separate VLAN that is not behind the load balancer. The problem is that in my current design I have to route through the F5 Load balancer to access the NAS system(s).  Unfortunately the amount of bandwidth it takes supersedes the F5 ADC’s total throughput.  I would like to bypass this without adding extra network cards or recreating a new VLAN and would like preserve the IP addresses as much as possible.  Also the F5 ADC is sitting on a network design that participates in vPC within Cisco Nexus Datacenter gear.

Based on this description above you extrapolate a high-level logical network design as shown in Figure 1 ( I have removed vPC design for now as you read on you will see it introduced into the article):

In the figure 1, we VIP VLAN which is a routable VLAN. Node VLAN is a non-routable VLAN, which is strictly Layer 2.  Since the VLAN is non-routable no external devices except the F5 can access the Nodes directly.  Finally we have Server VLAN Z which is where the NAS system is connected to.  In order to have communication between Server VLAN Z and Node VLAN, the traffic must route through the F5 via VIP VLAN. This is done by a static route pointing to .11 on VIP VLAN which is the F5 floating address on VIP VLAN to reach node VLAN address block. In figure 1 you also have all servers in Node VLAN pointing to .1 as their default gateway which is the floating address of the F5. The F5’s default gateway is .1 on VIP VLAN. Now that we have described the current behavior of Figure 1, we can start looking at making some changes.

So how do we change the network to accommodate the result that is being looked for? It is actually much easier then you might think.

The first item you want to remove is the static route on the switch pointing to point to .11 on VIP VLAN to access NODE VLAN. You will not need this since the end result is to allow SERVER VLAN and NODE VLAN to communicate directly via the Cisco Nexus Switch router.

Next you will need to change NODE VLAN from a non-routable network to a routable network. Thus, NODE VLAN will have a gateway of .1 on the switch router. The F5 will then change its own floating address to say .11 and subsequently change the self-addresses to .12 and .13.  All the servers in NODE VLAN will continue to use .1  as the default gateway.

Thus the network will now look more like Figure 2:

At this point, you are thinking how is the traffic going to return to F5 load balancer when it’s traffic via VIP. The easy way is to apply SNAT Automap. Which works, but then you run into another problem where you lose the client IP address. Normally this might be work, but will make tracking clients more difficult especially around traffic that is not HTTP based.

The short answer to this is utilizing a Cisco’s Policy Based Route.  How does that work?

On a Cisco switch router, you can do the following configuration (NX OS Syntax):


ip access-list from_node_vlan_deny
10 permit ip y.y.y.0/24 z.z.z.0/24
ip access-list from_node_vlan_allow
10 permit ip y.y.y.0/24 any
route map to_node_vlan deny 10
match ip address from_node_vlan_deny
route map to_node_vlan permit 10
match ip address from_node_vlan_allow
set ip next-hop y.y.y.11
interface VIP_VLAN
ip policy route-map to_node_vlan


NOTE: You must have feature pbr enabled.

If you are a student of Cisco IOS you might notice that IP access-list does not contain deny statements.  This is because PBR statements in the Nexus OS was designed to ignore the deny statements within IP access-lists.  I haven’t received an official reason of why this happened, but the best case was that they wanted  to make the ultimate PERMIT/DENY decision at the route map level.   The good news is that this new behavior only exists when applied to the pBR. Meaning Deny statements within an IP access-list will not be ignored when applying as a standard ACL for security access.   Also you can use the same access-list for security access and route-maps so just keep in mind that that DENY statements will be ignored by the route-maps ONLY.

Looking at the configuration example above the behavior is that if the NODE VLAN traffic is destined to the SERVER VLAN, skip the route-map statement and use the internal routing table of the switch. Thus allowing NODE VLAN to communicate directly to SERVER VLAN and vice versa. Subsequently, if traffic from NODE VLAN is attempting to talk to the internet then it will match the IP access-list “from_node_vlan_allow” within route map “to_node_vlan permit 10”.  It will then apply the next command which is a next hop of y.y.y.11 (Floating address of the F5) within NODE VLAN.

If we left everything alone, this story would be complete.   Unfortunately the network example I used is also using vPC, which adds another layer of complexity which needs to be accounted.   Figure 3 shows us what a vPC topology would look like with an F5:

You see F5 had decided to optimize the Ethernet Frames.   To optimize F5 typically ignores the arp reply given by the HSRP primary and instead forwards Ethernet frames to which ever MAC address it receives frames from the result is a faster response time.   NAS storage vendors also do this and it’s wide spread.  Unfortunately this is not a nonstandard behavior.   If you are well versed enough on the F5 you would immediately think to turn off the auto Last hop feature would counteract this behavior.  Unfortunately, this does not work in Cisco Nexus OS world.  Cisco recognized that many vendors had this same issue so they introduced the command “peer-gateway” command. This command in affect disabled the optimization.

So basically you would introduce the command in the following configuration example, in our diagram it would be on Nexus 7010 MDF A and MDF B

vpc domain 1

role priority 10

peer-keepalive destination 10.1.1.2 source 10.1.1.1 vrf VPC-KeepAlive

peer-gateway

Of course this is still not end of the story because peer-gateway has a caveat as stated in the Nexus OS Layer 2 guide

Packets arriving at the peer-gateway vPC device will have their TTL decremented, so packets carrying TTL = 1 may be dropped in transit due to TTL expire. This needs to be taken into account when the peer-gateway feature is enabled and particular network protocols sourcing packets with TTL = 1 operate on a vPC VLAN.

This means that the traffic will be treated like a layer 3 hop which means we need to make small adjustment in our access list

From:

ip access-list from_node_vlan_deny
10 permit ip y.y.y.0/24 z.z.z.0/24
ip access-list from_node_vlan_allow
10 permit ip y.y.y.0/24 any
route map to_node_vlan deny 10
match ip address from_node_vlan_deny
route map to_node_vlan permit 10
match ip address from_node_vlan_allow
set ip next-hop y.y.y.11
interface VIP_VLAN
ip policy route-map to_node_vlan


To:

ip access-list from_node_vlan_deny
5 permit ip y.y.y.0/24 y.y.y.0/24
10 permit ip y.y.y.0/24 z.z.z.0/24
ip access-list from_node_vlan_allow
10 permit ip y.y.y.0/24 any
route map to_node_vlan deny 10
match ip address from_node_vlan_deny
route map to_node_vlan permit 10
match ip address from_node_vlan_allow
set ip next-hop y.y.y.11
interface VIP_VLAN
ip policy route-map to_node_vlan

If you have been following closely on the difference you might be wondering why should you have a permit for traffic between NODE VLAN to NODE VLAN?  After all the access-list looks at Layer 3, not Layer 2 traffic.   As I mentioned above “Packets arriving at the peer-gateway vPC device will have their TTL decremented…”  Which means that Layer 2 traffic under vPC Peer Gateway will treat any traffic within that VLAN as a layer 3 hop and it will be processed within the access-list.

Conclusion

If you are running a F5 ADC which routes through F5 Nexus devices, then you don’t need peer-gateway , but you will if you  if you are directly attached to a Nexus Device that is configured to use vPC.

I have yet to face any issues with this configuration so it might be a good idea to add Peer-gateway into your vpc configuration as a default.

I thought it would be fun/interesting/(useful?) for the Umbraco and F5 Networks community to create a series of posts based on my experiences in using the F5 BIG-IP to deliver this application in a fast, secure and highly available manner.

The first post that I want to throw out there for folks in both communities is related to security and iRules.  There are always “Best Practice” things that you want to do with every web application and Umbraco is no different.  I have two issues that I want to cover.

One of the first things that you will want to do is turn off access to the built-in debug feature included with Umbraco.  According to the official Umbraco documentation found here: http://our.umbraco.org/wiki/how-tos/hide-debugging-features-for-production-systems this feature cannot be turned off inside of Umbraco.  The documentation then goes on to contradict itself  and mentions that you CAN turn off debugging.  It is a bit confusing I know, but I guess we have to work with the information that we have right?

In that same document it also mentions that debugging can be blocked from within Umbraco using the built in URL rewriting feature, but if you are going to be doing some URL manipulation… well, I think you know where I am going with this!

The basic iRule below will keep hackers from being able to see what is going on behind the scenes on you production Umbraco servers which accomplishes our Best Practice goals.

when HTTP_REQUEST {
if { ([string tolower [HTTP::uri]] contains "umbdebug")} {
HTTP::redirect "https://mycompany.com/default.aspx"
}
elseif { ([string tolower [HTTP::uri]] contains "umbraco")} {
HTTP::redirect "https://mycompany.com/default.aspx"
}
}

The first part of this simply scans your incoming HTTP Request URI’s looking for “umbdebug” and when found it redirects the request back out to the homepage or whatever location you choose to send them.

The second part of the iRule I have added because it will prevent people from accessing the Umbraco Administration console.  This is not only a good idea for security but is also another Umbraco Best Practice.  It is important because it prevents your content developers from accessing that area via the load balanced URL.

If you are using DFS as your storage method on the backend of Umbraco and you attempt to use the load balanced URL to upload documents their experience will not be a pleasant one.  Documents will hang while they are uploading them and may even lock-up their web browser.  They will need to access one (and only one) server directly for site administration.

Like the first part of the iRule, it scans incoming HTTP Request URI’s but this looks for “umbraco” in the URI path and if it is found redirects the user to the location of your choosing.  You could also just drop the packets or something along that line, but I find dumping people out to the root of the site is adequate enough in most cases.

For those of you who do not have a lot of experience with HTML parameters you probably have heard to them referred to as fields in your web application.  For example, many web applications have username and password fields and these are essentially parameter fields.  There are sometimes hidden parameters and dynamic parameters that are not associated with a field on the page, but today I want to discuss the basic ones.  I have chosen the TARGET parameter because it is deprecated and it can be used in phishing attacks as a form of “Open Redirect” attack on your web sites.  The user parameter was chosen because it is a pretty common parameter/field name and it just seemed to make sense to include it in the discussion.

An open redirect type of attack will often consist of an attacker creating a URL that will redirect a victim to a site that they control.  This URL is then used in a phishing attack where a user is presented with a valid link in an email and companywebsite.com redirects the user to companywebsite-justgotowned.com… which is the site the attacker controls!  That’s just one type of open redirect attack though, another type focus’s on using the TARGET parameter to redirect a user behind the scenes to a malicious web site.

Needless to say, that’s not good.  What is good though is that protecting against the malicious use of parameters is very EASY to do with BIG-IP ASM.  The first thing that you will want to do, provided you already have an application security policy in place, is to create a Parameter.  Navigate to Application Security, Parameter, Parameters List, select the application policy that you want to modify and click the GO button.

Then click Create.  Give your parameter an explicit name (I used TARGET in my example), select Global Parameter, Data Type should be Alpha-Numeric and check the “Regular Expression” box.  Now you will need to come up with a regular expression that fits your environment.  In my example I am going to define two things.  First I will use the hostname of the web site that is valid and then after the pipe I will define a value for a URL that is still being called in our own code via the TARGET method.  Since it is a relative URL I have to include it because the regex for just the hostname will not cover it.  Below is a screenshot for reference:

The regex looks like this:

.*mycompany.com.*|.*myurlpath.*

Something very important to remember when creating these regular expressions is that whenever you create a parameter value and check the Regular Expression box it is automatically setup as a POSITIVE regular expression.  Therefore whatever is in this box defines what is legal for this parameter/field.  In the example above if a TARGET value is submitted to the web application it must contain “mycompany.com” or “myurlpath” or it will be shot down by the ASM.  This will prevent someone from setting a target of somewhere other than your web site.  This will stop a blatant open redirect attack but certainly not all.  Then click the create button.

Now you will need to tell your web application policy to be on the lookout for violations of this type.  Navigate to Application Security, Policy, Blocking, Settings.  Then scroll down the list until you see “Parameter value does not comply with regular expression”, check the Learn, Alarm and Block check boxes.  Save and then Apply the policy.  That’s it!

When ever a violation happens you will now see this in the manual traffic learning section:

Now to tackle the “user” parameter.  I am going to take a different angle on this one because like I mentioned before, once you understand the principal behind it you will see it can be used in a million different ways to protect your web application.

After looking over a few security logs you might notice that some hackers attempt to utilize the “user” parameter/field in your web application and they will try to throw all kinds of things in there.  One common element I have seen is that they will try to inject a username@yourdomain.com into the field.  Since that is not a valid character for the application I am looking to protect, I am going to block this kind of attack configuring the ASM to block based off of an invalid metacharacter value being placed in the parameter value.

Following the instructions above for creating a new Parameter, except this time instead of using a regular expression, click the Value Meta Characters tab.  Select “@ (0×40)” from the list on the right hand side of the page and then set the value to be disallowed using the drop down box under the set state heading.  Put a check mark in the check characters on this parameter value check box.  Now to configure your web application policy to listen, alarm and block on these kinds of attacks.  Navigate to Application Security, Policy, Blocking, Settings.  Then scroll down the list until you see “Illegal meta character in parameter value”.  Check the appropriate boxes, save and then apply.

Now whenever a would be hacker attempts to inject an invalid character into that field (the @ character in this case, but like I said you can use countless others) they will be smacked down by the ASM.

It’s a piece of cake really once you do it a time or two.  If you get hung up on the regular expression part have no fear!  The kind folks over at F5 Networks have thought ahead and have included a regular expression validator inside of the ASM module.  Just navigate to Application Security, Options, Tools and RegExp Validator.  You can use that tool to compile your regular expression if need be.

Remember when thinking about security related things it is best to take the defense in-depth approach.  Little things added here and there to your web application security policy that do no harm but can mitigate attacks can be very effective.

Now some of you may remember that last year’s Box of Awesomeness contained a wide variety of totally awesome gear (hence the name ).  This year is certainly no exception as the folks over at DevCentral have outdone themselves yet again!   This year it contained something so fantastic that I almost made the title for this entry “f5 Networks – The Box of Insanity”!  Yes, it is that crazy good folks. But first I have to say….

I can’t tell you how much I have enjoyed being a member of the f5 Networks MVP program in 2010. Thanks in large part to DevCentral, I have learned a tremendous amount about BIG-IP over the last year. I look forward to contributing to the community in 2011 and would like to thank f5 Networks for being so gracious to all of the f5 MVP’s. They have supplied us with knowledge through DevCentral and unprecedented access to the inner workings of BIG-IP and TMOS through the MVP Summit. Not only that, but they have also supplied us with all of the tools that we need in order to give back to the community.

It must be said and recognized that they supply all of this with no questions asked, no demands and no conditions attached.  It sounds unbelievable, but it’s true.

So my hat is off to all of you at f5 Networks and to my fellow MVP members. Thank you for the great gear and thank all of you for supporting the community like you do!  To those of you out in the community, whether you are just getting started or have some experience under your belt I have a message for you.  Contribute what you can, when you can and as often as you can.  There is no f5 Networks community without all of you.

Now for the pictures:

No, I am just kidding! They didn’t send us Chia Pets, it’s just that we are so close to April fools day I couldn’t resist. Now on to the real pics!

The side of this box is definitely going to be tacked up on the wall of my cubicle at work!!!!!!!

And yes, it is INSANELY AWESOME.

This was prompted by the fact that I noticed recently that there is not a lot of documentation available on the web regarding the F5 BIG-IP’s Web Scraping Protection mechanism and almost none regarding what it actually does to the underlying web page code presented to your end users.

Web scraping is defined as a computer software technique of extracting information from websites.  The people people running the web scraper program typically save the contents of what is scraped and use it for their own means.  Sometimes it is just for archiving purposes, such as Archive.org’s “WayBackMachine“.  Several companies even sell what is considered by many to be legitimate commercial web scraping software.  One such company is called Mozenda, who lists such clients as Microsoft, IBM and Citi.

But then there are the “Others” as I like to to call them.  This can range from hackers with bad intentions to companies simply seeking a competitive advantage over another company. One example of this that I  can think of dealt with a few websites who make their living by offering vacationing deals.  So these leaders of their industry would publish airfares for many popular destinations on their websites and their competitors would use a computer program to scrape the pricing off of their pages. They would then take this pricing, subtract a few dollars, load it into another program and update the pricing on their own website thereby making their vacation deal offerings just a little cheaper than their competitors!

Web scraping is not an illegal activity, but it can be against the “Terms of Use” for some websites.  Now, all of that being said, it is definitely nice to know that the BIG-IP ASM has a built in feature that you can enable to protect your own websites from being scraped.

It does this by attempting to determine whether a web client source is a human or if it is a headless computer program.  To do this it injects a piece of java script code into the headers of your HTTP traffic. I will not provide the full source code for the java script, but I will hopefully provide enough for those searching through Google to be able to find this page.

When you are viewing the web page being protected by an ASM and web scraping anomaly detection is being actively used to protect the web page you will see the following elements. To actually see these elements, open up Firefox, browse to the website in question and then right-click and select “View Source”. You should see a java script insert beginning very close to the top of the page that contains some of the following elements:

var jsepee
jsepee CSHUI_RANDOM_DATA_NODE
CSHUI_RANDOM_DATA_NODE’]!==undefined&&jsepee['
CSHUI_RANDOM_DATA_NODE
CSHUI_COOKIE_NAME']=jsepee['CSHUI_RANDOM_DATA_NODE
CSHUI_COOKIE_VALUE_TRUE']=’true’+'_’+jsepee
CSHUI_RANDOM_DATA_NODE
CSHUI_MONITOR_KEYBOARD’]=true;jsepee['CSHUI_MONITOR_MOUSE
CSHUI_MOUSEMOVE_EVENTS_TARGETCSHUI_MOUSEMOVE_LAST_X_LOCATION
CSHUI_MOUSEMOVE_LAST_Y_LOCATION']=0;
CSHUI_MOUSEMOVE_IS_CONTINUOUS
CSHUI_KEYBOARD_EVENTS_TARGET’]=1;jsepee
CSHUI_KEYBOARD_EVENTS_COUNTER

You can seen by looking at these events that it is looking for keyboard, mouse and other data to determine if the content is being looked at by a human or something that falls in the OTHER category. Once it has made a determination the web application security policy will follow whatever guidelines you have set under the policy settings.

So there you have it, yet one more reason why the F5 BIG-IP ASM is an excellent tool to be included in your defense in depth lineup.

I have used the WAM (WebAccelerator Module) to accelerate a few SharePoint 2007 sites in the past and have been able to achieve a 45%-55% reduction in the number of hits on our web front end servers.  To me that is a pretty dramatic reduction to say the least.  Those servers have since been upgraded to SharePoint 2010 so I will hopefully be doing another blog post in a few weeks where I will show you how we use WAM to accelerate SharePoint 2010 web applications.  In this post I am going to cover using the default WAM IIS template to accelerate our main web site and show you the results.

Previously I was using TMOS Version 9.x so starting off couldn’t be more simple in Version 10.2.  One very nice thing that I want to point out with this version is that when you click on the WebAccelerator section in the GUI it no longer opens up in a separate window.  That used to really annoy me and I was glad to see it is more cohesive in this version.  After clicking into there, click the “Applications” menu option and then click “Create”.  Type in a name, select the central policy template that you want to use (MS IIS in my case), type in your requested host name and click save.

You then create a Class Profile by clicking “Class Profiles” and “Create”.  Assign a name to it and leave the default values as they are.  That way if you decide to change  or modify something in that profile in the future you can easily do so and it will not effect any of your other profiles.  Then go back into the Local Traffic portion of the GUI, select the Virtual Server that you want to add the policy to, click “Resources” and then click the “Manage” button under HTTP Class Profiles, select the newly created acceleration profile in the list, click the << button to add it to the list and then click the finished button.

That’s it ladies and gentlemen!  You now have a accelerated web site.  How easy is that?!  I can’t imagine it being any easier than that, of course those folks at F5 Networks are always improving things.

So what kind of results can you expect from such a simple setup?  Well lets take a look.  From the graphs below you can see that the BIG-IP WAM has a response time of about 21ms for content requests.  This is the length of time it takes the WebAccelerator system to respond to a request from the client.

The second picture below shows you that the unit responded to 48,000 requests and the unit was able to successfully accelerate around 37,000 requests via Smart Cache.  That is a lot of happy users and represents 37,000 requests that our web servers did not have to respond to!  The errors that show up in the report are mostly my fault because I have not cleaned up my traffic reports like George Watkins explains how to do over on DevCentral http://devcentral.f5.com/weblogs/watkins/archive/2010/08/18/clean-up-those-webaccelerator-performance-reports.aspx.  Thanks again George for that excellent post!  Once I have had a chance to clean those reports up I will try to post a prettier picture.

Then last but certainly not least, is a picture that shows you my CPU utilization on that particular unit over the last 24 hours.  This picture was taken roughly 9 hours after implementing the web acceleration profile.  As you can see there has been only a slight increase (maybe 1%) in my CPU utilization.

Looking at those facts it is safe to say this was a very successful deployment of a WAM profile on a production web site that has generated some very positive results.  Faster responses for the end users and less load on the back web servers, it is a win-win solution in my book.

From time to time, I usually receive a request that goes something like this.

“I have a pair of F5 ADC in an Internet DMZ, where the servers behind the load balancer need to access NAS system(s) on a VLAN located in the same network on another VLAN that is not behind the load balancer.

The problem is that in my current design I have to route through the F5 Load balancer to access the NAS system(s).  Unfortunately the amount of bandwidth it takes supersedes the F5 ADC’s total throughput.  I would like to by pass this without adding extra network cards or recreating a new VLAN and would like preserve the IP addresses as much as possible.”

For the purposes of the blog we will call the person requesting this Keyser Söze

Based on this description above you extrapolate a high-level logical network design as shown in Figure 1.

Figure 1

In the figure 1, we VLAN 10 which is a routable VLAN. VLAN 12 is an empty VLAN, which is strictly Layer 2, and no other traffic allowed to it from the router itself.  Finally we have VLAN13 which is where the NAS servers is connected to .  In order to access VLAN12 you need to route through the F5 that is also connected on VLAN10. This is done by a static route pointing to .11 on VLAN10 which is the F5 floating address on VLAN 10 to reach VLAN12 address block. In figure 1 you also have all servers in VLAN12 pointing to .1 as their default gateway which is the floating address of the F5. The F5’s default gateway is .1 on VLAN10. Now that we have described the current behavior of Figure 1, we can start looking at making some changes.

So how do we change the network to accommodate the result that Kyser is looking for? It is actually much easier then you might think.

For the purposes of this explanation, let us assume the switches are connected on Cisco Switch routers

The first item you want to remove is the the static route on the switch pointing to point to .11 on VLAN10 to access VLAN12. You will not need this since the end result is to allow VLAN 12 and VLAN 11 to communicate directly via the Cisco Switch router.

Next you will need to change VLAN11 from a non-routable network to a routable network. Thus, VLAN 11 will have a gateway of .1 on the switch router. The F5 will then change its own floating address to say .11 and subsequently change the self-addresses. All the servers will continue to use .1 on VLAN11 as their default gateway.

Thus the network will now look more like Figure 2

Figure 2

At this point, you are thinking well if that is the case then how do we get traffic back to the F5 for Load balancing traffic. Well the easy way is to apply SNAT Automap across all the Virtual addresses. Which works, but then you run into another problem where you lose the client IP address. Normally this might be work, BUT if you are tracking clients for statistical purposes, this is not going to work.

The short answer to this is utilizing a Cisco’s Policy Based Route. How does that work?

On a Cisco switch you can do the following configuration (IOS Syntax):

ip access-list extended from_vlan11
Deny y.y.y.0 0.0.0.255 z.z.z.0 0.0.0.255
Permit y.y.y.0 0.0.0.255 any
route map to_lb_vlan11
Match ip address from_vlan11
ip default next-hop y.y.y.11
interface Vlan11
ip policy route-map to_lb_vlan11

What these statements mean is that any traffic from VLAN11 is destined to addresses on VLAN12, skip the route-map statement and use the internal routing table of the switch. Thus allowing VLAN11 to communicate directly to VLAN12 and vice versa. Subsequently, if traffic from VLAN11 is attempting to talk to the internet then it will match the permit statement in the IP access list “from_vlan11” then apply the route map statement and thus your next hope is .11, which is hosted on VLAN11.

That pretty much sums up how to use the switches throughput for VLAN to VLAN traffic and the F5 ADC continues to do what it does best while Kyser can go home happy.

Thanks,

CB

Not having a default monitor to turn to in this situation and having only tinkered with external monitors before, I began searching around for a way to setup an external monitor that could log on to the SharePoint sites to perform the health check.  Naturally I turned to DevCentral and did a little digging around on the forums.   That is where I found a wonderful post by stp1978 that laid out the basics of what I needed to do.

I will try to write this post in a way that will explain to someone who has never setup an external monitor how to set one up and who knows there may be someone out there who is looking for a way to monitor a SharePoint 2010 web site that uses NTLM.

The basic installation steps are:

1.  Prepare the script that will run.
2.  Create a service account so the BIG-IP can log on to the SharePoint Farm.  This will be used by the monitor to log into the various websites.
2.  Copy the script over to your BIG-IP and change the permissions so that it can be executed 0777.
3.  Log on to the BIG-IP GUI and create the external monitor.
4.  Apply the monitor to the pool.

If you are running a highly available pair in a sync group, it is ok to do this on the active unit and when you are done run a config sync.  This will copy the monitor and script over to the standby unit and you will be good to go if you have a failover event.  You don’t have to manually copy this over to the other unit.

The script (code supplied by stp1978)

#!/bin/sh
# This removes the IPv6/IPv4 compatibility prefix.  This has to be done because the LTM passes addresses in IPv6 format.
IP=`echo ${1} | sed 's/::ffff://'`
PORT=${2}
PIDFILE="/var/run/`basename ${0}`.${IP}_${PORT}.pid"
# This will kill off the last instance of this monitor if it is hung and logs current PID
if [ -f $PIDFILE ]
then
kill -9 `cat $PIDFILE` > /dev/null 2>&1
fi
echo "$$" > $PIDFILE
# This is the meat of the code, it is responsible for sending the request & checking for the expected response.
curl -fNs --ntlm -k -v --user 'YourUsername@YourDomain.com:YourPassword' http://${IP}:${PORT}/_layouts/RecycleBin.aspx -H "Host: YourWebsite.com" | grep -i "deleted" 2>&1 > /dev/null
# This part of the code will mark the node UP if the expected response was received.
if [ $? -eq 0 ]
then
echo "UP"
fi
rm -f $PIDFILE
exit

The code above is commented very well and explains what each step does so I will not reiterate it here.  The parts that you will have to modify are of course your username, password and domain.  I created a service account in the domain and I use it to log onto the site with.  That way you don’t have to worry about the password expiring and you can limit your security risk by giving the service account only enough access to be able to get to the recycle bin on the SharePoint 2010 site in question.

You will also need to modify the URL string and the text that the BIG-IP searches for when it logs in and opens the page.  I thought it would be good to search for something simple and something that will likely never change.  In SharePoint 2010, your safest bet is probably to utilize the RecycleBin.aspx and search for the word “deleted”.  The way I see it this is the safest thing to check for.  This way it doesn’t matter what content gets changed or deleted on the site by the users, they can’t accidentally delete the recycle bin!

A small suggestion at this point… I HIGHLY recommend that you use something like Textpad to edit the file.  Using wordpad can have unintended consequences and may even mess the file up so much that the monitor will not work correctly.  Also be sure not to include a file extension on the end as it does not need one to work properly.

Using a program like WINSCP, copy the script over to the BIG-IP into the /usr/bin/monitors folder.  Then right click the file you just copied over and click properties.  Edit the permissions on the file to allow root to execute the file.  I just set the permissions on the file to 0777 as seen in the screenshot below.

Then log on to the BIG-IP GUI and create a new monitor.  Click create new monitor, select external monitor from the drop down menu, give it a name and then in the “External Program” field type the name of the file you copied over.  You don’t need to include the directory or a file extension, just the name.  Adjust the timing settings to your preferred time settings, I use 10/32 as seen in the screen shot below:

Then go and apply the monitor to your pool.  That’s it!  Now you have a fully functional external monitor that can check the health of your NTLM SharePoint 2010 web sites.

Thanks again to stp1978 for his hard work on this and for putting it out there in the community for others to utilize.

TMOS is gaining a wealth of new functionality with each release and word of what you can achieve through using iRules is spreading even to those unfamiliar with the BIG-IP product line.  I have personally seen this discussion pop up more than once and we even grappled with it at the MVP Summit in Chicago. 

I can’t help but reflect back on the book “The Art of War” by Sun Tzu when thinking about this subject.  During the summit I realized that we were pretty much attempting to do the same thing that Sun Tzu did.  To come up with tactics and lay out truths that could be relied upon to come to a logical decision about how to proceed.

With Sun Tzu, his end goal was to win the battle or war that he was fighting.  He wrote roughly 80 pages of tactics and guidelines for fighting war.  I think the same thing could be done simply to answer the question to use an iRule or to not.  The problem is that for those of us in the F5 community, is that generally speaking, we all have our own goals.

That makes setting guidelines to follow a little harder unless you first define two very important aspects.  I think the first question you should ask yourself is what is your role in your organization?  Secondly, what is the role of the F5 BIG-IP device(s) in your organization?

Something that I know without a doubt is that we all fill different roles in our respective companies and so do our BIG-IP devices.  There is no one size fits all answer to this unfortunately.  For those of you who are new to working the BIG-IP product line and those of you who have yet to set any real company policies regarding your use of iRules I have one small word of advice.  I urge you to sit down with your boss and talk about what you stance will be regarding iRules moving forward.  If you ARE the boss then I suggest thinking about this matter in depth and reflect not just on how it effects you but also your team.  I have no doubt that doing this in advance will save you a lot of trouble.

What are the topics you should think about?  What are all the possible gotchas that might come up?  It is again different for us all.  After having pondered this question myself, here are a few things I think one should keep in mind and discuss with their peers/boss:

1.  K.I.S.S. – That’s right, keep it simple stupid.  It’s a best practice that we should all follow.  The question though is this, will using an iRule make something simpler for you or more complex?  If it makes something simple it’s a no-brainer right?  It it makes things more complex?  Where do you draw the line?

2.  If you do use an iRule and you decide to do some complex logic in it, are you legally required to keep track of that code in an application code repository?  Different regulatory items will obviously apply depending on the nature of your business.  I know that in a lot of places that if one were to write complex iRules that changed the data that a customer see’s, then they would most certainly have to keep track of that.  Sometimes though, it is not external regulatory compliance but INTERNAL regulatory compliance that you have to think about.

3.  Who will support it?  If you write a really complex iRule who will support it in the future?  Are you prepared to redo an iRule at two o’clock in the morning because of a production update that a developer pushed out changed the code that your iRule relies upon?

4.  Let’s say that an opportunity to use an iRule has already presented itself.  Is it more cost productive for the business for the iRule writer to craft an iRule to fix the problem or to have the application programmers fix the problem in the code?

5.  What about your physical environment variables?  Can you implement this new iRule code without slowing down everyone else’s application traffic (provided you delivering multiple apps through it of course)?

6.  Perhaps it will come down to your boss looking at you and saying, “How comfortable are you writing an iRule to try to do this?”.  If that is the case and you are uncertain, then by all means head on over to the DevCentral forums and create a post about it!  You would be AMAZED at the things that people have done with iRules and AMAZED at how simple some of those things are to pull off!  iRules, it slices, it dices, it… well you get the idea.  Use the community to bounce ideas around because it can definitely help make that decision much easier for you to make.

7.  What approach should you take in general to iRule or not to iRule?  Should you take the look before you leap approach, always say yes or always say no?  I am sure that most will pick the look before you leap approach just to make certain they can do what they need to do using an iRule programmatically, that they can do it efficiently and that doing so meets their other preset criteria.  It also may be that your role in the company and the role of your F5 BIG-IP device is strictly that of a networking device and iRules are not to be used or developed.  If that is the case, I would urge you to reconsider that stance and at least consider using some of the simpler iRules… please see comment #6 above.

I am sure there are a million more questions you can think of to ask that might be relevant to your current working conditions, this post is by no means a definitive guide.  Please feel free to add a comment to this post regarding things that may have helped you and your organization define your policy towards using or not using iRules.  I really would love to hear them.

It is wise to remember what Sun Tzu said of laying plans, “The general who wins a battle makes many calculations in his temple before the battle is fought.  The general who loses a battle makes but few calculations beforehand.  Thus do many calculations lead to victory, and few calculations to defeat; how much more no calculation at all.”

I have had the pleasure and honor of attending the DevCentral MVP Summit that was held in Chicago over the last few days and I am just blown away at how awesome it was.  Even the picture on the right doesn’t do it justice!

Whew!

The folks over at F5 Networks did an amazing job of putting together an outstanding series of events and the DevCentral folks put together one heck of a summit for all of us MVP’s.  I had such an outstanding time I want to extend my thanks to all of you involved and to all of my fellow MVP members.

Now to break things down a bit.  After flying up via American Airlines from Fort Worth to Chicago, I was finally able to locate my ride to the Chicago Swissotel.  We had a very pleasant conversation on the way to the hotel, the driver having previously driven Nelson Mandella around Chicago.  So we talked about that some and we talked about the downsides to film making, as they were currently shooting scenes for the movie “Transformers 3″ on the route that we needed to use to quickly get to the hotel!  I believe one MVP member saw Optimus Prime going down the road at one point and Colin mentioned looking out of his cab and seeing a guy with a high powered rifle crouching by some shrubbery!  I am sure the first was Transformers 3 related, the other well… we will assume it was to.

Upon arriving at the hotel I was really amazed at how nice the place was.  It was easily the nicest hotel I have ever stayed at and the veiw from my room was incredible.  I didn’t get any pictures of the room, but I did take several pictures of the surrounding buildings and landscape.  I will try to get a few of those uploaded sometime soon for everyone to check out.

The next morning we kicked off the MVP Summit close to 8:00 A.M., even though it was scheduled to begin at 8:30 A.M.  We were all pretty eager and excited and no one minded one bit.  We then proceeded to have deep dive technical conversations with all sorts of people.  We covered everything from the guts of the physical hardware, to the guts of the software responsible for squeezing every ounce of performance out of those units.  I can tell you without a doubt that F5 Networks is commited to delivering the best product that can be delivered on the market today.

We weren’t given sales presentations or anything remotely close to that.  We were given introductions to the very people responsible for doing the motherboard and chip designs, the folks responsible for creating new attack signatures for the ASM module and even the folks responsible for programming TMOS!  They came in, gave us intro’s to who they are, what they do and then it was an open floor to discuss EVERYTHING we and they could think of.  Can you imagine having unfettered access to tweak the brains of the folks creating the technology that you interact with daily?  To say it was exciting, fun and technical would be a severe understatement.  What really stood out beyond the all of this to me though was the fact that these very people were intensly interested in our feedback on their ideas.  I don’t know how many times we would break up into small side conversations where we could take turns extracting tidbits of information from one another.

Yes, there is more (like the fun little contests we had in between each major discussion) but most of it is covered by a NDA agreement so I can’t spill the beans about it.  It’s safe to say F5 Networks has a good future ahead of it and not just because of the plans they have already laid.  I walked away from the MVP Summit that evening feeling much more knowledgable and I have no doubt that several of the F5 folks walked away feeling the same way and making plans in their minds to tweak things based on things we discovered in our talks.

Then as it turns out, they had more surprises in store for us that evening!  We scored some awesome loot earlier at the MVP Summit, thanks again guys for the gear it is all fantastic (and will be featured in another post!).  So after we carried our loot upstairs we all walked over to a local pizza place and into a nice area that F5 Networks had reserved for us all to grab an adult beverage and chow down on some authentic Chicago style deep dish pizza!  The food was great and so were the conversations.  I am certain a good time was had by all.

The next day was just as great.  We were provided access to customer sessions, I met all kinds of people from F5 Networks and I even got a few compliments on my cowboy hat!  Hehehe… Each of the MVP’s also had a chance to do an interview and George from F5 Networks was kind enough to interview me.  You can check that out here:  http://devcentral.f5.com/weblogs/dctv/archive/2010/08/04/f5-customer-summit-ndash-nathan-abbott.aspx

The customer sessions “Meeting Users’ Needs”, “Managing Scale and Growth” and “Security and Control” were all very good that afternoon.  I can’t say that I saw them all, but I did hear from others that they were generally quite exceptional.  I bounced around a lot that afternoon talking with different people so I did miss out on some workshop goodness I guess, but I just couldn’t help myself.  Later that evening we where all jumped on a bus, a few busses actually, and went to The Field Museum Chicago.  F5 Networks reserved the whole museum so we had free run of the place!  I really enjoyed walking around talking about BIG-IP stuff, looking at mummies and Sue the T-Rex!

Two interesting facts I picked up at the museum, Sue’s head is actually on display on the second level of the building because it was just to heavy to mount with the rest of the skeleton.  Her head alone weighs over 600 pounds!  Second, they are still using Mac OS 9 on some of the interactive kiosks in the museum and I will leave it at that….

To cap the evening and the whole experience off, F5 Networks brought in reknown blues guitarist and singer Robert Cray.  I am not really into music, I do enjoy some classical and country music on occasion, but Robert Crays performance was outstanding.  We happen to be coming out of one of the exhibits as his keyboardist was just shredding it and it was great getting to see him tear it up.  The band was into it, the crowd was into it and it just made for a great time all around.

The last day finished up with a great general session for all.

That pretty much sums up my experience there at the DevCentral MVP Summit.  I do want to mention that on the plane ride home I happened to end up sitting by very nice fella that is a Product Manager for Alcatel-Lucent.  I apologize for not remembering your name, but I remember you said you would check out my blog and I wanted to tell you thank you for the great conversation!

It was mentioned at a few different points during the summit that we will hopefully get to perhaps hold another summit sometime in the future.  I certainly hope that I am lucky enough to be chosen to participate when the time comes.  Again, to all of you folks there at F5 Networks, the DevCentral Team and my fellow MVP’s, Thank You. My hat is off to you for making this such a grand MVP Summit!

For those of you wanting to view more pictures please feel free to go over to my Mobile Me gallery for more: http://gallery.me.com/nathanabbott/100130

One way to do this is called transparent header modification. How it works is a user will enter a URL in their browser such as “www.mycompany.com/bus/”, the request will come in to your BIG-IP and the information sent to your web servers can be redirected or rewritten to whatever you like. Here is an example:


when HTTP_REQUEST {
switch -glob [string tolower [HTTP::uri] ] {
"/bus/*" {
HTTP::uri "/greyhound/bus"
}
}
}


Using the iRule above, this is what happens to your incoming HTTP request. The request comes in and the URI is converted to lower case and then inspected to see if it begins with “/bus/”. The asterisk indicates a wildcard, so anything could come after “/bus/”. If it does begin with “/bus/” then the URI will be transparently modified or changed to “/greyhound/bus”. The clients browser will not be updated, but the URI that the BIG-IP passes on to the server will be “/greyhound/bus”. Basically it turns a request for this “www.mycompany.com/bus/myrequest” INTO “www.mycompany.com/greyhound/bus” Pretty cool huh?

Now lets say you want to do something a little more exotic. Lets use the iRule from above in a different way.


when HTTP_REQUEST {
set uri [HTTP::uri]
switch -glob [string tolower [HTTP::uri] ] {
"/bus/*" {
HTTP::uri "/greyhound/searchBus.do?stationName=[string range $uri 5 end]"
}
}
}


What is this one doing? Let say an HTTP request comes in for “www.mycompany.com/bus/texas”. Using the iRule above the web server would actually receive a request for “www.mycompany.com/greyhound/searchBus.do?stationName=texas”. The clients browser would still read “www.mycompany.com/bus/texas”. Like I said powerful and flexible.

If you are interested in more content regarding transparent header modifications a.k.a. redirecting users without changing their URL, then I recommend reading this article by Joe Pruitt on the DevCentral website http://devcentral.f5.com/weblogs/Joe/archive/2005/07/27/ModifyingUriWithoutRedirect.aspx

Setting Up BIG-IP and Live Meeting Portal Server

Prerequisites:

Please consult the Live Meeting Portal Server documentation and ensure that your servers meet all the perquisites before installation. All the examples in this guide are setup so that you will end up with a website at this URL: https://livemeeting.mycompany.com/lmportal. Please feel free to substitute your company’s name for “mycompany”.

IIS Setup:
1. Download the latest version of Office Live Meeting Service Portal. As of 4/20/2010 that can be found here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=429bb528-fd1b-45b7-af2b-cbbf4a8e65ff&displaylang=en

2. Create a basic website in IIS and name it Live Meeting. This empty shell of a website will be used by the Live Meeting installer and will basically be taken over by it after you run through the installation.

3. Create a folder named “Livemeeting” in the directory of your choice. In this example we will use ”E:\web\content\”

4. Double click the lmportal.exe to begin the installation and choose custom when the option appears. Then select the directory you created above so the files will be placed in your normal custom web content location.

5. Remote Desktop (RDP) to the web server and open IIS. DO NOT USE THE IIS CONSOLE ON YOUR LOCAL MACHINE as you will not have access to everything that you need.

6. The screenshots below will help guide you through the configuration of the web site in IIS. Things that do need to be changed:
a. Add 443 to the SSL port and select the unique IP address for the site to use. We will be terminating SSL on the F5 BIG-IP and then re-encrypting before sending it back on to the server.

b. Allow Scripts and Executables under execute permissions. Verify application pool is set to Live Meeting Intranet Portal AppPool.

c. Verify that ASP.NET is set to version 1.1.4.322.

d. Under Directory Security, click Edit and make sure there is a check mark on the “Enable anonymous access” and “Integrated Windows authentication” box.

e. Go to the application pool, right click and go to properties. Click the Health tab and uncheck “Enable Rapid-Fail protection”. Not including a screenshot of this one.

7. Navigate to “E:\web\content\Livemeeting\Portal” on the server. Then find the file named “Portal.config”, right-click it and click the Security tab. Click Add and then add the “Network Service” user account and give it full control. You have to do this or you cannot modify the configuration settings from the GUI.

8. Do the same thing listed in step 7 for the “PortalExport” folder located in the directory you should currently be in: “E:\web\content\Livemeeting\Portal”

9. Now you have to import the SSL certificate that you are going to use into IIS website that you just set up. You will need to obtain the .crt file for the SSL certificate and the .key file for that certificate. We terminate our SSL on the BIG-IP so these can both be obtained from there. I will skip the steps regarding purchasing an SSL certificate for a site if you do not already have one. It kind of falls outside the scope of this guide.

10. Use a search engine and search for OpenSSL. You should find their homepage at: http://www.openssl.org/

11. Download OpenSSL and install it on your Local machine. I don’t recommend installing it on the server for a wide variety of reasons. I installed my copy of OpenSSL into “C:\OpenSSL”.

12. Take the .key file and the .crt file and put them into OpenSSL’s “bin” directory. It’s just a folder inside of your OpenSSL folder called bin.

13. Open a command line and change directory over to C:\OpenSSL\bin. The example I am going to provide is for a fictitious company named “MyCompany” that is using a wildcard ssl certificate on a few of their websites.

14. Then type in the following command:

This all needs to be on one line. Spaces are ok, but no carriage returns or anything like that. This command is modeled after this example for future reference:

openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt -certfile CACert.crt

certificate.pfx = the name of the new pfx file you want to create
privateKey.key = the private key you got off of the F5 BIG-IP
certificate.crt = the crt file that you got off the F5 BIG-IP
CACert.crt = the crt file that you got off the F5 BIG-IP

15. After you type the command and hit enter, you will be prompted for a password. You can use any password that you like but you will need to remember it because IIS asks you for the same password when you go to import it.

16. OpenSSL will compile a new .pfx file for you in the C:/OpenSSL/bin directory. Take that SSL certificate and copy it over to your web server.

17. RDP over to the server and open IIS. Again here is the disclaimer, DO NOT USE THE IIS CONSOLE ON YOUR LOCAL MACHINE. Right-click on the Live Meeting web site that you created and click on the Directory Security tab. Under “Secure Communications”, click the “Server Certificate…” button.

18. Click Next and then click the “Import a certificate from a .pfx file” radio button and click next. Browse to the .pfx file that you uploaded to the web server. Click next and enter your password information that you used when you created the certificate. Then finish clicking through the wizard. Then restart IIS on the server and delete the certificate off of your local machine. This completes the IIS setup. Now move on to the Live Meeting Portal setup.
Live Meeting Portal Setup

19. Navigate to the URL:

https://livemeeting.mycompany.com/LMPortal/settings.aspx

Where livemeeting.mycompany.com is the name of the website you setup. The screen will look like the one shown on the next page. This is the Settings-Portal Configuration page. You will want to use the following settings which are also pictured in the screenshot on the next page.

Conference Center URL = https://www.livemeeting.com/cc/mycompany
Conference Center Administrator
User Id =
Password =
Email address for escalation =
Enabled Portal Services = Check the Account Create, Account Login, Account Update and Web Method Calls
Ticket Timeout = 300 Seconds
Directory Service Parameters = AccountNamePolicy=LogonUsername

20. Then click Save. If you receive an error at this point, refer back to step #7.

21. Click on the Roles link on the left side of the page. This will take you to the Roles-Portal Configuration page. Under “Live Meeting Administrators” add the users who will be the Live Meeting Administrators. Use domain\name format. IE: mydomain\username
22. Then under the “Live Meeting Organizers” settings I recommend adding the “Domain Users” from the varies domains on your network. So if you have three domains on you network named ABC, 123 and XYZ you would list ABC\Domain Users, 123\Domain Users and XYZ\Domain Users.

23. Then click the “Export Configurations Settings” link on the left hand side of the page. This is not really labeled right because what it actually does is back up your configuration. If you mess something up in the running configuration, simply click on the “Import Configuration Settings” to restore the last configuration that you exported.

24. Then click on the “Events” link on the left side of the page. Change the log file directory to a directory that you want to have all the logs written into. In this example I chose the E: drive of the server I was working on. Whether you create a new one or use an existing one you must make sure that the “Network Service” account has permissions on that folder to Read, Write and Modify. Otherwise you will receive a nasty .NET error when you go to save the changes you just made. Click Save.

Live Meeting Portal Server BIG-IP LTM Setup

The BIG-IP LTM set up for this can be very easy to configure. You will need to create nodes for each of your web servers, assign them to a pool named “Live_Meeting_Pool” and then create a Virtual Server for the application. I named my virtual server “Live Meeting” in the example pictured below. You may need to customize it to match your environment, but the basic settings are:

Service Port: 443
Type: Standard
Protocol: TCP
Protocol Profile (Client): tcp
HTTP Profile: http
SSL Profile (Client): wildcard
SSL Profile (Server): serverssl

I also assigned the Live_Meeting_Pool to the Virtual Server, set the Default Persistence Profile to “Cookie” and Fallback Persistence Profile to “source_addr”.

Before I came to the class I could build a security policy and assign it to a website and do some minor tweaking.  Now I can say with confidence that I can build a web application security policy that is PCI compliant and has a solid foundation.

One of the main ingredients for a successful training session/class is you really need an excellent instructor.  If the instructor doesn’t know his stuff or doesn’t really enjoy the subject matter it can have a negative and direct impact on the course.  The class I took was lead by a gentlemen named Keith Bowers who has worked for f5 Networks for 10+ years.  Granted, I could be wrong about number of years, but I think I am close.  I can say for certain thought that Mr. Bowers knows the material and he seemed to really enjoy teaching the class.

This wasn’t the kind of class where you go and read along with the teacher word by word out of the book.  Keith gave very concise and well thought out lectures regarding each subject that we touched on.  I say concise because he said everything that he needed to in order for you to comprehend the material and to be able to apply in a real world situation.  Then he would provide guidelines for the hands-on portion of the lab for that section and turn us loose on the BIG-IP box that each student gets to all to his or her self.  When a student had trouble getting through a lab he would sit beside them, provide information on things to look for and provide clarification on things until the student got through the lab.  He was really good about teaching you to fish rather than just giving you an answer out of the teachers edition of the manual

So what kind of goodness can one expect to learn at an ASM 10.x course?  Here is a brief list of the things that we covered:

Installation
Web Application Concepts
Web Application Vulnerabilities (with instructions on how to perform a few basic hacks)
ASM Application Configuration
Security Policy Building
Creating Custom Attack Signatures
Reporting
Traffic Learning
Protecting XML and Web Services
And more…

On the second day that I was there I also had the chance to meet up with a few members of the DevCentral Core Team!  I was able to bounce out of class a little early so Joe met me outside the training room and proceeded to give me a tour of the place.  At one point I tried to slip a VIPRION into my cowboy hat and almost made off with it but the 30+ blue ethernet cables sticking out from underneath my hat gave me away.  Alas, I had to put it back.  <Sigh>  Seeing that I was upset though Colin, Jeff and Joe provided me sneak peak of their latest TOP SECRET project to get my spirits up.  After the tour that I was given, my spirits were definitely lifted!  I wish I could tell, I wish I could tell…. but I can’t.  It was awesome though.

We then proceeded down to Buckley’s Pub for some lunch and along the way we went over a little bit of history, talked about things that a tourist like me should do when visiting Seattle, etc…  Jeff kindly wrote up a blog article about it and even included a picture that he took of Colin, Joe and I at the pub.  You can check it out here:

http://devcentral.f5.com/weblogs/JeffB/archive/2010/04/01/1088132.aspx

I can’t provide all the details of what we talked about, I was having to good of a time to remember them all.  I know we talked about Bear Grylls (Man vs. Wild), Mac keyboard shortcuts and the MVP Summit… How those are all interconnected I will leave up to you to ponder… Hehehehe… seriously, thanks for a great time fellas.  And also thanks for what you do every day.

Well, if you have made it this far into my blog post you deserve a treat!  Below is a snippet of some videos that I took on April 1st during the training class, some footage from the TOP SECRET stuff they showed me and some footage from the pub!  I had to try out my f5 Networks MVP branded FlipMINO after all!  Sorry if it is a little choppy in a place or two, I had to compress it before I uploaded it to YouTube.

Camera In Cowboy Hat Video

Now that the naming of the entry has been completed, on to the main topic!  I received said box from FedEX this last Friday from f5 Networks and I felt compelled to write a blog post about it and include some pics for your viewing enjoyment.

I can’t tell you how much I have already enjoyed being a member of the f5 Networks MVP program.  It has been awesome from day one and I look forward to contributing more to the community now that f5 Networks has so graciously supplied all of us f5 MVP’s with the tools to do just that.  Thank you for the great gear and thank you for supporting the community like you do!

Here is a list of what was in “The Box of Awesomeness”:

A SanDisk 16 GB USB Flash Drive
A Logitech QuickCam Deluxe for Notebooks for Business
A Logitech ClearChat Pro USB High Performance Audio Headset
A Blue Polo Shirt with f5 Networks logo on the chest
AND
A flip MinoHD Camcorder with a custom f5 Networks MVP skin!

Both of the previously mentioned deployment guides discuss editing files on the Citrix farms Web Interface servers so that it looks for the client IP address in the X-Forwarded-For HTTP header.  Otherwise, every connection will appear to be originating from the BIG-IP LTM and not from its true IP.  After reading both guides and looking at my current environment I was dismayed to find that the files and locations mentioned were no longer valid.  I then turned to my top three resources on the web in the search for an answer: AskF5, DevCentral and Google.

I struck out on the first two (which seldom happens) but my Google search did turn up some interesting results on the Citrix Forums.  I finally found some code posted by Sam Jacobs back in August 2009 that modifies the way the Citrix farm looks up the client IP address.  His method allows for the use of the X-Forwarded-For header.

The first file that you will want to find and edit is the Include.java file.  You will want to locate and change this file on every Web Interface XenApp server in the farm.  Speaking from experience, save a copy of the original file to a safe location such as your desktop or flash drive.  DO NOT copy the file and rename the original to Include.old and leave it on the server.  It may sound crazy, but doing that will not work.  I’m not a programmer, so I cannot tell you why that will not work, but I can tell you I know for a fact it will not.  That being said, here is the file path for the Include.java file:

“\Inetpub\wwwroot\Citrix\XenApp\app_code\PagesJava\com\citrix\wi\pageutils\Include.java”

Now that you have found the file, open it up with a text editor (I use Textpad) and find the Java routine named “getClientAddress”.  Replace the code for that routine with the code listed below.

public static String getClientAddress(WIContext wiContext) {
String ageClientAddress = AGEUtilities.getAGEClientIPAddress(wiContext);
String userIPAddress = wiContext.getWebAbstraction().getRequestHeader("X-FORWARDED-FOR");
if (userIPAddress == null) {
userIPAddress = wiContext.getWebAbstraction().getUserHostAddress();
}
return (ageClientAddress != null ? ageClientAddress : userIPAddress);
}

Save the file and wash/rinse/repeat this step on every Web Interface server in the farm.  The next thing that you will want to do is to modify the login page so that it displays the client IP address being obtained from the X-Forwarded-For header.  The file you will want to edit is called “loginView.ascx” and can be found in the following file path on your Web Interface Servers:

”\inetpub\wwwroot\Citrix\XenApp\app_data\include\loginView.ascx”

The code you will want to add is:

Client IP: <%= com.citrix.wi.pageutils.Include.getClientAddress(wiContext) %>

I added the code directly below the LoginPageControl viewControl line and it works well for me.  Save the file and repeat this step on every Web Interface server in the farm and reboot each Web Interface Server after you are done.  Then it is time for the moment of truth… fire up your browser of choice and navigate to the Citrix login page.  If you have successfully set everything up and have finished following the rest of the deployment guide you should see a screen similar to the one below:

If you receive an error message or the screen doesn’t load, then you might want to go back and check your settings again.  Then that’s it!  I am aiming to develop some custom monitors for the Web Interface Server and for the XML Broker Servers over the next few weeks.  Once I have those done I will put them out in the Devcentral forums for the community enjoy.

I am very happy to mention that the kind folks over at F5 Networks allowed me to submit this as a Tech Tip article which you can find on their site at:

http://devcentral.f5.com/Default.aspx?tabid=63&articleType=ArticleView&articleId=1082335

No problem, I identified all of the expired certificates, checked the box beside them and hit the delete button at the bottom of the page.  After verifying everything was still happy and the support tickets didn’t start flooding my inbox I decided to run a config sync and push the config changes over to the standby box.

The config sync ran without a problem and the gui showed Config Sync: OK.  I then proceeded to check my changes on the standby unit, just for verification purposes.  And that ladies and gentlemen, is when the fun began….

As I was verifying the changes I noticed something I thought was rather strange.  The old SSL certificates that I deleted on the Active unit, were still there in the Standby units SSL Certificate store!  My first thought, oops, my Trusted Device Certificates must be out of whack.  I then proceeded to delete the trusted device certs and ran the “big_ip add” command from the CLI on each unit.  I checked my trusted device certificates and like magic there they were.  I ran another Config Sync thinking that probably fixed the problem, but wait… no such luck.

The Config Sync ran and didn’t kick out any errors, but the old SSL certificates were still in there in all their expired glory.  Frustrated and humbled once again, I decided to run a quick test by deleting a VS on the Active Unit to see if it would be removed once I ran a Config Sync.  I blew away the VIP I use for testing and ran the Config Sync again.  The VS was deleted off of the Standby Unit.  Not knowing off the top of my head what to do next, I then proceeded to open a ticket with my good friends over at F5 Networks.  I didn’t have a lot of faith in my running configuration at the time so I went ahead and opened the ticket as a level 2 ticket (site at risk).

I quickly received a phone call from a Network Support Engineer named Kevin “CB” Midkiff.  We went through the standard procedure of qkview files and few other tests.  After going over the problem Mr. Midkiff proceeded to explain to me that while the SSL Certificates store is indeed carried over when you run a Config Sync IT DOES NOT DELETE SSL Certificates on the unit that you push the config to.  In my case it was the Standby Unit.  The Config Sync function only appends SSL Certificates.

Moral to the story?  If you are double checking your configurations and happen to see some lingering SSL certificates don’t worry, just select them and let the delete button work its magic on them.  Also as an FYI, “CB” was great to work with and very knowledgeable.  Thanks again for your help Mr. Midkiff.

So that got me to thinking… how can someone do this in an iRule?  I have to admit I haven’t really looked into it that much previously because we utilize an ASM module running on a 4100 unit.  The 4100 can do a lot of different things regarding cookies such as checking if a cookie has been modified and if the cookie was obtained in a previous session.  I figured I would hit the AskF5 database to see what I could turn up and I uncovered this little gem:

when RULE_INIT {
set ::key [AES::key 128]
}
when HTTP_RESPONSE {
set decrypted [HTTP::cookie "MyCookie"]
HTTP::cookie remove "MyCookie"
set encrypted [b64encode [AES::encrypt $::key $decrypted]]
HTTP::cookie insert name "MyCookie" value $encrypted
}
when HTTP_REQUEST {
set encrypted [HTTP::cookie "MyCookie"]
HTTP::cookie remove "MyCookie"
set decrypted [AES::decrypt $::key [b64decode $encrypted]]
HTTP::cookie insert name "MyCookie" value $decrypted
}

There is definitely more to this, so you may want to go check out the full solution article here:  SOL7784.  There is also an awesome 2009 iRule Contest entry that you should check out here. The iRule you will want to look at is the Cookie Tampering Prevention iRule written by Henrik Gyllkrans.

I am also very excited to say that I have been selected to be a F5 Networks MVP!

That’s right, TheF5Guy is now an F5 Networks MVP!  I consider it a great honor and am very excited to say the least!  I go by the alias “naladar” in the DevCentral Forums and you can check out my profile here:  http://devcentral.f5.com/Default.aspx?tabid=2242.  You have to be a member of DevCentral in order to view the page, but it is free to join!

Now that the announcement has been made public I wanted to share a few things about the MVP program.  To start with, what’s all of this mean?  It means F5 Networks takes their user community seriously and they want to give back to that community.  This isn’t just an honorary title.  Far from it actually, as there are a number of perks to being an MVP member.

I can’t go into all of them in detail, but here are a few things that I can share since they are mentioned in the podcast.  We will be having regular meetings or round table discussions to go over a wide variety of things relating to the F5 Networks community.  We are being provided profile pages on the DevCentral site to help increase our visibility in the community.  MVP members will be receiving a MVP Kit that was put together with the goal in mind of providing us tools to help us deliver more content to the community.  We will also be having an MVP Summit sometime this year so that we can all meet face-to-face to kick around issues and provide input into the direction of the BIG-IP product line.  Sounds awesome doesn’t it!

This post would of course not be complete without a complete list of the MVP’s so here it is:

hoolio
bhattman
hamish
hwidjaja
smp
naladar
mikejo

The best news is that they want to continue to grow the MVP program.  Do you want to be an F5 Networks MVP?   How do you get started?  Just join DevCentral and start contributing to the community.  They’re watching…..

So what was covered in this “Nexum LTM Workshop”?  Quite a bit actually and it was all very well planned out.  The workshop was lead by a gentlemen named Peter Maag, who is a Senior Security Expert with Nexum.  I believe that part of what made this event so much fun for me was that it was obvious that Mr. Maag knew his stuff and (of course) I like talking about the capabilities of the BIG-IP line. 

Peter began by giving a brief introduction, explaining who he was and the services provided by Nexum.  I have to admit that I was unaware that Nexum was such a versatile company.  I believe it is so versatile and one of the fastest growing private companies because of them hiring and keeping talent like Mr. Maag around.  But that is a different philosophical discussion that perhaps I will touch on at some other time.  If I ever take up being a philosopher.

Then after the intro… it was f5 time!  For those in the audience not familiar with the BIG-IP product line Peter gave an overview of products available from f5 Networks.  He took the time to provide a clear picture of each modules functionality and I feel that he did the products justice.  He then steered the presentation to the real meat of the workshop which was the LTM module.  Virtual Servers, Pool Members and Nodes were all explained as well as the basics of configuring load balancing.  We spent some time discussing the full proxy architecture of the LTM module and we where then guided through a load balancing demo.

This lead into a discussion about monitors, persistence profiles, SSL termination and ended with a demo over those concepts.  There were a few questions at this point, as members of the audience asked questions such as “How long are self signed certificates valid for if they are generated on the f5 BIG-IP?” and “What are the different methods available for Cookie Persistence?”.  All of which were answered concisely and followed up with live demonstrations performed on a BIG-IP unit running TMOS version 10.x.  How cool is that?

We then went into a discussion about iRules.  Peter provided a number of examples of how to use iRules to pull off complicated tasks very easily.  In one example he showed how you could direct web traffic coming from an iPhone to a different set of servers than the ones used to serve up content to standard desktop browsers.  To augment the workshop Nexum provided an excellent booklet which just so happens to have a very handy page that lists almost all of the iRule Events that can be used in iRule generation.

We went over several other things, but the jest of this entry isn’t to really rehash everything that we covered.  The purpose is to encourage everyone using the LTM module to go check one of these workshops out.  Peter Maag did a phenomenal job explaining things for newcomers and veterans alike, which is not an easy thing to do.  To summarize, if you have just recently purchased an f5 BIG-IP product or are looking into purchasing one, attend one of these workshops.  You will walk away a wiser person and I cannot think of a better way to sell someone on f5 BIG-IP products.  Once you see it in action you will be wondering why you have stuck with Brand X for so long.

My next entry will be over the value of attending the SecureWorld Expo.  Is it worth the cost if you had to pay for it out of your own pocket?  What are the driving reasons for one to attend such an event?  I will be asking those questions and more soon and you may be surprised by my conclusions.  Stay tuned.

For those who haven’t had a chance to examine the file at length, the globalfragment.xml file is basically a roadmap that the WebAccelerator module can read to know how to handle or classify different file types.  If you want the WebAccelerator recognize and classify a particular file type it would be good for it to be defined in this file.

If you take the upgrade path I mentioned above, it might be wise to make a backup copy of that particular file first.  The file can be found in the “/config/wa/” directory.  It is possible for that file to be overwritten during the upgrade and if you have custom entries defined in it… well you get the picture!

Unbeknownst to me, custom entries had been inserted into that file for .wmv files at some earlier date.  Shortly after the upgrade of course, .wmv files stopped streaming properly from virtual servers utilizing WebAccelerator based HTTP class profiles.  So I took the profiles off the virtual servers in question and contacted F5 Networks Support.  I was unaware of how the WebAccelerator module used the globalfragments.xml file until I was educated by an excellent F5 Networks Senior Network Support Engineer about it.

I added the entries back for the .wmv files, reapplied the HTTP class profiles that I had disabled during troubleshooting and everything worked like a charm.  The Engineer was also kind enough to create CR12834 to add .wmv files to the stock list of file types into future TMOS versions.  Thanks again to Dale Anderson for all your help!

If you are having trouble with certain file types after applying a WebAcceleration HTTP class profile then you might take a peek at the globalfragments.xml file and ensure the file type is defined correctly within.

This issue and the one mentioned in my previous post are the only two issues that I had from upgrading from version 9.4.4 to version 9.4.8!

The unit that was upgraded has three modules running on it, the GTM, LTM and WA modules.  The issue is caused by the WebAccelerator module logging to many messages out to the PVAC log, which can lead to excessive disk I/O and may cause the log file to grow so large it crashes the WebAccelerator module.  It is now a Known Issue and is being tracked in CR127854.  So if you have upgraded to TMOS 9.4.8 and you are running the WebAcceleration module you might want to keep an eye out for this!

If you believe you have a unit experiencing this issue I would advise you to contact F5 Technical Support and open a case with them.  An Engineering Hotfix can be provided to you that addresses this issue.  In the meantime, if you are able to stop using the WebAccelerator class profiles, then I would suggest not using those until you have downloaded and applied the hotfix.  Below is the text from AskF5.com regarding the issue.

Known Issue
Updated: 9/17/09 10:11 AM

I have to say, they really went out of there way to make me feel welcome.  I had fun (despite being a little nervous) and I think a good time was had by all.  It is weird listening to myself in the audio though.  I have never done that before and nobody told me that I have a southern accent!   Hehehe… just kidding of course.

The USTREAM video of the event can be found at http://www.ustream.tv/recorded/2359077.  If you would like to participate in a DevCentral LIVE event yourself, I am certain they would love to speak with you.  The DevCentral LIVE page is located at http://devcentral.f5.com/Default.aspx?tabid=197.  Events usually begin around 1:50 P.M. PST every Thursday.  Just log in and participate!

]]> http://www.TheF5Guy.com/blog/2009/10/devcentral-weekly-roundup-episode-107-the-f5-guy/feed/ 1 http://www.TheF5Guy.com/blog/2009/10/secureworld-expo-dallas/ http://www.TheF5Guy.com/blog/2009/10/secureworld-expo-dallas/#comments Sun, 11 Oct 2009 22:52:10 +0000 naladar http://www.TheF5Guy.com/blog/?p=438 Well, I am back from my vacation to Cozumel, Mexico.  A week full of sun, sand, scuba and margaritas.  Ahh…  The only downside was the 11 hour trip from Cozumel back to the DFW airport.  Which is usually only a two and a half hour trip…  (Insert derogatory remark about American Airlines and Cozumel airport maintenance workers)  Anyway, after a mad dash through the MIAMI airport, I checked my e-mail and I am glad to say it looks like I will be fortunate enough to attend the SecureWorld Expo Conference in Dallas this year!  The conference, taking place November 4 – 5, will be held in the Plano Convention Centre and seems to have a number of  excellent conference sessions to check out.

On top of my list though is a F5 BIG-IP LTM related event (of course!) being hosted by Nexum.  The “Nexum LTM Workshop”, which will be November 4 from 1:00 PM to 4:30 PM, is free for all who register for the SecureWorld Expo.  Registration for the Expo is also free, so go register before it fills up!  You certainly can’t beat the price!

The agenda for this particular event shows that they will first give an Intro and Overview of Nexum.  Then move on to Load Balancing, Monitors, Profiles (Persistence and SSL Termination), iRules, Maintaining and Mastering the BIG-IP, discuss version 10.x and then wrap it all up with a Q&A session.  I am really looking forward to meeting some local F5′ers and will of course be doing a write up on my blog about the event.  The “Maintaining and Mastering the BIG-IP” part certainly sounds interesting.

I will also be attending a number of the other events at SecureWorld and will be posting a few blog entries regarding those.  The main purpose is not really to provide ALL of the information gleaned from each event, but to give a few highlights from each and share my overall thoughts on the value of the SecureWorld Expo Conference as a whole.

Go here to check out the SecureWorld Expo Dallas Conference Agenda.