In an article that TheF5Guy.com posted back in September 22, 2010 I explained a method about creating a F5-Cisco VLAN to VLAN Bypass for Cisco IOS gear.  With the introduction to Cisco Nexus and vPC (Virtual Port Channel) technology the configurations to make the VLAN-to-VLAN bypass would need to be updated.  (Previous article can be found here)

So now we have the following similar scenario with the added twist of Nexus and vPC.

I have a pair of F5 ADC in an Internet DMZ, where nodes behind the load balancer need to access NAS system(s) on a VLAN located on a separate VLAN that is not behind the load balancer. The problem is that in my current design I have to route through the F5 Load balancer to access the NAS system(s).  Unfortunately the amount of bandwidth it takes supersedes the F5 ADC’s total throughput.  I would like to bypass this without adding extra network cards or recreating a new VLAN and would like preserve the IP addresses as much as possible.  Also the F5 ADC is sitting on a network design that participates in vPC within Cisco Nexus Datacenter gear.

Based on this description above you extrapolate a high-level logical network design as shown in Figure 1 ( I have removed vPC design for now as you read on you will see it introduced into the article):

In the figure 1, we VIP VLAN which is a routable VLAN. Node VLAN is a non-routable VLAN, which is strictly Layer 2.  Since the VLAN is non-routable no external devices except the F5 can access the Nodes directly.  Finally we have Server VLAN Z which is where the NAS system is connected to.  In order to have communication between Server VLAN Z and Node VLAN, the traffic must route through the F5 via VIP VLAN. This is done by a static route pointing to .11 on VIP VLAN which is the F5 floating address on VIP VLAN to reach node VLAN address block. In figure 1 you also have all servers in Node VLAN pointing to .1 as their default gateway which is the floating address of the F5. The F5’s default gateway is .1 on VIP VLAN. Now that we have described the current behavior of Figure 1, we can start looking at making some changes.

So how do we change the network to accommodate the result that is being looked for? It is actually much easier then you might think.

The first item you want to remove is the static route on the switch pointing to point to .11 on VIP VLAN to access NODE VLAN. You will not need this since the end result is to allow SERVER VLAN and NODE VLAN to communicate directly via the Cisco Nexus Switch router.

Next you will need to change NODE VLAN from a non-routable network to a routable network. Thus, NODE VLAN will have a gateway of .1 on the switch router. The F5 will then change its own floating address to say .11 and subsequently change the self-addresses to .12 and .13.  All the servers in NODE VLAN will continue to use .1  as the default gateway.

Thus the network will now look more like Figure 2:

At this point, you are thinking how is the traffic going to return to F5 load balancer when it’s traffic via VIP. The easy way is to apply SNAT Automap. Which works, but then you run into another problem where you lose the client IP address. Normally this might be work, but will make tracking clients more difficult especially around traffic that is not HTTP based.

The short answer to this is utilizing a Cisco’s Policy Based Route.  How does that work?

On a Cisco switch router, you can do the following configuration (NX OS Syntax):


ip access-list from_node_vlan_deny
10 permit ip y.y.y.0/24 z.z.z.0/24
ip access-list from_node_vlan_allow
10 permit ip y.y.y.0/24 any
route map to_node_vlan deny 10
match ip address from_node_vlan_deny
route map to_node_vlan permit 10
match ip address from_node_vlan_allow
set ip next-hop y.y.y.11
interface VIP_VLAN
ip policy route-map to_node_vlan


NOTE: You must have feature pbr enabled.

If you are a student of Cisco IOS you might notice that IP access-list does not contain deny statements.  This is because PBR statements in the Nexus OS was designed to ignore the deny statements within IP access-lists.  I haven’t received an official reason of why this happened, but the best case was that they wanted  to make the ultimate PERMIT/DENY decision at the route map level.   The good news is that this new behavior only exists when applied to the pBR. Meaning Deny statements within an IP access-list will not be ignored when applying as a standard ACL for security access.   Also you can use the same access-list for security access and route-maps so just keep in mind that that DENY statements will be ignored by the route-maps ONLY.

Looking at the configuration example above the behavior is that if the NODE VLAN traffic is destined to the SERVER VLAN, skip the route-map statement and use the internal routing table of the switch. Thus allowing NODE VLAN to communicate directly to SERVER VLAN and vice versa. Subsequently, if traffic from NODE VLAN is attempting to talk to the internet then it will match the IP access-list “from_node_vlan_allow” within route map “to_node_vlan permit 10”.  It will then apply the next command which is a next hop of y.y.y.11 (Floating address of the F5) within NODE VLAN.

If we left everything alone, this story would be complete.   Unfortunately the network example I used is also using vPC, which adds another layer of complexity which needs to be accounted.   Figure 3 shows us what a vPC topology would look like with an F5:

You see F5 had decided to optimize the Ethernet Frames.   To optimize F5 typically ignores the arp reply given by the HSRP primary and instead forwards Ethernet frames to which ever MAC address it receives frames from the result is a faster response time.   NAS storage vendors also do this and it’s wide spread.  Unfortunately this is not a nonstandard behavior.   If you are well versed enough on the F5 you would immediately think to turn off the auto Last hop feature would counteract this behavior.  Unfortunately, this does not work in Cisco Nexus OS world.  Cisco recognized that many vendors had this same issue so they introduced the command “peer-gateway” command. This command in affect disabled the optimization.

So basically you would introduce the command in the following configuration example, in our diagram it would be on Nexus 7010 MDF A and MDF B

vpc domain 1

role priority 10

peer-keepalive destination 10.1.1.2 source 10.1.1.1 vrf VPC-KeepAlive

peer-gateway

Of course this is still not end of the story because peer-gateway has a caveat as stated in the Nexus OS Layer 2 guide

Packets arriving at the peer-gateway vPC device will have their TTL decremented, so packets carrying TTL = 1 may be dropped in transit due to TTL expire. This needs to be taken into account when the peer-gateway feature is enabled and particular network protocols sourcing packets with TTL = 1 operate on a vPC VLAN.

This means that the traffic will be treated like a layer 3 hop which means we need to make small adjustment in our access list

From:

ip access-list from_node_vlan_deny
10 permit ip y.y.y.0/24 z.z.z.0/24
ip access-list from_node_vlan_allow
10 permit ip y.y.y.0/24 any
route map to_node_vlan deny 10
match ip address from_node_vlan_deny
route map to_node_vlan permit 10
match ip address from_node_vlan_allow
set ip next-hop y.y.y.11
interface VIP_VLAN
ip policy route-map to_node_vlan


To:

ip access-list from_node_vlan_deny
5 permit ip y.y.y.0/24 y.y.y.0/24
10 permit ip y.y.y.0/24 z.z.z.0/24
ip access-list from_node_vlan_allow
10 permit ip y.y.y.0/24 any
route map to_node_vlan deny 10
match ip address from_node_vlan_deny
route map to_node_vlan permit 10
match ip address from_node_vlan_allow
set ip next-hop y.y.y.11
interface VIP_VLAN
ip policy route-map to_node_vlan

If you have been following closely on the difference you might be wondering why should you have a permit for traffic between NODE VLAN to NODE VLAN?  After all the access-list looks at Layer 3, not Layer 2 traffic.   As I mentioned above “Packets arriving at the peer-gateway vPC device will have their TTL decremented…”  Which means that Layer 2 traffic under vPC Peer Gateway will treat any traffic within that VLAN as a layer 3 hop and it will be processed within the access-list.

Conclusion

If you are running a F5 ADC which routes through F5 Nexus devices, then you don’t need peer-gateway , but you will if you  if you are directly attached to a Nexus Device that is configured to use vPC.

I have yet to face any issues with this configuration so it might be a good idea to add Peer-gateway into your vpc configuration as a default.

From time to time, I usually receive a request that goes something like this.

“I have a pair of F5 ADC in an Internet DMZ, where the servers behind the load balancer need to access NAS system(s) on a VLAN located in the same network on another VLAN that is not behind the load balancer.

The problem is that in my current design I have to route through the F5 Load balancer to access the NAS system(s).  Unfortunately the amount of bandwidth it takes supersedes the F5 ADC’s total throughput.  I would like to by pass this without adding extra network cards or recreating a new VLAN and would like preserve the IP addresses as much as possible.”

For the purposes of the blog we will call the person requesting this Keyser Söze

Based on this description above you extrapolate a high-level logical network design as shown in Figure 1.

Figure 1

In the figure 1, we VLAN 10 which is a routable VLAN. VLAN 12 is an empty VLAN, which is strictly Layer 2, and no other traffic allowed to it from the router itself.  Finally we have VLAN13 which is where the NAS servers is connected to .  In order to access VLAN12 you need to route through the F5 that is also connected on VLAN10. This is done by a static route pointing to .11 on VLAN10 which is the F5 floating address on VLAN 10 to reach VLAN12 address block. In figure 1 you also have all servers in VLAN12 pointing to .1 as their default gateway which is the floating address of the F5. The F5’s default gateway is .1 on VLAN10. Now that we have described the current behavior of Figure 1, we can start looking at making some changes.

So how do we change the network to accommodate the result that Kyser is looking for? It is actually much easier then you might think.

For the purposes of this explanation, let us assume the switches are connected on Cisco Switch routers

The first item you want to remove is the the static route on the switch pointing to point to .11 on VLAN10 to access VLAN12. You will not need this since the end result is to allow VLAN 12 and VLAN 11 to communicate directly via the Cisco Switch router.

Next you will need to change VLAN11 from a non-routable network to a routable network. Thus, VLAN 11 will have a gateway of .1 on the switch router. The F5 will then change its own floating address to say .11 and subsequently change the self-addresses. All the servers will continue to use .1 on VLAN11 as their default gateway.

Thus the network will now look more like Figure 2

Figure 2

At this point, you are thinking well if that is the case then how do we get traffic back to the F5 for Load balancing traffic. Well the easy way is to apply SNAT Automap across all the Virtual addresses. Which works, but then you run into another problem where you lose the client IP address. Normally this might be work, BUT if you are tracking clients for statistical purposes, this is not going to work.

The short answer to this is utilizing a Cisco’s Policy Based Route. How does that work?

On a Cisco switch you can do the following configuration (IOS Syntax):

ip access-list extended from_vlan11
Deny y.y.y.0 0.0.0.255 z.z.z.0 0.0.0.255
Permit y.y.y.0 0.0.0.255 any
route map to_lb_vlan11
Match ip address from_vlan11
ip default next-hop y.y.y.11
interface Vlan11
ip policy route-map to_lb_vlan11

What these statements mean is that any traffic from VLAN11 is destined to addresses on VLAN12, skip the route-map statement and use the internal routing table of the switch. Thus allowing VLAN11 to communicate directly to VLAN12 and vice versa. Subsequently, if traffic from VLAN11 is attempting to talk to the internet then it will match the permit statement in the IP access list “from_vlan11” then apply the route map statement and thus your next hope is .11, which is hosted on VLAN11.

That pretty much sums up how to use the switches throughput for VLAN to VLAN traffic and the F5 ADC continues to do what it does best while Kyser can go home happy.

Thanks,

CB

One way to do this is called transparent header modification. How it works is a user will enter a URL in their browser such as “www.mycompany.com/bus/”, the request will come in to your BIG-IP and the information sent to your web servers can be redirected or rewritten to whatever you like. Here is an example:


when HTTP_REQUEST {
switch -glob [string tolower [HTTP::uri] ] {
"/bus/*" {
HTTP::uri "/greyhound/bus"
}
}
}


Using the iRule above, this is what happens to your incoming HTTP request. The request comes in and the URI is converted to lower case and then inspected to see if it begins with “/bus/”. The asterisk indicates a wildcard, so anything could come after “/bus/”. If it does begin with “/bus/” then the URI will be transparently modified or changed to “/greyhound/bus”. The clients browser will not be updated, but the URI that the BIG-IP passes on to the server will be “/greyhound/bus”. Basically it turns a request for this “www.mycompany.com/bus/myrequest” INTO “www.mycompany.com/greyhound/bus” Pretty cool huh?

Now lets say you want to do something a little more exotic. Lets use the iRule from above in a different way.


when HTTP_REQUEST {
set uri [HTTP::uri]
switch -glob [string tolower [HTTP::uri] ] {
"/bus/*" {
HTTP::uri "/greyhound/searchBus.do?stationName=[string range $uri 5 end]"
}
}
}


What is this one doing? Let say an HTTP request comes in for “www.mycompany.com/bus/texas”. Using the iRule above the web server would actually receive a request for “www.mycompany.com/greyhound/searchBus.do?stationName=texas”. The clients browser would still read “www.mycompany.com/bus/texas”. Like I said powerful and flexible.

If you are interested in more content regarding transparent header modifications a.k.a. redirecting users without changing their URL, then I recommend reading this article by Joe Pruitt on the DevCentral website http://devcentral.f5.com/weblogs/Joe/archive/2005/07/27/ModifyingUriWithoutRedirect.aspx

Setting Up BIG-IP and Live Meeting Portal Server

Prerequisites:

Please consult the Live Meeting Portal Server documentation and ensure that your servers meet all the perquisites before installation. All the examples in this guide are setup so that you will end up with a website at this URL: https://livemeeting.mycompany.com/lmportal. Please feel free to substitute your company’s name for “mycompany”.

IIS Setup:
1. Download the latest version of Office Live Meeting Service Portal. As of 4/20/2010 that can be found here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=429bb528-fd1b-45b7-af2b-cbbf4a8e65ff&displaylang=en

2. Create a basic website in IIS and name it Live Meeting. This empty shell of a website will be used by the Live Meeting installer and will basically be taken over by it after you run through the installation.

3. Create a folder named “Livemeeting” in the directory of your choice. In this example we will use ”E:\web\content\”

4. Double click the lmportal.exe to begin the installation and choose custom when the option appears. Then select the directory you created above so the files will be placed in your normal custom web content location.

5. Remote Desktop (RDP) to the web server and open IIS. DO NOT USE THE IIS CONSOLE ON YOUR LOCAL MACHINE as you will not have access to everything that you need.

6. The screenshots below will help guide you through the configuration of the web site in IIS. Things that do need to be changed:
a. Add 443 to the SSL port and select the unique IP address for the site to use. We will be terminating SSL on the F5 BIG-IP and then re-encrypting before sending it back on to the server.

b. Allow Scripts and Executables under execute permissions. Verify application pool is set to Live Meeting Intranet Portal AppPool.

c. Verify that ASP.NET is set to version 1.1.4.322.

d. Under Directory Security, click Edit and make sure there is a check mark on the “Enable anonymous access” and “Integrated Windows authentication” box.

e. Go to the application pool, right click and go to properties. Click the Health tab and uncheck “Enable Rapid-Fail protection”. Not including a screenshot of this one.

7. Navigate to “E:\web\content\Livemeeting\Portal” on the server. Then find the file named “Portal.config”, right-click it and click the Security tab. Click Add and then add the “Network Service” user account and give it full control. You have to do this or you cannot modify the configuration settings from the GUI.

8. Do the same thing listed in step 7 for the “PortalExport” folder located in the directory you should currently be in: “E:\web\content\Livemeeting\Portal”

9. Now you have to import the SSL certificate that you are going to use into IIS website that you just set up. You will need to obtain the .crt file for the SSL certificate and the .key file for that certificate. We terminate our SSL on the BIG-IP so these can both be obtained from there. I will skip the steps regarding purchasing an SSL certificate for a site if you do not already have one. It kind of falls outside the scope of this guide.

10. Use a search engine and search for OpenSSL. You should find their homepage at: http://www.openssl.org/

11. Download OpenSSL and install it on your Local machine. I don’t recommend installing it on the server for a wide variety of reasons. I installed my copy of OpenSSL into “C:\OpenSSL”.

12. Take the .key file and the .crt file and put them into OpenSSL’s “bin” directory. It’s just a folder inside of your OpenSSL folder called bin.

13. Open a command line and change directory over to C:\OpenSSL\bin. The example I am going to provide is for a fictitious company named “MyCompany” that is using a wildcard ssl certificate on a few of their websites.

14. Then type in the following command:

This all needs to be on one line. Spaces are ok, but no carriage returns or anything like that. This command is modeled after this example for future reference:

openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt -certfile CACert.crt

certificate.pfx = the name of the new pfx file you want to create
privateKey.key = the private key you got off of the F5 BIG-IP
certificate.crt = the crt file that you got off the F5 BIG-IP
CACert.crt = the crt file that you got off the F5 BIG-IP

15. After you type the command and hit enter, you will be prompted for a password. You can use any password that you like but you will need to remember it because IIS asks you for the same password when you go to import it.

16. OpenSSL will compile a new .pfx file for you in the C:/OpenSSL/bin directory. Take that SSL certificate and copy it over to your web server.

17. RDP over to the server and open IIS. Again here is the disclaimer, DO NOT USE THE IIS CONSOLE ON YOUR LOCAL MACHINE. Right-click on the Live Meeting web site that you created and click on the Directory Security tab. Under “Secure Communications”, click the “Server Certificate…” button.

18. Click Next and then click the “Import a certificate from a .pfx file” radio button and click next. Browse to the .pfx file that you uploaded to the web server. Click next and enter your password information that you used when you created the certificate. Then finish clicking through the wizard. Then restart IIS on the server and delete the certificate off of your local machine. This completes the IIS setup. Now move on to the Live Meeting Portal setup.
Live Meeting Portal Setup

19. Navigate to the URL:

https://livemeeting.mycompany.com/LMPortal/settings.aspx

Where livemeeting.mycompany.com is the name of the website you setup. The screen will look like the one shown on the next page. This is the Settings-Portal Configuration page. You will want to use the following settings which are also pictured in the screenshot on the next page.

Conference Center URL = https://www.livemeeting.com/cc/mycompany
Conference Center Administrator
User Id =
Password =
Email address for escalation =
Enabled Portal Services = Check the Account Create, Account Login, Account Update and Web Method Calls
Ticket Timeout = 300 Seconds
Directory Service Parameters = AccountNamePolicy=LogonUsername

20. Then click Save. If you receive an error at this point, refer back to step #7.

21. Click on the Roles link on the left side of the page. This will take you to the Roles-Portal Configuration page. Under “Live Meeting Administrators” add the users who will be the Live Meeting Administrators. Use domain\name format. IE: mydomain\username
22. Then under the “Live Meeting Organizers” settings I recommend adding the “Domain Users” from the varies domains on your network. So if you have three domains on you network named ABC, 123 and XYZ you would list ABC\Domain Users, 123\Domain Users and XYZ\Domain Users.

23. Then click the “Export Configurations Settings” link on the left hand side of the page. This is not really labeled right because what it actually does is back up your configuration. If you mess something up in the running configuration, simply click on the “Import Configuration Settings” to restore the last configuration that you exported.

24. Then click on the “Events” link on the left side of the page. Change the log file directory to a directory that you want to have all the logs written into. In this example I chose the E: drive of the server I was working on. Whether you create a new one or use an existing one you must make sure that the “Network Service” account has permissions on that folder to Read, Write and Modify. Otherwise you will receive a nasty .NET error when you go to save the changes you just made. Click Save.

Live Meeting Portal Server BIG-IP LTM Setup

The BIG-IP LTM set up for this can be very easy to configure. You will need to create nodes for each of your web servers, assign them to a pool named “Live_Meeting_Pool” and then create a Virtual Server for the application. I named my virtual server “Live Meeting” in the example pictured below. You may need to customize it to match your environment, but the basic settings are:

Service Port: 443
Type: Standard
Protocol: TCP
Protocol Profile (Client): tcp
HTTP Profile: http
SSL Profile (Client): wildcard
SSL Profile (Server): serverssl

I also assigned the Live_Meeting_Pool to the Virtual Server, set the Default Persistence Profile to “Cookie” and Fallback Persistence Profile to “source_addr”.

Like a lot folks around the country I pre-ordered a 32 GIG iPad a few weeks ago and have been waiting eagerly to check out the new device.  I already have two Apple branded products in the house, so it was easy for me to drink the Kool-Aid and purchase another .

However, I was very disappointed with Apple on the day that I finally received my iPad.  I had updated my MacBook the night before and ensured it was ready to go, only to have my hard drive crash moments before I was able to sync up my new iPad!  There I was sitting in my cubicle at work shaking my fists in the air and screaming “NOooooo!!!!” In my mind anyways…

Well all was not lost and I do mean that literally.  I had my data backed up, but I did have to send the MacBook in for repair.  Thankfully I was still covered under my Apple Care plan.  As it turns out, I also received a new logic board, heat pipe assembly and top case replacement.  Evidently the now three year old MacBook had more wrong with it than I had guessed.

I decided it would be fun to post the User Agent String for the iPad and to list a few of the apps that I have enjoyed using so far.  I aimed the iPad over to a BIG-IP 6400 with an iRule that logs out the User Agent String and this is what was returned:


Mozilla/5.0 iPad U CPU OS 3_2 like Mac OS X en-us AppleWebKit/531.21.10 KHTML, like Gecko Version/4.0.4 Mobile/7B367 Safari/531.21.10

At least it mentions “iPad” in the User Agent String! This will make it a bit easier for traffic direction via an iRule if your company has a site that hosts content specifically for the iPad.

I have had the opportunity to check out a lot of different applications and games as well.  Some of my favorite applications so far are:

Plants Vs. Zombies HD – Addictive game
Fieldrunners – Nice tower defense game
Netflix – Great for streaming movies
Fargoal – Old School Dungeon Crawler
AirVideo – Great for streaming movies “Backed Up” on my Mac
TouchTerm – Decent for SSH
WinAdmin – Great app for Windows RDP functionality
MochaVNC – Decent app for Mac RDP functionality
Dragon Dictation – I was surprised by this one.
GoodReader – Hands down one of my favorite apps. I was able to pull down a lot of F5 BIG-IP manuals using this app!
The Weather Channel – You have to know what it is doing outside after all.
Citrix Receiver – Proven to be great for connecting to the Citrix Farm at work.
Backgrounds – A nice app to grab new backgrounds for your iPad.

Before I came to the class I could build a security policy and assign it to a website and do some minor tweaking.  Now I can say with confidence that I can build a web application security policy that is PCI compliant and has a solid foundation.

One of the main ingredients for a successful training session/class is you really need an excellent instructor.  If the instructor doesn’t know his stuff or doesn’t really enjoy the subject matter it can have a negative and direct impact on the course.  The class I took was lead by a gentlemen named Keith Bowers who has worked for f5 Networks for 10+ years.  Granted, I could be wrong about number of years, but I think I am close.  I can say for certain thought that Mr. Bowers knows the material and he seemed to really enjoy teaching the class.

This wasn’t the kind of class where you go and read along with the teacher word by word out of the book.  Keith gave very concise and well thought out lectures regarding each subject that we touched on.  I say concise because he said everything that he needed to in order for you to comprehend the material and to be able to apply in a real world situation.  Then he would provide guidelines for the hands-on portion of the lab for that section and turn us loose on the BIG-IP box that each student gets to all to his or her self.  When a student had trouble getting through a lab he would sit beside them, provide information on things to look for and provide clarification on things until the student got through the lab.  He was really good about teaching you to fish rather than just giving you an answer out of the teachers edition of the manual

So what kind of goodness can one expect to learn at an ASM 10.x course?  Here is a brief list of the things that we covered:

Installation
Web Application Concepts
Web Application Vulnerabilities (with instructions on how to perform a few basic hacks)
ASM Application Configuration
Security Policy Building
Creating Custom Attack Signatures
Reporting
Traffic Learning
Protecting XML and Web Services
And more…

On the second day that I was there I also had the chance to meet up with a few members of the DevCentral Core Team!  I was able to bounce out of class a little early so Joe met me outside the training room and proceeded to give me a tour of the place.  At one point I tried to slip a VIPRION into my cowboy hat and almost made off with it but the 30+ blue ethernet cables sticking out from underneath my hat gave me away.  Alas, I had to put it back.  <Sigh>  Seeing that I was upset though Colin, Jeff and Joe provided me sneak peak of their latest TOP SECRET project to get my spirits up.  After the tour that I was given, my spirits were definitely lifted!  I wish I could tell, I wish I could tell…. but I can’t.  It was awesome though.

We then proceeded down to Buckley’s Pub for some lunch and along the way we went over a little bit of history, talked about things that a tourist like me should do when visiting Seattle, etc…  Jeff kindly wrote up a blog article about it and even included a picture that he took of Colin, Joe and I at the pub.  You can check it out here:

http://devcentral.f5.com/weblogs/JeffB/archive/2010/04/01/1088132.aspx

I can’t provide all the details of what we talked about, I was having to good of a time to remember them all.  I know we talked about Bear Grylls (Man vs. Wild), Mac keyboard shortcuts and the MVP Summit… How those are all interconnected I will leave up to you to ponder… Hehehehe… seriously, thanks for a great time fellas.  And also thanks for what you do every day.

Well, if you have made it this far into my blog post you deserve a treat!  Below is a snippet of some videos that I took on April 1st during the training class, some footage from the TOP SECRET stuff they showed me and some footage from the pub!  I had to try out my f5 Networks MVP branded FlipMINO after all!  Sorry if it is a little choppy in a place or two, I had to compress it before I uploaded it to YouTube.

Camera In Cowboy Hat Video

Now that the naming of the entry has been completed, on to the main topic!  I received said box from FedEX this last Friday from f5 Networks and I felt compelled to write a blog post about it and include some pics for your viewing enjoyment.

I can’t tell you how much I have already enjoyed being a member of the f5 Networks MVP program.  It has been awesome from day one and I look forward to contributing more to the community now that f5 Networks has so graciously supplied all of us f5 MVP’s with the tools to do just that.  Thank you for the great gear and thank you for supporting the community like you do!

Here is a list of what was in “The Box of Awesomeness”:

A SanDisk 16 GB USB Flash Drive
A Logitech QuickCam Deluxe for Notebooks for Business
A Logitech ClearChat Pro USB High Performance Audio Headset
A Blue Polo Shirt with f5 Networks logo on the chest
AND
A flip MinoHD Camcorder with a custom f5 Networks MVP skin!

Both of the previously mentioned deployment guides discuss editing files on the Citrix farms Web Interface servers so that it looks for the client IP address in the X-Forwarded-For HTTP header.  Otherwise, every connection will appear to be originating from the BIG-IP LTM and not from its true IP.  After reading both guides and looking at my current environment I was dismayed to find that the files and locations mentioned were no longer valid.  I then turned to my top three resources on the web in the search for an answer: AskF5, DevCentral and Google.

I struck out on the first two (which seldom happens) but my Google search did turn up some interesting results on the Citrix Forums.  I finally found some code posted by Sam Jacobs back in August 2009 that modifies the way the Citrix farm looks up the client IP address.  His method allows for the use of the X-Forwarded-For header.

The first file that you will want to find and edit is the Include.java file.  You will want to locate and change this file on every Web Interface XenApp server in the farm.  Speaking from experience, save a copy of the original file to a safe location such as your desktop or flash drive.  DO NOT copy the file and rename the original to Include.old and leave it on the server.  It may sound crazy, but doing that will not work.  I’m not a programmer, so I cannot tell you why that will not work, but I can tell you I know for a fact it will not.  That being said, here is the file path for the Include.java file:

“\Inetpub\wwwroot\Citrix\XenApp\app_code\PagesJava\com\citrix\wi\pageutils\Include.java”

Now that you have found the file, open it up with a text editor (I use Textpad) and find the Java routine named “getClientAddress”.  Replace the code for that routine with the code listed below.

public static String getClientAddress(WIContext wiContext) {
String ageClientAddress = AGEUtilities.getAGEClientIPAddress(wiContext);
String userIPAddress = wiContext.getWebAbstraction().getRequestHeader("X-FORWARDED-FOR");
if (userIPAddress == null) {
userIPAddress = wiContext.getWebAbstraction().getUserHostAddress();
}
return (ageClientAddress != null ? ageClientAddress : userIPAddress);
}

Save the file and wash/rinse/repeat this step on every Web Interface server in the farm.  The next thing that you will want to do is to modify the login page so that it displays the client IP address being obtained from the X-Forwarded-For header.  The file you will want to edit is called “loginView.ascx” and can be found in the following file path on your Web Interface Servers:

”\inetpub\wwwroot\Citrix\XenApp\app_data\include\loginView.ascx”

The code you will want to add is:

Client IP: <%= com.citrix.wi.pageutils.Include.getClientAddress(wiContext) %>

I added the code directly below the LoginPageControl viewControl line and it works well for me.  Save the file and repeat this step on every Web Interface server in the farm and reboot each Web Interface Server after you are done.  Then it is time for the moment of truth… fire up your browser of choice and navigate to the Citrix login page.  If you have successfully set everything up and have finished following the rest of the deployment guide you should see a screen similar to the one below:

If you receive an error message or the screen doesn’t load, then you might want to go back and check your settings again.  Then that’s it!  I am aiming to develop some custom monitors for the Web Interface Server and for the XML Broker Servers over the next few weeks.  Once I have those done I will put them out in the Devcentral forums for the community enjoy.

I am very happy to mention that the kind folks over at F5 Networks allowed me to submit this as a Tech Tip article which you can find on their site at:

http://devcentral.f5.com/Default.aspx?tabid=63&articleType=ArticleView&articleId=1082335

No problem, I identified all of the expired certificates, checked the box beside them and hit the delete button at the bottom of the page.  After verifying everything was still happy and the support tickets didn’t start flooding my inbox I decided to run a config sync and push the config changes over to the standby box.

The config sync ran without a problem and the gui showed Config Sync: OK.  I then proceeded to check my changes on the standby unit, just for verification purposes.  And that ladies and gentlemen, is when the fun began….

As I was verifying the changes I noticed something I thought was rather strange.  The old SSL certificates that I deleted on the Active unit, were still there in the Standby units SSL Certificate store!  My first thought, oops, my Trusted Device Certificates must be out of whack.  I then proceeded to delete the trusted device certs and ran the “big_ip add” command from the CLI on each unit.  I checked my trusted device certificates and like magic there they were.  I ran another Config Sync thinking that probably fixed the problem, but wait… no such luck.

The Config Sync ran and didn’t kick out any errors, but the old SSL certificates were still in there in all their expired glory.  Frustrated and humbled once again, I decided to run a quick test by deleting a VS on the Active Unit to see if it would be removed once I ran a Config Sync.  I blew away the VIP I use for testing and ran the Config Sync again.  The VS was deleted off of the Standby Unit.  Not knowing off the top of my head what to do next, I then proceeded to open a ticket with my good friends over at F5 Networks.  I didn’t have a lot of faith in my running configuration at the time so I went ahead and opened the ticket as a level 2 ticket (site at risk).

I quickly received a phone call from a Network Support Engineer named Kevin “CB” Midkiff.  We went through the standard procedure of qkview files and few other tests.  After going over the problem Mr. Midkiff proceeded to explain to me that while the SSL Certificates store is indeed carried over when you run a Config Sync IT DOES NOT DELETE SSL Certificates on the unit that you push the config to.  In my case it was the Standby Unit.  The Config Sync function only appends SSL Certificates.

Moral to the story?  If you are double checking your configurations and happen to see some lingering SSL certificates don’t worry, just select them and let the delete button work its magic on them.  Also as an FYI, “CB” was great to work with and very knowledgeable.  Thanks again for your help Mr. Midkiff.

I am also very excited to say that I have been selected to be a F5 Networks MVP!

That’s right, TheF5Guy is now an F5 Networks MVP!  I consider it a great honor and am very excited to say the least!  I go by the alias “naladar” in the DevCentral Forums and you can check out my profile here:  http://devcentral.f5.com/Default.aspx?tabid=2242.  You have to be a member of DevCentral in order to view the page, but it is free to join!

Now that the announcement has been made public I wanted to share a few things about the MVP program.  To start with, what’s all of this mean?  It means F5 Networks takes their user community seriously and they want to give back to that community.  This isn’t just an honorary title.  Far from it actually, as there are a number of perks to being an MVP member.

I can’t go into all of them in detail, but here are a few things that I can share since they are mentioned in the podcast.  We will be having regular meetings or round table discussions to go over a wide variety of things relating to the F5 Networks community.  We are being provided profile pages on the DevCentral site to help increase our visibility in the community.  MVP members will be receiving a MVP Kit that was put together with the goal in mind of providing us tools to help us deliver more content to the community.  We will also be having an MVP Summit sometime this year so that we can all meet face-to-face to kick around issues and provide input into the direction of the BIG-IP product line.  Sounds awesome doesn’t it!

This post would of course not be complete without a complete list of the MVP’s so here it is:

hoolio
bhattman
hamish
hwidjaja
smp
naladar
mikejo

The best news is that they want to continue to grow the MVP program.  Do you want to be an F5 Networks MVP?   How do you get started?  Just join DevCentral and start contributing to the community.  They’re watching…..

I am stating the obvious here I know, but how many of you out there have worked at places where people guard their security knowledge like it’s KFC’s secret recipe for chicken?  Have you ever had to work with a security expert that can tell you every law of governance, but never truly explain WHY those laws are in place?  Ever talk to a business partner not in I.T. that just didn’t get why the web applications needed to be protected by a web application firewall or why ALL the ports on the firewall couldn’t be opened up?  I talked to a large number of people that worked at well known companies and each said that is the case where they work.  Of the group I talked to it was about 50% from the business arena and 50% from the IT side of the house, but they were all there for a common goal….

The SecureWorld Expo is a place where people can go to learn the WHY.  Not just I.T. folks, but people from all aspects of business as well.  They can talk to industry leaders and experts about things that are going down past, present and future.  It is all about translation and communication of the most up-to-date information available.  How up-to-date is the information that is covered?  The second day of the expo, the speaker Dan Greer came out to the podium and started talking about the SSL Man-in-the-Middle Renegotiation story that just broke in the news.  I have to say my hats off to the folks in the DevCentral community to, shortly thereafter, a way to mitigate the attack showed up on DevCentral (Lupo, thanks for your contribution!)… it can be found in the forums at http://devcentral.f5.com/Default.aspx?tabid=53&forumid=5&postid=86456&view=topic

Other than the open sharing and exchange of knowledge, the excellent speakers, free vendor loot and good food, the other thing that is great about the SecureWorld Expo is the fact that you get CPE credits for attending the various events.  Depending on the events that you sign up for you can either earn a 12 CPE or a 16 CPE Certificate of Attendance.  This is outstanding for those that have CPE requirements to meet and keep up with.  Not only can you obtain a lot of CPE’s in a short time, but it is also very cost effective.  You definitely get more bang for your buck at a SecureWorld event than you do at many others.

So what was covered in this “Nexum LTM Workshop”?  Quite a bit actually and it was all very well planned out.  The workshop was lead by a gentlemen named Peter Maag, who is a Senior Security Expert with Nexum.  I believe that part of what made this event so much fun for me was that it was obvious that Mr. Maag knew his stuff and (of course) I like talking about the capabilities of the BIG-IP line. 

Peter began by giving a brief introduction, explaining who he was and the services provided by Nexum.  I have to admit that I was unaware that Nexum was such a versatile company.  I believe it is so versatile and one of the fastest growing private companies because of them hiring and keeping talent like Mr. Maag around.  But that is a different philosophical discussion that perhaps I will touch on at some other time.  If I ever take up being a philosopher.

Then after the intro… it was f5 time!  For those in the audience not familiar with the BIG-IP product line Peter gave an overview of products available from f5 Networks.  He took the time to provide a clear picture of each modules functionality and I feel that he did the products justice.  He then steered the presentation to the real meat of the workshop which was the LTM module.  Virtual Servers, Pool Members and Nodes were all explained as well as the basics of configuring load balancing.  We spent some time discussing the full proxy architecture of the LTM module and we where then guided through a load balancing demo.

This lead into a discussion about monitors, persistence profiles, SSL termination and ended with a demo over those concepts.  There were a few questions at this point, as members of the audience asked questions such as “How long are self signed certificates valid for if they are generated on the f5 BIG-IP?” and “What are the different methods available for Cookie Persistence?”.  All of which were answered concisely and followed up with live demonstrations performed on a BIG-IP unit running TMOS version 10.x.  How cool is that?

We then went into a discussion about iRules.  Peter provided a number of examples of how to use iRules to pull off complicated tasks very easily.  In one example he showed how you could direct web traffic coming from an iPhone to a different set of servers than the ones used to serve up content to standard desktop browsers.  To augment the workshop Nexum provided an excellent booklet which just so happens to have a very handy page that lists almost all of the iRule Events that can be used in iRule generation.

We went over several other things, but the jest of this entry isn’t to really rehash everything that we covered.  The purpose is to encourage everyone using the LTM module to go check one of these workshops out.  Peter Maag did a phenomenal job explaining things for newcomers and veterans alike, which is not an easy thing to do.  To summarize, if you have just recently purchased an f5 BIG-IP product or are looking into purchasing one, attend one of these workshops.  You will walk away a wiser person and I cannot think of a better way to sell someone on f5 BIG-IP products.  Once you see it in action you will be wondering why you have stuck with Brand X for so long.

My next entry will be over the value of attending the SecureWorld Expo.  Is it worth the cost if you had to pay for it out of your own pocket?  What are the driving reasons for one to attend such an event?  I will be asking those questions and more soon and you may be surprised by my conclusions.  Stay tuned.

What a great way to segway into my first SecureWorld Expo blog entry!  Be afraid, be very afraid…  I am just kidding of course.  The Expo was excellent and I walked away from the event a wiser person.  It definitely helped me look at things differently and as Ralph Waldo Emerson once said, “Fear always springs from ignorance.” 

Man, oh man.  I think I may have committed a blunder of cosmic proportions.  Are you allowed to quote Yoda and Emerson in the same blog post?  Yes? No?  Anway, moving on…

The Expo started off with an awesome keynote by a very nice man named Jeff Bardin.  His topic was “Extremist Online Social Networks – Jihadis” and I was enthralled the whole time.  The banners that he had up on his very first presentation slide are the same web site banners that I have helped keep off of our network up at work.  When I saw those banners, I knew that he was going to be talking about a topic that hit close to home.

After taking the stage Mr. Bardin began explaining how Jihadis use resources provided by many American companies against America.  He talked about the Madrid train bombings, how Jihadis are using software like vBulletin and hacked copies of various software suites to pull off all kinds of nefarious acts.  He also discussed with great clarity how Jihadis are continuing to demonstrate their technical prowess.

Now I will not provide any more information about his presentation other than that.  Not because I do not want others to have the information, but because I cannot do the subject justice.  Mr. Bardin is an expert in his field and has spent countless hours researching, compiling information and teaching others.  I do not wish to diminish his work in any form or fashion.  Check out that link that I provided and the one at the bottom of this post for more information.

I would advise anyone, if Mr. Bardin is speaking at an event within a 12 hour driving distance, make the drive.  It really was that good.

After his presentation he stayed for a while answering questions.  I waited in the background for a bit, allowing others to ask questions as I listened in an attempt to take in as much information as I could.  When I did finally open my mouth he kindly gave me a his business card and answered all of the questions I had.  Anybody that will go out of there way to answer questions and share knowledge like Mr. Bardin did is a good man in my book.

For those seeking more information about Jeff Bardin and Treadstone 71, here is a link to some great information that will save you a trip to Google: http://blogs.csoonline.com/user/jeff_bardin

So what is coming up next?  Well I can’t go to long without talking about the F5 BIG-IP product line!  I am The F5 Guy after all.  My next post will be about the Nexum LTM Workshop that was lead by Peter Maag.

A friend of mine supplied the image to the left.  I am thinking that it may have to be the official logo for my website!  Of course, had I known he was taking pictures of me with his cell phone I would have flexed a bit more…

Not buying that are you?  Well OK, maybe that is just what I look like in my mind!  Coming next week to “The F5 Guy” website, news and reviews straight from the Dallas SecureWorld Expo!

The unit that was upgraded has three modules running on it, the GTM, LTM and WA modules.  The issue is caused by the WebAccelerator module logging to many messages out to the PVAC log, which can lead to excessive disk I/O and may cause the log file to grow so large it crashes the WebAccelerator module.  It is now a Known Issue and is being tracked in CR127854.  So if you have upgraded to TMOS 9.4.8 and you are running the WebAcceleration module you might want to keep an eye out for this!

If you believe you have a unit experiencing this issue I would advise you to contact F5 Technical Support and open a case with them.  An Engineering Hotfix can be provided to you that addresses this issue.  In the meantime, if you are able to stop using the WebAccelerator class profiles, then I would suggest not using those until you have downloaded and applied the hotfix.  Below is the text from AskF5.com regarding the issue.

Known Issue
Updated: 9/17/09 10:11 AM

I have to say, they really went out of there way to make me feel welcome.  I had fun (despite being a little nervous) and I think a good time was had by all.  It is weird listening to myself in the audio though.  I have never done that before and nobody told me that I have a southern accent!   Hehehe… just kidding of course.

The USTREAM video of the event can be found at http://www.ustream.tv/recorded/2359077.  If you would like to participate in a DevCentral LIVE event yourself, I am certain they would love to speak with you.  The DevCentral LIVE page is located at http://devcentral.f5.com/Default.aspx?tabid=197.  Events usually begin around 1:50 P.M. PST every Thursday.  Just log in and participate!

]]> http://www.TheF5Guy.com/blog/2009/10/devcentral-weekly-roundup-episode-107-the-f5-guy/feed/ 1 http://www.TheF5Guy.com/blog/2009/10/secureworld-expo-dallas/ http://www.TheF5Guy.com/blog/2009/10/secureworld-expo-dallas/#comments Sun, 11 Oct 2009 22:52:10 +0000 naladar http://www.TheF5Guy.com/blog/?p=438 Well, I am back from my vacation to Cozumel, Mexico.  A week full of sun, sand, scuba and margaritas.  Ahh…  The only downside was the 11 hour trip from Cozumel back to the DFW airport.  Which is usually only a two and a half hour trip…  (Insert derogatory remark about American Airlines and Cozumel airport maintenance workers)  Anyway, after a mad dash through the MIAMI airport, I checked my e-mail and I am glad to say it looks like I will be fortunate enough to attend the SecureWorld Expo Conference in Dallas this year!  The conference, taking place November 4 – 5, will be held in the Plano Convention Centre and seems to have a number of  excellent conference sessions to check out.

On top of my list though is a F5 BIG-IP LTM related event (of course!) being hosted by Nexum.  The “Nexum LTM Workshop”, which will be November 4 from 1:00 PM to 4:30 PM, is free for all who register for the SecureWorld Expo.  Registration for the Expo is also free, so go register before it fills up!  You certainly can’t beat the price!

The agenda for this particular event shows that they will first give an Intro and Overview of Nexum.  Then move on to Load Balancing, Monitors, Profiles (Persistence and SSL Termination), iRules, Maintaining and Mastering the BIG-IP, discuss version 10.x and then wrap it all up with a Q&A session.  I am really looking forward to meeting some local F5′ers and will of course be doing a write up on my blog about the event.  The “Maintaining and Mastering the BIG-IP” part certainly sounds interesting.

I will also be attending a number of the other events at SecureWorld and will be posting a few blog entries regarding those.  The main purpose is not really to provide ALL of the information gleaned from each event, but to give a few highlights from each and share my overall thoughts on the value of the SecureWorld Expo Conference as a whole.

Go here to check out the SecureWorld Expo Dallas Conference Agenda.

The great folks over at F5 Networks DevCentral have outdone themselves once again!  They have added a new section to their website called “Online Events” and rather than do injustice to the site with my own words, allow me to plagiarize their own description of the section:

Here of course is a direct link to the new content!

How awesome is that?  I have worked with a number of IT related companies over the years, but never have  I seen a technology based company so dedicated to their customers.  Post after post the experts driving the helm over at DevCentral continue to share and spread knowledge of all types.  They help people solve real world challenges and now they are doing it live!  Anyone can call or join them on chat during their weekly podcast and ask questions and seek their input about various things.  I have had to work with some companies in the past that… well, lets just say, you would consider yourself lucky if you got a call back from them within three days.  I must say it is a very refreshing approach and well done folks!

Also while you are there, go check out the latest DevCentral Weekly Roundup Podcast: Episode #104. They covered a variety of topics and I was fortunate enough that they discussed The F5 Guy website as well of a few of the posts that I have put up!  They discussed two posts in particular.  “Using not In An iRule” and “F5 Network Certification” which you can find below this post or just click on the links to follow of course.  Rather than rehashing what was said, read the two posts, leave some comments if you like and head on over to DevCentral to hear what they have to say.

I had to make a tweak or two here or there, but I have to say overall I really like it.

Coming soon, an interesting tale about BIG-IP™ TMOS Version 10.

Lightning, a very amazing thing… EXCEPT when experienced up close and personal!  The F5 Guy is not lightning proof (despite rumors to the contrary) nor is the equipment that I use on a day to day basis.

A few days ago a large storm system passed by the undisclosed location and during the height of the storm, a bolt of lightning came down from the sky and struck the ground outside.  That’s normal for any storm, but this particular bolt struck VERY CLOSE.  How close you say?  Well, it was close enough that I checked my body parts to verify that they were all present and accounted for.  After I performed my limb check was when the smell hit me.  The smell that no IT guy ever wants to smell… charred electrical components.

Even though all of my equipment was plugged into surge protectors and UPS units, they still got zapped.  I couldn’t believe it.  Thankfully though, I practice the regimen of backing up things that I so often preach to others.  I did however have to spend a few days coming up with parts so that I would have something to restore to.

All’s well that ends well.  I gave my old ASUS P5B Deluxe motherboard a crisp salute as the garbage man picked it up today, along with the charred remains of its companion a relatively new ASUS P5Q-SE motherboard.  I was indeed sad to see the P5B go.  I have had it for a number of years and it proved to be one of the most reliable motherboards I have ever owned.

I am working on an article about MOSS 2007 Alternate Access Mappings and the BIG-IP 6400.  I hope to have that cranked out in the next couple of days.  (Barring any unforeseeable disasters)