Now that the naming of the entry has been completed, on to the main topic!  I received said box from FedEX this last Friday from f5 Networks and I felt compelled to write a blog post about it and include some pics for your viewing enjoyment.

I can’t tell you how much I have already enjoyed being a member of the f5 Networks MVP program.  It has been awesome from day one and I look forward to contributing more to the community now that f5 Networks has so graciously supplied all of us f5 MVP’s with the tools to do just that.  Thank you for the great gear and thank you for supporting the community like you do!

Here is a list of what was in “The Box of Awesomeness”:

A SanDisk 16 GB USB Flash Drive
A Logitech QuickCam Deluxe for Notebooks for Business
A Logitech ClearChat Pro USB High Performance Audio Headset
A Blue Polo Shirt with f5 Networks logo on the chest
AND
A flip MinoHD Camcorder with a custom f5 Networks MVP skin!

Both of the previously mentioned deployment guides discuss editing files on the Citrix farms Web Interface servers so that it looks for the client IP address in the X-Forwarded-For HTTP header.  Otherwise, every connection will appear to be originating from the BIG-IP LTM and not from its true IP.  After reading both guides and looking at my current environment I was dismayed to find that the files and locations mentioned were no longer valid.  I then turned to my top three resources on the web in the search for an answer: AskF5, DevCentral and Google.

I struck out on the first two (which seldom happens) but my Google search did turn up some interesting results on the Citrix Forums.  I finally found some code posted by Sam Jacobs back in August 2009 that modifies the way the Citrix farm looks up the client IP address.  His method allows for the use of the X-Forwarded-For header.

The first file that you will want to find and edit is the Include.java file.  You will want to locate and change this file on every Web Interface XenApp server in the farm.  Speaking from experience, save a copy of the original file to a safe location such as your desktop or flash drive.  DO NOT copy the file and rename the original to Include.old and leave it on the server.  It may sound crazy, but doing that will not work.  I’m not a programmer, so I cannot tell you why that will not work, but I can tell you I know for a fact it will not.  That being said, here is the file path for the Include.java file:

“\Inetpub\wwwroot\Citrix\XenApp\app_code\PagesJava\com\citrix\wi\pageutils\Include.java”

Now that you have found the file, open it up with a text editor (I use Textpad) and find the Java routine named “getClientAddress”.  Replace the code for that routine with the code listed below.

public static String getClientAddress(WIContext wiContext) {
String ageClientAddress = AGEUtilities.getAGEClientIPAddress(wiContext);
String userIPAddress = wiContext.getWebAbstraction().getRequestHeader("X-FORWARDED-FOR");
if (userIPAddress == null) {
userIPAddress = wiContext.getWebAbstraction().getUserHostAddress();
}
return (ageClientAddress != null ? ageClientAddress : userIPAddress);
}

Save the file and wash/rinse/repeat this step on every Web Interface server in the farm.  The next thing that you will want to do is to modify the login page so that it displays the client IP address being obtained from the X-Forwarded-For header.  The file you will want to edit is called “loginView.ascx” and can be found in the following file path on your Web Interface Servers:

”\inetpub\wwwroot\Citrix\XenApp\app_data\include\loginView.ascx”

The code you will want to add is:

Client IP: <%= com.citrix.wi.pageutils.Include.getClientAddress(wiContext) %>

I added the code directly below the LoginPageControl viewControl line and it works well for me.  Save the file and repeat this step on every Web Interface server in the farm and reboot each Web Interface Server after you are done.  Then it is time for the moment of truth… fire up your browser of choice and navigate to the Citrix login page.  If you have successfully set everything up and have finished following the rest of the deployment guide you should see a screen similar to the one below:

If you receive an error message or the screen doesn’t load, then you might want to go back and check your settings again.  Then that’s it!  I am aiming to develop some custom monitors for the Web Interface Server and for the XML Broker Servers over the next few weeks.  Once I have those done I will put them out in the Devcentral forums for the community enjoy.

I am very happy to mention that the kind folks over at F5 Networks allowed me to submit this as a Tech Tip article which you can find on their site at:

http://devcentral.f5.com/Default.aspx?tabid=63&articleType=ArticleView&articleId=1082335

No problem, I identified all of the expired certificates, checked the box beside them and hit the delete button at the bottom of the page.  After verifying everything was still happy and the support tickets didn’t start flooding my inbox I decided to run a config sync and push the config changes over to the standby box.

The config sync ran without a problem and the gui showed Config Sync: OK.  I then proceeded to check my changes on the standby unit, just for verification purposes.  And that ladies and gentlemen, is when the fun began….

As I was verifying the changes I noticed something I thought was rather strange.  The old SSL certificates that I deleted on the Active unit, were still there in the Standby units SSL Certificate store!  My first thought, oops, my Trusted Device Certificates must be out of whack.  I then proceeded to delete the trusted device certs and ran the “big_ip add” command from the CLI on each unit.  I checked my trusted device certificates and like magic there they were.  I ran another Config Sync thinking that probably fixed the problem, but wait… no such luck.

The Config Sync ran and didn’t kick out any errors, but the old SSL certificates were still in there in all their expired glory.  Frustrated and humbled once again, I decided to run a quick test by deleting a VS on the Active Unit to see if it would be removed once I ran a Config Sync.  I blew away the VIP I use for testing and ran the Config Sync again.  The VS was deleted off of the Standby Unit.  Not knowing off the top of my head what to do next, I then proceeded to open a ticket with my good friends over at F5 Networks.  I didn’t have a lot of faith in my running configuration at the time so I went ahead and opened the ticket as a level 2 ticket (site at risk).

I quickly received a phone call from a Network Support Engineer named Kevin “CB” Midkiff.  We went through the standard procedure of qkview files and few other tests.  After going over the problem Mr. Midkiff proceeded to explain to me that while the SSL Certificates store is indeed carried over when you run a Config Sync IT DOES NOT DELETE SSL Certificates on the unit that you push the config to.  In my case it was the Standby Unit.  The Config Sync function only appends SSL Certificates.

Moral to the story?  If you are double checking your configurations and happen to see some lingering SSL certificates don’t worry, just select them and let the delete button work its magic on them.  Also as an FYI, “CB” was great to work with and very knowledgeable.  Thanks again for your help Mr. Midkiff.

So that got me to thinking… how can someone do this in an iRule?  I have to admit I haven’t really looked into it that much previously because we utilize an ASM module running on a 4100 unit.  The 4100 can do a lot of different things regarding cookies such as checking if a cookie has been modified and if the cookie was obtained in a previous session.  I figured I would hit the AskF5 database to see what I could turn up and I uncovered this little gem:

when RULE_INIT {
set ::key [AES::key 128]
}
when HTTP_RESPONSE {
set decrypted [HTTP::cookie "MyCookie"]
HTTP::cookie remove "MyCookie"
set encrypted [b64encode [AES::encrypt $::key $decrypted]]
HTTP::cookie insert name "MyCookie" value $encrypted
}
when HTTP_REQUEST {
set encrypted [HTTP::cookie "MyCookie"]
HTTP::cookie remove "MyCookie"
set decrypted [AES::decrypt $::key [b64decode $encrypted]]
HTTP::cookie insert name "MyCookie" value $decrypted
}

There is definitely more to this, so you may want to go check out the full solution article here:  SOL7784.  There is also an awesome 2009 iRule Contest entry that you should check out here. The iRule you will want to look at is the Cookie Tampering Prevention iRule written by Henrik Gyllkrans.

I am also very excited to say that I have been selected to be a F5 Networks MVP!

That’s right, TheF5Guy is now an F5 Networks MVP!  I consider it a great honor and am very excited to say the least!  I go by the alias “naladar” in the DevCentral Forums and you can check out my profile here:  http://devcentral.f5.com/Default.aspx?tabid=2242.  You have to be a member of DevCentral in order to view the page, but it is free to join!

Now that the announcement has been made public I wanted to share a few things about the MVP program.  To start with, what’s all of this mean?  It means F5 Networks takes their user community seriously and they want to give back to that community.  This isn’t just an honorary title.  Far from it actually, as there are a number of perks to being an MVP member.

I can’t go into all of them in detail, but here are a few things that I can share since they are mentioned in the podcast.  We will be having regular meetings or round table discussions to go over a wide variety of things relating to the F5 Networks community.  We are being provided profile pages on the DevCentral site to help increase our visibility in the community.  MVP members will be receiving a MVP Kit that was put together with the goal in mind of providing us tools to help us deliver more content to the community.  We will also be having an MVP Summit sometime this year so that we can all meet face-to-face to kick around issues and provide input into the direction of the BIG-IP product line.  Sounds awesome doesn’t it!

This post would of course not be complete without a complete list of the MVP’s so here it is:

hoolio
bhattman
hamish
hwidjaja
smp
naladar
mikejo

The best news is that they want to continue to grow the MVP program.  Do you want to be an F5 Networks MVP?   How do you get started?  Just join DevCentral and start contributing to the community.  They’re watching…..

I am stating the obvious here I know, but how many of you out there have worked at places where people guard their security knowledge like it’s KFC’s secret recipe for chicken?  Have you ever had to work with a security expert that can tell you every law of governance, but never truly explain WHY those laws are in place?  Ever talk to a business partner not in I.T. that just didn’t get why the web applications needed to be protected by a web application firewall or why ALL the ports on the firewall couldn’t be opened up?  I talked to a large number of people that worked at well known companies and each said that is the case where they work.  Of the group I talked to it was about 50% from the business arena and 50% from the IT side of the house, but they were all there for a common goal….

The SecureWorld Expo is a place where people can go to learn the WHY.  Not just I.T. folks, but people from all aspects of business as well.  They can talk to industry leaders and experts about things that are going down past, present and future.  It is all about translation and communication of the most up-to-date information available.  How up-to-date is the information that is covered?  The second day of the expo, the speaker Dan Greer came out to the podium and started talking about the SSL Man-in-the-Middle Renegotiation story that just broke in the news.  I have to say my hats off to the folks in the DevCentral community to, shortly thereafter, a way to mitigate the attack showed up on DevCentral (Lupo, thanks for your contribution!)… it can be found in the forums at http://devcentral.f5.com/Default.aspx?tabid=53&forumid=5&postid=86456&view=topic

Other than the open sharing and exchange of knowledge, the excellent speakers, free vendor loot and good food, the other thing that is great about the SecureWorld Expo is the fact that you get CPE credits for attending the various events.  Depending on the events that you sign up for you can either earn a 12 CPE or a 16 CPE Certificate of Attendance.  This is outstanding for those that have CPE requirements to meet and keep up with.  Not only can you obtain a lot of CPE’s in a short time, but it is also very cost effective.  You definitely get more bang for your buck at a SecureWorld event than you do at many others.

So what was covered in this “Nexum LTM Workshop”?  Quite a bit actually and it was all very well planned out.  The workshop was lead by a gentlemen named Peter Maag, who is a Senior Security Expert with Nexum.  I believe that part of what made this event so much fun for me was that it was obvious that Mr. Maag knew his stuff and (of course) I like talking about the capabilities of the BIG-IP line. 

Peter began by giving a brief introduction, explaining who he was and the services provided by Nexum.  I have to admit that I was unaware that Nexum was such a versatile company.  I believe it is so versatile and one of the fastest growing private companies because of them hiring and keeping talent like Mr. Maag around.  But that is a different philosophical discussion that perhaps I will touch on at some other time.  If I ever take up being a philosopher.

Then after the intro… it was f5 time!  For those in the audience not familiar with the BIG-IP product line Peter gave an overview of products available from f5 Networks.  He took the time to provide a clear picture of each modules functionality and I feel that he did the products justice.  He then steered the presentation to the real meat of the workshop which was the LTM module.  Virtual Servers, Pool Members and Nodes were all explained as well as the basics of configuring load balancing.  We spent some time discussing the full proxy architecture of the LTM module and we where then guided through a load balancing demo.

This lead into a discussion about monitors, persistence profiles, SSL termination and ended with a demo over those concepts.  There were a few questions at this point, as members of the audience asked questions such as “How long are self signed certificates valid for if they are generated on the f5 BIG-IP?” and “What are the different methods available for Cookie Persistence?”.  All of which were answered concisely and followed up with live demonstrations performed on a BIG-IP unit running TMOS version 10.x.  How cool is that?

We then went into a discussion about iRules.  Peter provided a number of examples of how to use iRules to pull off complicated tasks very easily.  In one example he showed how you could direct web traffic coming from an iPhone to a different set of servers than the ones used to serve up content to standard desktop browsers.  To augment the workshop Nexum provided an excellent booklet which just so happens to have a very handy page that lists almost all of the iRule Events that can be used in iRule generation.

We went over several other things, but the jest of this entry isn’t to really rehash everything that we covered.  The purpose is to encourage everyone using the LTM module to go check one of these workshops out.  Peter Maag did a phenomenal job explaining things for newcomers and veterans alike, which is not an easy thing to do.  To summarize, if you have just recently purchased an f5 BIG-IP product or are looking into purchasing one, attend one of these workshops.  You will walk away a wiser person and I cannot think of a better way to sell someone on f5 BIG-IP products.  Once you see it in action you will be wondering why you have stuck with Brand X for so long.

My next entry will be over the value of attending the SecureWorld Expo.  Is it worth the cost if you had to pay for it out of your own pocket?  What are the driving reasons for one to attend such an event?  I will be asking those questions and more soon and you may be surprised by my conclusions.  Stay tuned.

What a great way to segway into my first SecureWorld Expo blog entry!  Be afraid, be very afraid…  I am just kidding of course.  The Expo was excellent and I walked away from the event a wiser person.  It definitely helped me look at things differently and as Ralph Waldo Emerson once said, “Fear always springs from ignorance.” 

Man, oh man.  I think I may have committed a blunder of cosmic proportions.  Are you allowed to quote Yoda and Emerson in the same blog post?  Yes? No?  Anway, moving on…

The Expo started off with an awesome keynote by a very nice man named Jeff Bardin.  His topic was “Extremist Online Social Networks – Jihadis” and I was enthralled the whole time.  The banners that he had up on his very first presentation slide are the same web site banners that I have helped keep off of our network up at work.  When I saw those banners, I knew that he was going to be talking about a topic that hit close to home.

After taking the stage Mr. Bardin began explaining how Jihadis use resources provided by many American companies against America.  He talked about the Madrid train bombings, how Jihadis are using software like vBulletin and hacked copies of various software suites to pull off all kinds of nefarious acts.  He also discussed with great clarity how Jihadis are continuing to demonstrate their technical prowess.

Now I will not provide any more information about his presentation other than that.  Not because I do not want others to have the information, but because I cannot do the subject justice.  Mr. Bardin is an expert in his field and has spent countless hours researching, compiling information and teaching others.  I do not wish to diminish his work in any form or fashion.  Check out that link that I provided and the one at the bottom of this post for more information.

I would advise anyone, if Mr. Bardin is speaking at an event within a 12 hour driving distance, make the drive.  It really was that good.

After his presentation he stayed for a while answering questions.  I waited in the background for a bit, allowing others to ask questions as I listened in an attempt to take in as much information as I could.  When I did finally open my mouth he kindly gave me a his business card and answered all of the questions I had.  Anybody that will go out of there way to answer questions and share knowledge like Mr. Bardin did is a good man in my book.

For those seeking more information about Jeff Bardin and Treadstone 71, here is a link to some great information that will save you a trip to Google: http://blogs.csoonline.com/user/jeff_bardin

So what is coming up next?  Well I can’t go to long without talking about the F5 BIG-IP product line!  I am The F5 Guy after all.  My next post will be about the Nexum LTM Workshop that was lead by Peter Maag.