BIGIP Configuration for Prober reference on Generic host

    Today I worked on a GTM issue that happened because of configuration that was done not knowing the consequence. 

For Generic host we will have an option saying prober preference ---> outside datacentre. This will make any LTM or GTM outside the Datacentre configured in the GTM cluster. This  means there is a possibility that LTM that is behind the firewall can also poll and we might end up seeing member being marked down if there is no firewall rule to allow this traffic.

Netscaler password hashing With KEK for instance specific password

 

Before 13.1 version on Netscaler if we replace the ns.conf file from one Netscaler to another everything would normally work including the passwords. This means if I copy the command with the password in the hash, it would work fine.  However, after 13.1 they started to include additional hashing where the commands used to have an additional part that said KEK .  Here the password gets hashed again with keys local to the device. This means if you copy the command from one Netscaler and try pasting on another one it will fail.

It would throw an error saying the decryption failed. There is a way to get rid of this. If we go to /nsconfig folder there will be a folder with name keys. Try copying the files to the device where you wanted to import the password and this will work like magic. 

Netscaler Split tunnel hostname redirection

 After a Netscaler upgrade from 12.1.63.23 to 13.0.88.14 we started to see that users are not able to access a few IP segments specially 172.16.0.0/16 . Interestingly this is working if the client VPN plugin version is 22.2.1.103.

We raised a case with Citrix and after back and forth conversation it is realized that they have changed their design to use this segment for their intranet applications that are defined in split tunneling. This has been confirmed by the support that for some reason the VPN client is intentionally using 172.16.0.0 / 255.255.0.0 as it's Intranet Application hostname interception range. This means the client is pointing that range, on it's own, to itself to be used for hostnames that you list as Intranet Applications in the gateway config.

We will be able to see this clearly if we try to debug the traffic from the client plugin. The fix for this is configure a custom spoofed IP range so that this does not overlap the said I segment range. Below is how we do that. 

VPN Session Profile > Client Experience > Advanced Settings > "Spoofed IP Addresses for FQDN Based Tunneling". We used 169.254.0.0 / 255.255.0.0 as per Citrix support and now we can access 172.16.xxx.xxx on our VPN.

Big3d Timeout on F5

  After the F5 GTM upgrade to 14.1.4.6 it is observed that all the VIP on the GTM are flapping. We observed that this was matching multiple errors. Here the VIP will show a log saying it turned green to red because of big3d timeout. 

Basically, this is a timeout from GTM to LTM

Below were the articles that we saw matching our situation where we thought it is the problem is the resources on the LTM. Since it is pegging up we increased the resources on the LTM.

https://my.f5.com/manage/s/article/K35326235

We tried the recommended action as in below.  But this did not work. 

Recommended Actions

One or more actions can be taken on the impacted BIG-IP device to free up resources so that big3d can respond in a timely manner:

1. Add additional CPU cores to VM or vCMP guest
2. Reduce the size and/or complexity of the BIG-IP configuration

If the above actions do not work or are not feasible, then you may increase the BIG-IP monitor timeout value on BIG-IP DNS:

1. Create a new BIG-IP monitor and increase the probe timeout from 90 seconds (default) to a new value.  Increase in 5 second increments and test to see if the timeout messages stop.
2. Select the new BIG-IP monitor for the impacted BIG-IP server Health Monitor.
3. Click Update.

Timer Change :

Then we tried changing the timer. We created a monitor as in the below settings and configured it to all GTM. What i did not do is we have to apply the same monitor to all the LTM on the server. 

Further, there was a recommendation to update the timers. F5 recommends that the timeout value should be equal to 3 times the frequency ( interval) +1 . Example is below. Also they recommend that the interval value should be shorter. Below is the optimum settings that they suggested. 

Cert update in Netscaler with ADM

This post talks about Updating certificates in Netscaler. This is a tedious job when we have a bunch of Netscaler in the environment. Even the GUI-based option that we see in the Netscaler is not an easy one because it shows every certkey pair and we have to update it manually. I believe it is as similar to doing that on devices just that we don't login to each device. EOD it is still time-consuming. Instead, I thought running a job that executes the below command is easier as it runs on every device like a magic

First upload the cert and key via job as in below.  Here the hashed Item will be the certificate file. Try placing the files as PEM format and in the folder /nsconfig/ssl/

Then run a job to update the cerkey. You need to have the cert key details to do this.  Here in the below picture the command is simple 

update certkey <certkey name> -cert <pemfile> -key <pemfile>

Here the PEM file will be the file that is uploaded before. 

update ssl certKey <certkeyName> [-cert <string> [-password]] [-key <string> | -fipsKey <string>] [-inform <inform>] [-noDomainCheck]

Madhu Marriage Trip to Tirupathi

 It was kind of a sudden decision to attend the marriage. I say this because. I should have planned for darshan if I was sure about the travel. But I didn't.
             So the D day arrived.i booked tatkal tickets for me and Dayakar. We boarded train and started to talk. It was a nice journey. I never felt bored so I was glad in the morning that we reached renigunta early. Which is very near to tirupathi. Unfortunately my happiness didn't last for long time. It took 2 hours to reach tirupathi. Result is we missed marriage. Somewhere I felt they completed it too early. 1 hour from. Muhurtham time and every thing done.
We managed to click some pictures with the bride and groom.
             I understood that Madhu is concerned about Tulasi who didn't​ return after joining free Darshan queue yesterday. I tried to enquire and told him that it is 18 hr average time for Darshan.
              Tulasi finally came out and called us at 2 o clock which is 24 hrs. After seeing this we got scared about our Darshan plans. We went to CRO office and came to know about lucky dip for darshan ticket. We enrolled ourselves and came out. We managed to walk around and watched few things but we decided to start the return journey. This lucky dip was in our brain but we were pessimistic. At least I was.
             So when we came down to tirupathi. The moment I stepped at tirupathi I received a message that I got selected . I think  this is the first time I got selected in a lucky draw.  Then I was not sure about what to do. Next thing I did was I tried googling what is this seva about.
Understood that I will get a chance to watch closely and without standing in a big queue.
It happens in friendship that if u didn't get a chance ur friend will be unhappy but if u get a chance and not ur friend doesn't he will be more in happy.
               This happened to Dayakar I decided to go back and ask if Tulasi or Dayakar wants to come with me. I offered Daya that I am ready to walk if he agrees to come back. He wanted to go home instead.
              Dramatic things happened when I was on the way back. Travel from tirupathi to Tirumala generally takes 45 Min. I have to reach there by 8 and it is already 7;35 PM. I was loosing hope but I wanted to give a try. I stopped watching time and was just glaring around .  The view was fantablous. Tirupathi was glowing from the view with lights. When I reached I saw the time it took only half a hour. I was surprised. I rushed to the office and requested for my ticket showing my message. Next thing was again a speed breaker, he asked for enrollment slip which I misplaced. After repeated requests he told me to write a letter. I did and I managed to get one.
                   Madhu called me and when I told him that I need to stay back for the night he told he doesn't have bed but can sleep on floor. I decided to sleep on floor at waiting hall. I managed to sleep for couple of hours . After which I got my bath done and reached temple.


           The prime gist of the theme comes here I always believe about this enlightenment. With series of things that happened I thought something might happen. I wanted too give a try. I was exited to watch the diety very closely. I managed to reach there but for some reason there isn't anything that I felt. I was thinking that is this all? The. Feeling was not disappointed but it has no name, I just came out. Every one started say that i am lucky. No idea how was I.

       Then I started my marathon travel of 14 hours back to home and I will reach in an hour time from now

 

Last Day with Infosys

   Yesterday was my last day with Infosys. I have been working in General shift since one week and yes i almost stopped working for about 10 days from yesterday. I was going to office and spending time. I really did not have any emotional bonding with any of my fellow team mates they are just colleague for me and yes speaking frankly they were the best colleague i had. I never had any tiffs with any of them and i enjoyed working with this team. Only regret is that Ramana my manager has asked me to explain the network of the client CG. Some how i didn't want to do that and then i evaded but still i made myself available yesterday and today . He was too busy to catch me. We had a tea for one last time yesterday and then i bid the adieu. I did write an email straight from my heart. I got some nice words from Kaustubh initially and then Prem called me after seeing the email. It feels good and more emotional as the way he started talking showed it. He was talking about the team and was a long conversation. Then Ramana replied to this email. I am attaching along with this .