Interview Experience with S&P Global

 

This interview was for a Principal Network engineer role. The JD completely matched my profile but it had a little more into Network side of things. I still thought i can be a good match. I managed to get this referred via my friend Sai Pradeep who completed graduation in same college with me. His reference worked and I got a call from the HR. Interview was scheduled. 

I had around 3 days to prepare and i dont remember when was the last time that i prepared for an interview so well. I even managed to start learning AWS Networking as that was part of the JD. Surprisingly i was able to relate AWS networking so well that i am now preparing for AWS Certification CNS01 Advanced Networking. 

Coming to the case of actual interview, the person who took the interview Deepen Patel started the conversation as this is more of casual discussion and he was asking more about my work that i do. He then tried to ask some simple questions and interview was always comfortable. I have some below take aways that I though could have been done better

• I sounded little overconfident about certification when I said "probably i will do that after getting into S&P. 
• When the question was asked about how I troubleshoot I kept it very simple by saying look through the path and understand where the problem is. I could have elaborated it much better adding my experience in all sorts of technologies. For example, if something is not working I should have started from DNS to path tracing to handling firewall in path to troubleshooting on firewall and advanced troubleshooting on firewall and then including proxy load balancers etc.  Just so that i can prove my expertise. 
• I could have been little slow and explained things more detail so that it would have given a better chance to evaluate. This is because at the end of the day what we say is what they can judge upon. 

On a last note, I got an email from HR that I am not selected. Little disappointed but I want to take this positive as I think I have a better opportunity to work on migrations that I have in hand and customer is looking for me. 

Fixing Cannot complete your request after enabling two factor authentication

 

While performing a change on Netscaler to enable second factor authentication via RSA. I ran into an issue where in user would see an error that said "cannot complete your request STF:<storefront name>". The troubleshooting was difficult because the storefront server was not showing any error log that could indicate a possible issue.  

To clarify what is done here are high level steps of what is being done to enable 2nd factor

• Find the authenticaiton vserver that is mapped to authentication profile of the Gateway VIP.
• Check the Authentication policy and bind RSA as Next Factor authentication and dont forget the add END after RSA.
• Go to Login Schema and upload the new XML file that has the page that we see for 2 factor authentication. 
• Here this profile carries settings for SSO. If this is not enabled the credentials will not be passed on to Storefront.
• This last step is critical because previously SSO policy was configured to pass on the credentials to storefront now this is done as an option in login schema. 

Updating Login Portal on NetScaler with ADM

 

 Customizing the login portal on a Netscaler login page can be a bit tricky, especially since Citrix doesn’t officially support it. The challenge lies in updating multiple Netscaler devices across the environment. Unlike pushing CLI commands, which can be automated, portal changes are typically done manually through the GUI. These modifications often result in changes to specific files rather than the main ns.conf configuration file. For instance, when configuring background images or logos, the process involves adjusting these files directly. 🛠️🔒

Below is the path where the significant changes in custom portals will happen 

Logos

/var/netscaler/logon/themes/<custom theme name>/custom_media/

Configs Related to button/background header color/footer color/text file etc will happen for the below files in the path 

File: custom.css  and theme.css

path: var/netscaler/logon/themes/<custom theme name>/css/custom.css

           var/netscaler/logon/themes/<custom theme name>/css/theme.css

Also along with this please replace the custom_theme.json file which  is available under the custom theme folder

path: var/netscaler/logon/themes/<custom theme name>/custom_theme.json

The easiest way to handle this is to take one Netscaler and update the portal as needed and then the configs on that Netscaler should be on the 3 files along with logos. If we  do a config job on ADM to push the files to all Netscaler this will replace the files and changes will be effective immediately  ( no reboot needed)

The config job on Netscaler should look as below

Please note the custom_portal in the path is not a default. It is the name of folder if you create a custom theme.

Things to Note for the config job on ADM

1) Ensure we select "scp" for the dropdown before each command

2) Ensure we select the variable type as File. You should be able to see when you click on the variable or if you select the Preview Variable option on the right side top. 

3) Ensure we give the complete name of the file when specifying the path. Below is an example

Hope this helps 

cheeers!!!!

BIGIP Configuration for Prober reference on Generic host

    Today I worked on a GTM issue that happened because of configuration that was done not knowing the consequence. 

For Generic host we will have an option saying prober preference ---> outside datacentre. This will make any LTM or GTM outside the Datacentre configured in the GTM cluster. This  means there is a possibility that LTM that is behind the firewall can also poll and we might end up seeing member being marked down if there is no firewall rule to allow this traffic.

Netscaler password hashing With KEK for instance specific password

 

Before 13.1 version on Netscaler if we replace the ns.conf file from one Netscaler to another everything would normally work including the passwords. This means if I copy the command with the password in the hash, it would work fine.  However, after 13.1 they started to include additional hashing where the commands used to have an additional part that said KEK .  Here the password gets hashed again with keys local to the device. This means if you copy the command from one Netscaler and try pasting on another one it will fail.

It would throw an error saying the decryption failed. There is a way to get rid of this. If we go to /nsconfig folder there will be a folder with name keys. Try copying the files to the device where you wanted to import the password and this will work like magic. 

Netscaler Split tunnel hostname redirection

 After a Netscaler upgrade from 12.1.63.23 to 13.0.88.14 we started to see that users are not able to access a few IP segments specially 172.16.0.0/16 . Interestingly this is working if the client VPN plugin version is 22.2.1.103.

We raised a case with Citrix and after back and forth conversation it is realized that they have changed their design to use this segment for their intranet applications that are defined in split tunneling. This has been confirmed by the support that for some reason the VPN client is intentionally using 172.16.0.0 / 255.255.0.0 as it's Intranet Application hostname interception range. This means the client is pointing that range, on it's own, to itself to be used for hostnames that you list as Intranet Applications in the gateway config.

We will be able to see this clearly if we try to debug the traffic from the client plugin. The fix for this is configure a custom spoofed IP range so that this does not overlap the said I segment range. Below is how we do that. 

VPN Session Profile > Client Experience > Advanced Settings > "Spoofed IP Addresses for FQDN Based Tunneling". We used 169.254.0.0 / 255.255.0.0 as per Citrix support and now we can access 172.16.xxx.xxx on our VPN.

Big3d Timeout on F5

  After the F5 GTM upgrade to 14.1.4.6 it is observed that all the VIP on the GTM are flapping. We observed that this was matching multiple errors. Here the VIP will show a log saying it turned green to red because of big3d timeout. 

Basically, this is a timeout from GTM to LTM

Below were the articles that we saw matching our situation where we thought it is the problem is the resources on the LTM. Since it is pegging up we increased the resources on the LTM.

https://my.f5.com/manage/s/article/K35326235

We tried the recommended action as in below.  But this did not work. 

Recommended Actions

One or more actions can be taken on the impacted BIG-IP device to free up resources so that big3d can respond in a timely manner:

1. Add additional CPU cores to VM or vCMP guest
2. Reduce the size and/or complexity of the BIG-IP configuration

If the above actions do not work or are not feasible, then you may increase the BIG-IP monitor timeout value on BIG-IP DNS:

1. Create a new BIG-IP monitor and increase the probe timeout from 90 seconds (default) to a new value.  Increase in 5 second increments and test to see if the timeout messages stop.
2. Select the new BIG-IP monitor for the impacted BIG-IP server Health Monitor.
3. Click Update.

Timer Change :

Then we tried changing the timer. We created a monitor as in the below settings and configured it to all GTM. What i did not do is we have to apply the same monitor to all the LTM on the server. 

Further, there was a recommendation to update the timers. F5 recommends that the timeout value should be equal to 3 times the frequency ( interval) +1 . Example is below. Also they recommend that the interval value should be shorter. Below is the optimum settings that they suggested.