Hi,

I can´t count how many times I’ve been told that the Netscaler isn´t letting users log on, so no one can work.

In 99% of the cases it´s not the Netscaler that is failing, but the external authentication service we are using, so unless you work with local users on the Netscaler, then the Netscaler will ask an external authentication server to authenticate an user.

Let us have a look at what happens when an user tries to log on using an AGEE and they fail their login.

The user gets the message “Incorrect user name or password” When we have to figure out what is going on, we can turn to the auditing – syslog on the Netscaler

 

 

(Click the picture for a larger version of it)

 

The picture tells us the AAA module had a login_failed for the user mbptest the reason is “External authentication server denied access” this is tell a Netscaler admin, that it wasn´t the Netscaler itself that denied the user access to the system. However, it doesn´t say what authentication server was asked, what the reason for deny is, so the only useful information we got, was that it wasn´t the Netscaler itself.

 

Now if we want to a much better way to figure out what is going, we can use the aaad.debug module, this module is a pipe, so nothing is saved to disk, but require we do live monitoring of it.

 

To get access to the aaad.debug we need to use the command line of the Netscaler, so we can go System – diagnostics – command line interface, which will open a console on the Netscaler from the GUI, but it´s rather limited so I much rather start up my trusted SSH client and connect to the Netscaler.

 

Once we got access to the Netscaler, we have to go into NSCLI (Netscaler Command Line Interface) so type in shell and press enter, this will change the prompt from > to the user@hostname#

 

Go the /tmp folder using cd /tmp, and try to type ls -l, you will find aaad.debug in this folder, so now we just need to monitor the file, while we do a login, and to do that we can use the command cat, you can find the manual page for cat here http://unixhelp.ed.ac.uk/CGI/man-cgi?cat

 

So to monitor the aaad.debug, we will use cat aaad.debug, now we will see everything that touches the AAA daemon, ask the user to log on again and follow the authentication.

We can see that the user mbptest is starting an LDAP authentication against the server 10.10.10.11

 

The next thing that happens here is the connection to the server is using SSL/TLS, and the connection to the 10.10.10.11 is using SSL (ldaps port 636) then the bind event starts and finally the bind event is successful.

The bind is when the user we use to access the ldap server, so on our LDAP server we added a service account, that is used to access the ldap, now if the bind fails, then no one will be able to log on, because we can´t access the ldap server.

Common issues when bind fails are password expired, the account is logged out of the domain, and account is disabled.

The next that happens is a bind event for the user, where we will check the ldap for the user account, figure out what groups/nested groups the user is member of, and finally ldap will return the result of the bind event

We can see that the user is located, but the error is invalid credentials (i.e. wrong password)

Therefore, we checked that the Netscaler could communicate with the LDAP server, the service account works (the first bind is successful) but the user is typing in a wrong password.

If we have a primary and a secondary authentication server (like radius and ldap) then the auditing – syslog would still just say, “external authentication server denied access” but using aaad.debug we can check how far into the authentication the users gets.

For the next blog I will talk about using NSTCPDUMP.SH for live packet monitoring without the need for a wireshark.