I recently did a short 45 minutes talk at the #VirtualExpo event over at https://xenappblog.com/, about Bot management and Application firewall.
It was a short introduction to Bot management, with some live demo and then some tips and tricks for App Firewall.
If you didn’t catch it, the recording is now live, and I’ve uploaded my part here.
I wanted to create a blog post that could help the community, to use the App Firewall. Thit is one of the features that can help secure applications, running behind the ADC. But there seems to be a fear towards it, because admins have tried to enable it and something broke in the application behind it.
I think the biggest issue with App Firewall is understanding the application we want to protect, and one could argue that the ADC admin isn’t responsible for knowing how the application works.
In my experience the ADC admin is often the person that must figure out how the application works, because a lot of application owners only know how to use the application, not how it works.
So hopefully by the end of this series, you the reader, will have a better understanding on web attacks and how to counter them.
I will use an insecure web application called webgoat, that comes with lessons builtin allowing the reader to explore common vulnerabilities.
My most important advice will be to get to know the application developer. They should know how their application functions, what are expected requests and what should be blocked.
I run most of my web testing environment on a virtual Ubuntu server on my Intel Nuc.
The Ubuntu server is running a docker environment, which is perfect as webgoat got a docker image, that makes it easy to deploy.
If you want to run the webgoat/wolf using docker run, you can use the following cmd:
docker run -d -p 8080:8080 -p 9090:9090 -e TZ=Europe/Amsterdam webgoat/goatandwolf
But I like convert the run command into a yaml file, so I can use docker-compose. There is this little website that can create a yaml configuration from the run command, visit https://composerize.com/
I’ve changed the frontend port for the webgoat to port 8000, as I have something else running on port 8080.
mbp@mbp-virtual-machine:~/docker$ cat goat-with-reverseproxy.yaml
version: ‘3.3’
services:
goatandwolf:
ports:
– ‘8000:8080’
– ‘9090:9090’
environment:
– TZ=Europe/Amsterdam
image: webgoat/goatandwolf
Run it using docker-compose -f goat-with-reverseproxy.yaml up -d or add it to your docker-compose.yml
I’m running a Citrix ADC VPX 3000 Premium on my Intel nuc (long live partner licenses)
I’ve created 2 non-addressable load balancing servers on my ADC, pointing to the Ubuntu server on IP 192.168.0.10 and the ports 8000 for webgoat and 9090 for Webwolf. Then I created a couple of content switching policies, where I limit the traffic to only be accessable from my LAN and using the correct hostname.
There is also a responder policy bound on each LB to let the client know that requests against / should be to /WebGoat/ or /WebWolf/ depending on which LB the request landed at
The CLI configuration is as following:
#Server add server Ubuntu 192.168.0.10
#ServiceGroup add serviceGroup LB_SG_Webgoat HTTP -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO add serviceGroup LB_SG_WebWolf HTTP -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO
#Binding Servers to ServiceGroup bind serviceGroup LB_SG_Webgoat Ubuntu 8000 bind serviceGroup LB_SG_WebWolf Ubuntu 9090
#Load balancer add lb vserver LB_WebGoat HTTP 0.0.0.0 0 -persistenceType NONE -cltTimeout 180 add lb vserver LB_WebWolf HTTP 0.0.0.0 0 -persistenceType NONE -cltTimeout 180
#Binding ServiceGroup to LB bind lb vserver LB_WebGoat LB_SG_Webgoat bind lb vserver LB_WebWolf LB_SG_WebWolf
#Responder Action add responder action res_act_webgoat redirect "\"/WebGoat/\"" -responseStatusCode 302 add responder action res_act_webwolf redirect "\"/WebWolf/\"" -responseStatusCode 302
#Responder Policy add responder policy res_pol_webgoat "HTTP.REQ.URL.EQ(\"/\")" res_act_webgoat add responder policy res_pol_webwolf "HTTP.REQ.URL.EQ(\"/\")" res_act_webwolf
#Binding Responder Policy to LB bind lb vserver LB_WebGoat -policyName res_pol_webgoat -priority 100 -gotoPriorityExpression END -type REQUEST bind lb vserver LB_WebWolf -policyName res_pol_webwolf -priority 100 -gotoPriorityExpression END -type REQUEST
#Content Switch Action add cs action cs_act_Webgoat -targetLBVserver LB_WebGoat add cs action cs_act_Webwolf -targetLBVserver LB_WebWolf
#Content Switch Policy add cs policy cs_pol_Webgoat -rule "http.REQ.HOSTNAME.SET_TEXT_MODE(IGNORECASE).EQ(\"webgoat.netscaler.dk\") && CLIENT.IP.SRC.IN_SUBNET(192.168.0.0/24)" -action cs_act_Webgoat add cs policy cs_pol_Webwolf -rule "http.REQ.HOSTNAME.SET_TEXT_MODE(IGNORECASE).EQ(\"webwolf.netscaler.dk\") && CLIENT.IP.SRC.IN_SUBNET(192.168.0.0/24)" -action cs_act_Webwolf
#Binding CS policy to Content Switch bind cs vserver gw.netscaler.dk -policyName cs_pol_Webgoat -priority 200 bind cs vserver gw.netscaler.dk -policyName cs_pol_Webwolf -priority 210
WebGoat is a deliberately insecure application that allows interested developers just like you to test vulnerabilities commonly found in Java-based applications, that use common and popular open source components.
Now, while we in no way condone causing intentional harm to any animal, goat or otherwise, we think learning everything you can about security vulnerabilities, is essential to understanding just what happens when even a small bit of unintended cod,e gets into your applications.
What better way to do that than with your very own scapegoat?
Feel free to do what you will with him. Hack, poke, prod and if it makes you feel better, scare him until your heart’s content. Go ahead and hack the goat. We promise he likes it.
If you want to learn more about Webgoat, please visit https://owasp.org/www-project-webgoat/ and remember this application is made to be hacked, so having external access to it, isn’t a good idea.
When you access webgoat for the first time, you must register a new user, so click the register new user.
Make sure the username is in lowercase and accept the terms and conditions.
Now you can log on using the newly created user and start to explore webgoat. There is a lot of good information in the general area.
Let´s just jump directly into the first lession. It’s about injection and the focus for this first blog post, in the series.
There is of course a xkcd about injections.
Credit: https://xkcd.com/327/
The first test is learning how to extract information from a database, that our web application is using.
They want us to retrieve the department of the employee Bob Franco, which can be done using a Select query, like SELECT department FROM employees WHERE first_name=’Bob’
The statement can be translated to: please tell me what department users with the first name Bob, in the table employees, is a member of.
Our application got a nice form where we can submit our query, but it could a create user form, any input or query directly in the URL. How often have you seen something like ?id=5 in the URL? You could try to append your statement directly to the URL
Our statement returns the “Marketing” department. 
Great success, our insecure webapplication is insecure.
If we look with Inspect in our Chrome, we can see the query is sent.
To help against web attacks, there is a function on the ADC called Application firewall, which is a Premium licenses feature.
The App Firewall works by identifying pattern and behaviors in traffic. The simplest patterns are based on signatures. They can be found under Security – Citrix Web App firewall – Signatures
A signature is a string or a pattern that matches a known attack. The default signatures are derived from Snort (https://www.snort.org/), the ADC is able to use other lists if they are converted to XSLT files, which is the format the ADC supports.
The signatures can be updated by allowing the NSIP to contact Amazon AWS and a working DNS on the ADC. Now personally I don’t recommend allowing your NSIP to reach the internet. I would recommend you to setup a local webserver, and make it update from there.
To start with, I will create an appfw profile and policy that will log our sql injection attack. It’s a good idea to start with logging, because appfw can break web sites, that are doing something not common.
On my ADC I go the Security section and find Citrix Web app Firewall – Profiles
The profile is the collection of prevention methods we will use.
When you add the profile, you must know something about the web application, you’re going to protect.
Is a basic html application, a more web2.0 application (xml, soap) or json
For our little SQL blocking, a normal Web Application is enough (But the more ticks you make here, the more features you can enable in the profile, when we edit it)
I give it a name. I always use the naming methodology og appfw_profile_xxx (this could be the name of webapp, the function we’re protection against)
In this example it’s AppFW_prof_SQLi. Select Web application (html) and give it a comment, once you click OK, you get sent back to the previous screen. Next we have to edit it.
I select my newly created profile and edit it (I’ve put in a feature request, that we should be able to edit, when we create it)
When I open my profile, I select Security checks in the right panel, this will be the different areas we want to look at. A default policy looks like this.
I’ve removed the check marks on everything, except LOG and STATS on the HTML SQL Injection area, because I only want to focus on that part now. 
Next we have to create a policy (Remember that on the ADC there is policy that evaluates data, like a http request, response, client IP, destination port, source port and so forth) which then calls a profile that can apply an action.
I switch to the Policies – Firewall section in the Citrix Web app firewall, and add a new one.
My naming methodology here is AppFW_pol_XXX. This policy is appfw_pol_sqli, it points to my profile AppFW_prof_SQLi and the expression is true (which is an advanced policy compared to the old classic policies of ns_true), telling the ADC to always execute the profile, and I give it a comment. 
Now I have to bind it to my Load balancing server for WebGoat.
The policy is bound as an App Firewall policy on the request side.
My policy is selected. 
And the binding order is given. I select END as my goto expression, because right now I don’t have any other policies I want to evaluate.
Remember that our policy just logs and gives stats about it now, so how do we see if our policy is hit? Personally, I like to monitor the ns.log file using my ssh client.
When I log on my ADC I jump into the Shell and switch to /var/log folder, to keep on monitoring the ns.log. We can use the tail command, which default will show you the last 10 lines of a file. But I also want to keep on monitoring for any App Firewall log entry, so I use tail -f | grep APPFW, the option -f tells tail to watch for changes and update if anything is added within the last 10 lines.
Then when I try to SQL Injection on the Webgoat again, I can see it my log file.
The output is here: Feb 25 20:23:52 <local0.info> 192.168.0.240 02/25/2020:19:23:52 GMT MBP_ 0-PPE-0 : default APPFW APPFW_SQL 27820 0 : 192.168.0.3 10674-PPE0 – AppFW_prof_SQLi https://webgoat.netscaler.dk/WebGoat/SqlInjection/attack2 SQL Keyword check failed for field query=”SELECT(‘)” <not blocked>
I can see that my ADC on IP 192.168.0.240 (The NSIP) reported that client 192.168.0.3, triggered the AppFW_prof_SQLi profile with a SQL keyword check for query=”SELECT(‘)” and it wasn’t blocked, because my profile only logs and stats it.
if I want to see the same in the GUI, I can open my syslog audit message (Under System – Auditing or Security – Citrix Web App firewall – Policies – Auditing – syslog messages)
I will select the module APPFW to limit the output of the ns.log 
Now it’s time to block the SQL injection. To do so, all we have to do, is change our profile security, to block also.
When I try to enter my SQL Injection on the Webgoat, nothing happens, when I click submit. 
But I can see in my log, that the request was blocked. 
Or in the GUI.
Now a select statement might not be as dangerous you say, but they could do insert, delete or update data. Let´s look at that, while the App Firewall profile is set to LOG.
In the 3rd Injection lesson they want us to change the department of the employee called Tobi. Tobi is in Development, so maybe he created the application and therefore I don’t feel pity for him, as I change his department to sales.
My query is as following: UPDATE employees SET department=’Sales’ WHERE first_name=’Tobi’
Congratulations Tobi, you’re now in sales. Maybe Tobi should have learned how parameterized the sql statements.
And our trusted tail -f ns.log shows the following entry
Mar 1 13:37:11 <local0.info> 192.168.0.240 03/01/2020:12:37:11 GMT MBP_ 0-PPE-0 : default APPFW APPFW_SQL 107534 0 : 192.168.0.3 96299-PPE0 – AppFW_prof_SQLi https://webgoat.netscaler.dk/WebGoat/SqlInjection/attack3 SQL Keyword check failed for field query=”UPDATE(‘)” <not blocked>
Being a good ADC administrator, I noticed we had a SQL injection that wasn’t blocked. I quickly switch the profile back to block, before our friendly hacker feel pity for Tobi and want to move him back to Development department with the following query: UPDATE employees SET department=’Development’ WHERE first_name=’Tobi’
Sadly, it doesn’t work, and Tobi is stuck in Sales forever.
Mar 1 13:56:43 <local0.info> 192.168.0.240 03/01/2020:12:56:43 GMT MBP_ 0-PPE-0 : default APPFW APPFW_SQL 107967 0 : 192.168.0.3 97181-PPE0 – AppFW_prof_SQLi https://webgoat.netscaler.dk/WebGoat/SqlInjection/attack3 SQL Keyword check failed for field query=”UPDATE(‘)” <blocked>
If you ever find yourself in the need to make it hard to read headers and base64 encoding isn’t enough, because lets be clear, base64 encoding isn’t encryption and everyone can decode it, then you might find yourself in the need of encrypting the header value with a RSA public key, that you can later decode on the backend server using the private key.
First, we need to generate our public and private key pair, to do I will load up my trusted openssl to create the private key
openssl genrsa -aes256 -out private.pem 1024
You will be prompted to enter a pass phrase, this will needed to load the key
Next, we need to create a public key, that can use to encrypt the headers on the ADC on, I will do using openssl again and outputting the public key directly in my terminal
openssl rsa -in private.pem -RSAPublicKey_out
To read the key (private.pem) we will need to enter the pass phrase we created before
So now we have our private key and the public key which is:
—–BEGIN RSA PUBLIC KEY—–MIGJAoGBAMGf546XMUpqjoCx6GcZBZKTMdf2OUGlpAyD2RJ9fTMxjo79WvkA69T9I9d5JERG21dzDcBSPp2kmnEmwXPss+/3EnQIDAX4gtSS0zj/3WNd3h2ljrHkfqbWLJtVzz93JqfNVB3b8uMRK34usgIp78Da25yYrij/0ZtJAP12vJQ/AgMBAAE=
—–END RSA PUBLIC KEY—–
Next we need to create our expression on the ADC, so we can call it later on.
add policy expression RSA_Pub “\”—–BEGIN RSA PUBLIC KEY—–MIGJAoGBAMGf546XMUpqjoCx6GcZBZKTMdf2OUGlpAyD2RJ9fTMxjo79WvkA69T9I9d5JERG21dzDcBSPp2kmnEmwXPss+/3EnQIDAX4gtSS0zj/3WNd3h2ljrHkfqbWLJtVzz93JqfNVB3b8uMRK34usgIp78Da25yYrij/0ZtJAP12vJQ/AgMBAAE=—–END RSA PUBLIC KEY—–\””
It will look like this when you add it on the ADC (notice that my nsroot asks for a passphrase for a key, but that is another blog post)
Now that we our public RSA part is available on the ADC as an expression, we can start to use it.
To showcase it i will create a simple header insert called RSAENCRYPTED that will include the TXID “TXID
Returns the HTTP Transaction ID. The value is a function of an internal transaction number, system boot time and system mac address. HTTP.REQ.TXID is same as HTTP.RES.TXID.”
First I will create a rewrite policy and action that will insert the header on the RESPONSE (This is cheating a little, but it’s easier to show using inspect in a browser)
My action:
add rewrite action rw_act_encrypt_header insert_http_header RSAENCRYPTED http.RES.TXID
My policy:
add rewrite policy rw_pol_encrypt_header true rw_act_encrypt_header
And then I will bind it to a server I have running, which is just a default apache on an ubuntu
bind lb vserver name -policyName rw_pol_encrypt_header -priority 100 -gotoPriorityExpression END -type RESPONSE
Now when I make a request to against the server, the response will insert the header RSAENCRYPTED with the value of the transaction ID
(You can click on the picture to get a bigger view)
Now the whole idea was that our headed should be encrypted, so lets add expression to our rewrite
It’s done by adding the following bid of expression: PKEY_ENCRYPT_PEM(RSA_PUB)
The RSA_PUB is the name of our expression, the action would look like this, if we wanted to enable the encryption, when we added it the first time (Else you have to update the expression)
add rewrite action rw_act_encrypt_header insert_http_header RSAENCRYPTED “http.RES.TXID.PKEY_ENCRYPT_PEM(RSA_Pub)”
Lets take a look in the inspect again
(You can click on the picture to get a bigger view)
It’s unreadable now.
Now let’s keep in mind that I’m a major newbie to the whole azure, so there might be smarter way of doing some of the stuff I’m going to show, but it works.
In my Azure tenant for Netscaler.dk I’ve added my work account from edgemo.com, so my validation will happen with my edgemo account.
In the Azure portal go to Azure Active Directory.
Click on Enterprise Application, where we can have a look at my “Citrix Apps” application, which is just a SAML Single sign-on application (It’s a Non-gallery Application, so make sure to select that when you add our new application).
Since I’ve already created my App, I will just click on the MyCitrixApps, to get the properties.
The properties of the Application allows you to do different stuff, like controlling Single sign-on and Conditional Access, and who gets the use the new Application. Since I want to allow single sign-on from my Citrix Unified Gateway, I have to setup the SAML information.
The information that I filled out was:
Identifier (Entity ID): https://gw.netscaler.dk
Reply URL (Assertion Consumer Service URL): https://gw.netscaler.dk/cgi/samlauth
Notice that there is an App Federation Metadata URL, which will make the setup of the SAML server on the Citrix ADC much easier. This I will show you later on.
Now to use setup Conditional Access we have to setup a new Policy, so in your App go to Security – Conditional Access – Click New.
Next up is the conditional Access. Since this is just a Proof of Concept, my conditional access will be very simple.
If the User Mads logs on to the environment using Windows, he will be required to provide MFA. If he doesn´t use Windows, a password will be enough. Now let’s look at how that is configured.
Under Conditional Access click New Policy.
I name my policy Netscaler.
Next I will select the User/group that it will be applied too.
Next I will have to setup the conditions that will apply, in my case it will be limited to Windows.
After you selected the conditions that are suited in your environment, you have to do the Access Control. For this setup it will Grant Access, but require MFA.
Lastly remember to enable the Policy and Create it.
Remember to assign your new application to a group/user
Now I won´t go into most of the Citrix ADC configuration, since most of it will be like my last blog post, which you can read here: https://netscaler.dk/?p=67
But I will cover the feature in Citrix ADC 12.1 build 51.30 which is
Metadata reading and generation support for SAML SP and IdP configuration
Citrix ADC appliance now supports metadata files as means of configuration entities for both SAML Service Provider (SP) and Identity Provider (IdP). The metadata file is a structured XML file that describes the configuration of an entity. The metadata files for SP and IdP are separate. Based on deployment, and at times, one SP or IdP entity can have multiple metadata files.
As an administrator, you can export and import (SAML SP and IdP) metadata files on Citrix ADC.
For more information, see https://docs.citrix.com/en-us/Citrix ADC/12-1/aaa-tm/saml-authentication.html#metadata-reading-and-generation-support-for-saml-sp-and-idp-configuration
This will allow us to create a SAML server, just by pointing to the XML file. If you remember when we created the Azure Enterprise application, there was a field called “App Federation Metadata URL” that contained an URL.
Keep in mind that your Citrix ADC will have to be able to browse the URL. If it only has internal Access, you can download the XML file and put in on a server that the ADC can reach.
Now to setup a new SAML policy on the ADC, go to Security – AAA Appication Traffic – Policies – Authentication – Basic Policies – SAML – Servers and click Add.
Notice the new Export and Import. We will use the Import here, since we need the iDP information.
Like always give it a name that makes sense. I normally use the following format auth_location_type, so in this case it will be auth_azure_saml.
You will get the URL for the federation metadata URL under the Single Sign-on in the Azure App.
This is the URL you will copy to the Citrix ADC. My configured SAML server looks like this.
Under More I’ve changed the Signature Algorithm to RSA-SHA256 and Digest Method to SHA256.
Next we have to make the Advanced Policy (So the workspace App can trigger validation) and bind it to the AAA server we’re using. This was all explained in the previous blog post.
Now to try something new, I’ve recorded a video. If this is something I should continue with, please let me know.
Like I started this blog post with, I’m using my work account from Azure in my private Azure Tenant. This is why my user is redirected to a local ADFS from edgemo
This is how the Workspace App looks like.
Type in the URL/Email.
Enter the username.
Same as in the video, I’m redirected to our company ADFS, because I’m using my edgemo account.
And the conditional Access prompts to for MFA.
For a smoother user experience, you can click Yes (Ja) on the following screen, to reduce the amounts of times you must re-autenticate.
Now let’s try the same on a Mac, which is my primary machine.
I enter https://gw.netscaler.dk and get redirected at once and I can select my account.
I’m redirected to the ADFS server of edgemo.
After I enter my password, I get the same question “if I want to reduce the number of logins”.
And since I’m running on a mac, I get direct access to the unified gateway, without running MFA.
Let’s take a look on the workspace app on mac. I start by adding my account.
Notice that the “browser” doesn’t remember my account, so I have to enter my email.
And I get redirected to the ADFS, where I can enter my password.
Like the last time, I get the prompt to reduce the number of logins (I keep clicking no, because I want to be able to show the whole process on request without issues).
And I get access to my applications.
And this is how my launched desktop looks like.
To get the information about the FAS, you can go to powershell. First we load the Snap-in.
Add-PSSnapin Citrix.Authentication.FederatedAuthenticationService.V1
Now we can use a command to get the certificates issues by the FAS server.
Get-FasUserCertificate -Address Citrix.netscaler.local
For the first setup of the workspace app, there will be a popup, where you can enter information about the environment you will connect to. Once you enter the URL/Email, it will contact the Citrix ADC if you’re from the outside, and hopefully the StoreFront directly, if you’re on the inside. But since I want to show the SAML at once, this will be a setup from external.
I’m using OKTA as my SAML provider, therefore am i redirected to Okta login 
After i sign in, i will get access to my apps
I can launch my desktop
And on my FAS server I can see my certificate is created
Now to the fun part, how do we make it work.
I will mainly focus on the Citrix ADC part, as there are other great guides out there for the Citrix FAS, like the one Carl Stalhood made, which can be found here: https://www.carlstalhood.com/citrix-federated-authentication-service-saml
The Citrix ADC needs to be upgraded to firmware 12.1.50.28nc, It’s suppose to work in .49 also, but there is a kernel error which will make your Citrix ADC reboot, from what I’ve been told, by some fellow PTEC guys.
I’ve tested it both with a normal Citrix Gateway and Unified Gateway. To make it work it will require that you have an Advanced License (The old Enterprise), because we need to make use of nFactor.
On the Citrix Gateway we will have to configure an authentication Profile, so we can point our authentication traffic to an AAA server, to trigger the nFactor.
You can create the Authentication under Security – AAA Application Traffic – Autentication Profile – Add, now an authentication profile is just a pointer to the AAA server
This is how we move the authentication traffic from the Unified Gateway to AAA, where we want to do the SAML authentication.
I’ve created a SAML server for my Okta account. Like I wrote in the earlier blog post, there are some requirements for setting up SAML. We need the Redirection URL (Where will users do the authentication), the iDP signing certificate, the User Field (What will we receiver from the iDP), to sign our request from the service provider and finally the issuer name, which is the name the Citrix ADC sends to the iDP to uniquely identify the ADC.
My SAML server looks like this
Next we need a policy to call our server (remember that a policy is how we call a server/action). I’m using an Advanced Policy, which can be created under Policies – Authentication – Advanced Policies – Policy – Add
Since I want SAML to be used for everyone and everything, I will just use the policy expression true and point the action to my SAML server created earlier
Now it’s time to bind it to my AAA server, I will bind it under Advanced Authentication policies
That is basicly all you need to change, if you used a SAML policy directly on the Citrix Gateway before, which will work with browsers. But the workspace app needs nFactor to kick in.
The CLI can be found here
SAML:
add authentication samlAction auth_okta_saml -samlIdPCertName Okta -samlSigningCertName wildcard_netscaler_dk -samlRedirectUrl “https://netscaler.okta.com/app/citrixnetscalergateway_saml/secretsauce/sso/saml” -samlUserField “Name ID” -samlIssuerName “http://www.okta.com/secretsauce” -signatureAlg RSA-SHA256 -digestMethod SHA256 -logoutURL “https://netscaler.okta.com”
add authentication Policy authadv_pol_okta -rule true -action auth_okta_saml
Authentication:
add authentication authnProfile AAA-AUPL-SERVER -authnVsName AAA-SERVER -AuthenticationHost aaa.netscaler.dk -AuthenticationDomain netscaler.dk -AuthenticationLevel 3
add authentication Policy authadv_pol_okta -rule true -action auth_okta_saml
bind authentication vserver AAA-SERVER -policy authadv_pol_okta -priority 100 -gotoPriorityExpression END
Unified Gateway:
set vpn vserver UG_VPN_gw.netscaler.dk -authnProfile AAA-AUPL-SERVER
If you have issues with authentication, we can head over to the good old aaa.debug in /tmp folder, remember this file is a pipe, so we’re looking into it as traffic is coming through.
A sample SAML authentication will a little like this:
Sun Feb 3 13:34:11 2019
/home/build/rs_121_50_16_RTM/usr.src/netscaler/aaad/naaad.c[900]: process_kernel_socket 0-64: partition id is 0
Sun Feb 3 13:34:11 2019
/home/build/rs_121_50_16_RTM/usr.src/netscaler/aaad/naaad.c[2221]: process_kernel_socket 0-64: saml_canon: preamble size is 10782
Sun Feb 3 13:34:11 2019
/home/build/rs_121_50_16_RTM/usr.src/netscaler/aaad/naaad.c[2223]: process_kernel_socket 0-64: saml_canon: read 10758 bytes from socket
Sun Feb 3 13:34:11 2019
/home/build/rs_121_50_16_RTM/usr.src/netscaler/aaad/naaad.c[2232]: process_kernel_socket 0-64: saml_canon: canon_req size is 10722
Sun Feb 3 13:34:11 2019
/home/build/rs_121_50_16_RTM/usr.src/netscaler/aaad/naaad.c[2233]: process_kernel_socket 0-64: saml_canon: input string is <saml2:Assertion xmlns:saml2=”urn:oasis:names:tc:SAML:2.0:assertion” ID=”id243710244642880201145219498″ IssueInstant=”2019-02-03T12:34:10.919Z” Version=”2.0″><saml2:Issuer Format=”urn:oasis:names:tc:SAML:2.0:nameid-format:entity” xmlns:saml2=”urn:oasis:names:tc:SAML:2.0:assertion”>http://www.okta.com/SecretSauce</saml2:Issuer><ds:Signature xmlns:ds=”http://www.w3.org/2000/09/xmldsig#”><ds:SignedInfo><ds:CanonicalizationMethod Algorithm=”http://www.w3.org/2001/10/xml-exc-c14n#”/><ds:SignatureMethod Algorithm=”http://www.w3.org/2001/04/xmldsig-more#rsa-sha256″/><ds:Reference URI=”#id243710244642880201145219498″><ds:Transforms><ds:Transform Algorithm=”http://www.w3.org/2000/09/xmldsig#enveloped-signature”/><ds:Transform Algorithm=”http://www.w3.org/2001/10/xml-exc-c14n#”/></ds:Transforms><ds:DigestMethod Algorithm=”http://www.w3.org/2001/04/xmlenc#sha256″/><ds:DigestValue>6pq1us+JQQXPIWJ8R/p+ZH7Vb1DYUNyUCZvhg4WaFoY=</ds:DigestValue></ds:Reference>
You can also use a SAML tool in the browser to decode. I like SAML Message Decoder for Chrome myself and the Developer tools in the browsers are also useful.
The flow is as following
Now the Client will talk with the SAML iDP and here it will do a SAMLRequest, which is a base64 encoded message (Look for a POST against the iDP which contains the SAMLRequest) as seen here:
Now this big block of data, is something you can decode online. I like https://www.samltool.com/decode.php, where you can paste the whole SAMLRequest, and it will return something like this
So there is a lot of information to troubleshoot SAML on the Netscaler and in the browser
Hey,
It’s been a while since i updated my blog, but i thought it would be time to pick it up again.
So i was playing around with the native receiver (workspace app) and SAML/FAS, as i’m having some issues getting this to work, so i wanted to set it up my own little test environment at home.
For that i had to setup some SAML authentication on my Netscaler running build 12.1.50-28.nc and i kept getting an error while trying to add the SAML server
To setup a basic SAML policy we need to add the SAML iDP server which you can do under Citrix Gateway – Policies – Authentication – SAML – Servers – Add
To make the SAML server you need a couple of things
But whenever i tried to add the server i got the following message
Arguments cannot both be specified [samlIdPCertName, metadataUrl]
I have to admit there have been a lot of GUI issues in the Netscaler lately (Like the Invalid Argument AES256 from last patch) so i jumped into the CLI to see how little i should add before Netscaler would accept it and the GUI would allow me to work with it.
The CLI for adding a SAML server is something like this: add authentication samlAction auth_Okta_saml -samlIdPCertName Okta -samlSigningCertName -samlRedirectUrl “https:///adfs/ls/” -samlUserField “Name ID” -samlIssuerName
But if you don’t prefer the CLI and want to use the CLI, the least amount of configuration i could get the Netscaler to accept and allow me to edit the server in GUI was this: add authentication samlaction auth_Okta_saml -samlIDPCertName Okta -samlSigningCertName Cert -samlredirectUrl https://fqdn
After that i could edit the server and change any setting, so my guess is that the Netscaler got an issue in regards to the Redirect URL
p.s i know the Netscaler is Citrix ADC now, but i love the old name to much, sorry Citrix 🙂
I have just returned from a customer (a municipality here in Denmark) they are using an iPad application that a lot of municipalities are using, this application requires the user to be onsite, because connection to the service is firewalled, so only certain IPs can access it.
Now I´m doing a lot of Citrix XenMobile, so of course I thought that I would just wrap, upload it to the Citrix AppController, then that advantage of the power of the mighty Netscaler and mVPN feature.
This application required some iOS hacking before I was able to wrap it successful, sadly the AppController didn´t want to save the policy configuration, so I had some help from Citrix Denmark and we finally was able to get an MDX version of the application that the AppController accepted.
Now getting the application to respect the MDX policies was a major pain, I could see in the debug log of the application that it wasn´t intercepting the FQDN, but after a lot of debugging I was finally able to get application to function as we wanted.
The next path was creating a LB on the Netscaler, so the FQDN was sent through the mVPN and into the company network, so we can send the traffic from the application towards the public service from the correct IP. We got this part to work today and my customer told me that getting this one application to work was major for him, as they had no good way of handling the application. They had talked about using some VPN software on the iPad, but that wasn´t application based, it would be annoying for the users to remember to start the VPN.
In the end we managed to make a seamless solution, the application is deployed automatic when the iPad is enrolled in the MDM solution, our friends over at SMS Passcode protect the connection, and the users are finally able to use that application from wherever they are, whenever they want to. This is what Citrix is all about.
Hi all,
It´s been a while since my last blog post, but I have been busy at work, working on a very fun XenMobile project.
Last post I promised that I would explain how we can use the built-in tcpdump function in the Netscaler, without having to take a packet capture and open our trusted Wireshark.
I´ve been on a lot of assignments, where I had to setup something for the customer, like a SSL Offload for Outlook Web Access, and once I’ve done the service part, I quickly notice the service is in a downstate as shown in this picture.
Now we can check why the service is in a downstate by double clicking the service, and checking the monitor
This tells me that when the Netscaler tries to make a TCP connection to the service using the monitor HTTP, it sent the packets, but didn´t couldn´t make a proper TCP connection, something is blocking the connection. Since I almost never touch the backend service, like the OWA in this example, I would ask the contact person on site, if they were sure that the subnet IP of the Netscaler is able to communicate with the backend – hence, is there firewall blocking the communication. I often get an answer like “Yeah, we check the firewall and nothing is blocking the traffic, the problem must be on your end”
Now I make sure that I have the correct IP address, that I’m trying to connect to it on the correct port number, if there are a lot of different interface i.e. a lot of different SNIP, I make sure the Netscaler will send the traffic using the correct SNIP.
Now a way to check that the traffic is flowing like I want it, I could fire up the packet capture on the Netscaler, download the pcap file and run it in my trusted Wireshark, this is a very effective way of debugging, but seeing as this is a simple issue, I can just use the built-in tcpdump function.
I SSH to the Netscaler, change to shell and fire up my nstcpdump.sh
The OWA backend service ran on IP 10.10.200.5, so I want to monitor the traffic flowing from the Netscaler to that IP.
In my shell cmd I enter: nstcpdump.sh dst host 10.10.200.5, this will show me traffic sent to the destination host 10.10.200.5, the output will look like this
We can see that 10.10.200.16 is sending a packet to 10.10.200.5, but the ack 0, this means that the Netscaler have not received a reply from destination, and the pattern is the same in the following packets.
Just a note “the first packet sent wouldn´t be able to have an ack number, since the source haven’t communicated with the destination yet.”
The output tells me the follow, the Netscaler is trying to communicate with the backend server from SNIP 10.10.200.16, it´s connecting to the backend from a random TCP number, but the destination port number is 80/http like expected. I can now go back to my contact person, saying that I can see the Netscaler is behaving as I expected.
I would say from experience that 9 out of 10 times the traffic is being blocked by a firewall.
Once we get the service in an UP state, the output of nstcpdump.sh DST host 10.10.200.5 would look like
It´s easy to see the difference between a down and up service using nstcpdump.sh
There are lot of other useful filters, but take a look at the CTX article located at http://support.citrix.com/article/CTX118185.
That is all for now, next time we will take a look at XenMobile App Wrapping, I just did a fun job where I had to hack an iOS application, so I could wrap it and upload it to the Citrix AppController.
Take care out there.
Hi,
I can´t count how many times I’ve been told that the Netscaler isn´t letting users log on, so no one can work.
In 99% of the cases it´s not the Netscaler that is failing, but the external authentication service we are using, so unless you work with local users on the Netscaler, then the Netscaler will ask an external authentication server to authenticate an user.
Let us have a look at what happens when an user tries to log on using an AGEE and they fail their login.
The user gets the message “Incorrect user name or password” When we have to figure out what is going on, we can turn to the auditing – syslog on the Netscaler
(Click the picture for a larger version of it)
The picture tells us the AAA module had a login_failed for the user mbptest the reason is “External authentication server denied access” this is tell a Netscaler admin, that it wasn´t the Netscaler itself that denied the user access to the system. However, it doesn´t say what authentication server was asked, what the reason for deny is, so the only useful information we got, was that it wasn´t the Netscaler itself.
Now if we want to a much better way to figure out what is going, we can use the aaad.debug module, this module is a pipe, so nothing is saved to disk, but require we do live monitoring of it.
To get access to the aaad.debug we need to use the command line of the Netscaler, so we can go System – diagnostics – command line interface, which will open a console on the Netscaler from the GUI, but it´s rather limited so I much rather start up my trusted SSH client and connect to the Netscaler.
Once we got access to the Netscaler, we have to go into NSCLI (Netscaler Command Line Interface) so type in shell and press enter, this will change the prompt from > to the user@hostname#
Go the /tmp folder using cd /tmp, and try to type ls -l, you will find aaad.debug in this folder, so now we just need to monitor the file, while we do a login, and to do that we can use the command cat, you can find the manual page for cat here http://unixhelp.ed.ac.uk/CGI/man-cgi?cat
So to monitor the aaad.debug, we will use cat aaad.debug, now we will see everything that touches the AAA daemon, ask the user to log on again and follow the authentication.
We can see that the user mbptest is starting an LDAP authentication against the server 10.10.10.11
The next thing that happens here is the connection to the server is using SSL/TLS, and the connection to the 10.10.10.11 is using SSL (ldaps port 636) then the bind event starts and finally the bind event is successful.
The bind is when the user we use to access the ldap server, so on our LDAP server we added a service account, that is used to access the ldap, now if the bind fails, then no one will be able to log on, because we can´t access the ldap server.
Common issues when bind fails are password expired, the account is logged out of the domain, and account is disabled.
The next that happens is a bind event for the user, where we will check the ldap for the user account, figure out what groups/nested groups the user is member of, and finally ldap will return the result of the bind event
We can see that the user is located, but the error is invalid credentials (i.e. wrong password)
Therefore, we checked that the Netscaler could communicate with the LDAP server, the service account works (the first bind is successful) but the user is typing in a wrong password.
If we have a primary and a secondary authentication server (like radius and ldap) then the auditing – syslog would still just say, “external authentication server denied access” but using aaad.debug we can check how far into the authentication the users gets.
For the next blog I will talk about using NSTCPDUMP.SH for live packet monitoring without the need for a wireshark.
Hello,
There is an quick and easy way to see what policies applied in realtime using the command line
If you havn´t had time to check out the nsconmsg command, this post will help you master it.
The command can be used with the AGGE, rewrite and responder policies, and i find that it´s the fastest way to debug what is going on.
The first thing you need to know, is that you have to be in shell mode, for it to work, so after you SSHd into the Netscaler (I prefer to use Putty http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html on Windows and just use the buildin on my Mac)
So after you access to the netscaler, you have to type Shell like this:
After you got access to the shell, you can use nsconmsg, the parameters that i mostly use is the this:
nsconmsg -d current -g pol_hits
When an user logs on to the AGEE, it will display which authentication policies that was used and the session policies (if the login was successful of course) an output will look like this:
This picture shows what policies was hit in realtime.
There are a couple of other paramets that are helpful:
nsconmsg –d current | egrep –i rewrite/responder depending if you want check for rewrites or responder policies.
Hopefully this quick post will help Netscaler administrators to debug AGEE, rewrite and responder policies in realtime.
My next blog post will be about authentication troubleshooting in realtime also.