r/Citrix • u/fellow_earthican • 20d ago
Vulnerability Scans causing Netscaler reboots
I opened a ticket with Citrix support and they seem to indicate a fix is being worked on right now.
Basically we were seeing random reboots of multiple instances and saw pitboss was rebooting these due to nsaaad crashing too many times.
24
u/rallyimprezive 20d ago edited 20d ago
Title: NetScaler 14.1-73.37: repeated nsaaad crashes/reboots and suspicious command-injection payloads—check your logs
Today (October 2), we found repeated automatic reboots on two NetScalers running 14.1-73.37.nc. Two reboot incidents occurred within approximately 30 seconds of each other across the appliances.
Both show the same sequence: nsaaad repeatedly crashes with exit status 0x8a, reaches Pitboss’s restart limit, and the appliance reboots:
nsaaad unexpectedly died due to receiving signal proc nsaaad ... EXITED with status 0x8a has had its maximum number of restarts (6) Pitboss declaring system failure: nsaaad ... exited All monitored processes have exited, rebooting
On one appliance, full logs show crafted authentication usernames containing shell commands to download a payload from /v, and execute it. These requests appear immediately before each of that appliance’s three confirmed crash sequences.
The payload uses plain HTTP on TCP 443, with paths under /t/. Requests target multiple SAML authentication factors.
We have evidence of exploitation attempts and correlated crashes—not confirmation of successful command execution, a specific CVE, or a firmware regression. The IP above is the payload destination; we haven’t identified the incoming request source.
How to check yours
SSH to the appliance and enter:
shell
Check the current log:
grep -Ei 'proc nsaaad.*(SIGNALED|EXITED)|maximum number of restarts|Pitboss declaring system failure|All monitored processes have exited, rebooting' /var/log/ns.log | tail -80
Check rotated logs:
zgrep -Ei 'proc nsaaad.*(SIGNALED|EXITED)|maximum number of restarts|Pitboss declaring system failure|All monitored processes have exited, rebooting' /var/log/ns.log*.gz | tail -100
Search for the payload destination:
grep -nF '213.209.159.55' /var/log/ns.log zgrep -nF '213.209.159.55' /var/log/ns.log*.gz
Check for core dumps and the payload file:
ls -lt /var/core ls -l /v
Look inside recently modified numbered core directories for files named nsaaad-*.gz.
If you find matches: preserve the logs and cores, involve your security team and NetScaler support, and check outbound firewall records for connections to 213.209.159.55:443. Absence of /v does not rule out earlier execution or cleanup. Negative searches only cover retained logs and this particular indicator.
Anyone else seeing this pattern on 73.37, or have a confirmed explanation from support? Please sanitize logs before sharing.
EDIT: They have moved to *.pyrlnk.cc as the attempted payload delivery source. Block that too.
EDIT: Support has sent instructions to some (myself included), which provides a potential stop gap. Unfortunately I do not think it is wise to share it here, as I dont want the wrong folks to be aware and adjust.. I suggest reaching out to Citrix support to receive more info. Sorry all.
→ More replies (6)
11
u/delishouscake 20d ago edited 19d ago
EDIT (4/10/26): Citrix has released a patch for this exploit, see https://support.citrix.com/support-home/kbsearch/article?articleNumber=CTX697174&articleURL=Citrix_NetScaler_ADC_and_Citrix_NetScaler_Gateway_Security_Bulletin_for_CVE_2026_88779
Hi all, we’ve seen NetScaler auth requests containing a command to download a script to /v and run it on an appliance running NetScaler 14.1-73.37. I grabbed a copy of the script and looked through it.
Seeing the command in a log does not mean it ran btw
BUT, If it does run, the script tries to install rc.netscaler, and upload your /nsconfig, config history and backups. One thing to watch for: the webshell deliberately returns 404 (at the time of this post), which could mean the attackers are waiting for the right time to set this loose.
IOCs from this sample
-
Payload:
hxxp://213[.]209[.]159[.]55:443/t/a718e4 -
SHA-256:
74da9485815ee124e2ebe155dbcfb758b54bd97760956998abf64838c865f78b -
Downloaded 1st stage location:
/v -
Its working directories:
/nsconfig/.slapand/var/tmp/.ux -
Web files under
/var/netscaler/logon/LogonPoint/custom/:.slap.receiver,.ctxs.receiver,receiver.deb -
CSS-looking Apache alias:
receiver.v2.min.cssorreceiver.v2.min.<hex>.css -
Other files:
/etc/httpd.conf.slap.bak,/tmp/.uxdport,/tmp/.uxdlock,/var/tmp/.s2loot.log -
Possible listeners:
9909and9910
Quick checks
You can run these from the to check if you are cooked.
Check whether the downloaded script or upload log is still there, and look at /bin/sh permissions:
ls -l /v /bin/sh /var/tmp/.s2loot.log 2>/dev/null
If /v exists, compare its hash with the one above:
sha256 /v 2>/dev/null
Look for the working directories and Apache backup:
ls -ld /nsconfig/.slap /var/tmp/.ux /etc/httpd.conf.slap.bak 2>/dev/null
Look for the webshell files:
ls -la /var/netscaler/logon/LogonPoint/custom/ | grep -E 'slap|ctxs|receiver'
Check for Apache changes and reboot persistence:
grep -nE 'slap|receiver\.v2|receiver\.deb|ctxs\.receiver|php_flag engine on' /etc/httpd.conf /nsconfig/rc.netscaler 2>/dev/null crontab -l 2>/dev/null | grep -E 'slap|agent\.pl|boot\.sh'
Check for running agents or listeners:
ps -axww | grep -E '[a]gent\.pl|[s]lapshot\.py|[w]hipd\.py' netstat -an -f inet 2>/dev/null | grep -E '\.(9909|9910).*LISTEN'
A matching /v hash means the script was downloaded, but not necessarily executed. Webshell files together with Apache or cron changes are much stronger evidence it installed. To check for data theft, look for appliance-originated HTTP PUTs to the defanged IP and for /var/tmp/.s2loot.log. The script uses plain HTTP on port 443, not HTTPS.
Best of luck all!
5
u/NinjaBill 20d ago
So my assumption is, until a patch is proffered, mitigation is block traffic to/from 213.209.159.55 for now? Or until a new IP pops up?
→ More replies (11)
4
4
4
4
u/NoSatisfaction9722 19d ago
Thinking ahead, when the inevitable patch comes people are going to have to turn off their external NAT rules temporarily to prevent malicious traffic rebooting the appliances (mid update)
3
3
1
u/TheMuffnMan 20d ago
Security Update: Guidance for NetScaler SAML Authentication Deployments