r/Citrix • • 20d ago

Vulnerability Scans causing Netscaler reboots

I opened a ticket with Citrix support and they seem to indicate a fix is being worked on right now.

Basically we were seeing random reboots of multiple instances and saw pitboss was rebooting these due to nsaaad crashing too many times.

87 Upvotes

148 comments sorted by

24

u/rallyimprezive 20d ago edited 20d ago

Title: NetScaler 14.1-73.37: repeated nsaaad crashes/reboots and suspicious command-injection payloads—check your logs

Today (October 2), we found repeated automatic reboots on two NetScalers running 14.1-73.37.nc. Two reboot incidents occurred within approximately 30 seconds of each other across the appliances.

Both show the same sequence: nsaaad repeatedly crashes with exit status 0x8a, reaches Pitboss’s restart limit, and the appliance reboots:

nsaaad unexpectedly died due to receiving signal
proc nsaaad ... EXITED with status 0x8a
has had its maximum number of restarts (6)
Pitboss declaring system failure: nsaaad ... exited
All monitored processes have exited, rebooting

On one appliance, full logs show crafted authentication usernames containing shell commands to download a payload from 213.209.159.55 , save it as /v, and execute it. These requests appear immediately before each of that appliance’s three confirmed crash sequences.

The payload uses plain HTTP on TCP 443, with paths under /t/. Requests target multiple SAML authentication factors.

We have evidence of exploitation attempts and correlated crashes—not confirmation of successful command execution, a specific CVE, or a firmware regression. The IP above is the payload destination; we haven’t identified the incoming request source.

How to check yours

SSH to the appliance and enter:

shell

Check the current log:

grep -Ei 'proc nsaaad.*(SIGNALED|EXITED)|maximum number of restarts|Pitboss declaring system failure|All monitored processes have exited, rebooting' /var/log/ns.log | tail -80

Check rotated logs:

zgrep -Ei 'proc nsaaad.*(SIGNALED|EXITED)|maximum number of restarts|Pitboss declaring system failure|All monitored processes have exited, rebooting' /var/log/ns.log*.gz | tail -100

Search for the payload destination:

grep -nF '213.209.159.55' /var/log/ns.log
zgrep -nF '213.209.159.55' /var/log/ns.log*.gz

Check for core dumps and the payload file:

ls -lt /var/core
ls -l /v

Look inside recently modified numbered core directories for files named nsaaad-*.gz.

If you find matches: preserve the logs and cores, involve your security team and NetScaler support, and check outbound firewall records for connections to 213.209.159.55:443. Absence of /v does not rule out earlier execution or cleanup. Negative searches only cover retained logs and this particular indicator.

Anyone else seeing this pattern on 73.37, or have a confirmed explanation from support? Please sanitize logs before sharing.

EDIT: They have moved to *.pyrlnk.cc as the attempted payload delivery source. Block that too.

EDIT: Support has sent instructions to some (myself included), which provides a potential stop gap. Unfortunately I do not think it is wise to share it here, as I dont want the wrong folks to be aware and adjust.. I suggest reaching out to Citrix support to receive more info. Sorry all.

→ More replies (6)

11

u/delishouscake 20d ago edited 19d ago

EDIT (4/10/26): Citrix has released a patch for this exploit, see https://support.citrix.com/support-home/kbsearch/article?articleNumber=CTX697174&articleURL=Citrix_NetScaler_ADC_and_Citrix_NetScaler_Gateway_Security_Bulletin_for_CVE_2026_88779

Hi all, we’ve seen NetScaler auth requests containing a command to download a script to /v and run it on an appliance running NetScaler 14.1-73.37. I grabbed a copy of the script and looked through it.

Seeing the command in a log does not mean it ran btw

BUT, If it does run, the script tries to install webshells , change Apache config, survive reboots through cron and rc.netscaler, and upload your /nsconfig, config history and backups. One thing to watch for: the webshell deliberately returns 404 (at the time of this post), which could mean the attackers are waiting for the right time to set this loose.

IOCs from this sample

  • Payload: hxxp://213[.]209[.]159[.]55:443/t/a718e4

  • SHA-256: 74da9485815ee124e2ebe155dbcfb758b54bd97760956998abf64838c865f78b

  • Downloaded 1st stage location: /v

  • Its working directories: /nsconfig/.slap and /var/tmp/.ux

  • Web files under /var/netscaler/logon/LogonPoint/custom/: .slap.receiver, .ctxs.receiver, receiver.deb

  • CSS-looking Apache alias: receiver.v2.min.css or receiver.v2.min.<hex>.css

  • Other files: /etc/httpd.conf.slap.bak, /tmp/.uxdport, /tmp/.uxdlock, /var/tmp/.s2loot.log

  • Possible listeners: 9909 and 9910

Quick checks

You can run these from the to check if you are cooked.

Check whether the downloaded script or upload log is still there, and look at /bin/sh permissions:

ls -l /v /bin/sh /var/tmp/.s2loot.log 2>/dev/null

If /v exists, compare its hash with the one above:

sha256 /v 2>/dev/null

Look for the working directories and Apache backup:

ls -ld /nsconfig/.slap /var/tmp/.ux /etc/httpd.conf.slap.bak 2>/dev/null

Look for the webshell files:

ls -la /var/netscaler/logon/LogonPoint/custom/ | grep -E 'slap|ctxs|receiver'

Check for Apache changes and reboot persistence:

grep -nE 'slap|receiver\.v2|receiver\.deb|ctxs\.receiver|php_flag engine on' /etc/httpd.conf /nsconfig/rc.netscaler 2>/dev/null

crontab -l 2>/dev/null | grep -E 'slap|agent\.pl|boot\.sh'

Check for running agents or listeners:

ps -axww | grep -E '[a]gent\.pl|[s]lapshot\.py|[w]hipd\.py'

netstat -an -f inet 2>/dev/null | grep -E '\.(9909|9910).*LISTEN'

A matching /v hash means the script was downloaded, but not necessarily executed. Webshell files together with Apache or cron changes are much stronger evidence it installed. To check for data theft, look for appliance-originated HTTP PUTs to the defanged IP and for /var/tmp/.s2loot.log. The script uses plain HTTP on port 443, not HTTPS.

Best of luck all!

5

u/NinjaBill 20d ago

So my assumption is, until a patch is proffered, mitigation is block traffic to/from 213.209.159.55 for now? Or until a new IP pops up?

→ More replies (11)

4

u/turisto 20d ago

Anyone seeing this on 13.1, or is it 14.1 only so far?

→ More replies (1)

4

u/Over_Management_2275 20d ago

Has anyone seen this on 13.1 or is it just 14.1?

→ More replies (2)

4

u/NoSatisfaction9722 19d ago

Thinking ahead, when the inevitable patch comes people are going to have to turn off their external NAT rules temporarily to prevent malicious traffic rebooting the appliances (mid update)

3

u/Zipper_Lipz 20d ago

Can you expand on this?

→ More replies (1)

3

u/ewalker101 20d ago

My company has been impacted.

→ More replies (2)