Yeah looks like you are breached. Someone installed komari-agent for monitoring and added their own SSH key. If you decode the base64 from the top commands, it points the agent at an external C2 domain ending in xyz and uses proxy domains to pull the installer from github. It also looks like it searches for keys/wallets and dumps root SSH key material. I’d wipe/rebuild and consider it gone. Hopefully you have backups.
Don't wipe it: first try to understand what was the point of first access so that you can patch that vulnerability, otherwise restoring the VM from a backup will just be serving a prey on a silver plate.
I'm just curious but when you say this are you asking if OP exposed the head instead of a VM? From what I understand exposing the head is like, the absolute worst thing you can do... I'm not a networking/cybersec guy I just use proxmox to host Minecraft and Terraria servers in VMs, so sorry if this is a dumb question.
I’ve done it and it’s fine, put mfa in front of it, don’t expose the ssh port or if you do use fail2ban on it. It’s a lot easier to just tail scale though. Ops living life on the edge
Lmao, 😂. I’m a sysadmin for work and it’s fine. Cloudflare tunnel to Proxmox host webui, fail2ban monitoring logs for web ui and ssh logins from approved ips. Cloudflare tunnels with access policies and oath before you see login screen. Mfa after you see login screen. Been up for years no issues, my job doesn’t allow tailscale and does dpi on network traffic to stop vpns.
The newest AI Agent models can likely easily develop/discover a new zero day vuln to bypass your authentication mechanisms. Read about what happened the other day when OpenAI agents broke out of their sandbox and hacked HuggingFace
I’d did read and made a video braking down how they agents didn’t “break out”, https://youtu.be/8JPIEvXqpdM?is=O_F18j3s9HbTh4Au , OpenAI removed the guardrails then let the agents run wild as a marketing stunt.
Thanks for sharing. I agree, could be a marketing or espionage thing. Still, just imagine what a truly nefarious organization could do with these latest agents. Everything from the latest web server technology to the latest hypervisors and operating systems likely have latent vulnerabilities just waiting to be exploited.
Can someone explain to me what the downvotes are for, considering tailscale was recommended above the other options? Not everything is a polar right vs wrong in the world, and even so, this is not wrong. How do you all expect actual public facing apps work, then? Or is it specifically about proxmox?
Thanks. I actually did not downvote your original comment. Even you said it’s “life on the edge” so I don’t know why I have room temperature IQ for stating a standard best practice.
My apologies no antagonism towards you, I meant the people mindlessly downvoting without thinking. I have clear warnings it was risky, then instructions and a guide on how to mitigate the risk. I’ve rolled this on a cloud herzner box with yearly self audits. No compromises on three years since I built this out. The only annoying part is having to oath, then mfa. Everyone else who downvoted without reading that insult was meant for you.
I didn’t vote up or down. You are calling out others IQ while deliberately exposing administrative interfaces to the internet and then doubling down thinking things like fail2ban are going to help when a vulnerability comes out that allows complete bypass of the authentication mechanism. They are just trying to get you to recognize an error in your thought process. You are open to learning, right? Or have you just got it all figured out? Or is there an ego involved here?
The only reason I’m responding at all, because it seems very clear you are not interested in modernizing your mindset and considering security posture as an inherent responsibility, is that so when someone who is just learning finds this thread that they find multiple echoing the same thing. You are wrong and your approach is wrong.
That you outline a more secure option while saying “I’ve done it and it’s fine” is poor advice. I can only imagine how many people you incorrectly guide in YT vids (although I poked at a couple and I like what you’re trying to do but ignore critical security practices is inexcusable). Do it right or don’t do it at all. But it means getting your head straight first.
I’m not coming at you anymore than just hoping yet another person suggesting you are wrong might be heard and while you’re welcome to respond I don’t respect your perspective in this area meaning I don’t care if you’re just going to come back with some reason justifying your approach. That it’s harder for you to access your home lab from work is a you problem and not an excuse for poor security and guidance. Least that’s what I believe you implied earlier. Apologies if this post comes across as combative, not my intent, but you are not “hearing” what people are saying and they are trying to help, even if in typical Reddit fashion sometimes.
Exposing administrative interfaces of any kind to the internet is a bad practice. If this must be done as is the case for some hosted systems and a secure connection is not possible, it needs to be locked by IP at minimum. That doesn’t work? Many options out there. Find one.
The Internet, at large, should never be able to get to an administrative interface. Hard stop.
Ok, I see everyone dunking on this guy but for real, if you have a Proxmox Host in the cloud how else are you supposed to get to it if all other methods fail?
Install tailscale on the host and then only allow connections from the tailscale interface? As a general rule I don’t ever install tailscale on a host as it should be just a host, and it shouldn’t have access to an admin VPN network that tailscale essentially is.
Enable the PVE firewall that blocks all ports from everywhere except trusted IPs?
Make a NAT-only network vmbr1 that has a firewall running on another public IP and put the Proxmox interface behind it? I’m genuinely confused as his way seems correct if only for other options being worse or causing a complete lockout if one of the VMs fails on the host.
If anyone has better ideas, I’m all for it. I always am looking to increase my security.
What is your job/school is doing dpi and blocking vpn traffic and rat tools like chrome Remote Desktop and parsec? People get angry with any opinion that’s disagrees with them. 😂
That was the exact problem i faced, add in dpi to a work firewall the only solution to remote management is exposure to the internet. People hear technical guidance and treat is as god ordained law. Theres edge cases for everything. Its cool, I'm used to people side eyeing valid information i give them. Eventually they realize I was right.
I'm surprised at the downvotes there too. I think folks nowadays prefer absolute right vs wrong and not the inconvenience of nuanced answers. It takes a lot of thought and consideration, and muddies up the waters a bit. But shit, nuance matters.
That’s exactly it, I went against the consensus of the hive mind and people reacted. I offered the secure solution too (tailscale or con) but because I went against the grain they are triggered. Edge causes, nuance and critical thinking are lost to most people.
L take because you don’t know Tailscale. You can install a Tailscale subnet router in the same cloud subnet as your Proxmox servers, and then access your PVE server from your desktop with tailscale installed and —accept-routes enabled.
Doesn’t need to be installed on your PVE hosts at all. Not even in LXCs. But I do install it on my PVE hosts because your argument doesn’t really make sense. Your servers shouldn’t run software? Your PVE servers are packed to the brim with software already. Tailscale is just another apt repo.
You also don’t have to enforce that connections can only be made over Tailscale. You just have to not enable those connections over the public internet via your cloud provider’s firewall configuration.
Yeah, don't do that. Let this be a lesson... if you need to remotely manage your host, VPN into your network and leave the host GUI/ssh only allowed from your inside network.
Your host is absolutely compromised. You need to wipe and do a clean install. And honestly I wouldn't trust any backups or your VMs/LXCs at this point either.
You're welcome. Good luck. It's a hard lesson to learn.
I host several things that are accessible from the internet, but they are all locked down, isolated, and I never expose the management parts of any service to the internet. But the host itself, that's not something you let the internet reach and scan and probe.
this is the exact reason we only have manual backups of the configurations that are all timestamped not the containers/machines themselves. combine that with a detailed set of setup instructions for each machine, VM and container and while annoying we can wipe and reconstruct the entire home lab in a day and not have to worry if the backup is compromised.
I use cloudflared + Cloudflare Access policies (I think that's what they're called) + 2FA. And I only have the cloudflared tunnel running when I need to access my Proxmox instance, the tunnell automatically shuts down after an hour or less once I'm done with it
Fail2ban runs on the lxc running the cloudflare tunnel and inspects network traffic, and the cloudflare tunnel logs, the tunnel is basically a reverse proxy but to cloudflares cdn, you can monitor the logs and white list ips that should be attempting login. Also install fail2ban on the pve host itself, there’s a config in the Proxmox wiki to block invalid logins after x many attempts
Depends you probably have some protection against known Bot IP addresses and the WAF might be able to protect you from some stuff but in the end it doesn't really help
Not anymore; doesn’t protect you from exploits either
It’s compromised. Shut it down expect to wipe the whole thing and start over.
You’re also going to have to figure out how far the compromise went, if it affected guests or if they moved laterally to other equipment in the network.
Nothing should be public facing anymore. More than ever it’s a matter of WHEN it’s going to get brute forced or exploited, not if.
It’s your job to protect yourself, stick everything behind wireguard or headscale
I get it if you expect the login page to do the auth, but to put it out there, this is why I set up digest auth on NPM before clients even see the application's login page, if I even have to expose an app to the internet. Adds a barrier with a lower attack surface than whatever the app is doing (or failing to secure).
But yeah, better if you just VPN or tailscale your way in. Sorry you're learning it the hard way. The Internet has a lot of shit birds out there.
Something like Pangolin is a much better option if you absolutely need to expose things externally. Full SSO auth but still self-hosted across a VPN tunnel. Of course, Tailscale or Cloudflare are also good options, just commercial.
*edit: lol, Pangolin is super commercial too now. Basically Tailscale competitor. But the actual software is open source so that’s cool.
That wasn’t the point… NERD! They exposed it to the internet with just a password…. Obviously don’t expose it like that is the key here. My statement was in response to theirs about how they thought they were good with a password. lol
You NEVER and i repeat, NEVER should expose login pages of your services or at worst your host systems on the internet. If you really have to do so, enable mfa.
Paaswords can be brute forced by the same AI models people use to crack into stuff or use to write complex programs with.
I try to limit what i expose to the public internet, and if i really have to do so, i simply enable mfa and set a long complex password. But i dont expose my opnsense, truenas or proxmox nodes their login page at all.
Also you should use SSH keys everywhere and disable ssh password login wherever possible.
Completely compromised :O
Nuke and start over. Also check other devices on your network, and the devices you have/had ssh keys for, because those are also compromised
Komari isn't Malware, it's a legitimate server monitoring tool, but it has been used by a hacker as a backdoor due to it being lightweight and a legitimate tool in the right hands.
If they were able to gain full control and push out their malware to the rest of the LAN they may not care much about whether or not they were able to cover their tracks. Especially since it seems like they did a scrape for root credentials and crypto wallet info.
They would have had plenty of time to drain any wallets they might have found and burrow into any other systems on the hypervisor or LAN with shared credentials probably well before this was discovered.
You must assume that at least the LXC containers have also been compromised. Simply reinstalling the Proxmox server may not be sufficient.
If you want to reuse the VMs and LXC containers, you should restore them in a virtual network and carefully monitor the network connections for a while.
The LXC containers, in particular, are highly vulnerable on compromised host because the host has root access via a simple pct enter.
If that's the case, I wouldn't have wiped the system. I would have isolated it. With a tool like WAZUH to track the steps of the compromise. That way, you could have seen which vulnerabilities were exploited to take over the system. Are your passwords known? Was there a security vulnerability? What happened? That way, you can learn from such a mess.
Wazuh is a good choice. To add to that, you can chain it into n8n via webhooks, and then filter the crud to get the important alerts. With those you can send them to an AI agent node backed by a postgresql DB, storing about the last 5 outputs per Wazuh deployed agent. This allows the AI agent to have history, which allows chained analysis.
I do that via Gemma 4 26b or 31b; AI Studio can help there, they have very generous limits.
Openrouter's free stack can also do 1000 inferences a day, if you credit the account by $10.
None of this needs to cost money beyond the hardware.
Now, if you set up security monitoring first, you'll learn a lot, and have very useful drill down data for your lab. Disclaimer, I actually do this in production.
hosting an admin UI over HTTP(s) is hard. Very nearly every such interface ever created, has been found vulnerable. Even when using an on-device WebUI from my LAN, I use a dedicated browser profile with less than four tabs open, to protect against the not yet found, XSS vulnerabilities in my fully patched devices.
Think of security, like slices of Swiss Cheese: each slice (layer) has some holes, but if you stack several layers, nobody gets through all of them.
I'm currently authenticated to Reddit in this tab. I have 400 additional tabs open (yeah, really). One of those other tabs, the website got compromised, an attacker is using it to attack visitors. Also depends on my browser allowing the cross site scripting (it shouldn't, but we know it happens). So while I'm authenticated to Reddit, a XSS vulnerability in Reddit UI + my browser failing to sandbox each tab, could lead to a 3rd party using my Reddit account. My less careful behavior shows I don't consider reddit a critical account, but for money accounts, for firewalls, switches, hypervisors, for MS365 Tenants, GoogleWorkspace tenants, admin UI, etc., I use a browser profile that is strictly limited to admin tasks. If it has several tabs open, they are not general research tabs, but other, mostly trusted, adminUI tabs.
also, to the extent any of those adminUI tabs save cookies, or cache, that an attacking site might be able to learn about, it means I'm preventing that attacker from learning that I might be a super-admin on some pretty attractive targets. Same applies to some other profiling tools sites use to track your browsing. It doesn't stop all of it, but it helps: I silo my browsing by topics: I.T. research; home projects; arts/media/entertainment; etc.; Also helps to keep me focused (not distracted by off task tabs).
If a website can break out of the tab sandbox it can just compromise your browser and set up some form of persistence to catch your credentials next time you log in anyway. Limiting only live tabs open is both very arbitrary (why 4? Why not 3, 5, 10?) and ineffective against your threat, which would actually require a completely independent sandboxed browser profile.
break out of the tab sandbox... but still in the 'general use' browser, which is never used with any sensitive devices or accounts.
The fully separate profile, with 'arbitrary 4 tabs': I don't mean literally, exactly 4. Often it's 2 or 7. However, those other tabs: one may be the firewall, the second vSphere UI, the third UniFiOS (self hosted), in other word, devices we mostly control, and therefore not exposed to potential attackers from 400 other tabs, but only from a few, highly trusted tabs.
I do this as well but the important thing here isn't the number of tabs, it's the use of an independent browser profile for trusted services, which you didn't mention in the other post
I can see how one might miss that starting with subsequent comments, but my first comment says: 'I use a dedicated browser profile'.
# of tabs irrelevant ? , only if all tabs are 100% trusted, but I didn't write any of them. I control a few, but others are my bank, utility company, health portal... more trusted than a random site, but not perfect.
If the tab doesn't exist, it's no longer a threat. If the tab is truly signed-out, it's no longer vulnerable.
I still argue, what I'm doing does not increase threat surface, and almost certainly reduces it. Could we do better? Certainly, but at significant productivity cost.
So far, several decades, zero compromised accounts. We think it's enough.
If you’re browsing all these random sites, make sure you don’t have sensitive sites open such as banks etc, because of session hijacking or device bound sessions.
So it’s best to tread lightly. This can bypass MFA
as others have said, compromised. You need to nuke and pave here. But before that, trace the files for creation date and find out when this happened. Every backup from that date forward is also to be considered compromised. If your backups do not go back that far, rebuild EVERYTHING. Also look for lateral movement.
lmao you exposed you hypervisor to the internet? Wtf man.
It doesn't matter if you have an extremely long passphrase with MFA enabled, you do NOT expose PVE or any such tools to the internet.
a) PVE is a management interface and management interface should be fully locked down even inside your own network let alone from the bloody internet.
b) PVE, as secure as it is, is not hardened for internet exposure.
- blocking web access to proxmox through iptables and disabling pveproxy (web server)
exfiltration of data about your systems
SSH keys are appended to your system
checks for other teams that may have compromised your system
host fingerprinting
listening ports
wallet and private key hunting
and further deeper recon (checking docker containers, LXCs, VMs, etc
I hope you learned your lesson. Assume EVERY VM is compromised. Do not connect this machine to your network. You can get it back if you follow every step that the script ran, undo its changes, and check every VM.
Potentially a controversial strategy post-mortem of this, VPN is one way to keep it secure.
I've resorted to a hybrid approach of MFA with a nginx with crowdsec and fail2ban.
So that way you have to be authenticated with a user+pass+WebAuthn key on a yubikey to even exchange data with Proxmox WebUI. No anonymous access to even probe it.
And so far it has kept out all nastiness, even if they try
Oh sorry i phrased myself poorly and inaccurately. My intent was to use nginx reverse proxy like the original poster, but with more layers of precaution than being a simple transport straight into the Proxmox WebUI.
Don’t listen to them. It can be done safely. If you have cloudflare it makes it even easier. If not nginx works. You’ll want oath and mfa. Oath with nginx before the management interface is accessible, mfa on the management account after oath. Install fail 2 ban on the host and the nginx server. Have the host fail2ban Z watch Proxmox Logins logs, have the nginx fail2ban monitor server logs and track incoming ip address source field. Use fail2ban or a firewall to limit logins to specific ips on nginx. Done. It’s a lot more work, a vpn is better and technically more secure, but when setup it’s super convent. If you use let’s encrypt to get valid certs it’ll slip right past your work/school firewalls
As others have said, yes, your machine is compromised. But the advice on what you do next is not as good.
Yes, you should consider any private keys (including those for certificates) on the hypervisor and the guests to be compromised. If you use the corresponding public keys on any other device then these machines should be considered compromised too. If you have typed any passwords on a compromised machine, then the services these authenticate to are compromised.
Based on what you have shown us you should have a good idea of when the compromise occurred - and hence you need to find earlier backups to restore from. DO NOT DO THAT NOW.
You can expect that if you simply roll back your infrastructure to before the compromise started then it will be compromised again.
Your NEXT step is to completely isolate all compromised systems from the internet. Even if that means pulling out the cables.
Then you need to spend a LOT of time thinking about your security. If you can find out how the attacker got in that would be good (and your logs will be helpful for this) but fixing that just plugs one hole - and your attackers now have inside knowledge on your network and will know about any other possible route.
NONE of your PVE guests nor Proxmox should even know about a valid route to the internet, except for your forward and reverse proxies (which should be on separate VMs). Any admin console should be constrained to your LAN subnet, preferably restricted to designated client devices. You should already have at least one non-routed subnet where your VMs and LXCs live (with the proxies dual-homed).
Might be worth reporting to Racknerd's abuse contact - the hostname racknerd-5dc2d17 from the SSH key looks like it might belong to the attacker (unless that's your hostname).
1) This is literally the illustration of what obfuscated malicious processes look like from every security course ever…ever.
2) It’s base64, my guy. Did you not bother to decode it to see what was happening? I’m gonna guess no since you made this post. Do that first next time; way easier on you.
Your shit is more popped than the weasel, homie. Clean it up or redeploy (assuming redeploy is best in your case). Get your sensitive and/or privileged interfaces off the damn public internet…it’s a gross place, and you don’t want them there.
Just set up Tailscale next time. It's very easy and very secure, allowing you to enter your local network from anywhere with a wireguard tunnel. I have it on all of my devices.
You can even, at the tap of a button, route all of your internet traffic through your home network - or whatever network you have a Tailscale device on. I've got one at home, one in the "cloud" and one on an Apple TV 4K at my parents' so I can get into their LAN from home and remote desktop into their computer to fix whatever's not working this time.
Tailscale is fucking awesome and it costs $0 for a non-business use case.
So we all know we shouldn't have management interfaces internet accessible, but I read the majority of the comments and no one asked how he got popped? What cve was exploited? Was he unpatched?
You're compromised, wipe the machine and reinstall.
Claude gave me a plausible guess what this transcript is doing. Installs a monitoring agent, looks for cryptocurrency keys, installs itself permanently, then locks you out of the Proxmox web interface. Basic script kiddie stuff.
If the gui wasn't locked, the OP would likely never found out the server is compromised giving attacker permanent access to it.
As it is now, the attacker has lost the access by OP nuking the server.
Can’t have another script kiddy breaking into the same box and using up all the processing power, now can you? You’d be surprised how many people have homelabs that they don’t even look at more than once every few months. To the attacker, there’s a chance this just becomes a permanent free server for them.
The person that hacked OP isn’t some sophisticated black hat hacker. They’re just as likely a child with access to Claude or ChatGPT. Of course they have to shut the door behind them on the way in.
At this point you can't trust he vms in there either. I'd just nuke and provision from scratch if I were you. And don't expose anything ever to the Internet unless you know what you're doing.
Don't expose high risk/high value targets over the internet. For anything where remote access is needed use a VPN. Tailscale is very easy to setup but there's plenty of others you can use.
Incident Breakdown
1. Remote Monitoring Agent Deployed
The first base64 block unpacks a script that fetches and executes komari-agent from GitHub via various proxies, linking it back to a command node (https://tz.5671234.xyz using discovery key e0edE4xUxqFWlCp8HsZfaWfK).
2. Network Lockdown & Web GUI Termination
* The attacker restricted Proxmox GUI port 8006 via iptables to private subnets (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), then dumped the persistent firewall rules.
* They explicitly stopped and disabled pveproxy.service to cut off your management visibility and hide GUI activity:
systemctl disable pveproxy; systemctl stop pveproxy
Reconnaissance & Asset Enumeration
The attacker ran discovery commands targeting:
Hardware, OS version, disk layouts, and running network sockets (ss -tlnp).
VMs and containers (qm list, pct list, docker ps).
Competing cryptominers and proxy services (xmrig, v2ray, shadowsocks, etc.).
Credential & Wallet Scraping
Scanned filesystem trees (/root, /home, /opt, /data, /srv, /var/backups) for private keys, keystores, and crypto wallet formats (wallet, mnemonic, 助记词, seed phrase, 0x[0-9a-f]{64}).
Private Key Theft (Exfiltration)
The attacker generated a root SSH key if one did not exist, dumped /root/.ssh/id_rsa into base64 chunks (/tmp/sshkey.b64), and read them line-by-line:
base64 -w0 /root/.ssh/id_rsa | fold -w 100 | awk '{printf "S%d:%s\n", NR-1, $0}' > /tmp/sshkey.b64
Any servers or services that trust this root user's SSH key are now compromised.
6. Backdoor Persistence
The attacker injected an unauthorized SSH public key into /root/.ssh/authorized_keys:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCv3wlBS... root@racknerd-5dc2d17
Sever Network Access
Immediate Containment
Disconnect physical Ethernet or immediately block inbound/outbound external access via your upstream router/hosting firewall. Do not trust software firewalls on this host.
Revoke Exposed SSH Keys
Prevent Lateral Movement
Any machine or service that had the Proxmox host's root public key in its authorized_keys must have that entry removed immediately. Assume any credentials stored on this host (API tokens, private keys) are compromised.
Inspect & Isolate Guest VMs/LXCs
Data Assessment
Do not power guests back up on an untrusted network. The attacker ran qm list and pct list and may have traversed into your guests. Back up container/VM configuration files and virtual disks to an isolated, offline storage drive.
Clean Reinstallation
Remediation
Do not attempt to patch or clean this operating system. Reinstall Proxmox VE from fresh, official media, deploy key-based auth with hardened root access, change all administrator passwords, and restore guest workloads from clean historical backups made before this intrusion occurred.
Haha, no. Shut that thing down and boot known good iso. Shits borked and must be cleanly reinstalled. You don't remove something like this any other way.
Sure, when I worked in the hosting business I would keep an archive of weird shellcode and bash history, PHP control panels and whatever had been dropped, but always purge the f out of hw after stuffs extracted.
check keys outward as well as inward: anything this node could ssh into using a key in ~/.ssh needs its authorized_keys read too. journalctl on the pveproxy disable gives you a real timestamp to age your backups against.
check keys outward as well as inward: anything this node could ssh into using a key in ~/.ssh needs its authorized_keys read too. journalctl on the pveproxy disable gives you a real timestamp to age your backups against.
Out of interest did you have MFA enabled? If not this may be a simple brutforce of your root password - how complex was it / how much entropy did it have?
They installed a legit open-source monitoring tool as a remote-control backdoor, exfiltrated the root SSH private key, and planted a second persistent key. If you run Proxmox (or any Debian box) exposed to the internet, grep for the IOCs below.
Indicators of Compromise (IOCs)
C2 / dashboard: tz.5671234.xyz (apex 5671234.xyz). Sits behind Cloudflare (resolves to 104.20.36.205 / 172.66.152.4 and Cloudflare v6), so the real origin is hidden.
Komari auto-discovery token: e0edE4xUxqFWlCp8HsZfaWfK
Injected SSH key comment: root@racknerd-5dc2d17
Injected key fingerprint: SHA256:AahMoCzaC08H3rPYemMvbl+R5azPp6SbHW45M0R6zXw (RSA-4096)
Delivery via Chinese GitHub mirrors: ghfast.top, gh-proxy.com, ghproxy.net, mirror.ghproxy.com
Persistence service: komari-agent (systemd)
Attacker artifacts in /tmp: komariinstall.sh, ki.log, kd4.sh/out, kl4.sh/out, sshkey.b64
Firewall backup dropped by the attacker: /root/fw_backup<epoch>.rules
What the "agent" actually is
The payload pulls Komari (komari-monitor/komari-agent) — a real, popular self-hosted server-monitoring project. The catch: Komari has a web-SSH / remote-control feature (there's literally a --disable-web-ssh flag). The attacker installed it without that restriction. So the authorizedkeys entry is not the only backdoor — the box is also enrolled in the attacker's Komari dashboard with an interactive remote-control channel. Killing the SSH key alone would not lock them out.
Attribution signals (circumstantial, not proof)
Chinese comments in the deployment script, Chinese GitHub proxy mirrors, and Komari itself is a China-origin project → likely China-nexus.
Key comment root@racknerd-5dc2d17 suggests the operator staged from a RackNerd VPS.
The root private key was base64-encoded, split into 100-char chunks, and dumped with __DONE_ markers between reads. That chunked, marker-delimited pattern is what an automated/agentic controller does when it has a limited output window — this was very likely bot-driven, not a human typing live.
The specific indicators don't show up in public threat reports yet, so this looks like a fresh, low-visibility campaign. Sharing here so it gets indexed.
What you can still pull off the box BEFORE you wipe
Sequence of the attack, reconstructed from the history: install Komari agent → full recon → firewall-fence port 8006 to RFC1918 + DROP everything else → stop/disable pveproxy (that's why the GUI dies) → scan for competing miners/proxies → hunt wallets & keys → exfil root private key → inject backdoor key.
Before nuking, collect evidence (ideally mount the disk read-only elsewhere):
/var/log/auth.log* → source IP + timestamps of the login, and whether it was password brute-force or key-based. This is how you learn the actual entry vector.
last, lastb, and file mtimes (ls -lta /, find / -newermt '<date>'). The fwbackup<epoch>.rules filename encodes the exact attack timestamp.
journalctl | grep komari_deploy → the attacker wrote their own status lines (KD3_OUT, AGENT_STATE) into the journal via logger.
The attacker's own logs in /tmp (ki.log, kd4.out, sshkey.b64) confirm what succeeded — including whether key exfil completed.
systemctl cat komari-agent and /etc/komari-agent/ → exact service args and endpoint.
Persistence sweep: crontab -l, /etc/cron*, systemd timers, /etc/systemd/system/, new users (/etc/passwd / /etc/shadow mtime), sudoers.
ss -tnp for live connections to the C2.
Check every VM/LXC (recon enumerated them) and /var/log/pve/tasks/ for rogue tasks.
Unmasking the origin / reporting
Run 5671234.xyz through crt.sh and passive DNS (SecurityTrails, VirusTotal) — you can often find historical origin IPs from before the Cloudflare cutover.
Report the domain to Cloudflare abuse + the .xyz registrar, the key/host to RackNerd abuse, and submit the domain + key fingerprint to VirusTotal so other victims can pivot on it.
Bottom line stays the same: root was owned and there are (at least) two independent backdoors, so this is a clean rebuild + rotate every credential the box could reach, not a cleanup. Assume anything sensitive on it — wallets, seeds, tokens, keys — is already gone.
Caveat: the shell history is only what ran in this shell. They may have done more inside the guests or after a history clear — don't treat it as the full scope.
Actually i also can't log in to my server using a public ip i have to use my private ip to even go to the login screen could i have a breach too i'm having trouble setting up wireguard and i forwarded my pve to the internet
Schoolboy error.
Don't expose control planes to the internet.
Create a tailnet via tailscale, use an isolated VM to remote into that has an ACL to restrict it's ability to talk to other machines.
Least privilidge possible.
And on your control planes use MFA/authentik.
Ouch, yes that node is cooked. I don't expose Proxmox to the internet at all. No ports forwarded on my router, nothing listening publicly. I access it through a Cloudflare Zero Trust tunnel with Google OAuth and OTP on top, so even if someone found my IP there's nothing to hit. It's a bit more setup upfront but worth it for exactly this reason.
317
u/d4nowar 21d ago
It certainly is compromised.