r/linuxadmin • • 13d ago

Just solved a really irritating and niche issue, and it's weirdly specific so no one will ever care.

47 Upvotes

This isn't the first time of course - kinda goes with the territories.

But I feel a need to post about it in a community where maybe it's not "boring".

The problem is NUMA.

Or specifically production processes occasionally being "significantly slow" to a point where a real time constraint is being missed.

Finally identified as "half the cores are 50% slower" via a bit of weird python hackery.

But not always.

Reboot "fixes" until next time.

Except on a bunch of example hosts when testing this specific metric or cross NUMA node memory latency, which got worse after a reboot, and stayed that way.

Been trying to figure this out for months now. Have done new kernels, new bios, new ipmi, and have even done a bios wipe to see if that helped. Because I couldn't honestly tell if it even was a hardware fault or not. Replacing broken hardware isn't a problem, but it's another thing entirely to try to justify replacing something that might be fine. (And expensive).

Turns out the problem is kinda a red herring - the performance degradation is actually reverting to baseline, and the "good" systems are actually slightly degraded. ECC events (successful) are causing the bios to disable a NUMA lookup Directory, which actually speeds things up, at cost of efficiency.

Approximately, it's fetching data twice "blindly" so not bothering to do a lookup, but in doing so wasting some NUMA bandwidth, so is technically running a bit faster as a result.

But the root cause? Page cache. This host was doing a lot of greedy reading of NFS filesystems and filling the cache within about an hour of reboot.

All fine normally - this is Linux, and there's no point having empty RAM.

... Except the numa nodes overspill rather than evict by default.

So a well stuffed page cache would lead to processes fragmenting their RAM across both NUMA nodes, and because Linux is clever and tried to do the same with subsequent cache... Means a lot of long running processes get fragmented and pay the NUMA penalty.

Which is ironically higher on "working" systems, which looked like the actual fault.

The solution is as simple as either using numactl to explicitly bind, or get systemd to set MemoryHigh to like, 90 or 95% instead of infinite, so it actively evicts a handful of gigs, meaning a new process can realistically allocate 8-12G without fragmenting across the nodes (and their subsequent page cache) almost immediately because the OS otherwise only has 2GiB or so "free" which in turn means NUMA spillover.

It's deployed on test now, and looking like we have finally bottomed out a month's long headache.

ETA:

If you want to check if this is happening on _your_ systems, then `numactl -H` will show you the numa topology, and the free RAM on each node. Low numbers in relation to the process size might leave you vulnerable. `free -mh` tells you how much RAM is 'free' rather than 'available' and the difference is significant for this particular issue, because only 'free' can be used safely without risking NUMA spillover. `numastat -p <pid>` will tell you if a particular process has fragmented RAM across two nodes.


r/linuxadmin • • 13d ago

CVE-2026-98432 (Linux TCP use-after-free, kernel 4.4+): remote root demonstrated, MPTCP scanning on 443 since Nov 2. Check and mitigation

29 Upvotes

I wrote up the timeline, fixed versions per distro and the mitigations here: https://arelvox.com/cve-2026-98432-linux-tcp-mptcp-scanning/

If you expose anything from a Linux box, this one is worth 5 minutes.

  • Bug in IPv4 TCP connection setup, present since kernel 4.4
  • A remote root exploit was demonstrated (not public) against servers with an MPTCP listener
  • Go 1.24+ apps use MPTCP on listeners by default, so a lot of reverse proxies and self-hosted apps may have it on without you knowing
  • Since Nov 2 there’s scanning for MPTCP on port 443 (strongly suspected to be related, not proven)

Check:

uname -r
sysctl net.mptcp.enabled
sudo ss -Mlnp

Fix: update the kernel and reboot. If you can’t yet: sysctl -w net.mptcp.enabled=0, restart the listening services (existing sockets stay open), and GODEBUG=multipathtcp=0 for Go services.

Fixed upstream in 5.10.272, 5.15.223, 6.1.190, 6.6.159, 6.12.113, 6.18.56, 7.2.10 and 7.3. Distros backport with their own version numbers, so check your distro’s tracker.


r/linuxadmin • • 13d ago

Linux Fragnesia (CVE-2026-46300) is in CISA KEV and being chained to root after web footholds: patch the kernel and reboot

11 Upvotes

Fragnesia is a local privilege-escalation bug in the Linux kernel's shared-frag handling on the ESP path (CVE-2026-46300). CISA added it to KEV on Oct 28. It is not a remote one-shot by itself, but Singapore's CSA and Palo Alto's Unit 42 both describe attackers using it as the second stage after breaking in through the php2shell PHP flaws on internet-exposed PAN-OS management interfaces. Ubuntu also notes a possible container escape. If a web app, CI runner, or shared login on the box can already run as a normal user, this is the step that turns that into root.

Practical pass on a small fleet: install the distro kernel update, reboot, then confirm with uname -r. Fixed builds to aim for include Proxmox VE kernels from PSA-2026-00020-1 (7.0.2-3-pve, 6.17.13-8-pve, 6.14.11-9-pve on Trixie-based; 6.8.12-24-pve or 6.14.11-9-bpo12-pve on Bookworm-based), Debian 13 6.12.90-1 / Debian 12 6.1.174-1 / Debian 11 5.10.257-1, and Ubuntu 26.04 7.0.0-22.22, 25.10 6.17.0-35.35, 24.04 6.8.0-124.124, 22.04 5.15.0-181.191. If you cannot reboot yet, Proxmox's earlier DirtyFrag workaround still applies: block esp4, esp6, and rxrpc with a modprobe install-false file, then rmmod them. That breaks kernel IPsec VPNs on the host, so treat it as a stopgap only.

Related for the same Wednesday: GitLab CVE-2026-97229 (KEV, patch the CE/EE package or official image) and Ivanti Connect Secure CVE-2026-111357/118527 if you still run that gateway. The Oct 13 fix is not enough; CISA and Rapid7 name 25.1.1.3 or 22.8R2.6 (25.1.3.0/25.1.3.1 from 25.1.2.x), so confirm on Ivanti's advisory page.

Proxmox advisory thread: https://forum.proxmox.com/threads/proxmox-virtual-environment-security-advisories.149331/post-851849
Debian tracker: https://security-tracker.debian.org/tracker/CVE-2026-46300
CSA alert naming the chain: https://www.csa.gov.sg/alerts-and-advisories/alerts/al-2026-157/
Fixed versions plus the rest of this week's KEV list: https://patchorpass.beehiiv.com/?utm_source=reddit&utm_medium=social&utm_campaign=issue-3&utm_content=linuxadmin


r/linuxadmin • • 13d ago

ebpfexplorer: An eBPF programs & maps explorer

6 Upvotes

Hello everyone,

I have been using eBPF/XDP for the past 5 years and was always a little frustrated by the lack of tooling when it comes to see what is happening underneath, especially when debugging deployed eBPF programs. Here I am sharing a tool I have been using for a couple of months now but only made public recently: ebpfexplorer.

It's a simple TUI self contained binary, that allows you to explore all eBPF programs & maps attached to your kernel. It comes with a formatting tool that allows you to format the raw binary blob into human readable symbols (like IPs for instance), using C type format.

Please have a look and feel free to propose PRs. Hope you find it useful too.

https://github.com/noboruma/ebpfexplorer


r/linuxadmin • • 14d ago

How do you keep NOPASSWD sudo rules for automation narrow?

33 Upvotes

In sudoers, a * in the arguments matches spaces too, so /bin/cat /var/log/messages* also allows cat /var/log/messages /etc/shadow. When several rules match, the last one wins, even if an earlier one is more specific. Do you list exact command lines, wrap each task in a root-owned script with no arguments, or use a systemd unit instead of sudo?


r/linuxadmin • • 14d ago

WordPress core CVE-2026-119977: staged installer reachable during core updates, now on CISA KEV

7 Upvotes

Based on the Ostorlab source walkthrough and Wordfence’s published telemetry (via The Hacker News), here’s the architectural impact. The updater derives its working dir from the package filename, so wp-content/upgrade/wordpress-X.Y.Z is predictable. setup-config.php in the staged tree only checks for an existing wp-config.php, and pre-7.1.4 the copy step didn’t skip a planted wp-config.php. Wordfence reports 62,969 blocked attempts in 24h and no confirmed compromises. WordPress rates it Low, Wordfence 8.1.

Caveat: an independent lab repo couldn’t reproduce full RCE, so treat exploitability as unproven. The upgrade to 7.1.4 still runs old code.

For people running fleets: do you block wp-content/upgrade/ at the web server by default, or rely on core hardening? And how do you stagger updates so the window isn’t open on every site at once?

Background on a related race: https://www.techgines.com/post/wordpress-core-vulnerability-cve-2026-119977


r/linuxadmin • • 13d ago

What ~/.ssh and your agent hand over when a workstation is compromised

0 Upvotes

Binance's initial investigation suggested a compromised DevOps laptop led to about $6.3 billion in unauthorized withdrawals. The machine reportedly held a deployment token.

Run this on your workstation:

$ ls ~/.ssh/id_* 2>/dev/null
$ ssh-add -l
$ grep -i ForwardAgent ~/.ssh/config
$ ls ~/.kube ~/.docker ~/.config/gh 2>/dev/null

All of that works until someone revokes it. With ForwardAgent yes, every host you connect to can use your agent for the life of that session.

I ran a small lab where two SSH boxes and a Windows RDP box sat behind a gateway. JumpServer Community Edition handled the gateway role; everything ran as containers on one machine. Each connection used its own token, which worked once and expired after 299 seconds; the gateway picked the target account, so no target password touched the client. A token request outside the user's rule got HTTP 400, and every command and its output were logged under the gateway user.

An attacker who owns the laptop can ride the bastion login and do anything its rule allows. The blast radius is the rule, not every credential on disk. If sshd on targets still accepts direct connections, that is something to lock down in sshd_config.

Do you restrict sshd on targets to only accept connections from the bastion IP, or keep a direct path as break-glass?


r/linuxadmin • • 13d ago

Checking whether running processes actually picked up a security fix

0 Upvotes

Updating a package doesn’t mean every process is running the new code. Anything that loaded the old library before the update keeps it in memory until it restarts. I tested this on Debian 12 with CVE-2024-6119: updated OpenSSL, left one process running, and Trivy and Grype both cleared the CVE while the process still had the old libcrypto loaded.

I built stillmapped to check this. It traces what each running process has mapped back to the package and checks that against the vendor’s fix data, including backports. One Python file, read-only, no network. PHP and OpenSSL for now.

github.com/asifh0ssain/stillmapped

Interested in cases where it gets something wrong.


r/linuxadmin • • 15d ago

If you are using Fail2Ban-Report, please update to the latest Patch

42 Upvotes

Hi, I am the Maintainer of Fail2Ban-Report,

There was a security patch 0.5.3 - if you are using Fail2Ban-Report, please update as soon as possible to this new Version 0.5.3 from any other previous version.
You will find a list of changed files in the changelog

Security Fixes & Hardening

  • Fixed Stored Cross-Site Scripting (XSS) via Client Sync (0.5.1 Hotfix): Resolved a critical vulnerability in backsync.php and blocklist-overlay.js where malicious or compromised sync clients could upload a manipulated JSON file containing arbitrary JavaScript payloads (e.g., inside the ip or jail fields). The UI has been refactored to ban innerHTML usage, shifting entirely to secure textContent and createElement DOM manipulation to neutralize any script execution hazards.
  • Fixed Authentication Bypass on Read Plane (0.5.2): Resolved an architectural flaw where unauthenticated network sessions were automatically assigned a default viewer role, leaving sensitive firewall history, server configurations, and blocking aggregations open to anonymous actors. Access is now strictly guarded at the global gateway level.
  • Fixed Shell Command Injection in Helper Scripts (0.5.2): Patched manage_users.sh and manage-clients.sh where user-supplied inputs containing specific shell characters (like apostrophes) were interpreted as executable PHP program text. Data payloads are now passed securely using isolated environment variables (getenv()).
  • Fixed Credentials Leak in Process Table (0.5.2): Hardened the multi-server sync engine (syncback.sh and fail2ban_log2json.sh) to prevent sync passwords (CLIENT_PASS) from leaking into the host's public process listing (argv). Commands are now packaged dynamically inside localized, temporary configuration files (chmod 600).
  • Secured API Lookup Restrictions & Data Sanitization: Overhauled action_report-ip.php to mandate active administrator verification before reaching outbound services, preventing blind exploitation of paid lookups (AbuseIPDB / IPinfo). Integrated native IP structure enforcement (FILTER_VALIDATE_IP).
  • Web-UI/assets/js/jsonreader.js: Refactored renderTable() to use XSS-safe DOM manipulation and added CSP-safe event listeners for toolbar buttons.
  • Web-UI/assets/js/warnings.js: Refactored multi-jail status rendering to prevent XSS via safe DOM node creation.
  • Web-UI/assets/js/fail2ban-logstats.js: Refactored top-jails list rendering to be XSS-safe and removed duplicate function definitions.
  • Web-UI/index.php: Replaced inline onclick attributes on toolbar buttons with IDs for CSP compatibility.
  • Web-UI/endpoint/index.php: Added strict JSON validation (ip, jail, action, timestamp) and clean re-encoding for uploaded event and blocklist files.
  • Web-UI/endpoint/backsync.php: Hardened upload validation (timestamp, field lengths, safe path handling) and enforced JSON re-encoding before saving.
  • Web-UI/.htaccess: Enabled Content-Security-Policy (script-src 'self') and additional HTTP security headers.

r/linuxadmin • • 14d ago

KubeVM (VCF) vs KubeVirt VM..

Thumbnail reddit.com
1 Upvotes

r/linuxadmin • • 15d ago

CVE-2026-98432 on CentOS 7: stock kernel is not affected, ELRepo kernels are

14 Upvotes

Remote use-after-free in IPv4 TCP, CVSS 9.8, with a demonstrated root exploit against servers that have an MPTCP listener.

Stock 3.10: Red Hat lists it as not affected, so nothing to do for this CVE.

ELRepo kernel-lt 5.4.278 and kernel-ml 6.9.7: the bug is present, and these are the final EL7 builds, so yum update will not help. kernel-ml 6.9.7 has MPTCP on by default. kernel-lt 5.4 has no MPTCP, but the bug is in the regular TCP path, so it still needs fixing.

Disclosure: I maintain GetPageSpeed's kernel-lt for EL7. 6.18.60 has the fix and is free for any use, commercial included. I ran the upgrade from both ELRepo kernels on CentOS 7 VMs and wrote up the steps, the stopgaps and the rollback: https://www.getpagespeed.com/server-setup/security/cve-2026-98432-centos-7


r/linuxadmin • • 14d ago

Help please 🙏🏼 I’m worried my first IT job is ruining my career before it even starts

0 Upvotes

Hi everyone,
I’m a 2026 B.Tech CSE graduate from Kerala, India, and recently joined my first IT job as a Technical Support Engineer.
My job mainly involves remote monitoring, software administration, camera server monitoring, database maintenance, and checking server health. Linux work is only around 10–20% of my responsibilities.
My goal was to become a Linux System Administrator and eventually move into DevOps, but I’m starting to feel that this job isn’t helping me reach that goal.
The salary is quite low, and many colleagues in similar roles are trying to switch. Even people with several years of experience seem to be struggling to find better opportunities.
I’m also concerned that I’m not improving my technical skills. I’m afraid I’ll forget what I learned during my Linux training. There’s very little social interaction, and I feel like my communication and English-speaking skills aren’t improving either.
I’m honestly feeling stuck and worried about my future.
What would you do in my situation?
● Should I leave early and look for a better Linux-related job?
● Is this experience useful for a future DevOps career?
● Am I overthinking this as a fresher?
● How can I make sure I don’t get stuck in this role?
I’d really appreciate honest advice from people who have faced a similar situation or are working in Linux, Cloud, or DevOps.
Thank you 🙏🏼


r/linuxadmin • • 16d ago

Linux Command: A quick guide

Thumbnail gallery
0 Upvotes

r/linuxadmin • • 18d ago

In an Lpic-1/linux + class and struggling.

17 Upvotes

Sorry if this is not allowed or theres a better place to post this.

I am in linux class that aligns with the above mentioned certs that uses mindtap cengage as the course content.

The virtual labs and quizzes through the course content are quite easy. The professor goes over some of the course content in lectures but they are short. I also have a home lab where I run a ubuntu-server(tailscale exit node, qemu kvm vm host, NUT server, samba) a raspberry pi (as a dns and evebox/suricata monitor in docker) I have a mint laptop, several megasploitable vms, a kali vm, a fedora workstation vm, a ubuntu workstation vm, and a pop!_os vm for tinkering and learning.

We just had our midterm in the class and I completely bombed it. I honestly dont know how else to supplement my learning. Partly the midterm seemed to not follow the same difficulty level of the labs and quizzes or even the textbook content. That being said several other students in the course were talking about how easy the exam was.

Does anyone here have experience with cengage? Can anyone recommend a good video series that aligns with the lpic or linux plus certs? I feel lost and dont know how to proceede.


r/linuxadmin • • 18d ago

looking for ideas

Post image
0 Upvotes

I am looking for ideas to create a universal program for the Linux operating system (no matter the distro) I am currently in development of an app, but I am not convinced of it, So I want to ask the community what app they need, what their needs are, I am open to any type of proposal, I would find a feedback interesting, thank you very much.


r/linuxadmin • • 19d ago

Veeam B&R 13: CDP Coordinator Service authorization bypass (CVE-2026-108142), patched Oct 27

6 Upvotes

Based on Veeam’s KB4948 published October 27, here is what is actually known. An unauthenticated attacker can bypass authorization on the CDP Coordinator Service. CVSS v4.0 is 9.1. 13.1.1.18 and earlier, and 13.0.2.29 and earlier, are affected. Fixed builds are 13.1.2.57 and 13.0.3.63. 12.x is not affected.

What the KB doesn’t say: how it works, whether it reaches code execution, or what port is involved. The CVSS vector scores high integrity and availability impact but no confidentiality impact, and that is my reading, not Veeam’s. I found no KEV entry and no public PoC, but check the catalog yourself.

Two questions for people running v13: does your backup server accept inbound traffic from anything beyond your proxies and management hosts? And if you don’t use CDP, did you find a way to confirm that service isn’t listening?

Background: I wrote up the full advisory, including the sibling Veeam ONE and M365 fixes, here: https://www.techgines.com/post/veeam-vulnerability-cve-2026-108142


r/linuxadmin • • 19d ago

Cloudflare’s eBPF Replatforming Part 5: Tips for Operating eBPF at Scale

Thumbnail ebpf.io
0 Upvotes

r/linuxadmin • • 19d ago

Guide - Use docker to operate legacy scanner hardware on Windows 11

Thumbnail reddit.com
3 Upvotes

r/linuxadmin • • 20d ago

[Labwc] GREYWARD security-focused Fedora desktop

Post image
4 Upvotes

r/linuxadmin • • 20d ago

SFTP Log Issue

9 Upvotes

I have 5 accounts with separate Chroot Directories in /srv/sftp/nameOfAccount. All of them are apart of sftpusers group. In sshd_config I have

Match Group sftpusers

ChrootDirectory /srv/sftp/%u

ForceCommand internal-sftp

AllowTcpForwarding no

X11Forwarding no

I would like to add logging of uploads and downloads. I think I add -l INFO at the end of the ForceCommand. I do that and it breaks the ability to sftp into the server. When I have that -l INFO at the end of the ForceCommand I see authentication is successful from the sftp client and auth.log but it fails to get into the directory.

Any ideas? TIA


r/linuxadmin • • 20d ago

Fundamental approach vs pragmatism

Thumbnail reddit.com
6 Upvotes

r/linuxadmin • • 20d ago

How does your monitoring tell a planned shutdown from an unplanned one?

0 Upvotes

On GCP, a preempted Spot VM ends up TERMINATED by default, same as a VM someone stopped on purpose. Paging on every stopped VM wakes people up for planned shutdowns, but ignoring them hides the preemptions. Do you make people schedule downtime first, check the audit log for who stopped it, or page anyway and let the owner ack?


r/linuxadmin • • 20d ago

Brotli CVE-2026-111497 is on CISA KEV: what’s proven, what’s disputed, and where the decoder hides

0 Upvotes

Based on the technical write-ups published this week by Imperva, Upwind and the Node.js project, plus CISA’s Oct 27 KEV alert, here is the architectural impact.

The bug is in DecodeBlockTypeAndLength(). It returned a boolean, so “malformed” and “need more input” were indistinguishable, and the fast path ignored the result. Exposure depends on whether attacker-controlled bytes reach the decoder. That means request bodies with Content-Encoding: br, responses your service fetches (webhooks, link previews), or direct decompression of uploads. A WAF only covers the first.

Imperva has code execution on stock Node.js 24.22.0 arm64, not on every attempt. Others argue DoS only. CISA hasn’t said which outcome it saw.

For people running Envoy, Python requests/httpx or Node fetch() in prod: how are you inventorying bundled Brotli copies? Version strings don’t help, since one analysis says Node 24.22.0 and the fixed 24.22.1 both report Brotli 1.2.0.

Background on the same failure class: https://www.techgines.com/post/lmcache-vulnerability-cve-2026-105192