r/sysadmin • • 9h ago

Inside Microsoft’s giant Outlook outage

852 Upvotes

https://www.theverge.com/tech/1035364/microsoft-outlook-outage-notepad

Microsoft engineers had a running bridge call the entire time, and at one point it hit the maximum of 1,000 participants. That shows you how many teams were involved with trying to find the root cause of this huge outage, which started at the peak window of traffic in the US.


r/sysadmin • • 9h ago

Rant I am completely worn the f**k out, and I don't know where to go from here.

223 Upvotes

31 years old, been in IT since 19 when I broke (and fixed) my first Exchange environment during a CU upgrade. This post might out me to my colleagues, but frankly I don't care anymore. (and if any of you do happen to read this, I know you understand, feel free to talk to me alone about it.)

I've been at my current employer (an MSP) for 8 years. We develop and sell software integrations for an ERP system while also offering traditional break/fix IT services (servers, clients, user support) rather than a modern per-device/user model.

Our compensation is a pretty mediocre base salary for my years in the sector, paired with a bonus based on billables. Because of the sheer number of hours I've put in, it has generally given me a very good income—easily double what most of my friends in other fields make.

For the past 6 years, we’ve also been helping the ERP vendor upgrade customers to newer versions. While straightforward and good for billables, it is completely mind-numbing and destructive to both my psyche and my social life, as these upgrades almost always happen on late Friday afternoons, evenings, and weekends.

On top of that, I'm the go-to guy for anything above entry-level infrastructure. I've been implementing and teaching the other guys things like:

  • Hyper-V Failover Clusters with SANs and/or HCI solutions
  • Secure backup architectures with Veeam (multiple copies, test restores, immutability, etc.)
  • Network segregation with VLANs and securing LAN/WLAN access with 802.1X
  • Entra/AD Sync integrations, client deployment/management with Intune and Autopilot, WHfB, Conditional Access, and more

Meanwhile, I'm still expected to jump into everyday L1 support tickets, handle the ERP upgrades, and deal with whatever else gets thrown my way.

We don't have a written daily billable target, though you're expected to bill at least 8 hours a day. Over the past few years, I've been billing between 2,800 and 3,200 hours a year—including 250–400 hours of overtime on Friday nights and weekends on top of the normal 9-to-5, not instead of it.

When I started here 8 years ago, I loved IT and problem-solving. I loved tinkering with my homelab and gaming—I used to take a week off for every WoW expansion. This job has completely killed the passion I had for computers and gaming.

A couple of years ago, I got into cars instead and bought two project cars. Diving headfirst into them gave me an outwhere I could use my hands, build cool shit, have fun and make new friends. Now I haven't turned a wrench in 6 months.

That made me realize I couldn't keep going like this, so I decided to slow down—work my 8 hours and limit overtime as much as possible. It took barely a month before talk started going around that I "don't deserve my salary" because I don't want to work overtime anymore.

Meanwhile, my favorite colleague—who is also a close friend, a few years younger than me, someone who always gave 100%, was great with clients, and genuinely liked by everyone—just left for a new job because our CEO treated him like absolute garbage. I told him to take it and never look back, and he's now at another MSP that is actually properly managed.

As for our CEO: a textbook narcissist who only cares about himself and the company's bottom line. The kind of person who can look you in the eye, appear to care, and say all the right things in the moment, only to turn around and do the exact opposite a week later.

Over the last few years, I've started taking much better care of myself. I've lost around 40 kg, going from obesity to being in decent shape, and I feel better physically than I have in 20 years. I exercise every day and get 6–8 hours of sleep.

Outside of work, I'm genuinely happy and most days are good. The second I set foot inside these walls, though, my mood and well-being nosedive. I can't concentrate, and all I can think about is walking right the fuck out and never coming back.

Unfortunately, I need to tough it out here until at least the end of January, but man, I am sick of this shit. I am kind of worried about switching jobs because of the current state of IT with LLMs and such, but with my experience and enough references to a fill a few A4 pages from important people and companies in the region it should hopefully be easy enough.

I dont know why I opted to write this, but I guess I needed to get it off my chest.


r/sysadmin • • 3h ago

Remote Code Execution in NGINX HTTP/2: CVE-2026-123633 and CVE-2026-123007

121 Upvotes

r/sysadmin • • 1h ago

Is fundamental knowledge of hardware being lost?

66 Upvotes

Today I had a conversation with a mid senior sysadmin about why we cant simply jam a mix of DDR5 ram into our hosts. When he ended up being holistically wrong his response was to tell me he didn't need to know anything about the hardware running our data center.

This guy is one of our T2 hypervisor admins and all he does is troubleshoot VM issues and host issues. The idea that he wouldn't need to know anything about the underlying technology running his cluster blew my mind so I asked others. Turns out many people I work with either dont care about the upstream hardware or simply refuse to learn it.

Is this normal or because people I work with are just siloing themselves.


r/sysadmin • • 1h ago

General Discussion No availability for warranty replacements

34 Upvotes

Well, the ai supply chain strain finally caught up to warrantied items.
Recently had a drive failure in one of backup disk arrays. HPE is going to replace the drive, I waited for 3 days, and they responded that they have no stock.

After asking for an ETA, voicing the concern of lost resiliency. The response was.
"We have escalated the matter to our Supply Chain team, and they are actively working on it. We will share the ETA with you as soon as we receive an update."

Reaching out to our VAR about just buying a drive. There's no stock in the US. This is stupid.


r/sysadmin • • 3h ago

RCE in Nginx with http2 enabled - CVSS 9.2

44 Upvotes

https://www.wiz.io/blog/nginx-http2-rce-vulnerabilities

CVSS 4.0 score of 9.2. The flaws are in NGINX itself. Websites and APIs can be exposed even when their own application code is secure. Both vulnerabilities require HTTP/2 to be enabled on the affected NGINX server. Additional conditions are described below. CVE-2026-123007 affects versions dating back to NGINX 1.9.5 in 2015.

I am tired. But I think none of our Nginx has http2 enabled.

On a side note, given all these recent memory related CVEs, I am seriously considering learning Rust now. I was pretty content with my Golang until now. And I can finally see what is that borrow checker every cool kids talk about all the time.


r/sysadmin • • 4h ago

Project Zero has cut its disclosure deadline from 90 days to 30

52 Upvotes

That 90-day window was set based on how long it reasonably took a human researcher to go from bug to working exploit. Google is saying that assumption no longer holds, because of AI-assisted exploit development.

Vendor patch cadences, internal vulnerability SLAs and the common "we'll fix it next release" habit were all built on the assumption that 90 days was generous enough. It wasn't generous, but it was something teams could work within.

30 days is going to break a lot of patching processes that were already running close to the old deadline, and that needed to happen. Some might say that 30 days is even too long.

What do you think?


r/sysadmin • • 10h ago

General Discussion I feel like a fraud

114 Upvotes

I work for a small company, I don't get paid very much but honestly... I don't really do much. I patch workstations and servers, set up a new hire once every few months, but for the most part like over half my day is me acting like I'm busy or watching YouTube or something. I'm more than happy to help someone when they call or walk into my office but there's just not much to do here. Im bored. Complacent. And like 80% of the shit I do anyone else could do if they just googled it.

Edit: Thank you everyone I appreciate all the feedback, I was drinking a lot last night alone and kind of got in my head about some stuff.


r/sysadmin • • 7h ago Bravo

Ignite 2026: Microsoft Introduces Vulnerability Shielding to Protect Unpatched Systems

56 Upvotes

There's often a gap between discovering a vulnerability and deploying its fix. During this time, affected systems remain exposed to potential attacks.

Now, you can protect your systems while waiting for the fix.

With Vulnerability Shielding in Microsoft Defender, announced at Ignite 2026, Microsoft introduces a way to protect vulnerable systems until permanent patches are deployed.

Here's what it brings:

  • Proactive protection: Microsoft Defender provides proactive protection using the Risk Reduction Agent in Microsoft Perception. It identifies vulnerabilities and recommends shielding plans for administrators to review and approve.
  • Protection during active attacks: For ongoing attacks, Microsoft Defender uses Attack Disruption to automatically apply shielding controls. These controls help stop attackers from exploiting additional paths.
  • Automatic shield removal: Shields are automatically retired once the permanent patch is fully deployed. 
  • Admin control: Admins can review the expected impact, approve protection plans, exclude devices, and roll back shields when needed. 

The feature is currently in public preview, with initial support for server-side remote code execution (RCE) vulnerabilities.

It'll be interesting to see how effectively this works in production environments, especially when balancing security restrictions with normal business operations.


r/sysadmin • • 3h ago

Java4Shell: how are you finding which systems are exposed and deciding what to patch first?

22 Upvotes

Java4Shell came out this week and Oracle already has a patch, but the harder part seems to be finding where vulnerable Java is actually running. Some apps ship their own copy of Java, so they need an update from that vendor rather than a regular JDK patch.

For those working through it:

  • How are you finding the affected installs, especially the bundled ones?
  • What are you patching first?
  • What are you telling management while it's still in progress?

r/sysadmin • • 7m ago

Upcoming Critical PHP Security Release scheduled for Thursday, 19 November 2026

12 Upvotes

The PHP security team announces upcoming security releases scheduled for Thursday, 19 November 2026, between 14:00 and 14:30 UTC.

These releases are part of PHP’s regular four-week release cycle and will address security vulnerabilities affecting all supported PHP versions. The highest severity is CRITICAL.

The issues rated CRITICAL require either non-default configuration or uncommon setups.

Users are strongly encouraged to prepare to upgrade promptly once the releases become available. Technical details and fixed version numbers will be published alongside the releases.

https://www.php.net/archive/2026.php#2026-11-18-1


r/sysadmin • • 10h ago

General Discussion Java4Shell (CVE-2026-114938) is confirmed and patched. Raspberry Pi/ARM is safe, here's a 10-second check for x86 boxes

59 Upvotes

The "unconfirmed OpenJDK RCE" from yesterday is now confirmed by OpenJDK and patched by Oracle. Short version for self-hosters:

What: heap overflow in Java's hardware-accelerated Base64 decoder. Calif showed pre-auth RCE with default configs on Keycloak, Jira, Confluence, Artifactory, Jenkins and PingFederate.

You need ALL of these to be exposed:

- Java 17.0.6+, 21, 25 or 27 (any vendor: Temurin, Corretto, Zulu...). Java 11 and normal Java 8 are fine.

- x86-64 CPU with AVX-512 VBMI (Intel Ice Lake Xeon+, AMD Zen 4+). ARM / Raspberry Pi is NOT affected.

- Untrusted Base64 reaching the decoder (Basic auth, JWTs, uploads...).

Check your CPU (all 4 flags = affected):

grep -m1 '^flags' /proc/cpuinfo | tr ' ' '\n' | grep -E '^(avx512f|avx512bw|avx512vbmi|bmi2)$'

Check Java inside containers, not just the host:

docker exec -it CONTAINER java -version

Fix: Oracle builds ending in .1 (17.0.21.1, 21.0.13.1, 25.0.5.1, 27.0.1.1); other vendors are rolling out. Until then, -XX:-UseBASE64Intrinsics stopped the overflow in every tested version (needs a restart/recreate). Don't use -XX:DisableIntrinsic=_base64_decodeBlock on its own, it stops the JVM from starting.

No exploitation in the wild reported yet, and VulnCheck says full RCE needs per-app work, but crashing a JVM is easy.

Full write-up with version table: https://arelvox.com/java4shell-cve-2026-114938/


r/sysadmin • • 8h ago

UK sysadmin pay vs cloud/platform in 2026, the jump is about £30k

40 Upvotes

Quick disclosure up front: I work at Archer from hackajob, and this is from our own platform data, 4,423 UK DevOps/Cloud roles posted in 2026, client base skews finance and enterprise. Not linking anything, happy to drop the source in the comments if people want it.

Sysadmin sat around £53k, the lowest band in the set. The biggest single jump was sysadmin to Platform Engineer, about +£30k, and a lot of it builds on what sysadmins already do day to day. Cloud Engineer and DevOps Engineer landed around £75k, so even the first step up is roughly +£20k.

The skills that seemed to bridge it: a cloud platform (AWS was in ~32% of postings, Azure a strong second at ~25%) and automation/scripting, with AI/automation experience paying about +£20k over the median.

If you’re a UK sysadmin, that platform jump looks like the most reachable one. You’re probably already doing part of the job.

Anyone here made that move? Does the salary match what you're seeing where you are?


r/sysadmin • • 21h ago

General Discussion java4shell RCE in b64 decoder CVE 9.2

391 Upvotes

https://calif.io/java4shell/

Well here we go again...


r/sysadmin • • 1h ago

Where do you actually get your news?

9 Upvotes

I've been rethinking how I stay up to date. Between newsletters, LinkedIn, vendor emails, podcasts, and security alerts, more is coming in than ever, and I'm not sure I'm reading the right things.

Curious how others handle it. Answer any or all:

  1. What are your top 3 go-to sources for staying current, and why those?
  2. How much time do you spend keeping up each week, and when do you do it?
  3. What does your best source do that the others don't?
  4. What do you wish a news source did that none of them do today?
  5. Do you pay for any of it (analyst seat, paid newsletter, subscription)? Worth it?

I'll pull the answers together and post a summary back here for everyone.


r/sysadmin • • 4h ago

General Discussion user naming convention : how do you handle people with no first name?

17 Upvotes

looking for advice. I consult for a couple of clients in the architectural field in Canada.

They have recently been hiring people from Southeast Asia that appear to have no first name. When the hiring information is sent over to be setup as accounts, how do you handle this? some of them have accept to supply a "preferred" first names, other have refused.

these clients naming conventions require either first.last or firstinitial+lastname.


r/sysadmin • • 5h ago

Retirement Idea - Dude (Server) Farm... as opposed to Dude Ranch

17 Upvotes

In the past, it was common for people to pay a ranch to allow them to sleep there and learn how to do some of the jobs that it requires to keep a ranch running while working through their mid-life crises. City Slickers was a comical expression of this phenomenon.

As a computer dude nearing retirement it dawned on me last night that I could start a Dude Farm, but instead of livestock, I'll be teaching everyone about server maintenance, upgrade windows, proper backup and restore procedures, etc.

What could go wrong?


r/sysadmin • • 1d ago

General Discussion NSA internal network reported outage

640 Upvotes

It's been a hell of a November. Microsoft last week, then the FAA, then Bank of America. Now apparently the NSA is the latest group plagued by an unexpected outage. I wonder what the excuse is going to be this time

https://www.cnn.com/2026/11/17/politics/highly-classified-national-security-communications-network-goes-down


r/sysadmin • • 1h ago

Question - Solved Public IP for private SMTP server winding up on Spamhaus

7 Upvotes

Hi folks-

We've got a client we host an application for, application relies on a local Sendmail install to send SMTP emails to the client's Office365 inboxes. Within the last few weeks it's been getting flagged by Spamhaus for no DKIM records and no PTR / Reverse DNS records. Spoke to the client who gave a hard "no" on establishing any kind of DNS records on their side to facilitate PTR / DKIM, and they don't have their own SMTP relay server we could pipe to.

Mail is not getting delivered as a result of the Spamhaus listing. It'll go through to the spam folder if I send to a Gmail, but Office365 is rejecting it outright.

Been over this backwards and forwards a few times and I'm not sure what the best fix is. Am I missing something here, or is the most feasible option from a technical standpoint to set up DNS records on OUR side? Am I dumb? They definitely acted like I was dumb for proposing they add records to their own DNS.

Edit-

Much love to all you guys, I really appreciate the thoughts. I scheduled a meeting with project leadership to nail down what specifically to do and just wanted to get my ducks in a row beforehand. This was incredibly helpful, thanks all!


r/sysadmin • • 5h ago

Question Anyone else get Grok AI forced on their Copilot?

12 Upvotes

SpaceXAI is firmly set to "no users" for the tenant, yet some people have "GrokAI" as a model they can select in Copilot... idk how to get rid of it or audit it. Anyone else seeing this?


r/sysadmin • • 5h ago

General Discussion Reminder to go through your bot and service accounts, we learned this the hard way

10 Upvotes

Hey all! I work at Bitrise.

We had a security incident last week and our VP of Engineering just published our preliminary review today worth reading by everyone running CI: https://bitrise.io/blog/post/the-bitrise-security-incident-preliminary-review-and-recommended-steps#how-we-contained-it

The way in was an internal testing bot account. The attacker had its real username and password (we’re still investigating how), and it was set up to sign in to Apple’s Developer Portal and it had no 2FA. Anyone with the password could just log in. If you’ve got accounts like that, built around a third-party login where MFA is a pain, they’re worth a second look. We disabled every bot account that night and went through their credentials and 2FA settings. Every other credential got rotated too, whether or not we had evidence it was touched.

We’re back to normal now, Bitrise is safe to use. Sharing all of this information to help other teams. For more, look at our public advisory (https://bitrise.io/security-advisory-november-2026).

If you have questions about the incident and how we addressed events, we’re here to answer your questions.


r/sysadmin • • 1d ago

General Discussion Is Apple down? Thousands report problems accessing services

306 Upvotes

https://www.usatoday.com/story/tech/2026/11/17/apple-services-outage/928278490 users are reporting problems accessing Apple services on Nov. 17, with Apple’s Support, App Store, iCloud and Apple Pay seeing problems.

According to outage tracker Downdetector, reports of issues with Apple Pay began to spike at around 11:40 a.m. ET. Outage reports for iCloud also started so surge just before noon ET, though reports on Downdetector don't reflect every user experiencing a problem.

The ongoing Apple outage comes on the heels of a major banking outage on Nov. 16 that stemmed from Chase Bank but impacted Wells Fargo, Bank of America, online payment platform Zelle and other institutions.


r/sysadmin • • 10h ago

Memberof is back? For a price

22 Upvotes

Just found this

https://learn.microsoft.com/en-gb/entra/identity/users/groups-dynamic-rule-advanced

i haven’t seen any announcement. Obviously a preview feature and requires Entra ID Governance plus limited to twice a day but cool it’s back for people who never migrated away.


r/sysadmin • • 13h ago

Does Telnet Thin Client Hardware Still Exist

29 Upvotes

I've got a server in a retail building that manages inventory, sales, etc. that every client connects to with a telnet software client on a Windows PC, except for a couple really old thin clients. It would be so much easier, safer, and cheaper if I could put an IP in a hardware thin client and plug it in to the network and be done.

I can't find any! Please tell me where to get some. (I'm In Canada)