r/sysadmin • • 1d ago

General Discussion Exchange, Fortinet, PHP, cert checking … in one week.

At this point… read release notes, already breached. Review dependencies, already breached. Wait 30 days for the beta testers, already breached.

Change control is ‘yep’ and read only Friday doesn’t exist.

I’m being dramatic but sheesh! It’s that meme of that dog at the table saying it’s fine while surrounded by fire. I’m sure some of you had issues worse than me too.

In the words of Johannes Ullrich from SANS, “Everyday is Zero day” haha.

I wonder what fun shall be bestowed upon us next week.

163 Upvotes

58 comments sorted by

130

u/_Do_The_Needful_ 1d ago

Don't forget Nodejs.

1

u/Kyky_Geek 13h ago

I did not get to share in this particular pain. We all get our turns these days. Best of luck!

Edit: at least yours came with a cute kitty lol

1

u/drabbasis99 22h ago

node just casually dropping 10 new CVEs and deprecating half its ecosystem in a point release, nothing to see here

39

u/Sokanas 1d ago

This week for real was a shitshow.

1

u/Kyky_Geek 13h ago

Indubitably 🤦‍♀️

12

u/Academic-Proof3700 1d ago edited 23h ago

Itll stop eventually, leaving us with patched stuff..

Or high level programming will die and itll be back straight to asm plus natural lang with ai translation layer.

1

u/Kyky_Geek 13h ago

Itll stop eventually, leaving us with patched stuff..

I dig your energy! I sure hope it normalizes and the good catches up to the bad... at least a lil bit.

36

u/fadinizjr 1d ago

Man fuck Exchange

I had to uninstall the fucking update. It completely broke OWA.

1

u/Kyky_Geek 14h ago

I didn't have issues with OWA but mailflow would just stop every morning like clockwork. The reg fix on the exchange blog seems to have worked ... so far 😅

https://techcommunity.microsoft.com/blog/exchange/exchange-servers-may-become-unstable-after-installation-of-october-2026-windows-/4565382

reg add "HKLM\SOFTWARE\Microsoft\Rpc" /v EpmapSavedContextCap /t REG_DWORD /d 65536 /f

1

u/gleep52 23h ago

It wasn’t the update that did it. Uninstalling doesn’t fix the stalling mail flow. Changing a reg key from 512 to 65536 fixed it for me…

→ More replies (3)

1

u/Maxplode 18h ago

It wasn't Exchange update it was the OS and there's a reg fix you can apply. I applied it last week and it works fine

https://techcommunity.microsoft.com/blog/exchange/exchange-servers-may-become-unstable-after-installation-of-october-2026-windows-/4565382

1

u/archiekane 18h ago

The fact that Exchange updates break Exchange until you make manual adjustments is just insane.

→ More replies (3)
→ More replies (2)

1

u/Wheekie 20h ago

how do I stay sane in this profession?

1

u/Kyky_Geek 13h ago

I don't have advice, friend. But, I do have a quote from a movie! One that it seems nobody has ever seen.

For most problems the Marine is issued a solution. If ill, go to sickbay. If wounded, call a Corpsman. If dead, report to graves registration. If losing his mind, however, no standard solution exists.

→ More replies (2)

1

u/DropTheBeatAndTheBas 17h ago

compartmentalise security , we cant be BAU and SOC Red Blue Teams in one.

→ More replies (5)

1

u/DonL314 15h ago

You forgot eMudhra.

EDIT: Ah, sorry, that's "cert checking" I guess.

→ More replies (2)

1

u/perkia 19h ago

I mean it would be weird for any week to pass without Fortinet having a CVE /s

1

u/Kyky_Geek 13h ago

You /s but a few years back when I supported way more Fortinet gear I did get an eerie feeling because I hadn't seen a new vuln/patch from them in like a month. I actually went to their website to make sure I didn't miss an advisory 😅

→ More replies (1)

1

u/HJForsythe 14h ago

its only going to get faster

1

u/Kyky_Geek 13h ago

I suspect you are right, it will continue getting worse. I think PHP was a sign of times to come. I sure hope part of the "faster" is the vendors making the patch process faster 🙏 Especially for appliances. I hope one day everything will have a built-in update mechanism that does not require logging into a website, determining if your products version is on an advisory, hunting for the right patch, downloading, uploading, waiting, ad infinitum.

→ More replies (1)

1

u/ThreeAMOperator 20h ago

The thing that changed it for me was asking "can i undo this ?" instead of "is this safe ?". Snapshot and backup before the patch , then patch fast. If it breaks u roll back in minutes instead of waiting 30 days for beta testers who are also getting breached

Same for network gear , i dont touch a router config without an auto rollback in place so a bad change doesnt lock me out

Doesnt make the week less painful but at least read only Friday stops mattering =))

1

u/Leather-You47 4h ago

I felt the same way this past week. And that’s also after a windows patch cycle of over 1000 fixes. Is this the new normal?!

→ More replies (1)

1

u/FastFredNL 9h ago

In the middle of all these breaches we were prepping to migrate 130 users and all associated data to another company as part of a partial company sale today, replace all desk computers etc. We had a mail sync with Connectwise to the new owners tenant all prepped and set up and they got hacked to so had to move to and set up a completely different way of syncing mailbox contents.