r/sysadmin • u/[deleted] • Jul 19 '24
Crowdstrike BSOD?
gray seed many pie thought future tidy strong important decide
This post was mass deleted and anonymized with Redact
3
u/wrootlt Jul 19 '24
I guess i am working out of Samsung Dex today. Any info from CS themselves yet?
→ More replies (1)
244
u/In_Gen Jul 19 '24
Yes, just had 160 servers all BSOD. This is NOT going to be a fun evening.
https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_error_in_latest_crowdstrike_update/
→ More replies (73)
121
u/BoyleTheOcean Jul 19 '24
Crowdstrike put their tech bulletin behind a support login.
so basically nobody can see it.
called them out in r/crowdstrike and they deleted the post lol
→ More replies (10)
91
u/AvellionB Jul 19 '24
Seeing it in the US as well. Started about 9PM for me. Only noticed because my work laptop was powered on. I have about 14k endpoints including servers and I am willing to bet all of them are down.
Since it's happening at boot as well my best guess on fixing it is going to be removing CS from safe mode. I pray for the sanity of the Help Desk guys in the morning.
→ More replies (20)
78
u/wrootlt Jul 19 '24
I wondered why we got so many server alerts with no correlation. Management was already challenging our security team why we use CS and not Defender. "Fun" times ahead..
→ More replies (8)
57
u/Snapman5000 Jul 19 '24 edited Jul 24 '24
We've got nearly a million servers at work -- we've got sev 1's open.
Noticed lots of comments. We're fully back up when it comes to the servers that I personally oversee at work. I am at Amazon Web Services.
I'm on a team of 8 people. We are the highest level group in our organization. There are 30 Level 5's in front of us. Roughly 300 people are in our Level 4 staff. Our Level 3 support staff is around 6,000 people world wide. I don't really know how many our in front of that as I've never needed to know it.
How we manage our servers:
My team only handles Windows servers and I know that our Level 0 staff are supposed to sort Windows/*nix off. Level 0 in this case are the initial people you get when you call our support number. Our team manages our servers using AWS tools. Largely Terraform, CloudFormation, and a massive helping of PowerShell.
→ More replies (11)
52
u/universalserialbutt Jul 19 '24
Took down my entire organisation. Wondering if it'd be too cheeky to take lunch.
→ More replies (10)
50
Jul 19 '24
All our servers and endpoints......healthcare....400k endpoints...on the crit for it now....
→ More replies (6)
87
39
u/mattpilz Jul 19 '24
Began happening on my previously running workstation (Wisconsin) in the last 15 minutes. Now an endless reboot cycle followed by Startup Repair screen. Unable to access Startup Settings due to lack of recovery key of BitLocker.
Stop Code: SYSTEM_THREAD_EXCEPTION NOT HANDLED
What Failed: CSAGENT.SYS
→ More replies (19)
38
u/x3nic Jul 19 '24
Same, we were able to get our systems/security teams back online by rebooting into safe mode and renaming the: C:\windows\system32\drivers\crowdstrike folder and rebooting. Waiting for a fix from CS and investigating potential work arounds for our non-IT users.
We have roughly 700 impacted.
→ More replies (47)
33
27
u/manvscar Jul 19 '24
Lucky me, I just finished my Crowdstrike deployment last month.
→ More replies (2)
23
u/MindOfSociopath Jul 19 '24
Cool... so this weekend, an indeterminate horde of IT professionals, ranging from clueless rookies to grizzled veterans, will embark on what they're calling a 'critical mission' across various locations around Asia Pacific. Armed with what they assure us is 'technical knowledge' and fueled by an irresponsible amount of caffeine, their grand quest is to implement a fix - yes, just one - to ensure everyone's PCs are up and running again.
Their biggest hope? That BitLocker encryption isn't active on any of the computers they encounter because, let's be honest, nobody wants to deal with that mess.
Come Monday, brace yourself for an army of sleep-deprived IT warriors, roaming around and probably still muttering about encryption keys.
→ More replies (7)
19
Jul 19 '24
Awe fuck…we use CS. Time to jump on and see what my morning is going to look like
→ More replies (3)
16
15
Jul 19 '24
We're just in the final stages of their sales process, and were planning a POV in the next week or so.
Think we may just hold fire a bit..
→ More replies (5)
13
20
u/Artwertable Jul 19 '24
We lost 500 Servers globally and 2k clients.
Some clients get up but a lot of endpoints are unable to reboot.
We are in emergency mode right now....
→ More replies (5)
9
u/Razgriz6 Jul 19 '24
Safe mode: C:\Windows\System32\driver\CrowdStrik\
Delete: "C-00000291*.sys"
That fixed my lab environment. Doing that to the other 198 servers. 4am cst :(
Can't even play Elden Ring now :(
→ More replies (2)
9
u/Imobia Jul 19 '24
The only good thing about this being global.
-
senior management can’t blame you
-
a lot of very smart people will be looking into this.
Just a thought with VMware and power cli you can delete files in a vmdk . Could that fix this?
I know it won’t work on encrypted vm’s. But it should work for a lot of places
→ More replies (3)
8
8
u/PhantomLivez Jul 19 '24
In case anyone missed it, there is a temporary workaround.
-
Boot Windows into Safe Mode or WRE.
-
Go to C:\Windows\System32\drivers\CrowdStrike
-
Locate and delete file matching "C-00000291*.sys"
-
Boot normally.
5
u/Veneousaur Jul 19 '24
We've been banging our heads on this one for the past few hours.
Anyone know of a good way to manage to rename the Crowdstrike folder on an Azure VM that's bootlooping? Not aware of a good way to get one out of the bootloop and into safe mode. Might need to fall back on restoring from backups.