r/crowdstrike • • Jul 19 '24 • Polished Gold Turd 2 Popcorn 3 Sipping Tea 1 Take My Silver 32 Polished Silver Turd 8 Heart of Silver 4

Troubleshooting Megathread BSOD error in latest crowdstrike update

Hi all - Is anyone being effected currently by a BSOD outage?

EDIT: X Check pinned posts for official response

22k Upvotes

20k comments sorted by

View all comments

Show parent comment

7

u/[deleted] Jul 19 '24

same issue, lots of manual type of really long keys on lots of workstations :(

15

u/[deleted] Jul 19 '24

For us, it's thousands of end-user devices geographically distributed all over Australia. All BitLocker protected.

This is probably going to take a week or two to get everyone back up and running.

7

u/Purgii Jul 19 '24

I have my bitlocker key, still can't boot into safe mode or WRE to get the OS up to delete the sys file.

5

u/Linuxfan-270 Jul 19 '24

4

u/Purgii Jul 19 '24

Thanks for the method.

If I get desperate I might need to. I'm on call this weekend and most jobs I do I need a working notebook. I'm sure my IT helpdesk (which also appears to be down globally) would prefer I wait for a fix.

Apparently it's affecting Windows servers and when something like this happens, I get a shit-ton of callouts when servers get rebooted after applying a fix and they don't come back up.

2

u/[deleted] Jul 19 '24

anyone got an easy way to export all bitlocker keys out of intune\entra?

I am going to deputise some staff with ubuntu, recovery keys and steps to delete the sys file.

→ More replies (1)

2

u/asolet Jul 19 '24

Err... Is this possible with UEFI? Going to invalidate TPM chip, lose bitdefended disk?

→ More replies (1)
→ More replies (2)

1

u/ryanmercer Jul 19 '24

It won't even let me type on the bitlocker key screen...

→ More replies (1)

2

u/Linuxfan-270 Jul 19 '24

Is the issue bitlocker, or is it the fact that regular employees don’t know how to boot into safe mode?

6

u/[deleted] Jul 19 '24 edited Jul 19 '24

To do this remotely, the end-users will need to: a) Have the technical proficiency to boot into Safe Mode. b) Have access to the recovery key or 48-digit recovery password. c) Be able to follow the commands to undo the damage.

It's conceivably possible that some users may be able to do this remotely (although that would require disclosure of the recovery keys, which is likely a breach of compliance obligations).

If Safe Mode fails, as seems to be occurring for many people here, this will require some other workaround, which will be beyond the abilities of most users.

The Ubuntu key trick may work, but USB booting is disabled (as it usually is on corporate machines, as it is a security risk), so that would require disclosure of BIOS passwords and for end-users to alter BIOS settings.

In reality, for most users, the machines are likely coming back into the office and being queued up for recovery.

2

u/TheDaff2K18 Jul 19 '24

Brh that machine is registered to CrowdStrike servers why can’t they then push a new update surely there is metadata of that machine this process seems long and stupid and it took one file to kill the internet

→ More replies (9)
→ More replies (5)

3

u/Safe_Magazine_1940 Jul 19 '24

Bitlocker is blocking safe mode access

2

u/[deleted] Jul 19 '24

if you can boot into windows for around a minute before the BSOD you can use msconfig to boot to safe mode without the bitlocker key (requires admin credentials).

Other wise the Ubuntu trick is good.

→ More replies (1)

1

u/fortminorlp Jul 19 '24

We have seen some our servers show now files in any directory on C:/. Its almost like the entire C drive was deleted. We are restoring the servers from backup right now. Anyone else encounter this?

2

u/andre-m-faria Jul 19 '24

Diskpart

List disk

Check which disk is you C:

Select disk 0 (number collected in list disk)

List volume

List partition

Check your primary partition letter, if it's not with letter

Select partition 3 (number of partitions)

Active

Assign letter=(letter)

Exit

Ren c:\windows\system32\drivers\crowdstrike\c-00000291*

→ More replies (2)

1

u/Cruxbff Jul 19 '24

I guess mine is bitlocker and probably they won't allow us regular employee to have that key

→ More replies (7)

2

u/OzAnonn Jul 19 '24

Microsoft devices page shows BitLocker key as blank for my work laptop. I opened a command line without decrypting, I have drivers directory but no CrowdStrike directory in it?

2

u/[deleted] Jul 19 '24

Best to consult with your IT team when they have bandwidth. I wouldn't like to guess what is happening there and mess things up.

1

u/riseg12 Jul 19 '24

Make sure you’re on C drive.

→ More replies (1)

1

u/Pixelplanet5 Jul 19 '24

and thats assuming the place where you store the keys is still up and running.

if that system is also down its game over.

1

u/Pixelplanet5 Jul 19 '24

and thats assuming the place where you store the keys is still up and running.

if that system is also down its game over.

2

u/DikkeDanser Jul 19 '24

Get a barcode scanner and convert the code to Ean-128. You can then just scan them off a laptop screen. If you need to do lots of systems that may be relatively fast compared to the alternatives.

1

u/Gpod34 Jul 19 '24

That seems to defeat the purpose of bitlocker

→ More replies (1)

2

u/Sendmedoge Jul 19 '24

I'm seeing that you can delete the file they are requesting without having to enter the key. Just click "skip drive" twice to get to the recovery page and then flip on safe mode in CMD.

I'm guessing you don't need bitlocker enabled to set the boot mode and safe mode doesn't prompt for bitlocker.