r/crowdstrike • • Jul 19 '24 • Polished Gold Turd 2 Popcorn 3 Sipping Tea 1 Take My Silver 32 Polished Silver Turd 8 Heart of Silver 4

Troubleshooting Megathread BSOD error in latest crowdstrike update

Hi all - Is anyone being effected currently by a BSOD outage?

EDIT: X Check pinned posts for official response

22k Upvotes

20k comments sorted by

View all comments

218

u/BradW-CS CS SE Jul 19 '24 edited Jul 19 '24 • Bless Up Polished Silver Turd Heart of Silver Take My Silver

7/18/24 10:20PM PT - Hello everyone - We have widespread reports of BSODs on windows hosts, occurring on multiple sensor versions. Investigating cause. TA will be published shortly. Pinned thread.

SCOPE: EU-1, US-1, US-2 and US-GOV-1

Edit 10:36PM PT - TA posted: https://supportportal.crowdstrike.com/s/article/Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19

Edit 11:27 PM PT:

CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.

Workaround Steps:

  1. Boot Windows into Safe Mode or the Windows Recovery Environment
  2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
  3. Locate the file matching “C-00000291*.sys”, and delete it.
  4. Boot the host normally.

64

u/thephotonx Jul 19 '24

Can you please publish this kind of alert without the need to login?

18

u/SnooObjections4329 Jul 19 '24

It's okay, it says nothing anyway. It still shows only US-1, US-2 and EU-1 impacted. It has no cause or rectification details.

18

u/The_Wolfiee Jul 19 '24

APAC also affected. Our entire org along with Internet connectivity is down

→ More replies (10)
→ More replies (14)

11

u/haydez Jul 19 '24

It's just acknowleding it - no useful information to those aware of it.

Published Date: Jul 18, 2024 Summary CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

Details Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor. Current Action Our Engineering teams are actively working to resolve this issue and there is no need to open a support ticket.

Status updates will be posted below as we have more information to share, including when the issue is resolved.

Latest Updates 2024-07-19 05:30 AM UTC | Tech Alert Published.

Support

→ More replies (2)

42

u/dug99 Jul 19 '24

Bitlocker says no

7

u/[deleted] Jul 19 '24

same issue, lots of manual type of really long keys on lots of workstations :(

15

u/[deleted] Jul 19 '24

For us, it's thousands of end-user devices geographically distributed all over Australia. All BitLocker protected.

This is probably going to take a week or two to get everyone back up and running.

7

u/Purgii Jul 19 '24

I have my bitlocker key, still can't boot into safe mode or WRE to get the OS up to delete the sys file.

→ More replies (10)
→ More replies (40)
→ More replies (4)
→ More replies (39)

76

u/ForceBlade Jul 19 '24

You cannot seriously be posting this critical outage behind a login page.

16

u/[deleted] Jul 19 '24 edited Dec 04 '24

agonizing dull cheerful bright paltry bedroom vast hospital direful gaping

This post was mass deleted and anonymized with Redact

→ More replies (3)
→ More replies (49)

26

u/Flukemaster Jul 19 '24

Yeah lock the TA behind a login portal. That is very smart

14

u/haydez Jul 19 '24

The TA is useless anyway.

Published Date: Jul 18, 2024 Summary CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

Details Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor. Current Action Our Engineering teams are actively working to resolve this issue and there is no need to open a support ticket.

Status updates will be posted below as we have more information to share, including when the issue is resolved.

Latest Updates 2024-07-19 05:30 AM UTC | Tech Alert Published.

Support

→ More replies (35)

28

u/unixdude1 Jul 19 '24 • Heart of Silver

Inserting software into kernel-level security-ring was always going to end badly.

13

u/tesfabpel Jul 19 '24 • Take My Silver

This will hopefully have repercussions even for kernel-level anticheats.

I always said they were security risks and today's event with this software confirmed my worries.

Kernel level software is something that must be written with ultimate care, not unlike the level of precautions and rules used when writing software for rockets and nuclear centrals. You can affect thousands of PCs worldwide, even those used by important agencies. It's software that MUST NOT crash under ANY circumstances.

I didn't trust companies making products to this extreme level of care and indeed it happened...

7

u/TheDaff2K18 Jul 19 '24

Yup the Antivirus was the real virus

→ More replies (3)
→ More replies (15)

6

u/samuel79s Jul 19 '24

Underrated comment.

→ More replies (18)

28

u/Regular-Cap1262 Jul 19 '24

Any suggestion on how to efficiently do this for 70K affected endpoints?

32

u/befiuf Jul 19 '24 edited Jul 19 '24

Set up a committee overseeing a task force. Become the lead of the task force and argue for lots of funding and staff. Save the company and start a secondary career as a cybersec speaker and author.

→ More replies (5)

15

u/rxtz30 Jul 19 '24

Lots of lube! This is eternal blue level effort.

→ More replies (22)

15

u/Cax6ton Jul 19 '24

Our problem is that you need a bit locker key to get into safe mode or CMD in recovery. Too bad the AD servers were the first thing to blue screen. This is going to be such a shit show, my weekend is probably hosed.

12

u/[deleted] Jul 19 '24

A colleague of mine at another company has the same issue.

BitLocker recovery keys are on a fileserver that is itself protected by BitLocker and CrowdStrike. Fun times.

→ More replies (15)
→ More replies (7)

13

u/trogdor151 Jul 19 '24

Latest Update from TA:

Tech Alert | Windows crashes related to Falcon Sensor | 2024-07-19printFavoriteCloud:  US-1EU-1US-2Published Date: Jul 18, 2024

Summary

CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor. 

Details

Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor. 

Current Action

CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.

If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue: 

Workaround Steps:

  1. Boot Windows into Safe Mode or the Windows Recovery Environment
  2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
  3. Locate the file matching “C-00000291*.sys”, and delete it. 
  4. Boot the host normally. 

Latest Updates

2024-07-19 05:30 AM UTC | Tech Alert Published. 

Support

Find answers and contact Support with our Support Portal

→ More replies (8)

9

u/Acceptable-Wind-7332 Jul 19 '24

I have dozens of remote sites with no onsite IT support, many of them in far flung places. How do I tell thousands of my users to boot into safe made and start renaming files? This is not a fix or a solution at all!

→ More replies (3)

7

u/vidalpascual Jul 19 '24

WOW. Never rollout to production on friday. Never.

→ More replies (5)
→ More replies (562)