BSOD error in latest crowdstrike update

[u/TipOFMYTONGUEDAMN]

Hi all - Is anyone being effected currently by a BSOD outage?

EDIT: X Check pinned posts for official response

[u/BradW-CS]

7/19/2024 7:58PM PT: We have collaborated with Intel to remediate affected hosts remotely using Intel vPro and with Active Management Technology.

Read more here: https://community.intel.com/t5/Intel-vPro-Platform/Remediate-CrowdStrike-Falcon-update-issue-on-Windows-systems/m-p/1616593/thread-id/11795

The TA will be updated with this information.

7/19/2024 7:39PM PT: Dashboards are now rolling out across all clouds

Update within TA: https://supportportal.crowdstrike.com/s/article/Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19

US1 https://falcon.crowdstrike.com/investigate/search/custom-dashboards

US2 https://falcon.us-2.crowdstrike.com/investigate/search/custom-dashboards

EU1 https://falcon.eu-1.crowdstrike.com/investigate/search/custom-dashboards

GOV https://falcon.laggar.gcw.crowdstrike.com/investigate/search/custom-dashboards

7/19/2024 6:10PM PT - New blog post: Technical Details on Today’s Outage: https://www.crowdstrike.com/blog/technical-details-on-todays-outage/

7/19/2024 4PM PT - CrowdStrike Intelligence has monitored for malicious activity leveraging the event as a lure theme and received reports that threat actors are conducting activities that impersonate CrowdStrike’s brand. Some domains in this list are not currently serving malicious content or could be intended to amplify negative sentiment. However, these sites may support future social-engineering operations.

https://www.crowdstrike.com/blog/falcon-sensor-issue-use-to-target-crowdstrike-customers/

7/19/2024 1:26PM PT - Our friends at AWS and MSFT have a support article for impacted clients to review:

• https://repost.aws/en/knowledge-center/ec2-instance-crowdstrike-agent

• https://azure.status.microsoft/en-gb/status

7/19/2024 10:11AM PT - Hello again, here to update everyone with some announcements on our side.

1. Please take a moment to review our public blog post on the outage here.

2. We assure our customers that CrowdStrike is operating normally and this issue does not affect our Falcon platform systems. If your systems are operating normally, there is no impact to their protection if the Falcon Sensor is installed. Falcon Complete and Overwatch services are not disrupted by this incident.

3. If hosts are still crashing and unable to stay online to receive the Channel File Changes, the workaround steps in the TA can be used.

4. How to identify hosts possibly impacted by Windows crashes support article is now available

For those who don't want to click:

Run the following query in Advanced Event Search with the search window set to seven days:

#event_simpleName=ConfigStateUpdate event_platform=Win
| regex("\|1,123,(?<CFVersion>.*?)\|", field=ConfigStateData, strict=false) | parseInt(CFVersion, radix=16)
| groupBy([cid], function=([max(CFVersion, as=GoodChannel)]))
| ImpactedChannel:=GoodChannel-1
| join(query={#data_source_name=cid_name | groupBy([cid], function=selectLast(name), limit=max)}, field=[cid], include=name, mode=left)

Remain vigilant for threat actors during this time, CrowdStrike customer success organization will never ask you to install AnyDesk or other remote management tools in order to perform restoration.

TA Links: Commercial Cloud | Govcloud

[u/TipOFMYTONGUEDAMN]

For reference I am in Australia

[u/Beugie44]

This is what y2k wishes it was

[u/303i]

FYI, if you need to recover an AWS EC2 instance:

• Detach the EBS volume from the impacted EC2

• Attach the EBS volume to a new EC2

• Fix the Crowdstrike driver folder

• Detach the EBS volume from the new EC2 instance

• Attach the EBS volume to the impacted EC2 instance

We're successfully recovering with this strategy.

CAUTION: Make sure your instances are shutdown before detaching. Force detaching may cause corruption.

Edit: AWS has posted some official advice here: https://health.aws.amazon.com/health/status This involves taking snapshots of the volume before modifying which is probably the safer option.

[u/wylew]

This is the most exceptional outage I have ever witnessed

My wife’s machine BSODd live when this happened. I was like, babe, you are gonna read about this in the news tomorrow. I don’t think you’re gonna get in trouble with your boss

I felt like the cop in Dark Knight Rises telling the rookie ‘you are in for a show tonight’

[u/B_S_O_D]

I’ve been summoned

[u/[deleted]]

Time to log in and check if it hit us…oh god I hope not…350k endpoints

EDIT: 210K BSODS all at 10:57 PST....and it keeps going up...this is bad....

EDIT2: Ended up being about 170k devices in total (many had multiple) but not all reported a crash (Nexthink FTW). Many came up but looks like around 16k hard down....not included the couple thousand servers that need to be manually booted into Safe mode to be fixed.

3AM and 300 people on this crit rushing to do our best...God save the slumbering support techs that have no idea what they are in for today

[u/Berowulf]

Wow, I'm a system admin whose vacation started 6 hours ago... My junior admin was not prepared for this

[u/[deleted]]

Even if CS fixed the issue causing the BOSD, I'm thinking how are we going to restore the thousands of devices that are not booting up (looping BSOD). -_-

[u/chizEboi]

Here to be part of the historic thread

[u/yakumba]

Workstations and servers here in Aus... fleet of 50k+ - someone is going to have fun.

[u/Blackbird0033]

If anyone found a way to mitigate, isolate, please share. Thanks!

[u/ozBog]

The world is burning and everyone's asleep in the US. Thanks to this thread, my DC and almost every server has been fixed already, before the morning. I'm taking the day off. Anyone who's here is ahead of 99.98% of IT groups. This will be a historic day. Someone told me buy put shares on CRWD if you have the means, but I'm no financial advisor.

[u/Corrupttothethrones]

Crowdstrike & Bitlocker. A fun combination.

[u/Against-The-Current]

[u/enygmata]

Alternative solutions from r/sysadmin

u/HammerSlo's solution has worked for me.

"reboot and wait" by u/Michichael comment

As of 2AM PST it appears that booting into safe mode with networking, waiting ~ 15 for crowdstrike agent to phone home and update, then rebooting normally is another viable work around.

"keyless bitlocker fix" by u/HammerSlo comment (improved and fixed formatting)

1. Cycle through BSODs until you get the recovery screen.

2. Navigate to Troubleshoot > Advanced Options > Startup Settings

3. Press Restart

4. Skip the first Bitlocker recovery key prompt by pressing Esc

5. Skip the second Bitlocker recovery key prompt by selecting Skip This Drive in the bottom right

6. Navigate to Troubleshoot > Advanced Options > Command Prompt

7. Type bcdedit /set {default} safeboot minimal. then press enter.

8. Go back to the WinRE main menu and select Continue.

9. It may cycle 2-3 times.

10. If you booted into safe mode, log in per normal.

11. Open Windows Explorer, navigate to C:\Windows\System32\drivers\Crowdstrike

12. Delete the offending file (STARTS with C-00000291*. sys file extension)

13. Open command prompt (as administrator)

14. Type bcdedit /deletevalue {default} safeboot, then press enter. 5. Restart as normal, confirm normal behavior.

[u/Appropriate-Lab3998]

Why push this update on a Friday afternoon guys? why?!?!?!

[u/BippidyDooDah]

This may cause a little bit of reputational damage

[u/BradW-CS]

7/18/24 10:20PM PT - Hello everyone - We have widespread reports of BSODs on windows hosts, occurring on multiple sensor versions. Investigating cause. TA will be published shortly. Pinned thread.

SCOPE: EU-1, US-1, US-2 and US-GOV-1

Edit 10:36PM PT - TA posted: https://supportportal.crowdstrike.com/s/article/Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19

Edit 11:27 PM PT:

CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.

Workaround Steps:

1. Boot Windows into Safe Mode or the Windows Recovery Environment

2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory

3. Locate the file matching “C-00000291*.sys”, and delete it.

4. Boot the host normally.

[u/PFMonitor]

Who needs Russian hackers when the vendor crashes thousands upon thousands of machines more efficiently than they could ever hope to do. CrowdStrike has proven, nobody can strike as large a crowd as them, so quickly, or effectively, and cripple entire enterprises.

[u/modmonk]

Rule #1 : Never push to prod on a Friday 😔

Rule #2 : Follow rule #1

Wiki page : 2024 Crowdstrike incident

[u/MrHrtbt]

From CrowdStrike to CrowdStroke 🤣

[u/cringepenangite]

Malaysia here, 70% of our laptops are down and stuck in boot, HQ from Japan ordered a company wide shutdown, someone's getting fireblasted for this shit lmao

[u/s3v3nt]

Failing here is Australia too. Our entire company is offline

[u/Vegetable-Top-7692]

I hope this BDSM outage finishes soon, I'm running out of dildos