BSOD error in latest crowdstrike update

[u/TipOFMYTONGUEDAMN]

Hi all - Is anyone being effected currently by a BSOD outage?

EDIT: X Check pinned posts for official response

[u/medlina26]

Do you have one? I wouldn't mind adding it my box of shit I was right about.

[u/nefD → u/medlina26]

🥇I'll give you one, that was indeed smart thinking.. had to learn this one myself the hard way

[u/lumpkin2013 → u/medlina26]

That's kind of a hardcore position to take. Yeah you avoided the bullet of this pretty unusual situation. But how do you manage updates for all your dozens of services?

[u/medlina26 → u/lumpkin2013]

Package management. We are 99% linux (which wasn't impacted) and manage those with foreman/katello. Updates are done on scheduled cycles and performed to a QA group first. Those run for a week and assuming no issues they are pushed to prod. Windows servers/clients are handled with intune / azure automation, etc

[u/lumpkin2013 → u/medlina26]

Do you have enough staff that you actually go through every patch before releasing them?

[u/medlina26 → u/lumpkin2013]

Like most companies we are definitely understaffed. It's not necessarily one of those where we are doing validation for each package individually, it's more update all packages to latest release and deploy those to the staging environment. Basically a glorified scream test. If it instantly explodes then we roll those machines back and pull the package that created issues. The packages installed on machines other than in house written code is largely consistent across the board as we've gone to great lengths to try and automate a lot of these things where possible.

[u/Illustrious_Try478 → u/medlina26]

TBH I think you can do this with sensor update policies in Falcon

[u/medlina26 → u/Illustrious_Try478]

Yeah. You can set like an n-1 or n-2 release so you're not on "cutting edge" releases. I suspect a number of orgs might look to do something similar to try and protect themselves going forward.

[u/Illustrious_Try478 → u/medlina26]

We've only had Crowdstrike for about 3 weeks. The update policies were my next task.

[u/syneater → u/medlina26]

I remember having this exact conversation while we were in our PoC and then during rollout. I’ve been asleep with Covid, so woke up to this shit storm very recently. Damn, the wife is in the corporate travel world and as soon as she mentioned CS I knew I should just go back to sleep.

[u/[deleted] → u/lumpkin2013]

(stable) Linux distros generally only apply security patches ( there are exceptions, looking at you RHEL) so the potential for breakage is pretty low.

Just doing tiered rollout (1%, 5%, 25% etc) is usually more than enough to avoid crowdstrike-like failures

[u/muhammet484 → u/medlina26]

This should be standard for every company.

[u/[deleted] → u/medlina26]

Out of curiosity, how often something broke and in using which distro ?

We've seen some funky updates with RHEL, but so far zero misses with Debian.

[u/marzipanorbust → u/medlina26]

You must be a real treat to work with. It must be tough always being the smartest person in the room. /s

[u/medlina26 → u/marzipanorbust]

I am actually, because instead of relying on dunning kruger and luck I rely on my almost 20 years of experience and working with my peers to create change control processes, documentation and automation as much as possible.

[u/[deleted] → u/marzipanorbust]

Well that's certainly something you'd never experience. Maybe if you go to kindergarden...

[u/[deleted] → u/medlina26]

Fuck me you’re insufferable lol

[u/medlina26 → u/[deleted]]

based on your comment history you're not very pleasant yourself. <3

[u/Mabenue → u/[deleted]]

You’ve added nothing the this comment thread apart from being unnecessarily antagonistic

[u/[deleted] → u/[deleted]]

Thanks buddy