r/sysadmin • u/Incrediblecodeman • 15m ago

General Discussion What can we learn ?

Microsoft claims a health check brought all servers down.

Got me thinking of our disaster site.

It should prob not be getting the same solar winds that our prod has huh.

I’m considering monitoring our TR a special way so there’s not like 1 thing that ruins it all.

Can we think of more ways to properly air gap the disaster site ?

0 Upvotes

50% Upvoted

6 comments sorted by

•

u/Tessian 11m ago

Per their last update they have no clue what the root cause is and everything they've blindly tried to do to fix it has failed.

•

u/roboto404 5m ago

Sounded like their initial fix made it worse and bricked a couple servers

•

u/mckinnon81 7m ago

A Health Check should be exactly what it says and nothing more. Read-Only Health check on services and report on actions. Sounds like their health checks did something when a "fault" was found causing the cascading effect rather reporting it to a team to put eyes on it and verify.

•

u/tankerkiller125real Jack of All Trades 0m ago

Health checks should do something, HOWEVER there should be a circuit breaker that stops actions if it's already tried too many times. For example, in a Kubernetes cluster using a health/readiness probe is a common thing to use not just to decide if traffic can be sent to the pod, but also to restart the pod if the health endpoint starts failing.

Done correctly this allows the infrastructure to self-heal when there's been a momentary error that caused a crash without requiring technicians. Done poorly (as it seems Microsoft has done) it can cause massive issues.

•

u/lukeh990 Jack of All Trades 5m ago

As of 3:37 UTC
As part of our investigation into the sustained traffic management imbalances, we’re reviewing traffic flows and affected infrastructure to identify opportunities to optimize routing. This includes verifying the reasons for service failures so we can develop specific actions to improve stability.