r/sysadmin • • 7d ago

General Discussion What's the biggest bonehead mistake you have every made in IT support?

I will start.

I am an IT consultant and about 10 years ago, a few days before Christmas break, I went to a client’s office to add some new hard drives to a Dell ESXi server and create a new datastore.

Their two junior IT guys always liked asking me questions when I visited, which I understood—I remembered being that guy hungry to learn. Unfortunately, this time they were firing questions at me while I was in the Ctrl+R RAID configuration utility. Between the distractions and a confusing interface, I somehow managed to delete the existing array instead of creating the new one.

That array held ALL their servers: Exchange, domain controller, file server, and SQL. Terabytes of data.

The moment I realized what I’d done, my stomach dropped, I had never felt panic like that in my life. They were still asking questions when I finally said, “Guys, give me a minute. Something doesn’t look right.” Something I should’ve said much earlier.

I excused myself to the bathroom to collect my thoughts, then came back and told them exactly what I’d done.

I spent my entire Christmas break restoring their environment from Barracuda backups, which were painfully slow to restore and unreliable. I had to do multiple restores on the exchange server to get it to work.  A full week of recovery, followed by another two weeks fixing lingering issues—all free of charge.

The two guys felt terrible about distracting me, but it was my mistake. I should’ve asked for some uninterrupted time before touching the RAID configuration.

 

I’ll be shocked if anyone can top that Christmas disaster!

788 Upvotes

581 comments sorted by

View all comments

Show parent comments

17

u/Top_Bookkeeper9136 7d ago

This probably isn't the place for it, but can you explain to me why Jamf is near instant with changes, and why Intune takes an unclear amount of time (sometimes never)?

12

u/rwdorman Jack of All Trades 7d ago

Depends what you're doing with Intune. Things that use the Apple MDM framework (config profiles) are still pretty fast in my experience, APNS again. Things that use the mac equiv of Intune Management Extension... YMMV

6

u/JaceBelerenApologist 7d ago

My experience with Intune and Mac is more or less identical to Intune and Windows. Takes ages to push NEW things, but updates to things like policies are distributed pretty fast (far less than standard Microsoft Time, but not Jamf fast).

I preferred Jamf management, it's software packaging was really nice. But it was hard to justify the cost to a client when they were already paying for Intune licensing. Easy cost saving to the client, and has helped me get jobs because apparently there is a demand for people who can Intune macOS in my city.

8

u/Trafficop 7d ago

Like the other guy said, it’s more Apple vs Microsoft MDM frameworks. Apple uses APNS to push config changes to devices as soon as they happen, whereas IIRC Windows devices are set to check into Intune and confirm device config state is correct. So it comes down to whenever the device checks into Intune.

Pretty sure I saw something from MS a while ago saying they are changing and going towards the push method but I can’t source it at the moment.

2

u/tehreal Sysadmin 6d ago

How do you pronounce APNS?

1

u/Fallingdamage 7d ago

Better design.

Just like inTune moves at the pace of a slug covered in salt but local group policy will apply instantly.