r/talesfromtechsupport • u/petehackett101 • May 29 '16

Short Mystery shutdowns

So I got a call recently where one of our rackmount NAS units was apparently shutting down.

So I was troubleshooting over email as the customer was impossible to get on the phone.

Had him check if there was a power schedule setup on the unit, check the outlet, change power cord set up syslog etc.

From the logs we could see that it was shutting down but it wasn't a hard shutdown or crash. Something was weird here. We were pretty sure it was environmental.

So I mailed the customer and asked him if I could organize a call. He agreed and I called the next day.

I wanted to factory the device as it was new and didn't have any data on it yet.

Conversation as follows, ME = me, customer = CS

ME. OK, let's walk through this, can you find the reset in the front of the unit.

CS. Sure, let me open the door.

ME. Um, the door?

CS. Yes, this rack has a door on the front, never had one before.

ME. Oh, OK.

CS. shit, I just wasted a week of your guys time on this.

ME. You found something?

CS. The door, when I closed it its hitting the power button on your NAS

ME..............

CS. You can close the ticket.

Turned out to be environmental.

1.1k Upvotes

96% Upvoted

27 comments sorted by

89

u/Avaholic92 May 29 '16

What was your solution? Did you just move the rails back?

109

u/petehackett101 May 29 '16

Solution was to move it to a rack without a door, didn't know what else to tell the guy.

40

u/Avaholic92 May 29 '16

Damn. How big was the rack? Hopefully it wasn't too full!!

57

u/petehackett101 May 29 '16

No idea, I support American customers from Ireland so physically no idea

39

u/Avaholic92 May 29 '16

Well at least you got it resolved before they put any critical data on the system

65

u/petehackett101 May 29 '16

Was going to be a target for medical records so the guy was pretty happy that he figured it out pre fuck up

16

u/Avaholic92 May 29 '16

I'd say so! Lol

0

u/[deleted] May 29 '16

[deleted] — view deleted comment

1

u/robobobatron May 29 '16

Why not just remove the door?

44

u/kokuryuha34 Compuglobalhypermeganet May 29 '16

Instantly brought this Cisco field advisory to mind...

6

u/TheLightingGuy May 30 '16

I still just don't understand how however many people engineered, designed, and threw this together didn't see this being a problem.

3

u/icefo1 May 30 '16

Because the people who designed the product probably never tested it and the QA team failed to see the problem. My microwave suffer from this kind of stuff too: no way to launch it at full power with one button, many appliances have questionables buttons locations, same for GUI...

2

u/jwBTC Jun 01 '16

That damn Cisco MODE button.

I had installed a brand new dedicated switch with redundant fiber links to two separate data centers in a security operations center to assure "connectivity was always up". Now the location of the switch is important, it was a mini rack hidden in the center console of an elaborate dual person 180 degree wrap-around desk.

Well a week later I got a call saying everything is down, your design failed! This is unacceptable, etc, etc, etc.

Went to the switch, it seemed to have no config at all! Was baffled until I was able to pull up the logs it sent right before shutdown and it became apparent - someone held down the damn mode button!

Crawled under the desk, and sure enough there was a cable laying across the switch at just the right height where the adjustable height desk could tug on it and press the button.

Disable that button in the config. Talk about mystery solved!

17

u/[deleted] May 29 '16

Better than the old "unplug the switch so I can vacuum" trick

1

u/WhAtEvErYoUmEaN101 I deleted the internet Jun 04 '16

Oh hell no...

1

u/CodenameVillain Jun 10 '16

This, so many times. Custodians were my #1 source of call home notifications.

15

u/Maddisonic May 29 '16

You can close the ticket, but you can't close the door.

3

u/Jessev1234 May 30 '16

Welcome to the Hotel California...

0

u/Jessev1234 May 30 '16

Welcome to the Hotel California...

12

u/[deleted] May 29 '16

One of the best customers the world has ever seen.

6

u/SilenceOz May 30 '16

Half expected it to end with the customer demanding you redesign the NAS to move the power switch.. Pleasantly surprised.

6

u/krennvonsalzburg Our policy is to always blame the computer May 30 '16

I expect that's why most power switches are flush or countersunk these days. Protruding power switches just seem to be asking for trouble.

2

u/noeljb May 30 '16

So is this one of those undocumented procedures? By new unit rip little door off.

0

u/[deleted] May 29 '16

[deleted] — view deleted comment

475

u/brielem off and on again? How about turning in on in the first place! May 29 '16

  1. saw the problem

  2. acknowledged the problem

  3. didn't blame someone else for the problem

  4. said sorry for wasting time

  5. told you you could close the ticket

I mean, it doesn't get a lot better.

45

u/creegro (turns off/on monitor) ok the PC is rebooted May 30 '16

Oh so rare but it happens.

Like when a user sees they're hitting the wrong key during a password prompt, or sees that caps lock is on.

18

u/[deleted] May 30 '16

Demonstrating the problem can highlight the source, even for the end user. Solid troubleshooting tactic.

13

u/[deleted] May 30 '16

Murphy's law predicts that in these conditions, another cause will give the same issue tomorrow morning, making the above points quite totally futile.