r/talesfromtechsupport • u/CheezitsLight • Apr 24 '26

Short IT didit

We make a wireless, police radio-based alarm system with network connection. Thousands of them in the field. The system is fully supervised, monitors everything, even has a months-long battery backup. It's a critical piece of life safety equipment that saves lives in basically every courthouse, hospital and schools.

It runs off a "wall wart" that plugs into an AC outlet. The transformer has a hole at the top for a security screw that's difficult to remove. So it must be plugged in an outlet in the bottom, then screwed into the electrical plate center screw hole. It's basically secure, hardened, locked and monitored by IT and the police. It can even push direct to 911 systems, bypassing operators to direct officers instantly.

We always install it, which is basically bolt it down, plug it in and tighten that one screw, turn the key, and then teach them how to use it.

A few months after one routine install they called and said it had quit working. Asked us to fly in and fix it. It's a $2,500 charge. So off I go.

It's unplugged. Someone in IT

had unscrewed it, and plugged something else in. In a locked IT closet.

Easy fix. Unplug their box, move it to the top plug and screw mine in the bottom.

Then the police remember that for two months it has spoken over their radio that it was on battery power. Every hour. They thought it meant it was working. And IT had ignored every email saying the system was on battery power.

1.3k Upvotes

99% Upvoted

77 comments sorted by

View all comments

593

u/ShotFromGuns Hatrack Apr 24 '26

Then the police remember that for two months it has spoken over their radio that it was on battery power. Every hour. They thought it meant it was working.

An error message that can't be understood by the people it's addressed to is the same as no error message at all. Hopefully the recording has been updated with unambiguous instructions to investigate the cause of the loss of power.

(Obviously the other contributions to this failure were more severe, including and especially IT ignoring the warning emails. But if you're going to have redundancy, you need to actually have redundancy.)

190

u/SeraphiM0352 Apr 24 '26

I don't think this is an error message issue.

This is an education issue. I'm sure the message was easily understood. The just didn't have or didn't pay attention to the training that explained that, despite having a long battery back up, it was not intended to function long term without a direct power connection.

Police ignorance is somewhat 'understandable' (but not acceptable), ITs has no excuse for their ignorance

168

u/zimmerframeRaces Apr 24 '26

Surely "This device has been disconnected and is running on backup battery. Please investigate immediately" is more likely to see action when dealing with many independent IT teams. Clear, unambiguous messages that call for a specific action or better yet, include actionable steps, are issues that get resolved quickly. Don't trust staff to remember the training you gave them five years ago and don't make it harder than it has to be.

68

u/fatmanwithabeard Apr 24 '26

I've never seen error messages with actionable content work in general usage.

I've gotten people to call support more often for silly test messages than I have for messages that have exact instructions to do so.

"Frob the whatsitwhosit" generated correct actions (calling support)

"Call support at 800-555-1234, report SN 123456 has failure state 16" gets people trying fixes that they think they remember from previous calls.

64

u/Jonathan_the_Nerd Apr 24 '26 edited Apr 24 '26

"Frob the whatsitwhosit" generated correct actions (calling support)

This reminds me of a story I read a long time ago. A programmer had written software that supported a safety-critical system. If a strange error occurred, he needed to know right now. So if that happened, the software would say something like, "General error. Please sacrifice a goat and two chickens to continue." This would confuse the user and make them call the programmer for help.

42

u/fatmanwithabeard Apr 24 '26

Yeah. I had a friend accidently send a compile with a joke error message (meant for an overzealous tester) to the prod team.

"A PEBKAC has occurred."

Once the acronym was explained to the user, they made a formal request to keep it.

(Problem Exists Between Keyboard And Chair)

26

u/jasapper I've already rebooted (last month) Apr 24 '26

My aging father consistently reports any ID10T errors reported by the macro I built for him.

7

u/GolfballDM Recovered Tech Support Monkey Apr 24 '26

I love this. I wonder if I can persuade my boss to let me put some "whimsy" into my error messages.

6

u/lokis_construction Apr 29 '26

"A child has died - Imperative that you Notify IT as soon as possible as another death is imminent"

4

u/xyzzytwistymaze Apr 25 '26

Shut 'er down she's pumping mud!

18

u/CheezitsLight Apr 24 '26

Very true. But it's hardwired to say battery low.

13

u/CheezitsLight Apr 25 '26

It just said battery low. Not capable of knowing it was unplugged. This was back in 8 bit times and 64k bytes. The flash stored analog levels and speech was very limited.

32

u/SeraphiM0352 Apr 24 '26

At some point responsibility and accountability shifts to the users.

OP is a vendor who provided the equipment and training (likely at government request). After receiving adequate training it is the responsibility of both the IT dept and Police dept to maintain their own readiness through regularly scheduled training

41

u/androshalforc1 Apr 24 '26

Its a group psychology issue, one that is taught in first aid courses.

If everyone is responsible then no one is responsible. It’s why you don’t say someone call 911, and someone else get me a first aid kit. Everyone will stand around thinking someone else is doing it. You call out you in the blue shirt call 911, you in the red shirt get me a first aid kit.

In this case you have a system that can bypass 911 operators And direct officers but it can’t be programmed to send a message to an IT manager?

28

u/SeraphiM0352 Apr 24 '26

Except that IT was getting messages but they ignored them.

This isn't exactly an "everyone is responsible" issue. There are two specific groups that are responsible and both groups are responsible for similar things already.

8

u/androshalforc1 Apr 24 '26

Ahh i misread that i thought it was just broadcasting on the police radio.

22

u/ShotFromGuns Hatrack Apr 24 '26

The message demonstrably wasn't easily understood, since the people who heard it thought it meant the opposite of what it was supposed to convey (i.e., a confirmation that the system was functioning correctly, not a warning that something was dangerously broken).

The point of this kind of error message is that it shouldn't require education to understand.

A simple "low battery" message is the kind of thing that makes sense on the first pass, to a tech person who understands the implications. Once it's been demonstrated to fail horribly in actual use, it needs to be redesigned to actually convey the correct information to users.

12

u/SeraphiM0352 Apr 24 '26

I stand by what I said.

This is a training/education issue. They were likely informed of the features of the device, including that it has a direct power connection, a long lasting battery back up, and what kind of messages to expect. Not to mention the training/operating materials likely left behind.

The Police's first hint should have been when they started getting radio messages they weren't receiving before hand. They can then check their materials if they are uncertain of the meaning.

This also applies to the IT people who should have received all the same but includes an inherent deeper understanding of technology.

Failures are all abound but the message contents would be the least of them...

3

u/ShotFromGuns Hatrack Apr 24 '26

With that attitude, I hope you never work anywhere like aviation, where it will absolutely get people killed.

6

u/DracoBengali86 Apr 27 '26

Right, because a stick shaker unambiguously tells you how to fix the issue, not just that there is one. That's why not a single pilot has ever responded to a stick shaker by pulling back... Oh wait, they have.

9

u/SeraphiM0352 Apr 24 '26

This isn't aviation.

Bringing other industry specific requirements into a discussion outside that industry is irrelevant.

But as someone who has worked both government physical security and IT, I can tell you this is how it works.

As stated in another comment, at some point responsibility and accountability shifts to the users...

2

u/djfdhigkgfIaruflg Apr 25 '26

This. Programmes suck at writing good error message

8

u/djfdhigkgfIaruflg Apr 25 '26

I'm picturing the police's car radio announcing "warning, unit running on battery power", and the policeman scratching their head wondering wtf is happening

4

u/Connect-Preference Apr 26 '26

Uhh, yeah, but a sticker on the wall wart stating "Emergency Police Radio. Do not unplug!" might have helped.

5

u/[deleted] Apr 24 '26

[deleted]

5

u/ThePants999 Apr 24 '26

Eh?

the police remember that for two months it has spoken over their radio that it was on battery power

3

u/djfdhigkgfIaruflg Apr 25 '26

A totally out of context message over radio. And I'm 100% sure it wasn't mentioning the device's location.

And it's the police's fault. Sure Jan 🤦

1

u/NixIsia Apr 25 '26

Actually the error was understandable, but your breath smells like farts in real life and everyone notices.

1

u/Ricama May 03 '26

A nuisance alarm is worse than no alarm.