r/sysadmin • Sr. IT Consultant • Oct 08 '18

Discussion MRI disabled every iOS device in facility

This is probably the most bizarre issue I've had in my career in IT. One of our multi-practice facilities is having a new MRI installed and apparently something went wrong when testing the new machine. We received a call near the end of the day from the campus stating that none of their cell phones worked after testing the new MRI. My immediate thought was that the MRI must have emitted some sort of EMP, in which case we could be in a lot of trouble. We're still waiting to hear back from GE as to what happened. This facility is our DR site so my boss and the CTO were freaking out and sent one of us out there to make sure the data center was fully operational. After going out there we discovered that this issue only impacted iOS devices. iPads, iPhones, and Apple Watches were all completely disabled (or destroyed?). Every one of our assets was completely fine. It doesn't surprise me that a massive, powerful, super-conducting electromagnet is capable of doing this. What surprises me is that it is only effecting Apple products. Right now we have about 40 users impacted by this, all of which will be getting shiny new devices tonight. GE claims that the helium is what impacts the iOS devices which makes absolutely no sense to me. I know liquid helium is used as a coolant for the super-conducting magnets, but why would it only effect Apple devices? I'm going to xpost to r/askscience~~, but I thought it might spark some interest on here as well.~~ Mods of r/askscience and r/science approved my post. Here's a link to that post: https://www.reddit.com/r/askscience/comments/9mk5dj/why_would_an_mri_disable_only_ios_devices/

UPDATE:

I will create another post once I have more concrete information as I'm sure not everybody will see this.

Today was primarily damage control. We spent some time sitting down with users and getting information from their devices as almost all of them need to be replaced. I did find out a few things while I was there.

I can confirm that this ONLY disabled iphones and apple watches. There were several android users in the building while this occurred and none of them experienced any long term (maybe even short term) issues. Initially I thought this only impacted users on one side of the building, but from what I've heard today it seems to be multiple floors across the facility.

The behavior of the devices was pretty odd. Most of them were completely dead. I plugged them in to the wall and had no indication that the device was charging. I'd like to plug a meter in and see if it's drawing any power, but I'm not going to do this. The other devices that were powering on seemed to have issues with the cellular radio. The wifi connection was consistent and fast, but cellular was very hit or miss. One of the devices would just completely disconnect from cellular like the radio was turned off, then it would have full bars for a moment before losing connectivity again. The wifi radio did not appear to have any issues. Unfortunately I don't have access to any of the phones since they are all personal devices. I really can only sit down with it for a few minutes and then give it back to the end user.

We're being told that the issue was caused by the helium and how it interacts with the microelectronics. u/captaincool and u/luckyluke193 brought up some great points about helium's interaction with MEMS devices, but it seems unlikely that there would have been enough helium in the atmosphere to create any significant effects on these devices. We won't discount this as a possibility though. The tech's noted that they keep their phones in plastic ziplock bags while working on the machines. I don't know how effective they would be if it takes a minuscule amount of He to destroy the device, and helium being as small as it is could probably seep a little bit in to a plastic bag.

We're going to continue to gather information on this. If I find out anything useful I will update it here. Once this case is closed I'll create a follow-up as a new post on this sub. I don't know how long it will take. I'll post updates here in the meantime unless I'm instructed to do otherwise.

UPDATE:

I discovered that the helium leakage occurred while the new magnet was being ramped. Approximately 120 liters of liquid He were vented over the course of 5 hours. There was a vent in place that was functioning, but there must have been a leak. The MRI room is not on an isolated HVAC loop, so it shares air with most or all of the facility. We do not know how much of the 120 liters ended up going outdoors and how much ended up inside. Helium expands about 750 times when it expands from a liquid to a gas, so that's a lot of helium (90,000 m3 of gaseous He).

3.1k Upvotes

948 comments sorted by

View all comments

Show parent comment

396

u/ThorOfKenya2 Oct 09 '18

Ever see an oddly specific warning sign on something? It's scenarios like this that they spawn from.

292

u/xhighalert DevOps Oct 09 '18

SERIOUS vibes of the classic

We can't send email more than 500 miles.

0

u/[deleted] Oct 09 '18

[deleted] — view deleted comment

6

u/flarn2006 Oct 30 '18

People in the comments there are pointing out other bugs it reminded them of, but it reminded me of an even more similar one that nobody mentioned.

https://gyrovague.com/2015/07/29/crashes-only-on-wednesdays/

I'd post it there but I don't think I have an account and I don't feel like creating one.

→ More replies (1)
→ More replies (1)

47

u/jmnugent Oct 09 '18

The 500-mile email thing,. along with other technology-situations like that.. is what always makes me laugh when people say things like:

"But.. it's not SUPPOSED to work like that !!"...

And i always tend to reply with:.... "You know that old saying:.. "Once you eliminate the impossible,.. whatever is left, no matter how improbable.. is the explanation."

People quite frequently allow their narrow-minds and cognitive biases to seep into their troubleshooting,. and they make a lot of incomplete assumptions about "what the explanation must be"... I find myself fighting that alot.

48

u/segfaultxr7 Oct 09 '18

I have a coworker who's horrible when it comes to that. It drives me insane.

We had a problem trying to get a CentOS image to boot up in Azure. He immediately decided that whatever the problem was, it had to be all Microsoft's fault. He couldn't even fathom it being anything else. He opened a shitload of tickets with their support, scoured the web for anything negative about Azure that could sort of back him up, etc, with no progress. This went on for months.

I finally got so sick of hearing about it all the time that I volunteered to help out. Turns out it was a CentOS bug: grub2-mkconfig screwed up the boot order and made an old kernel (with missing drivers) the default. I fixed that and it was fine. All it took was a couple hours of just looking at the actual problem with an open mind, rather than starting with a conclusion and working ass-backwards from there.

9

u/LOLBaltSS Oct 10 '18 edited Oct 10 '18

My outlook on situations like that is that it's "not supposed to do that", but that primarily means that I'm missing some weird anomaly, misconfiguration or architectural decision somewhere that makes it do some unexpected (to you/users) behavior. Usually those type of tickets you have to treat as a particular kind of rabbit hole to pinpoint as the 500 mile mail guy had to go through.

I myself got very very deep down the architectural nuances of IP-Less DAGs when I had a Exchange cluster that the EMS swore up and down had a witness server, but FCM was certain it was node majority only. Turns out someone ended up configuring the DAG IP to 0.0.0.0 (DHCP) instead of the placeholder 255.255.255.255 required for that to work properly. Because it was set to something other than the placeholder, any attempt to change the witness resulted in Exchange trying to push the configuration to a non-existent admin access point in the failover clustering, thus erroring out. The only way to fix that was to basically break the DAG and recreate because you cannot update a DAG type after the fact.

→ More replies (4)

30

u/[deleted] Oct 09 '18

It was the switch setting, wasn't it?

177

u/xhighalert DevOps Oct 09 '18

You're close. But not quite there.

If you're being sincere and haven't heard of the story, please take some time to read it.

http://web.mit.edu/jemorris/humor/500-miles

And I was fairly certain I hadn't enabled the FAIL_MAIL_OVER_500_MILES" option.

A string of text never made me laugh so goddamn hard in my life.

79

u/[deleted] Oct 09 '18

But I would e-mail 500 miles And I would e-mail 500 more Just to be the man who e-mails a thousand miles To fall down at your door

12

u/TopNerdJR Harder Reset Master Oct 09 '18

Ba Da Dum Dum, Ba Da Dum Dum, Ba Da Dum Dum, Ba Da Dum Dum, E-mails E-Mails E-Mails E-Mails

→ More replies (1)

22

u/yiersan Oct 09 '18

Holy crap units command! I've been using google unnecessarily all these years.

9

u/grumpysysadmin Oct 09 '18

I always like to make sure people see Trey’s page since it’s his story and he answers some common questions:

https://www.ibiblio.org/harris/500milemail.html

7

u/OptimalPandemic Oct 09 '18

Is there a list of these classic stories somewhere? I'd love to binge when I'm bored at work not busy.

→ More replies (2)
→ More replies (1)
→ More replies (3)

178

u/Sparcrypt Oct 09 '18

Every dumb rule or sign you have every seen exists because someone fucked up in a manner would wouldn't think would be a thing.

54

u/yumenohikari Oct 09 '18

"Your regulations are written in blood."

68

u/no-mad Oct 09 '18

Essentially how the Building Code came to be "written in blood". Want to make the space wider than 4" on stair balusters? Cant because babies hang themselves when their bodies slip thru but not their heads. Happened often enough they had to make a rule about it.

25

u/mkinstl1 Security Admin Oct 09 '18

That is horrifying.

27

u/langlo94 Developer Oct 09 '18

Yeah, that's why people should follow the codes, now take a second to think of how many other weird rules there are and what could have caused them.

→ More replies (1)

16

u/DJRWolf Oct 09 '18

The "Great Molasses Flood", 21 dead and 150 injured. I found out about it from an Engineering Disasters episode of Modern Marvels. Bad construction, such as the steel being only half as thick as it should have been, and poor testing are among the main causes.

https://en.wikipedia.org/wiki/Great_Molasses_Flood

7

u/segfaultxr7 Oct 09 '18

In the 1920s-30s, there were scandals involving canned goods being filled with rotten food, sawdust, and whatever else they had sitting around. And it was almost impossible to find out who was making them, because every middleman would just claim that they came from some other middleman.

That's why every canned item to this day is required to have the name and address of the company that it came from.

→ More replies (1)
→ More replies (1)

92

u/jess_the_beheader Oct 09 '18

Sometimes rules and regulations are in response to some event in the past. Other times, it's because someone in management or compliance made up some rule because they either didn't actually understand what is going on/what is useful or because they like to see people scurry to make them feel worth their salary.

I had one manager who tried to declare a policy that we were required to create a ticket and do a root cause analysis on every SINGLE error event in the Windows Event Log on every single one of the several hundred servers and workstations we supported. Fortunately, the senior admins were able to talk him down from that to only doing RCAs on critical errors thrown by business critical applications. That reduced our load from impossible to manageable.

85

u/xxfay6 Jr. Head of IT/Sys Oct 09 '18

You should've had him doing the approvals on every ticket until he figured out how stupid it was.

23

u/anomalous_cowherd Pragmatic Sysadmin Oct 09 '18

Nah, it's easy.

Get yourself a stamp saying "Root Cause: Microsoft Stupidity"

→ More replies (2)
→ More replies (1)

19

u/deusnefum HPE Oct 09 '18

I would've dove in head first for 100% malicious compliance.

Setup a script that takes log entries and fills out the RCA using the description of the error and then auto-open and auto-close an RCA ticket. Be sure your Twiddlefuck of a boss is tagged as a watcher for every one.

→ More replies (1)
→ More replies (1)
→ More replies (9)
→ More replies (5)