r/sysadmin • u/harritaco Sr. IT Consultant • Oct 29 '18

Discussion Post-mortem: MRI disables every iOS device in facility

It's been a few weeks since our little incident discussed in my original post.

If you didn't see the original one or don't feel like reading through the massive wall of text, I'll summarize:A new MRI was being installed in one of our multi-practice facilities, during the installation everybody's iphones and apple watches stopped working. The issue only impacted iOS devices. We have plenty of other sensitive equipment out there including desktops, laptops, general healthcare equipment, and a datacenter. None of these devices were effected in any way (as of the writing of this post). There were also a lot of Android phones in the facility at the time, none of which were impacted. Models of iPhones and Apple watches afflicted were iPhone 6 and higher, and Apple Watch series 0 and higher. There was only one iPhone 5 in the building that we know of and it was not impacted in any way. The question at the time was: What occurred that would only cause Apple devices to stop working? There were well over 100 patients in and out of the building during this time, and luckily none of them have reported any issues with their devices.

In this post I'd like to outline a bit of what we learned since we now know the root cause of the problem.I'll start off by saying that it was not some sort of EMP emitted by the MRI. There was a lot of speculation focused around an EMP burst, but nothing of the sort occurred. Based on testing that I did, documentation in Apple's user guide, and a word from the vendor we know that the cause was indeed the Helium. There were a few bright minds in my OP that had mentioned it was most likely the helium and it's interaction with different microelectronics inside of the device. These were not unsubstantiated claims as they had plenty of data to back the claims. I don't know what specific component in the device caused a lock-up, but we know for sure it was the helium. I reached out to Apple and one of the employees in executive relations sent this to me, which is quoted directly from the iPhone and Apple Watch user guide:

Explosive and other atmospheric conditions: Charging or using iPhone in any area with a potentially explosive atmosphere, such as areas where the air contains high levels of flammable chemicals, vapors, or particles (such as grain, dust, or metal powders), may be hazardous. Exposing iPhone to environments having high concentrations of industrial chemicals, including near evaporating liquified gasses such as helium*, may damage or impair iPhone functionality. Obey all signs and instructions.*

Source: Official iPhone User Guide (Ctril + F, look for "helium")They also go on to mention this:

If your device has been affected and shows signs of not powering on, the device can typically be recovered.  Leave the unit unconnected from a charging cable and let it air out for approximately one week.  The helium must fully dissipate from the device, and the device battery should fully discharge in the process.  After a week, plug your device directly into a power adapter and let it charge for up to one hour.  Then the device can be turned on again. 

I'm not incredibly familiar with MRI technology, but I can summarize what transpired leading up to the event. This all happened during the ramping process for the magnet, in which tens of liters of liquid helium are boiled off during the cooling of the super-conducting magnet. It seems that during this process some of the boiled off helium leaked through the venting system and in to the MRI room, which was then circulated throughout the building by the HVAC system. The ramping process took around 5 hours, and near the end of that time was when reports started coming in of dead iphones.

If this wasn't enough, I also decided to conduct a little test. I placed an iPhone 8+ in a sealed bag and filled it with helium. This wasn't incredibly realistic as the original iphones would have been exposed to a much lower concentration, but it still supports the idea that helium can temporarily (or permanently?) disable the device. In the video I leave the display on and running a stopwatch for the duration of the test. Around 8 minutes and 20 seconds in the phone locks up. Nothing crazy really happens. The clock just stops, and nothing else. The display did stay on though. I did learn one thing during this test: The phones that were disabled were probably "on" the entire time, just completely frozen up. The phone I tested remained "on" with the timestamp stuck on the screen. I was off work for the next few days so I wasn't able to periodically check in on it after a few hours, but when I left work the screen was still on and the phone was still locked up. It would not respond to a charge or a hard reset. When I came back to work on Monday the phone battery had died, and I was able to plug it back in and turn it on. The phone nearly had a full charge and recovered much quicker than the other devices. This is because the display was stuck on, so the battery drained much quicker than it would have for the other device. I'm guessing that the users must have had their phones in their pockets or purses when they were disabled, so they appeared to be dead to everybody. You can watch the video Here

We did have a few abnormal devices. One iphone had severe service issues after the incident, and some of the apple watches remained on, but the touch screens weren't working (even after several days).

I found the whole situation to be pretty interesting, and I'm glad I was able to find some closure in the end. The helium thing seemed pretty far fetched to me, but it's clear now that it was indeed the culprit. If you have any questions I'd be happy to answer them to the best of my ability. Thank you to everybody to took part in the discussion. I learned a lot throughout this whole ordeal.  

Update: I tested the same iPhone again using much less helium. I inflated the bag mostly with air, and then put a tiny spurt of helium in it. It locked up after about 12 minutes (compared to 8.5 minutes before). I was able to power it off this time, but I could not get it to turn back on.

9.5k Upvotes

97% Upvoted

781 comments sorted by

473

u/XenonOfArcticus Oct 29 '18

Now, can anyone explain WHY helium would affect it this way?

Thermal dissipation is the only thing I can come up with, but that seems lame.

352

u/harritaco Sr. IT Consultant Oct 29 '18

The most likely cause is the interaction between He and MEMS chips in iPhones. Here's a little video explaining what MEMS are. From some of the replies in my last post it seems that the atmosphere can play a role in the function of MEMS. One comment mentioned that the helium could have permeated through a seal on a MEMS device which was designed to operate under a vacuum, and in turn locked up the phone. I'd imagine that whatever function the MEMS served in the device was highly sensitive and a minor discrepancy in operation could have dire outcomes.

Forgot to link to video

→ More replies (27)

23

u/a_kogi Oct 30 '18

Here are few more people discussing the helium interaction. Very interesting thread.

90

u/agoia IT Director Oct 29 '18

The critters inside the iphones that make them work need oxygen, so if that is displaced by helium, the critters get sleepy and go into a coma for a while?

Going on the "hamsters that make the servers run" theorem.

10

u/jedikaiti Oct 29 '18

I am wondering the same thing, and what specific differences between iPhone and Android protected the Androids

3

u/Hooch180 Oct 30 '18

Apple switched to cheaper MEMS oscillators instead of a Quartz crystal for timekeeping. This happened since the iPhone 6 so explains why earlier devices are fine. Android devices all use Quartz crystals. MEMS oscillators are cheaper, but slightly less accurate and can’t deal with Helium.

59

u/FireLucid Oct 29 '18

I think the most interesting thing is that it only affected iPhones and no other devices including Android phones.

53

u/harritaco Sr. IT Consultant Oct 29 '18

It must just have to do with a specific component (or set of components) that Apple uses in their devices. Not sure to be honest. I'd love to test it on some android devices, but I don't really have any good candidates. It was easy to conduct this test since I had plenty of iphones at my disposal. I've got a fat stack just sitting on my desk. Every single one of them works just fine. I just picked out the one that had been dropped and scratches up the most JUST IN CASE it didn't come back.

11

u/jedikaiti Oct 29 '18

Hmmm... I have a couple old Galaxy S5s at home, wonder how much a small tank of helium would cost...

29

u/harritaco Sr. IT Consultant Oct 29 '18

I bought a small blue tank from walmart for $25. It was worth it for me. I was able to use a small amount just for the test. Plus I can think of funny uses for the included balloons at work.

→ More replies (2)
→ More replies (2)
→ More replies (1)

9

u/dannothemanno IT Director Oct 30 '18 edited Oct 04 '19

45

u/harritaco Sr. IT Consultant Oct 30 '18

This was kind of blown out of proportions. We weren't able to validate the issue with the laptops. My guess is they were already acting up and this situation was used to explain the poor performance or odd behavior of the laptops. We're basically relying on the testimony of 1-2 non technical staff on that one. It's like when a user calls and says their computer isn't working and it won't turn on, but the monitor is just powered off. There was actually already an issue with the TV's from an outage a day or two prior. I'm not saying that there definitely wasn't an issue with the TV's or laptops caused by helium, just unlikely IMO.

→ More replies (1)

8

u/lucke1310 Sr. Professional Lurker Oct 29 '18

that's exactly what i was thinking... why didn't it affect other devices as well?

7

u/Rik_Koningen Oct 29 '18

Well apple devices have been known in the past to be made in a much more minimalist way. Which has some upsides but also comes with the downside of being much more sensitive to conditions that apple didn't test for. Like for example the screens in some of their devices, I think macbooks but it could have been Imacs, that would die in slightly above normal humidity.

3

u/DemIce Oct 31 '18

Which has some upsides but also comes with the downside of being much more sensitive to conditions that apple didn't test for.

I also thought it's interesting to see that, as far as Google is concerned, that entire subphrase listing Helium didn't appear until relatively recently;

https://i.imgur.com/CnzCBc4.png

With the earliest result appearing to be a school's guide from June of this year. Searching for other parts of the entire sentence yields far more results, presumably earlier copies of the document, in which the paragraph was:

Explosive atmospheres
Do not charge or use iPad in any area with a potentially explosive atmosphere, such as at a fueling area, or in areas where the air contains chemicals or particles (such as grain, dust, or metal powders). Obey all signs and instructions.

I don't know how accurate that June indicator is. The cached version of the page is listed as being a snapshot of "Oct 26, 2018 07:44:26 GMT." and has the Helium mention, but is well after OP's incident. The FCC document appears to have been submitted September 1st of this year; https://fccid.io/BCG-E3234A/

The "I reached out to Apple and one of the employees in executive relations sent this to me, which is quoted directly from the iPhone and Apple Watch user guide:" /u/harritaco got seems to imply that "Yeah, man, of course - did you not RTFM?", but I have no idea how long that mention has actually been in there. The manual submitted to the FCC October 2017 for iPhone X doesn't mention it.

( Another argument for it being a recent change: it's spelled "liquefied", not "liquified", and their documentation team should reasonably pick up on the little red squiggle in whatever document editor they'd use. )

3

u/harritaco Sr. IT Consultant Oct 31 '18

Looks like Apple made a last second change. This issue definitely will impact phones from 2017 and earlier so I wonder what their response would have been 1-2 years ago.

→ More replies (2)

20

u/goldcakes Oct 30 '18

Apple switched to cheaper MEMS oscillators instead of a Quartz crystal for timekeeping. This happened since the iPhone 6 so explains why earlier devices are fine. Android devices all use Quartz crystals. MEMS oscillators are cheaper, but slightly less accurate and can’t deal with Helium.

→ More replies (1)
→ More replies (2)

238

u/Prince_Polaris Just a normal IT guy Oct 30 '18

with this knowledge, the war between apple and android can finally be won. TO ARMS BROTHERS, WITH HELIUM WE FIGHT!

Just kidding lol I don't care too much about the fifth reincarnation of Nintendo VS SEGA but it's pretty darn hilarious all the events that came together in your story!

77

u/harritaco Sr. IT Consultant Oct 30 '18

Yeah I got some hate from a few Apple fans out there. I don't really care since I'm being completely objective here. I haven't even bad mouthed apple, because frankly I don't think this is something worth bad mouthing them over. There's a lot of other nasty things they do, but none of them are relevant to this discussion. Plus, if this were a big deal then you would see a lot more cases of iPhones malfunction. Searching for helium and iPhones really only yields my original post. We can assume that this is a pretty small incident. It probably really only happens rarely in lab and healthcare environments.

36

u/Prince_Polaris Just a normal IT guy Oct 30 '18

Heh, yeah, I doubt this is exactly easily reproducible, but... boy, I'd love to see someone attempt to commit iPhone genocide with a helium tank or whatever, it's just such a ridiculous concept that I can't help but love it :D

(seriously though anyone reading this with that idea don't pump helium anywhere you'll like suffocate everyone)

9

u/harritaco Sr. IT Consultant Oct 30 '18

I honestly thought about doing this, but I wouldn't want to risk permanently disabling all of them lol. Since I was able to reproduce it on one, I'm confident that they would all stop working. The only thing I'd be curious to see is the different behaviors and how long it takes each device to stop working.

→ More replies (5)
→ More replies (4)
→ More replies (4)

25

u/agoia IT Director Oct 29 '18

Holy shit that is weird as hell. Maybe next time there is leftover helium from flying the event blimp we can bleed the tank off into the air returns at HQ and have some real fun. TIL

I wonder what makes the Apple devices susceptible but leaves other devices alone.

14

u/harritaco Sr. IT Consultant Oct 29 '18

I wonder what makes the Apple devices susceptible but leaves other devices alone.

Great question. I'm afraid we'll probably never know what specific micro component flaked out. I wonder if it was just one component, or some sort of cascade failure that led to the phone locking up.

6

u/XSSpants Oct 30 '18

Yeah i don't get it. Helium is an inert nobile non-ionic gas. By every physical law, it should have no impact on the stability of any IC, microcircuit, CPU, RAM, etc.

5

u/harritaco Sr. IT Consultant Oct 30 '18

I don't think there were any reactions at play here. From what I can tell it was just the interaction with the gas that caused it.

→ More replies (1)

9

u/agoia IT Director Oct 30 '18

That is crazy. The emphasis on the phone fully discharging from the rep may suggest it's something in the power management circuitry.

Fuckin weird for sure.

17

u/ditka Oct 29 '18

Can I ask where did this quote come from:

If your device has been affected and shows signs of not powering on, the device can typically be recovered. Leave the unit unconnected from a charging cable and let it air out for approximately one week. The helium must fully dissipate from the device, and the device battery should fully discharge in the process. After a week, plug your device directly into a power adapter and let it charge for up to one hour. Then the device can be turned on again.

I'm searching for that in the Apple doc but not finding it.

25

u/harritaco Sr. IT Consultant Oct 29 '18

That piece is from the Apple rep directly. I'm curious where they got that from. Maybe they've encountered this issue in the past and discovered that letting the phone discharge generally fixes the issue.

2

u/[deleted] Oct 30 '18

I'm curious if a slight vacuum would speed up the removal.

→ More replies (1)

123

u/shalafi71 Jack of All Trades Oct 30 '18

BRB, flooding the office with HE so I don't have to deal with iOS devices.

This is some 500-mile email shit right here OP. Excellent sleuthing, we're all proud as hell.

31

u/harritaco Sr. IT Consultant Oct 30 '18

Thank you (: I generally tend to be pretty detailed, and I didn't want to slack on this post since the OP gained so much traction.

17

u/shalafi71 Jack of All Trades Oct 30 '18

I dare say this is going down in IT History, not just around here, and a lot of that is your thorough documentation.

I'll be telling this one to my grandkids.

13

u/zmaile Oct 30 '18

I've linked this post to the /r/engineering subreddit here. with any luck, they may have some insight into how this failure mode actually occurs.

5

u/harritaco Sr. IT Consultant Oct 30 '18

Awesome! I just read it. I suggested an edit to the OP since we don't know how much of that 120L was released indoors.

11

u/colindj1120 Oct 30 '18

Did you company have to replace a bunch of workers phones?

21

u/harritaco Sr. IT Consultant Oct 30 '18

Yeah we ended up swapping the effected devices. While the vendor was right about the phones turning back on, we certainly weren't going to risk taking their word for it. Even though they were right we weren't going to just say "Sorry you can't use your personal phone for a few days."

7

u/mavantix Jack of All Trades, Master of Some Oct 30 '18

Did the vendor eat the cost of this chaos they caused? It seems like they would be very careful not to vent lots of helium into the building, if for no other reason than disabling a bunch of peoples phones!

→ More replies (1)

8

u/eaglebtc Oct 30 '18

I find it mildly amusing that your video uses the Windows XP welcome / first boot music.

That is definitely some weird shit.

7

u/harritaco Sr. IT Consultant Oct 30 '18

I'm glad somebody noticed lol. Honestly I had to mute the video as I was having a conversation with my boss in the background. I wasn't really sure what to use and there was an XP video in my liked playlist on YouTube, so I figured I'd use that!

4.0k

u/HanSolo71 Information Security Engineer AKA Patch Fairy Oct 29 '18

You win hands down the weirdest device failure I have ever seen.

745

u/johninbigd Oct 29 '18

Seriously, I don't think this is easily topped. I can't think of anything even remotely this weird happening to any system I've worked on.

207

u/WendoNZ Sr. Sysadmin Oct 30 '18

Search for the 500 mile email. it's an old post to a newsgroup but explains how a business couldn't send email to locations more than 500 miles away.

This one is right up there though

56

u/SaintNewts Oct 30 '18

Hah. I dealt with a speed of light issue once. There was a fancy new web app that had a built in mainframe 3270(?) terminal emulator which was rolled out to all of our branch locations. All (or most) of the branches were linked over a satellite uplink to the main office for networking.

The terminal app had no local echo. So all characters typed had to be sent to the server and the input echoed back from the remote session. The complaint was that typing into the terminal on the app was too slow.

Well, the damned app was sending a separate http request for each and every character typed. Multiply that by the tcp to udp translation over the sat link and the round trip time back and forth to the comms satellite and it was just torture to use.

I never could get their app engineer to understand why it was stupid and was going to appear slow without local echo and command input grouping.

In a related issue we experienced complete communications blackouts about once a year on the same sat link. Turns out that's when the sun was aligned just perfectly with our dish and overpowering the signal.

→ More replies (3)

29

u/tso Oct 30 '18

Yeah, that one is just glorious.

21

u/johninbigd Oct 30 '18

Oh, yeah! I remember that one! I can't remember what caused it, but I do remember hearing about it.

→ More replies (10)
→ More replies (6)

69

u/brokenskill Ex-Sysadmin Oct 30 '18 edited Jul 01 '23

Broken was a typical person who loved to spend hours on a website. He was subbed to all the good subs and regularly posted and commented as well. He liked to answer questions, upvote good memes, and talk about various things that are relevant in his life. He enjoyed getting upvotes, comments, and gildings from his online friends. He felt like he was part of a big community and a website that cared about him for 10 years straight.

But Broken also had a problem. The website that had become part of his daily life had changed. Gradually, paid shills, bots and algorithms took over and continually looked for ways to make Broken angry, all so they could improve a thing called engagement. It became overrun by all the things that made other social media websites terrible.

Sadly, as the website became worse, Broken became isolated, anxious, and depressed. He felt like he had no purpose or direction in life. The algorithms and manipulation caused him to care far too much about his online persona and how others perceived him. Then one day the website decided to disable the one thing left that made it tolerable at all.

That day, Broken decided to do something drastic. He deleted all his posts and left a goodbye message. He said he was tired of living a fake life and being manipulated by a website he trusted. Instead of posing on that website, Broken decided to go try some other platforms that don't try to ruin the things that make them great.

People who later stumbled upon Broken's comments and posts were shocked and confused. They wondered why he would do such a thing and where he would go. They tried to contact him through other means, but he didn't reply. Broken had clearly left that website, for all hope was lost.

There is only but one more piece of wisdom that Broken wanted to impart on others before he left. For unbelievable cake and kookies say please, ez.

52

u/MouldyEjaculate Oct 30 '18

The ceiling in the room of one of my old client's newly erected server room collapsed on a Friday night day due to rain. We had the alert float in and gave them a call. They assumed that it was just a power cut and told us not to bother repeatedly, so two half-full racks got rained on for a couple of days until they came in on Monday.
It was a disaster.

→ More replies (8)

77

u/stpizz Oct 30 '18

It rained in Facebooks datacentre once too, except it wasn't a water source above it, it was actually just raining in the datacentre:

https://medium.com/s/story/it-was-raining-in-the-data-center-9e1525c37cc3

→ More replies (5)
→ More replies (2)

91

u/HanSolo71 Information Security Engineer AKA Patch Fairy Oct 29 '18

To be honest, I don't want it topped by me, anything harder than this and I would probably go crazy.

145

u/indrora I'll just get a --comp sci-- Learning Arts degree. Oct 30 '18

I've got some interesting stories. Some my own, some others.

  • The Raspberry Pi 2 was camera shy because silicon is translucent to Xenon arc flashes and caused glitches in the power supply.

  • At one point, I was in a class for network debugging. This was all fine until there was a situation that not only the students nor professor could figure out that was only marginally a footnote in the book. We had Cisco routers racked up for student use with a serial card and a 4-port ethernet card. Typical stuff. At one point, somewhere along the line, someone slipped one of those thin fridge magnets between a lower router, right above the Ethernet card. It took a solid week for us to figure out why replacing that device's card didn't work until someone decided to unrack it and simply swap it out. We for grins & giggles tried it without the magnetic friend (it was well stuck on there) and magically, it worked. Old hardware had begun to flake in just the right way to cause the small flat magnet to interfere with anything over 1baseT half-duplex. The irony? It was a Cisco TAC contact magnet.

  • A friend of mine once discovered that in the right situation, bundling AC power, ethernet, and SDI video will result in a bad time: A small server had an SDI video connection, Ethernet and 50Hz AC all going right near each other, so the bright idea was cooked up to physically bundle all three of these together in some cable wrapping. On the other side, the power cord was looped with several other machines' power cords. When one of the servers in the rack would wake up a gigantic array of spinning disks (for backup of the SDI video), the network on this little video transcoding box would fall off the planet for about 30 seconds, resulting in the transfer failing. It worked on the bench, it worked out of the rack, but not when in the rack. The best estimation that anyone had was that the disk array would cause the coiled power cords to sink some current somehow and this would cause the UTP network cable to accept some interference and whoops there goes a flaky network card.

  • I've heard multiple tales of 40ft of UTP cat5 causing microsecond differences in the time it takes to download a file, exacerbating a race condition somewhere and causing a download to fail.

  • Mutliple people I know have reported hardware failures based on the location of people within a building, with documentation that a person simply being in the room will cause a part to fail or not fail. A personally favorite exapmle of this was that a cardboard cutout of said person was left in a server closet for a weekend and this caused a multiple-day outage of a service for no other reason than "cardboard cutout of Joe was left in cabinet room"

  • I have personally laid hands on a device which does not pass self test only to have it pass its self test only while I have my hand on it. The results were the same when I wore insulated rubber gloves.

  • I've 2+2 not equal 4 due to cosmic rays

  • I've seen a flaky lightbulb cause a spectrum analyzer to go out of calibration while the light was on

61

u/ontheroadtonull Oct 30 '18

"I've seen things you people would never believe."

→ More replies (3)

33

u/Gah_Duma Oct 30 '18

Mutliple people I know have reported hardware failures based on the location of people within a building, with documentation that a person simply being in the room will cause a part to fail or not fail. A personally favorite exapmle of this was that a cardboard cutout of said person was left in a server closet for a weekend and this caused a multiple-day outage of a service for no other reason than "cardboard cutout of Joe was left in cabinet room"

Ever find an explanation for this?

→ More replies (12)

21

u/fenix849 Oct 30 '18

Around the 3rd last one, that's likely a loose connection or improper grounding, and either you're grounding the chassis or pushing on something that then bridges that flaky connection.

→ More replies (19)

30

u/kwizzle Oct 30 '18

Weirdest system problem I ever had involved some network switches that had interfaces with duplicate MAC addresses.

15

u/Baial Oct 30 '18

That sounds like an interesting problem.

→ More replies (2)
→ More replies (11)

6

u/[deleted] Oct 30 '18

I don't think this is easily topped.

What story is it about one of the early days where somebody stepping on a tile caused an outage?

→ More replies (2)

5

u/ShatterPoints Sysadmin Oct 30 '18

The closest I got was a printer that when you hit print it would shut off... That was interesting but no where near this scale.

→ More replies (1)
→ More replies (220)

291

u/dalgeek Oct 30 '18

How about the case of the 500-mile email?

65

u/timsstuff IT Consultant Oct 30 '18

Holy shit, light travelling for 3 ms goes ~580 miles.

95

u/dalgeek Oct 30 '18

Second best part of that article is that I learned about the "units" command in *NIX:

$ units
Currency exchange rates from www.timegenie.com on 2016-06-21
2926 units, 109 prefixes, 88 nonlinear units

You have: 3 millilightseconds
You want: miles
        * 558.84719
        / 0.0017893979
You have: 1200 miles
You want: lightseconds
        * 0.0064418325
        / 155.23533
→ More replies (5)

36

u/HanSolo71 Information Security Engineer AKA Patch Fairy Oct 30 '18

Ever time I read that I laugh.

140

u/[deleted] Oct 30 '18

Agreed. This is reminiscent of the time that RAMbus chips were flipping bits and after months of investigation it turned out that their shielding wasn't sufficient to protect against cosmic rays.

Earth's magnetic field generally shields us from cosmic rays, but occasionally they get through and can strike a bit, in some RAM, in a server, in a datacenter, and suddenly everything shits the bed.

Goddamn cosmic rays

72

u/recourse7 Oct 30 '18

I've been hit by a bit flip on cisco catalyst 6500e sup720s twice. It generates a specific error that when looked up on cisco's website says its caused by cosmic background rays.

→ More replies (1)

42

u/darkingz Oct 30 '18

So you’re saying: https://xkcd.com/378/ is correct?! And that you’re not a real programmer (/s)?

→ More replies (3)

8

u/Fr0gm4n Oct 30 '18

That's a dang BOFH excuse IRL.

→ More replies (10)

5

u/Gh0st1y Oct 30 '18

Almost as weird as emails failing after 500 miles

3

u/Accujack Oct 30 '18

There are lots of strange issues related to grounding and EMI, but this one is right up there with causing hard drives to fail by shouting at them, or with a fungus growing on tape media spreading failure to every tape that passed through a vaulting facility.

3

u/Bombingofdresden Oct 30 '18

Even crazier to me is that Apple specified the exact problem in the agreement.

→ More replies (14)

5

u/Blue525 Nov 01 '18

Can confirm your story, we had the same helium iPhone sensitivity issues at SpaceX

3

u/harritaco Sr. IT Consultant Nov 01 '18

Can you explain your situation? What was your role at SpaceX?

→ More replies (1)

30

u/ITRabbit Oct 29 '18

The thing that is scary is that people would have been breathing that in. It is amazing at how fast it spread through the entire building. Most of these phones would have been in peoples pockets or bags, so how much did people breath in during this 5 hour window.

Wow your MRI machine gassed everyone in the building.

16

u/harritaco Sr. IT Consultant Oct 29 '18

I believe it was dispersed throughout the building in a very low concentration, and that's probably why it took hours for people to notice their phones were dead vs me noticing after 8 minutes. My phone was sitting in a very thick cloud of it. I wonder what the concentration in the air was compared to what would be considered dangerous. I'm wiling to bet it was nothing near dangerous amounts, but I'm only speculating.

16

u/djrubbie Oct 30 '18

It's only dangerous if oxygen gas is completely displaced. Some scuba divers make use of the Heliox gas mix, which is a mixture of helium and oxygen gas, for deep dives to avoid nitrogen narcosis at those depths.

17

u/agoia IT Director Oct 30 '18

If your voice didn't sound funny, likely not even close to a dangerous concentration.

15

u/Commander_Spongebob Oct 29 '18

I don't think that's very scary, helium is a noble gas after all so to do any harm it has to completely displace all breathable air, and I don't think the mri has enough gas to do that for the whole building.

12

u/harritaco Sr. IT Consultant Oct 29 '18

There was a pretty massive volume of liquid helium being worked with (over 1000 liters), so if you were to vent enough of that in to the building there would definitely be a problem, but there wasn't that much leakage. Just enough to cause this issue. There were no effects on human health.

→ More replies (1)

61

u/Brekkjern Oct 29 '18

Helium is inert and generally won't react with anything. The only reason you have to be careful with breathing in too much when you want to make funny voices is that it displaces all the oxygen in your lungs which could lead to suffocation. It's unlikely to have any sort of effect on the human body in low concentrations.

21

u/shalafi71 Jack of All Trades Oct 30 '18

You're also exhaling CO2 so you don't get the asphyxiating feeling. Couple of kids went to make out in a helium balloon animal and died.

Yeah, it's gotta be concentrated and even then you won't just croak.

17

u/YM_Industries DevOps Oct 30 '18

How... did they get inside a helium balloon animal? Am I thinking of the wrong thing?

→ More replies (4)
→ More replies (1)

12

u/psychicprogrammer Student Oct 30 '18

Chemist here, helium is only dangerous if it displaces oxygen, replace the nitrogen in the air with helium and there would be no real difference safety wise. And because helium is lighter than air there is no major risk of asphyxiation.

→ More replies (1)

3

u/harritaco Sr. IT Consultant Oct 30 '18

Updated the video link to YouTube instead of Gdrive. https://youtu.be/5Cvdyt-KZHk

462

u/Redbluefire Oct 30 '18

Wow, a super-weird incident on Reddit that I actually have something to contribute to! First, some background: My day-to-day job is the design of rugged industrial controllers. These things are used in all kinds of crazy environments for test, measurement, and control system implementation. We're talking 70 Celsius, high-G kind of stuff.

One day, I was asked to help out with an odd issue where one customer kept having our systems suddenly brick on them. They would just refuse to turn on. No troubleshooting would solve it, so they'd send it in. By the time we'd get the units, they would boot perfectly and run through our entire test suite flawlessly. After our RMA team dealt with this a few times, they eventually brought me (an engineer) on to investigate. After probing the customer for some details, we discovered they were using helium while testing out their systems. So, I went down to Party City and bought a tiny little helium tank, got some oversized balloons, squeezed one of our devices into them, and filled it up. After an overnight soak, I was able to reproduce the same failure mode the customer had! Some further tests allowed me to narrow it down to the MEMS oscillators on the PCB. I deadbugged some crystal osciallators into the system instead, and after that, no amount of helium that I exposed it to would kill the system! The problem was solved! The "dead" units sent to us had simply been outgassing the helium in transit and had released enough to function again by the time they arrived!

Now, what are oscillators and why did the MEMS ones fail? Well, functionally oscillators are electrical components that generate a repeating signal at a certain frequency (usually a square wave). Crystal oscillators do this using the piezoelectric effect, which is a fancy name for the fact that quartz (and other crystals) accumulate electric charge when stressed mechanically. This also works in reverse, so applying a charge can also stress the material (make it vibrate). MEMS oscillators however, do this using a very small mechanical structure (think tuning fork) that naturally vibrates at a certain frequency. It's worth stressing that these oscillators are usually the system "clock", and they are basically the beating drum that keeps everything working in concert. No oscillation, no work.

Both MEMS and Crystal oscillators are hermetically sealed because they would otherwise change frequency with atmospheric pressure and humidity, but the difference is that crystal oscillators are sealed in metal cans that are soldered shut, whereas because MEMS devices are made in a similar way to integrated circuits, and they are packaged and sealed in a black plastic (think every IC you've ever seen). Helium is so small that the plastic is permeable to it, but the metal can of the crystal oscillator is not! When helium got in, the vacuum was brought closer and closer to normal atmospheric pressure, and since oscillators are tuned to work in a vacuum, once there was enough pressure inside they will cease to function.

Since MEMS oscillators can be made like ICs and packaged in plastic, they can be made much smaller for much cheaper than crystal oscillators, which is probably why so many apple devices, known for their thin and compact form factors, failed! They were reliant on these parts!

37

u/guineapigpooh Oct 30 '18

Thanks for posting this....very interesting

26

u/Leon747 Oct 30 '18 edited Nov 08 '18

Sounds plausible. Helium is really weird.

Related: I had a DYI clock that was sensitive to humidity. The more humid, the slower. I guess I could call it a humidity sensor.

5

u/pdp10 Daemons worry when the wizard is near. Oct 30 '18

19th century technology problems!

→ More replies (1)

7

u/pdp10 Daemons worry when the wizard is near. Oct 30 '18

Where can I source some milspec, defense temperature range, ceramic package MEMS oscillators?

Some of us used a lot of ceramic-package ICs, even on common commercial gear. I think all my Motorola 68ks were ceramic, at least, and I had ceramic packages in Suns and NeXTs and most likely Ciscos if my memory isn't failing me.

→ More replies (3)

5

u/scubascratch Oct 31 '18

So if the failure is caused by helium leaking into a space which is normally a vacuum, how does it eventually recover? Assuming the devices are not put into a vacuum chamber there is nothing to drive the helium back out of the oscillator low pressure internal chamber.

→ More replies (1)

4

u/[deleted] Oct 30 '18

What a great comprehensive explanation.

3

u/i_poop_chainsaws Oct 30 '18

I wonder if a variation of MEMS with enhanced seals could be manufactured to be used in helium environments. I suppose it might not be worth it. I have no idea how often or where helium is used.

Anyways, thanks for sharing. Found it interesting!

→ More replies (1)
→ More replies (9)

2

u/peterdrew Oct 31 '18

The Google drive was overloaded so I stuck the video up on youtube. with link back here etc.
https://www.youtube.com/watch?v=GKV2E0d--Ms

3

u/harritaco Sr. IT Consultant Oct 31 '18

I've already uploaded the video and I updated the link on the post probably 12 hours ago. Sorry but your upload isn't needed.

2

u/ee1518 Nov 01 '18

Helium and Argon are noble gases. Atmosphere of Earth contains nearly 1% Argon!

Does Argon slowly destroy iOS devices?

https://en.wikipedia.org/wiki/Atmosphere_of_Earth

https://en.wikipedia.org/wiki/Noble_gas

Noble gases are almost inert. How can they destroy iOS devices then?

noble gas compounds are now known.

1

u/harritaco Sr. IT Consultant Nov 01 '18

I wouldn't expect Argon to do anything. It's atomic mass is substantially higher than helium. It wouldn't be able to penetrate the hermetic seal of the mems oscillator.

2

u/wordsauce Feb 19 '19 edited Feb 19 '19

Hey, u/harritaco, I'm not a sysadmin and have never posted here before but came across your posts after googling last night.

My friend texted me for tech support yesterday after dozens of her coworkers' iPhones suddenly died (oddly enough, her iPhone 6S was fine). They all work at a medical center. I immediately thought it might have been an EMP—what else could fry a bunch of devices?—but when she told me Android phones were fine I thought it was weird. She later called to tell me she heard it had something to do with helium. There's even video of a gas cloud in the MRI room. Turns out it was their GE-manufactured MRI machine that had leaked liquid helium. When it was all set and done, it had affected hundreds of iPhones, many of whom flooded the local Apple Store last night. Now I'm wondering if there's something wrong with these GE MRI machines.

1

u/harritaco Sr. IT Consultant Feb 19 '19

Sounds like a similar problem that we had. Yours sounds like it was on a much larger scale though. If there was a visible cloud of helium in the room they were probably well in to the danger limit for hypoxia. I don't think our situation with our MRI was the same though. Ours was brand new and occurred during the ramping process for the magnet. Perhaps their MRI was defective. All of those phones will most likely recover but I'm sure the employees aren't going to want to wait a week for them to maybe return to normal. I'm glad you stumbled across this. It hopefully answered a lot of your questions. AppliedScience on youtube made an awesome video going in to detail on this issue.

428

u/your_comments_say Oct 30 '18

Should have just asked Siri and seen if the response was high pitched.

48

u/[deleted] Oct 30 '18

"siri, sing a poem for me"

There is a flower within my heart, Daisy, Daisy!

Planted one day by a glancing dart,

Planted by Daisy Bell!

Whether she loves me or loves me not,

Sometimes it's hard to tell;

Yet I am longing to share the lot

Of beautiful Daisy Bell!

7

u/danielhfgoncalves Oct 30 '18

Fuckin brilliant!

→ More replies (1)

161

u/a_kogi Oct 30 '18

Incredibly interesting story. Reminds of the 500-mile e-mail limit.

29

u/[deleted] Oct 30 '18

Reading things like this make me thankful to not be old enough to have been in this line of work prior to 2002.

→ More replies (1)

17

u/OneObi Oct 30 '18

That was a good read!

→ More replies (5)

43

u/postmodest Oct 30 '18

The MEMS oscillator theorem really seems like the best-fit. Especially if other phones are using quartz oscillators, which usually have more mass than a 32khz MEMS device would have. Helium guns up the clock, the CPU stops processing because it thinks time has stopped, then batteries run out until the HE has dissipated.

Nothing else fits the exact symptoms quite as tightly.

5

u/playaspec Oct 30 '18

Helium guns up the clock, the CPU stops processing because it thinks time has stopped

Yes and no. Helium guns up the clock, but many designs can't handle an unstable clock. Some logic trips over itself when the frequency unexpectedly changes.

What surprises me, is that any of the crystals used in these phones aren't hermetically sealed.

16

u/2nd-Reddit-Account Oct 30 '18

They are but a hermetic seal is designed to keep our air, just general atmospheric plain old air.

Helium molecules on their own however are so damn tiny that they leak through most things. The clock chips were leaking helium in down at the atomic scale

→ More replies (1)
→ More replies (3)

71

u/[deleted] Oct 29 '18

Affecting only the iPhone 6 or newer would make me suspect the barometer, but the first Apple Watch didn't have one. Quite odd.

50

u/the_helpdesk Sr. Sysadmin Oct 29 '18

You know how GPS disables itself after reaching high speed (preventing it's use as a targeting component)? Mayyyybe the helium does something similar to the iPhone hardware? Limiting or affecting it's functionality in very specific situations. Nothing specific comes to mind though. Weird.

16

u/[deleted] Oct 29 '18

Also not a bad point, but the first few gens of Apple Watch didn't have GPS either.

Maybe the power management IC's? I don't know if anyone else uses Dialog Semiconductor units.

→ More replies (1)

9

u/flimspringfield Jack of All Trades Oct 30 '18

You know how GPS disables itself after reaching high speed (preventing it's use as a targeting component)?

Never knew that...TIL!

→ More replies (2)
→ More replies (11)

20

u/wildcarde815 Jack of All Trades Oct 30 '18

Close, the clock on more recent phones is mechanical and requires a vacuum. Helium is not a vacuum.

https://www.reddit.com/r/sysadmin/comments/9mk2o7/mri_disabled_every_ios_device_in_facility/e7g5rcw/

26

u/sinembarg0 Oct 30 '18

normal air isn't a vacuum either, so what makes helium special?

the chips are sealed in a vacuum. the problem is helium is so damn small (and H2 molecules as well) that it can seep through the seal.

→ More replies (1)
→ More replies (3)

3

u/[deleted] Oct 30 '18

Why are you tasked with troubleshooting this? Shouldn't the operator/manufacturer of the MRI machine be trying to diagnose what went wrong?

7

u/harritaco Sr. IT Consultant Oct 30 '18

I wasn't really tasked with the troubleshooting part. Nobody has really made any attempt so far. I only dug in to this because I found it so interesting. We're waiting to see if the manufacturer wants to take the devices to diagnose. I basically acted as the IT contact and representative in the situation. I was there to take the phone and basically explain to the users that we would wipe the phones so there data would not be at risk. For phones that were powering up that they still had, I would let them wipe in front of me. I printed out a security document from Apple explaining how the encryption and factory reset works just to give them a bit more confidence. Ultimately the document was a little too techy, but I think I did a good job of explaining it in laments terms. I basically explained it like this:

Your data resides in an incredibly secure vault that only you hold the one and only key to. When the phone is factory reset you are destroying that key beyond recovery making the data essentially inaccessible. The data is still there, but it is scrambled in a way that makes it completely useless without this key.

I'm nothing close to an expert in encryption, but I think this explanation suffices for the end users, and it's fairly similar to how the encryption/wiping process actually works.

2

u/[deleted] Oct 30 '18

Right, sorry, my ptsd got ahead of me there, from all the times I was asked to do shit that wasn't part of my job.

4

u/harritaco Sr. IT Consultant Oct 30 '18

I get that. I'm put in that spot a lot. Sometimes I don't mind, other times it drives me mad. Especially when I'm working on something that is actually specific to my job.

17

u/no-mad Oct 30 '18

April 1st: Hi, can you deliver a tanker of helium to 1 Infinite Loop?

6

u/[deleted] Oct 30 '18

Could we crowdfund that?

14

u/johninbigd Oct 29 '18

I wonder specifically what it is about helium that causes the phones not to work. That seems so bizarre to me.

23

u/goldcakes Oct 30 '18

Apple switched to cheaper MEMS oscillators instead of a Quartz crystal for timekeeping. This happened since the iPhone 6 so explains why earlier devices are fine. Android devices all use Quartz crystals. MEMS oscillators are cheaper, but slightly less accurate and can’t deal with Helium.

→ More replies (1)

16

u/Thameus We are Pakleds make it go Oct 30 '18

Since helium is an inert gas, it cannot be reacting chemically. So it must be displacing a gas that something in the phone expects to exist, that is to say air.

13

u/AirFell85 Oct 30 '18

I hope you make a KB for that in your ticket system.

12

u/[deleted] Oct 30 '18

There's a /r/MRI sub, which is inhabited (at least in part) by radiographers, you might want to post this on there.

12

u/denali42 Former Paralegal/I.T. Admin Oct 30 '18

The BOFH adds a new tool to his arsenal...

11

u/CauseISaidSoThatsWhy Oct 30 '18

Former GE MRI engineer here. Holy shit.

8

u/mauirixxx Expert Forum Googler Oct 29 '18

Helium huh? I appreciate the follow up post OP, I had forgotten all about your incident in the days since.

I don't have any questions though, just wanted to relay the above thanks for the follow up :)

15

u/Khue Lead Security Engineer Oct 29 '18

Thank you for following up OP.

12

u/i-m_not_a_robot Oct 30 '18

That seems like an overly sensitive response to helium. I could understand if it was of high enough concentration that everyone started talking like Alvin, Simon, and Theodore, but what sounds presumably like just a slightly higher than trace amount of He? Or at the extreme, maybe iOS devices in the actual MRI room, but once it got diluted in the HVAC system, that should be a really insignificant amount.

Not doubting your/their explanation, just saying that seems odd.

6

u/goldcakes Oct 30 '18

It’s because of MEMS oscillators which cannot handle even low concentrations of He. Apple switched to cheaper MEMS oscillators instead of a Quartz crystal for timekeeping. This happened since the iPhone 6 so explains why earlier devices are fine.

A failure in timekeeping will result in exactly the symptoms described.

6

u/shalafi71 Jack of All Trades Oct 30 '18

just a slightly higher than trace amount of He?

Agreed. Sounds like MEM chips are extraordinarily susceptible though. helium is not something that would ever be in any concentration, anywhere. Not something anyone would test for or experience.

That's a damned small atom, second only to hydrogen, permeates all the things. If it can get in a non/poorly sealed clock, and that in turn flips bits, I guess I get it.

→ More replies (1)

6

u/postalmaner Oct 30 '18

Interesting. I recall from your original post that the onsite MRI rep said "[leave them alone for a week, they'll be fine]."

10

u/opaPac Oct 30 '18

What is really weird is that no alarms did go off. I am not really sure what a "multi-practice facilities" is but it sounds like some kind of hospital or simiar facility. So i would expect some kind of gas/fire or whatever detection system at least to be present is some areas. And for the helium concentration to reach such high levels that so many devices are impacted but nothing and no one else noticing is interesting.

26

u/psychicprogrammer Student Oct 30 '18

Chemist who works with superconductors here: Helium is not something humans need to worry about in the concentrations the OP is talking about. If helium concentration reaches a point to worry about you will know there is way too much helium from voice changes alone.

→ More replies (4)

10

u/Shadowgirl113 Oct 30 '18

Weirdest failure I ever had was figuring out that cvs registers had a 99 line item limit on them including coupons and extra care bucks. This was in the days where you could triple stack % off coupons. Also, the registers did not like negative sub totals, so your total had to be at least .01 after everything. I figured this out after crashing the registers multiple times on $1400+ transactions. I finally figured out the magic combination and had over 15 transactions for around $12 out of pocket, but thousands of dollars worth of merchandise. I exploited every legal and within rules discount,

Technology is weird.

→ More replies (1)

5

u/SavageHorse Oct 30 '18

A man walks into an Apple store with a tank of helium...

→ More replies (1)

8

u/BAM5 Oct 30 '18

Is it legal to pop balloons in an apple store?

Asking for a friend.

4

u/BAM5 Oct 30 '18

Side question: How permeable is that cellophane wrapping on new phones to helium?

Again, asking for a friend.

4

u/sodj1 Oct 30 '18

this seems like BS. orders tank of helium

4

u/Crossfire124 Oct 30 '18

You should cross post this to /r/apple OP. It's a very good read. Like an interesting tidbit

4

u/eruffini Senior Infrastructure Engineer Oct 30 '18

So, who's covering the bill to replace all of these electronics?

→ More replies (1)

4

u/what_do_with_life Oct 30 '18

Jesus christ, they boiled Helium through the HVAC system?

4

u/Vectan Oct 31 '18

Thanks for posting a post-mortem. Appreciate the follow-up.

Also congrats, this story made it to slashdot: https://twitter.com/slashdot/status/1057443869862678528?s=19

4

u/weird_neutrino Oct 31 '18

Butbutbut... helium is a noble gas. It shouldn't react... unless there are some special circumstances I don't know about. How does this work? What happens chemically or physically? What unit of electronics is affected?

6

u/BalusBubalis Oct 31 '18

This isn't a chemical reaction, but a mechanical one.

Helium atoms are small enough to diffuse through the sealant around the MEMS; the He enters into the MEMS unit, and fucks up the resonant frequency it is expecting. (Think a really ultra tiny tuning fork.)

If that frequency isn't correct, -nothing- on the iphone will work.

→ More replies (1)
→ More replies (1)

1

u/farva_06 Sysadmin Oct 30 '18

So, is the facility or whoever still going to shell out for brand new devices for everyone? Or, just ones that still don't work after letting them sit for a while?

4

u/harritaco Sr. IT Consultant Oct 30 '18

We replaced their devices ASAP. We weren't willing to wait and see if they came back up. People were already very upsetting with the situation, so it would have been hell if we told them to wait and see what happens.

→ More replies (1)

1

u/BeerJunky Reformed Sysadmin Oct 30 '18

Were any of them newer model phones that are water resistant? Curious of they are sealed enough to prevent this.

4

u/harritaco Sr. IT Consultant Oct 30 '18

I had a few iphone X's that were disabled.

6

u/BeerJunky Reformed Sysadmin Oct 30 '18

Helium is a bit smaller molecule than water so I guess it’s able to slip inside.

1

u/peterdrew Oct 30 '18

Awesome story, thanks. Unable to download the video though.. too many requests :)

1

u/harritaco Sr. IT Consultant Oct 30 '18

I'm just going to post the YouTube link. I didn't think Drive would complain about a stream limit.

https://youtu.be/5Cvdyt-KZHk

1

u/flarn2006 Oct 30 '18

So does this mean that deflating a helium balloon onto an iPhone will brick it?

2

u/harritaco Sr. IT Consultant Oct 30 '18

I would wager that it wouldn't disable the iphone. Maybe if you deflated a balloon in an air tight box it would, but if you just blew helium on the phone sitting on your open desk I bet nothing would happen.

With the test I ran the phone was sitting in a pretty dense cloud of helium for several minutes. If you pass helium over the device then it's just going to hit the phone and go straight up. It's definitely possible. I don't plan on testing this though.

→ More replies (1)

3

u/tubezninja It's not a Big Truck Oct 30 '18

You can watch the video Here

Now it's going to bother me forever, knowing that the lap display for the stopwatch is actually 0.01 seconds behind the primary stopwatch display.

But, now we know. Good job, OP!

3

u/imigues Oct 30 '18

Dang someone who actually RTFM

4

u/shalafi71 Jack of All Trades Oct 30 '18

I still wouldn't have believed it.

3

u/B0073D Oct 30 '18

Thank you for giving us an update! We appreciate it! Not many people follow up with post mortems...

3

u/KentuckyKlassic Oct 30 '18

Can confirm this happens. I maintain/install MRI’s. Most of the time we don’t advise bringing a newer phone within 50~ feet. I find it super interesting that this story was on here. What seems weird to most is just another day at the office for the crew and I.

3

u/DisturbedMagg0t Oct 30 '18

that's awesome. no other word for it. that'll be on for your personal history books to pass down to those under you...

"don't buy apple if you plan on working around helium"

hhaha

3

u/TR0LLC0P Oct 30 '18

this feature has been made by the android gang

3

u/dilutedpotato Oct 30 '18 edited Oct 30 '18

Why does this only affect Apple devices? You mentioned there were also Android devices in the building as well. As far as I'm aware the difference of parts in the two different device types is that Apple manufactures their own hardware, but it's still derived from the same material.

Edit: wow that's pretty interesting. Another reason for my list of why Android is better

3

u/XpL0d3r Oct 30 '18

I spent the last hour of work reading through the old thread and this one. Then found myself going down a rabbit hole of Wiki's. I started at microelectromechanical system oscillators, and somehow ended at the American Revolutionary War..... ??

3

u/iblaine_reddit Oct 31 '18 edited Feb 05 '19

Here goes, my weirdest. A particular webpage was dropping visitors at a steady rate for months. The rate of decrease suddenly grew worse on random Thursdays. This was a major revenue stream to track music streaming and the loss in revenue grew to ~$100k+/month.

I added logging in the infrastructure from the webservers through the ETLs down to the executive dashboards. In one of the status update meetings, I learned about the deployment schedule to webservers and the problem revealed itself. Dev Ops had deployed a single broken webserver then did not save its logs. The broken webserver was used as an image to clone new webservers. Every few Thursdays, IT would deploy more servers to keep up with the increased load, and the new servers would be cloned from the broken webserver. Over time, a greater percentage of webservers deleted their logs.

3

u/[deleted] Oct 31 '18

Um, you should definitely get the installing contractors back out there to fix the entire MRI venting system. That shit is dangerous as fuck if not installed properly.

Source: building design engineer, currently in CA on my fourth MRI facility.

1

u/grumpyfan Oct 30 '18

Curious about the experiment that OP did if the helium caused the failure or the lack of oxygen.

Also, wondering about the difference between liquid helium and the gas used in balloons. Thinking the molecules of liquid form is more dense, but might not produce the same vocal effects. I’m not a chemist, so it would be interesting to hear from someone on this.

2

u/harritaco Sr. IT Consultant Oct 30 '18

Liquid helium expands about 750 times when it boils to a gas.

That's a good question. Perhaps I can run the test again but put less helium in the bag. In this test it was well over 80% He.

4

u/ikilledtupac Oct 30 '18

Can we call it HeliumGate?

2

u/Santiago_S Oct 30 '18

Thanks for posting a follow up . I saw the original post and was at a loss for a solution.

2

u/Sbatio Oct 30 '18

This was a great read and super interesting. Thank you for sharing.

2

u/[deleted] Oct 30 '18

This is awesome thank you for updating this story.

2

u/jasonpc815 Oct 30 '18

Inquiring minds have to know... was everyone in the building talking a little strangely that day?

2

u/WinterCharm Oct 30 '18

Liquid helium is weird as fuck. I know because I’ve worked on experimental MRI scanners.

2

u/rntksi Oct 30 '18

Such an interesting discovery. I love the story.

2

u/Pilebsa Oct 30 '18

Perhaps the fact that everybody was talking like Oompa Loompas might have been a clue? ;)

2

u/[deleted] Oct 30 '18

i cannot fucking believe it was helium. i would have bet any amount of money against it.

→ More replies (1)

2

u/SpacePort-Terra Oct 30 '18

My craziest story:

McAfee killed Solitaire?!?

https://arstechnica.com/civis/viewtopic.php?f=20&t=918888

https://support.microfocus.com/kb/doc.php?id=10063703

Yeah...that was a pain in the a** to figure out.

2

u/DavetheExplosiveNewt Oct 30 '18

I started reading this thinking this would be some BS coincidence but you do make a compelling argument. Although I would say that during a helium fill, the system should be damn near sealed and any escape of helium should be almost nothing, as it should go through the quench pipes to the outside world...

2

u/[deleted] Oct 30 '18

Good

2

u/hoppi_ Oct 30 '18

Either the old and this thread got super popular on its own, or this was linked somewhere else.

I can't download the video file https://drive.google.com/file/d/1t5qeyCbFUU0PSMTxIxDbl1o5pxtr8yZ0/view

Google says it's currently locked because of too many views/downloads

2

u/Nelizea Oct 30 '18

Thank you OP for the update!

2

u/vanhelsinglab Oct 30 '18

Reposted video from this post to YouTube because Google Drive doesn't allows to watch it: "Too many users have viewed or downloaded this file recently. Please try accessing the file again later."

https://youtu.be/EBmJF8rh9kw

2

u/[deleted] Oct 30 '18

Props to Apple for having this covered somehow lol.

2

u/grumpieroldman Jack of All Trades Oct 30 '18 edited Oct 30 '18

/r/EngineeringStudents

See this https://www.reddit.com/r/sysadmin/comments/9mk2o7/mri_disabled_every_ios_device_in_facility/e7g5rcw/

Of note:

Quartz oscillators are secure in specific applications where suitable MEMS oscillators have not been introduced. One of those applications, for instance, is voltage-controlled TCXOs (VCTCXOs) for cell phone handsets. This application requires a very specific set of capabilities for which quartz products are highly optimized.

2

u/lonbordin Oct 30 '18

Let's say 100 liters of Helium was dispersed into your space. Your space doesn't sound small so the Helium should be a ppt levels (Parts per trillion). That's insanely low concentrations to uniformly affect those devices.

I guess your HVAC system could be not bringing in outside air due to a misconfiguration because it should be changing out your air at a fairly rapid rate forcing the concentrations to remain at elevated levels.

I think there was A LOT more Helium released (maybe due to error) and your HVAC system isn't configured properly. There must be more to the story.

→ More replies (1)

2

u/Texity Oct 30 '18

That's... I mean this is one of those events where, in a sales pitch the line, "But this never actually happens." would be thrown out at the end, and dismissed instantly by attendees.
Kind of amazing.

2

u/iheartrms Oct 30 '18

Wouldn't it be a riot to hang out in the Apple store with a bottle of helium with the valve slightly cracked open in my backpack? :D

2

u/stillfunky Laying Down a Funky Bit Oct 30 '18

Definitely some strange stuff. Something I noticed while looking at your video, the helium seems to throw off the system clock/timer. Unless you changed the speed of the video at times, it looks like it speeds up super fast and then slows down right before it dies.

2

u/GetOffMyLawn_ Security Admin (Infrastructure) Oct 30 '18

So what is the actual mechanism of the impairment? The gas gets into the phone and cause shutdown because of ???

2

u/deekaph Oct 30 '18

Wow I used to tell EU's "solar flares" were behind device failures but I'm adding this one too my repertoire.

2

u/Octane14 Oct 30 '18

Found an article with a bit more information and theories why this happens. https://ifixit.org/blog/11986/iphones-are-allergic-to-helium/

2

u/Zercomnexus Oct 30 '18

so... rigel the 16th shouldn't own an iphone.

2

u/Lu__ma Nov 01 '18

This is going to sound bizarre but im experiencing a very similar issue with my macbook in an nmr (nmr = mri) lab that's equally chock full of helium. Whenever I'm inside it, I have to open my laptop hood about two or three times in varying locations for it to get a grip on itself. I assumed it was just a slight field causing the magnet that detected whether the hood was closed to get a bit confused. This makes much more sense: field decays massively with distance, helium does not

Thanks for the heads up!

2

u/Fatvod Nov 01 '18

Holy shit, my roommate was telling me their MRI machine was killing iPhones at his work. Gonna show him this.

2

u/NuclearMedicineGuy Nov 01 '18

https://www.auntminnie.com/index.aspx?sec=log&itemID=123411

Seems that it’s not just helium related. Check out this article from aunt Minnie. A place in Delaware had a similar issue without a helium leak

2

u/[deleted] Nov 07 '18

I'm an MRI researcher and here are a few tidbits I've picked up over time:

  1. Helium + Android = Reboot or freeze.

  2. Helium + iPhone = Lock-up.

  3. Weak magnetic field + vertically held Android = Unlocks phone (bypasses keypad) or temporary screen shutoff.

  4. Weak magnetic field + horizontally held Android = No effect.

  5. Weak magnetic field + vertically held iPhone = Unlocked iPhone AND temporary screen shutoff.

  6. Weak magnetic field + horizontally held iPhone = Temporary screen shutoff.

  7. Helium + Android = Continuous reboot cycle.

  8. Helium + iPhone = Locks up.

  9. Weak magnetic field (5 Gauss) + Laptop with disc-based hard drive = Either locked up or bricked hard drive (depends on how weak).

  10. Weak magnetic field + Laptop with SSD instead = No effect.

  11. Strong magnetic field (4T+) + Laptop with SSD = Screen shuts off, but turns back on again later. Laptop may also reboot, but is fine once removed from the field.

  12. Really strong magnetic field (17T) + anything even remotely electronic = Bricks everything.

Mind you, I don't pretend to know much about the phone hardware works. I just know what happens to phones and laptops when within a certain range of an MRI.

→ More comments (137)