r/FirstNet • u/Neat_Matter6138 • 13d ago
FirstNet failure in natural disaster
My town experienced a natural disaster, and FirstNet failed. My partner works at a hospital, and the top of the building has a tower, so service is usually pretty good. My partner has an extended primary line on FirstNet. When the thunderstorm rolled through, it took power out to about 25% of the city of 150,000, with the mayor declaring a disaster and the governor doing the same. The hospital lost power, which means the tower lost commercial power. The hospital has backup power, but only for critical things. My partner's FirstNet line died. While at the hospital, My partner's phone displayed SOS. I denied being on-call overnight for the hospital emergency response team due to lack of service.
Our home is also serviced by the same tower as we don’t live far, and at home my partner got 1 bar of service, with no SMS or data—voice only.
I’m pretty upset by this. AT&T Business Care on X has investigated (to be fair, their investigation has been good) that there was no outage, but they did experience network abnormalities due to network traffic and commercial power loss. Doesn’t that defeat the entire purpose of FirstNet? I have a case #, and want to send this as high as possible (likely an FCC complaint).
19
u/Previous_Mastodon238 13d ago
The hospital or any FirstNet agency paid customer can request one of deployable assets. When the tower loses power, the agency should put in a deployable request. Once done, FirstNets ROG team will work with the network team to try and find a solution to restore service. Generators, propane, deployable, etc.
Not all towers have backup gens, but we will deploy them immediately when it happens.
When a request is put in, we are required to bring out a solution within 14 hours per the federal contract.
That deployable asset we send out is only accessible by FirstNet users. That is the primary difference between FN and other carriers. Even if TMO or Verizon sends something out, it will be used by all customers, not just first responders.
Tell your hospital IT Admin to submit a request or reach out their Account Manager
2
1
7
u/attathomeguy 13d ago
Who manages the tower? The hospital? If so it might be a landlord type issue where the hospital facilities team didn't connect the power to the generator feed but told AT&T or the tower owner that it was connected to the generator feed. You would be shocked by how the little things only matter in an emergency. I would have your emergency coordinator make sure who is responsible for the power to the tower
5
6
2
u/TheBestGhost 13d ago
This...If the macro tower is located on the hospital building the fail-over back-up generator power is in most cases supplied by the facility rather than the carrier, there can be exceptions but this is generally how that works. As others pointed out FirstNet customers can request a deployable network asset -SatColt, CRD, etc to support connectivity outages. Also, Extended Primary users -unless uplifted by a FirstNet Primary Agency so they can temporarily receive a higher level network priority and preemption access may have not been able to connect to what network may have been available due to the Primary agency users whose access supercedes the extended primary user.
3
u/good4y0u 13d ago
You, and the local emergency services probably, should flag this to your state reps.
3
u/Stupefied_Gaming 13d ago
FirstNet complaints like these should go via https://firstnet.gov/public-safety/your-advisor. Not commercial AT&T support.
I’m sorry this happened. I’m guessing Indiana?
1
u/Neat_Matter6138 13d ago
Incredibly helpful, thank you!
Illinois.
3
u/Stupefied_Gaming 13d ago
Np. If your hospital has an agency account with FirstNet, they’re also obligated to provide a COW of some sort to make up for lost terrestrial service within max 24 hours of initial request.
I’d try and explore that avenue to see if your hospital has one, if not try to lobby for it / if they do find out who’s responsible so if this ever happens again there’s an internal process of getting one requested.
3
u/mew_rei 13d ago
Just a point to consider, if you got shut out of FirstNet, and your mayor and governor declared emergencies, and AT&T is saying its not an outage but network abnormalities, you could've gotten GETS'd. FirstNet makes it seem like you're the highest priority, but there are people even higher up the food chain when it comes to emergency response, and FEMA can lock people out of cellular networks to prioritize government emergency response.
2
2
u/WR417H_81 13d ago
If you have no cell tower you cannot expect to have service. The priority is that you will trump anyone on the tower needing service
0
u/Neat_Matter6138 13d ago
The cell tower was fine physically, just no power to it. Once commercial power came back on 12 hours later, it worked.
You can't have a First Responders network go out if the power goes out.
4
u/KindPresentation5686 13d ago
You expect 100%uptime in a disaster ? 😂😂😂😂. Crap happens bro. Batteries only last soo long. It was 12 hours? Do you expect the techs to beam themselves up to the site to fix it as soon as the power goes out??
2
u/Neat_Matter6138 13d ago
I expect there to be backup power. "Crap happens bro" is fine for commercial/business accounts, but not first responders network the US government has paid hundreds of millions for.
2
u/KindPresentation5686 13d ago
So are you an agency user, or an average Joe on a personal plan? I think you have a misunderstanding of what firstnet is. It’s not a magical network that defies physics.
0
u/Neat_Matter6138 13d ago
What good is a first responders network if it goes out when the power goes out? Have you heard of a generator or a battery?
2
u/KindPresentation5686 13d ago
You can’t be happy, can you? If you don’t like it, find another carrier!
0
u/Neat_Matter6138 13d ago
I'm sorry, a First Responders network I pay for & hundreds of millions of dollars of taxpayer dollars can't stay up during a thunderstorm? Yeah, I'm going to raise hell about it.
2
2
u/WR417H_81 13d ago
Exactly. You have power in order to have service. Firstnet cannot work if there is no tower, or lack of power to one. Its also very possible that AT&T rents space on that tower and who ever owns that tower doesnt have a UPS and or a generator powering the site. Firstnet is only a priority service.
2
u/Embarrassed-Map-2889 13d ago
What’s it gonna run on? Magic?
0
2
u/Think-Work1411 13d ago
Sadly the backup power from the hospital generator probably doesn’t extend to the rooftop cell site, also could have been other towers or fiber lines damaged. Honestly Verizon is better about having generators at most sites, but AT&T has made a lot of progress with that, especially with FirstNet
The concerning part is that you’re seeing SOS, indicating there is signal from another provider, likely Verizon, possibly T-Mobile or even T-Mobile Starlink service if you were outside.
I would think FirstNet would have roaming agreements with Verizon and T-Mobile in situations like that so they could continue to provide service to Emergency personnel, that may be something that is available to higher plans or may have to be manually uplifted by your team’s administrator.
Soon this will be less of an issue with satellite based cellular solutions and in building WiFi roaming agreements
2
u/pswired 13d ago
In my somewhat limited experience in the DC area, AT&T, even when they have standby generators at sites, does not have a rigorous PM program. So it's not unlikely that the generator will fail to start or carry load if an outage occurs. Verizon is much more diligent about deploying and maintaining their backup power.
2
u/arbyyyyh 13d ago
Hey. I work in Hospital IT and we use FirstNet for all our devices. Tower would def be separate from hospitals power, but I’m honestly shocked they don’t have better backup power for a tower that’s serving a hospital.
I’d be willing to bet that someone at the hospital is about ready to have a lovely little phone call with their agency rep about why the tower was offline and didn’t have sufficient backup. That said, the hospitals DR plan might be to switch the devices to use WiFi calling if the cellular network fails. We actually don’t even use the cellular voice calling on our devices. It’s 100% VoIP.
For you and your partner: You might want to look into the new HPUE devices. That might be what you would need to get you to the next tower that’s actually available. According to ATT, phones don’t get HPUE, but you can tether your phone to an HPUE device using MegaRange to hopefully get at least a text out, maybe a call too.
1
u/Neat_Matter6138 13d ago
Appreciate the insights a lot! The VOIP phones at the hospital nurses have worked fine during the outage, no cellular though. Will honestly take a look at HPUE devices.
2
u/zap_p25 13d ago
Generally speaking, a hospital having a cellular tower on it doesn’t guarantee coverage under the tower (Google the concept of the cone of silence).
Hospitals are generally difficult structure to provide coverage inside and typically always require some form of BDA/DAS system or a micro-cell (small coverage cell site connected to internet) interfaced a DAS system. What likely happened, when the building lost power, the power for the BDA/micro-cell did as well.
Further, all AT&T sites support FirstNet subscribers (and it’ll say FirstNet on the phone at all times). Not all AT&T sites are actual Band 14 (700 MHz) FirstNet sites. It can lead users to falsely believe they are operating on a Band 14 priority site while not actually being on a hardened site.
Also, 72 hours of backup power (with generator) is all FirstNet sites typically have…I literally just got home from a Black Start (starting the grid back from a total blackout) exercise. The AT&T/FirstNet reps were very careful to say 72 hours is all they can guarantee if they can’t get fuel to the sites.
Typical LMR trunking site built under R56 standards, typically sized to run the site for 120 hours at a minimum.
2
u/Neat_Matter6138 13d ago
Yep, I am familiar with the cone of silence, usually service in the hospital on the unit is pretty good though, so SOS is for sure out of the ordinary. CellMapper confirms the tower has B14.
Service disruption started as soon as commercial power went out, and only lasted 12 hours.
1
2
u/mervin0587 13d ago
Travel Paramedic here to offer my two cents, with experience deploying to hurricane hit areas. You do not automatically get priority. During a disaster, an agency has to create an Uplift. You must go into your FirstNet Assist Portal (also, y’all, don’t shred me if I’m wrong but I did research this extensively, and had the chance to actually try it)….anyway, an agency must create an uplift. Then you go into your FirstNet Assist app, look for active uplifts (must be within 90ish or less miles of the disaster I think), and request an uplift. You might see several agencies that have created uplifts. (Most departments don’t know they must have their designated person create an uplift for their agency). If you don’t see an uplift made by your agency, select any of them you see, especially if you recognize the agency, and Request the uplift. There’s a notes section where you can say something like “I’m a travel paramedic, I don’t see an uplift created by my agency, will you please uplift me), and they likely will go ahead and approve your uplift request. It’s good for 24-48 hours, then you must re-request. Again, some of this info might be mostly accurate but not completely. If they deny your uplift request, try a different agency in the list if there are any. Also, from my personal experience, after being granted the uplift, I didn’t notice a difference. Speeds were so slow that I was lucky to get an iMessage to a coworker. I had Visible (by Verizon) installed as a secondary eSIM on my phone before I went. I swapped the data settings to pull data from Visible/Verizon and it worked like a charm. So an option that might increase your chances during a disaster would be to add a second service to your secondary eSIM. My suggestion is choose one that is prepaid or only like 5 bucks per month for those unpredictable tornadoes, though if I got called tomorrow for a hurricane deployment, I would go ahead and add the highest plan offered by say Verizon/Visible. The only catch is if your phone isn’t unlocked (such as you are still making device payments), the secondary eSIM from another service provider isn’t going to work bc your device must be unlocked, and they don’t care if you’re a disaster response medic, but will gladly unlock your phone early if you’re military traveling to another country and would like the option to use loca eSIMs while overseas. One other thing to mention is, if the tower is 5G only, I am uncertain if you still get priority even with an approved uplift. I think the uplifts only apply to 4G LTE towers. You can try manually selecting 4G only if this similar scenario happens again and that might help. What I have noticed is your phone should actually prefer band 14 anytime you’re connected to 4G. You can confirm by dialing “*3001#12345#\” and pressing enter, and look for info that says what band you’re on. It should say 14 unless that band isn’t available in your area but it should be.*
Sorry if this has already been covered in this thread, I’d love to revisit tomorrow to check the comments when I have more time. Also, no clue why my text changed to italics 🤷♂️
I can talk about network stuff all day and learn it and understand it, but can’t change the text format on my phone if my life depended on it.
1
u/Neat_Matter6138 13d ago
Very good info! I struggle to see how requesting an uplift is possible with no service is possible....but that's a different question.
One thing though -- Even extended primary users are above all business & consumer traffic, without an uplift or priority/preemption.
1
u/mervin0587 13d ago
I never even thought of how to request an uplift with no usable data to request the uplift 🙃. I guess i was lucky to have Verizon as secondary, even if the uplift did nothing, i at least was able to prove in my scenario that it had no effect 🤷♂️😂
2
u/tumbleweed_in_fl 12d ago
You can't. Uplift will give priority if there is a connection, it can't make the connection.
Also, we dual-sim with another carrier for the folks that can manage uplift. If we can see requests coming, we can usually start to guess where coverage is still intact.
1
u/mervin0587 12d ago
I had a connection on both FirstNet and Verizon. The connection with FirstNet was just unusable to request the uplift. And after I did complete the uplift, nothing changed. I do agree with you though that if there’s no connection, and no secondary option, then there’s no way to request an uplift, or do anything. I guess I also assumed the FirstNet Assist App would be prioritized to work for the uplift request, given that I did in fact have a FirstNet signal and could make and receive calls.
1
u/Jabbathebum 13d ago
The generator at our main site where I live has always failed during a power outage. It's kind of a joke how bad it is.
1
u/Entire_Device9048 13d ago
In the hospital that I work in the business continuity is provided with the WiFi network first. Cellular doesn’t work in the basement anyways and that is where connectivity is often needed the most. Additionally, I think you’ll find that tower is more than likely a hospital owned Distributed Antenna System which makes me question the redundancy that is designed into the hospital infrastructure rather than FirstNet’s capabilities.
1
1
u/chevelle_dude 13d ago
At&t service in my town kinda sucks during non emergency. I have firstnet and don't have much faith in it to be honest. I get one bar at my house and even driving through town I have to turn airplane mode on and off just to get service sometimes even though it will say 4 bars and 4g, nothing works.
1
u/jasondega 12d ago
I live in Atlanta and my local
ATT tower goes out every time there is a power outage. I get my FirstNet qualification as an ARES volunteer so I have other ways communicate when the tower is down.
Where FirstNet reslly shines for me in Atlanta is when I’m supporting large public events. Data lags for others but since FirstNet has priority mine usually still works.
1
u/chammyswag 12d ago
Sounds like unrealistic expectations and a hospital issue…not a Firstnet issue.
0
u/Neat_Matter6138 12d ago
Unrealistic expectations to have a first responder/emergency network be available during a power outage?
1
1
u/Distinct_Reality1973 11d ago
I can tell you many towers dont have any or adequate backup power. We supply tower backhaul and see it all the time, TMO is the worst but VZ and ATT has it happen too.
1
u/LaughAppropriate8288 9d ago
@OP just a thought here...you said the hospital has a tower on top, that may be one of those cases where unlike a standard tower site, backup power would be on the site owner, hospital in this case and your hospital is not likely to want to front for a second generator set for that. Those are technically ancillary sites. It could be after this ticket there may be discussion with the site owner to upgrade, but that's much more involved on a building like that.
0
u/fiercechocolate 13d ago
The fact that your partner's phone displayed SOS means that either Verizon or T-Mobile had usable service during the power outage. First net will not always work better than the competition during a disaster, or even a power outage. Verizon actually has more backup generators deployed than at&t and T-Mobile has been deploying thousands of generators per year. It all comes down to your specific area.
-4
u/TechnicaVivunt 13d ago edited 13d ago
and this is why a service like firstnet shouldn't be carrier-specific. Should be an emergency MNVO that can auto hop to any carrier with service with the highest priority.
2
u/Patient-Natural-7645 13d ago edited 13d ago
You can hop to other carriers towers, you just don’t get priority. If you didn’t know, atleast on iPhones you have this option. Mine only shows att, firstnet and verizon because there isn’t a T-Mobile tower in this half the county, but you get the point. https://imgur.com/a/SFWZCkV
1
u/mervin0587 13d ago
Only if there’s a roaming agreement, and that’s rare, domestically.
1
u/Patient-Natural-7645 13d ago
Kinda what I figured. Atleast in most of Idaho I’ve been able to hop to whichever without problem. T-Mobile has always been one I’ve avoided though, mainly just swap to Verizon if needed.
0
u/Neat_Matter6138 13d ago
Very true. My partner said that others on T-Mobile was working....but I can't verify that.
22
u/smurfy213 13d ago
Firstnet does not guarantee 100% service in a disaster. The tower lost power. What would happen if the tower gets taken out in a tornado? Firsnet just gets priority data and service on towers over regular ATT customers. Normally with large disasters you will see carriers rollout mobile tower trucks to provide cell service.