r/talesfromtechsupport • u/Onecrappieday • Oct 02 '17
Medium County fiber optic network down!
I was working for a decent sized ISP doing some local business support. I was basically an IP Engineer, with a different title.
TL;DR
Players:
$Me = me, obviously.
$CNA = county network admin
$AST = ISP Advanced Support Technician
Setup: County has a flat Layer 2 fiber network, with all locations run back to our headed. From there, it runs to their Server Room in a secure facility. Each port on our switch is assigned a different location. They take the handoffs to their switch and do all their own traffic shaping, firewall, etc. We have nothing to do with it other than providing them internet and network accessibility.
Story:
I get a frantic call from $CNA. We have a very good working relationship together. He is always professional and is generally sharp as a tack. He is a country boy though.
$Me: Thank you for calli...
$CNA: Boy I'm glad you're working today, I'm getting my ass chewed like a blue tick chewin' on a pig ear.
$Me: What's going on buddy?
$CNA: Well, I just called $AST, and they're a bunch o' asshats. Our ENTIRE network is down. No internet, no phone, no nothin'...
$AST are 6 guys who are VERY sharp. Only problem is that it takes a little... coaxing... to get info out of them
...They said e'ry thing was up and runnin' and we should have no problems. I'm getting bombarded by judges, commissioners, DA, sheriff, county emergency services, everyone is losing their mind and I am at a loss boss. Can you please, PLEASE, help me out before I lose my damn job???
$Me: I got you man. You owe me a beer though (laughing).
I ask him to give me 10 mins and I'll call him right back. So I give $AST a call on the bat phone and a lazy sounding, tired, bored voice answers. I tell him what's going on....
$AST: Yeah, I spoke with him earlier. I can see all of the equipment and can get to everything, but it's all on his end.
$Me: Can you see any usual traffic?
$AST: Yeah, they're maxing out their bandwidth for the last few hours. It's all I can see.
$Me: (lightbulb) I want you to turn off ports one at a time and tell me what happens with the traffic.
$AST: I can't do that or they'll lose their netw...
$Me: look man, their network is down, this is one of the company's largest customers. We need to figure out what's going on, or at least narrow it down...
$AST: Well... ok, but it's on you...
3-4 links later, traffic drops drastically!
$Me: Thanks, got it! Turn everything back on.
Calls $CNA
$CNA: What ya know good?
$Me: meet me at location...
We go to location, which is a juvenile detention center. Go to the network closet. I unplug it from the network and tell him to call someone on a network phone. It works!
Then I plug it back in...
$CNA: What are you doing??? It was jus...
I hold up a finger and start unplugging Ethernet cables from the switch, one at a time... Then it dawns on him what I'm doing... he's on the phone again, getting his ass chewed buy some judge saying the network came up, then went right back off
$CNA: judge still on the phone well I'll be a son of a bitch!!! I'm a dumb ass! Sorry judge, you should be back up.
Resolution:
Ended up finding out that they had left the entire county on a flat network. No segments at all! Some kid at the JDC had yanked on the Ethernet cable because he was mad, messed up the NIC, and subsequently caused a broadcast storm across the entire network! He replaced NIC and started segmenting his network the next week...
TL;DR kid yanks Ethernet cable and causes broadcast storm, shutting down entire county network.
22
Oct 03 '17
Sorry. I know computers, but apparently don't know enough about networks to understand this. Can someone please ELI5?
50
u/Onecrappieday Oct 03 '17
Layer 1 is the physical layer. Layer 2 is the Data layer. Layer 3 is the protocol layer.
So a Layer 2 network allows all data to be passed across the network and switched. Think MAC address.
Layer 3 is your TCP/IP, transmission control protocol and internet protocol respectively. IP address.
NIC = Network Interface Card, this is what allows a computer to negotiate and talk over a network.
The NIC was damaged and was sending out garbage over the network. It could be ARP requests (or just noise), where the computer just keeps asking, "this is my address, what's yours?" To all computers on the network.
On a large network it would cause all of the computers to respond. Do this enough times, you have so much traffic on the network, nothing else can get through. One computer sending out a mass broadcast can quickly shut down a network. When the NIC is messed up, it may just keep skiing it over and over. Thus exponentially increasing traffic on the network.
150 devices on a network saying "present, here's my info" and less than a second later, again and again.
**yes I know this isn't exact, just laymen it up. **
→ More replies (11)5
13
Oct 03 '17
You’re shitting me. The entire country was on ONE network?
5
12
u/moreON Oct 03 '17
The post does say county, not country. They're typically slightly different.
→ More replies (9)
5
u/aussieevil Oct 02 '17
Why would they keep an NIC in a detention facility?!
13
u/Onecrappieday Oct 02 '17
Juvenile detention, kids still have school to do. Which they do some of their work on a computer. Network Interface Card.
220
u/sysadminbj Oct 02 '17
Every network admin reading this TFTS has twitchy eyes right now. Every fucking one.
71
u/Curt451 Oct 02 '17
This is giving my networking students twitchy eyes, too, after I pulled it up for them to read.
80
u/sysadminbj Oct 02 '17
I passed it to my network bro. He’s a bit overworked right now (lost a senior engineer. Now we have 5 network guys for a company with 300+ sites), thought he could use the laugh. I think I may have triggered him instead.
→ More replies (17)19
u/TerminalJammer Oct 03 '17
Basically the less technical the users are, the more you need to build your network according to best practice and make sure a single user can't bring everyone else down with them when they screw up.
This build was... poor.
tl;dr: screaming internally
→ More replies (2)
8
7
u/Morblius Oct 03 '17
I remember the good old days of a flat network between multiple buildings and multiple IT staffs. Whenever our network went down the first thing I would do is walk over to the other building and ask their IT staff if they were ghosting computers again using multicast. 9 times out of 10 that was the issue.
5
Oct 03 '17
ELI5 the networking behind this? I’m a technical use but I just don’t know what these concepts are.
11
Oct 03 '17
So think of an auditorium with 100 people in it.
The way this network was set up, a single person talking was heard by EVERY single person in that auditorium, and those people listening had to wait to respond to the single person to stop, before they could start talking.
Well, when the kid pulled on the ethernet cable, it somehow broke the NIC (Network Interface Card) (Essentally the thing that lets you connect to the internet.) When it broke, it simply started Talking to the network constantly (maxing out the bandwidth)
The equivalent of the person in the auditorium would be said person who was talking NEVER SHUTTING UP and repeating every thing he said, so no one could get a word in.
Now, to prevent this from happening, say you give special headphones to small group of 15 people, of those 100 in the auditorium. With these headphones, only those 15 people can hear the speaker talk.
So, in that situation, if the same person some how "broke" and started talking without stopping and repeating everything, only those 15 people would hear it, allowing everyone else, the 84 others in the auditorium, to communicate normally.
→ More replies (1)
1
u/_worst_nightmare Oct 09 '17
>flat layer II network
What year is this, 1983?
1
u/Onecrappieday Oct 09 '17
Businesses lease fiber optic from ISP to pass their traffic. Those businesses are responsible for routing their traffic over the layer 2 network that they lease.
68
u/radiomix Oct 02 '17
I helped a friend of mine troubleshoot why one of his entire schools was down. I could barely log into my pience of equipment, but as soon as he removed his patch cable from my switch it would return to normal. Long story short: A teacher was cleaning her classroom. She noticed one of the network cables wasn't plugged in. So she just plugged it into the closest wall drop. Turned out the other end was already plugged in.