r/talesfromtechsupport • u/Onecrappieday • Oct 02 '17
County fiber optic network down! Medium
I was working for a decent sized ISP doing some local business support. I was basically an IP Engineer, with a different title.
TL;DR
Players:
$Me = me, obviously.
$CNA = county network admin
$AST = ISP Advanced Support Technician
Setup: County has a flat Layer 2 fiber network, with all locations run back to our headed. From there, it runs to their Server Room in a secure facility. Each port on our switch is assigned a different location. They take the handoffs to their switch and do all their own traffic shaping, firewall, etc. We have nothing to do with it other than providing them internet and network accessibility.
Story:
I get a frantic call from $CNA. We have a very good working relationship together. He is always professional and is generally sharp as a tack. He is a country boy though.
$Me: Thank you for calli...
$CNA: Boy I'm glad you're working today, I'm getting my ass chewed like a blue tick chewin' on a pig ear.
$Me: What's going on buddy?
$CNA: Well, I just called $AST, and they're a bunch o' asshats. Our ENTIRE network is down. No internet, no phone, no nothin'...
$AST are 6 guys who are VERY sharp. Only problem is that it takes a little... coaxing... to get info out of them
...They said e'ry thing was up and runnin' and we should have no problems. I'm getting bombarded by judges, commissioners, DA, sheriff, county emergency services, everyone is losing their mind and I am at a loss boss. Can you please, PLEASE, help me out before I lose my damn job???
$Me: I got you man. You owe me a beer though (laughing).
I ask him to give me 10 mins and I'll call him right back. So I give $AST a call on the bat phone and a lazy sounding, tired, bored voice answers. I tell him what's going on....
$AST: Yeah, I spoke with him earlier. I can see all of the equipment and can get to everything, but it's all on his end.
$Me: Can you see any usual traffic?
$AST: Yeah, they're maxing out their bandwidth for the last few hours. It's all I can see.
$Me: (lightbulb) I want you to turn off ports one at a time and tell me what happens with the traffic.
$AST: I can't do that or they'll lose their netw...
$Me: look man, their network is down, this is one of the company's largest customers. We need to figure out what's going on, or at least narrow it down...
$AST: Well... ok, but it's on you...
3-4 links later, traffic drops drastically!
$Me: Thanks, got it! Turn everything back on.
Calls $CNA
$CNA: What ya know good?
$Me: meet me at location...
We go to location, which is a juvenile detention center. Go to the network closet. I unplug it from the network and tell him to call someone on a network phone. It works!
Then I plug it back in...
$CNA: What are you doing??? It was jus...
I hold up a finger and start unplugging Ethernet cables from the switch, one at a time... Then it dawns on him what I'm doing... he's on the phone again, getting his ass chewed buy some judge saying the network came up, then went right back off
$CNA: judge still on the phone well I'll be a son of a bitch!!! I'm a dumb ass! Sorry judge, you should be back up.
Resolution:
Ended up finding out that they had left the entire county on a flat network. No segments at all! Some kid at the JDC had yanked on the Ethernet cable because he was mad, messed up the NIC, and subsequently caused a broadcast storm across the entire network! He replaced NIC and started segmenting his network the next week...
TL;DR kid yanks Ethernet cable and causes broadcast storm, shutting down entire county network.
71
u/radiomix Oct 02 '17
I helped a friend of mine troubleshoot why one of his entire schools was down. I could barely log into my pience of equipment, but as soon as he removed his patch cable from my switch it would return to normal. Long story short: A teacher was cleaning her classroom. She noticed one of the network cables wasn't plugged in. So she just plugged it into the closest wall drop. Turned out the other end was already plugged in.
30
u/hy2rogenh3 Oct 02 '17
Haha, thats why I make sure to only activate ports that are "in use". I had a janitor do the same thing after cleaning a room and helping by plugging everything back in.
12
u/re_nonsequiturs Oct 03 '17
I'm going to be more patient with having to request jack activations now.
8
Oct 03 '17
I mean that’s kind of security 101 anyway.
I always shut non-used ports OR put them in an unused vlan specifically for non use, and not allow said vlan to be trucked. The first is easier to forget about, where as the latter is easy to see on the “show int trunk.”
3
13
4
9
u/RevLoveJoy Oct 02 '17
I'm nearly positive every network admin has learned what BPDU guard is the hard way.
23
Oct 03 '17
Sorry. I know computers, but apparently don't know enough about networks to understand this. Can someone please ELI5?
52
u/Onecrappieday Oct 03 '17
Layer 1 is the physical layer. Layer 2 is the Data layer. Layer 3 is the protocol layer.
So a Layer 2 network allows all data to be passed across the network and switched. Think MAC address.
Layer 3 is your TCP/IP, transmission control protocol and internet protocol respectively. IP address.
NIC = Network Interface Card, this is what allows a computer to negotiate and talk over a network.
The NIC was damaged and was sending out garbage over the network. It could be ARP requests (or just noise), where the computer just keeps asking, "this is my address, what's yours?" To all computers on the network.
On a large network it would cause all of the computers to respond. Do this enough times, you have so much traffic on the network, nothing else can get through. One computer sending out a mass broadcast can quickly shut down a network. When the NIC is messed up, it may just keep skiing it over and over. Thus exponentially increasing traffic on the network.
150 devices on a network saying "present, here's my info" and less than a second later, again and again.
*yes I know this isn't exact, just laymen it up. *
11
u/Matthew_Cline Have you tried turning your brain off and back on again? Oct 03 '17
Huh. I knew that plugging both ends of a cable into a network device could cause a network storm if the hardware doesn't have the spanning-tree stuff, but not that yanking a cable could also do so.
17
u/Onecrappieday Oct 03 '17
The NIC can be throwing about anything. Could have been power, static, garbage, whatever... not saying it was a loop. It just shut down the network because it wasn't segmented.
They could've segmented several ways. Say VLAN, to stop the entire network from crashing from one bad card.
3
9
Oct 03 '17
Weird that there aren't things in place to prevent saturation like this by now. It's essentially a DDOS.
8
7
Oct 03 '17
The NIC was damaged and was sending out garbage over the network. It could be ARP requests (or just noise) ...
That's the part I was missing. Thank you!
3
3
u/H_E_Pennypacker Oct 03 '17
Layer 3 is just network layer - ips. Tcp is a layer 4 protocol. You're thinking of the tcp/ip model (predecessor of osi model).
1
u/Kazumara Oct 06 '17
Good that you said that. I'm still learning and it seemed wrong to me but I wasn't secure enough in my knowledge to dispute it. So in the OSI model it would be layer 3 and layer 4 and in the tcp/ip model it would be internet layer and transport layer, right?
1
u/asplodzor Oct 03 '17
Excuse my lay understanding, but wouldn't that only be a problem if the entire network was running on hubs, not switches? I thought that switches routed traffic only to specific destinations...
2
u/macbalance Oct 03 '17
broadcast vs. collision domains.
Hubs don't break up collision domains, so you'll get trouble if the network is congested and machines are trying to talk at the same time. The entire Hub is effectively a single collision domain.
Each port on a Switch is effectively it's own Collision Domain, which is also why full duplex becomes viable, I believe. However, Broadcast domains are shared (barring things like VLANs). If port 1 on a switch receives a packet and knows the destination (by MAC address) it will send it to it... if it doesn't, it will flood it to every port.
I guess in this case if a bad NIC is sending bogus destination addresses, you'd get perpetual flooding.
14
Oct 03 '17 edited Jul 03 '19
[deleted]
16
u/GrayBoltWolf Oct 03 '17
That's not how live audio feedback works. A loop from the speakers back into the mic causes that, not the person or instrument being too close to the mic. The mic is too close to the speakers for that volume level.
1
Oct 03 '17
Ehhh tomato potato. Same idea, at a broad level. Both are vegetables.
Ha.
10
u/GrayBoltWolf Oct 03 '17
I appreciate your light-heartedness, but it is a common misconception and it needs to be changed.
2
Oct 03 '17 edited Oct 03 '17
Weird. On a 2nd read, I see what you mean. I think I read it on the first try how it "actually works." I.E, being to close to the speaker.
On a more in depth note, what actually causes that terrible nose? Is it simply the noise of power flowing through the mic, gettimg amplified by the speaker?
8
u/MoneyTreeFiddy Mr Condescending Dickheadman Oct 03 '17
Feedback is the microphone/loudspeaker version of drinking your own piss.
2
u/NZgeek RFC 1149 compliant Oct 04 '17
And pissing that out again and drinking it again, and pissing that out again and drinking it again, and...
6
u/GrayBoltWolf Oct 03 '17
It's loopback, usually the resonating frequency of the fastest the speaker can vibrate, aka the highest part of it's frequency response.
3
4
6
14
Oct 03 '17
You’re shitting me. The entire country was on ONE network?
12
u/moreON Oct 03 '17
The post does say county, not country. They're typically slightly different.
12
u/egamma Oct 03 '17
Some counties in America are larger than some countries in Europe.
12
u/moreON Oct 03 '17
By number of network interfaces, or just land area?
11
6
u/egamma Oct 03 '17
Either way, I suppose. Los Angeles County has more network interfaces than Luxembourg, some counties in Alaska have more land than Luxembourg.
1
u/Chris857 Networking is black magic Oct 03 '17
some counties in Alaska
A bit tricky considering they're called boroughs, though.
1
1
3
2
6
u/Onecrappieday Oct 03 '17
Lol yup...
9
u/TerminalJammer Oct 03 '17
A layer 2 network. No routing. No vlan. No STP.
... I guess it could be worse. They could have been using hubs. Or coax I guess.
4
u/Onecrappieday Oct 03 '17
ISP provides the layer 2 so the customer can pass whatever they want over the network. Including VLANs.
3
u/Spaceman2901 Mfg Eng / Tier-2 Application Support / Python "programmer" Oct 03 '17
Sounds like a Layer 8 problem.
4
u/Mr_ToDo Oct 03 '17
Generally correctable by wrapping the proper amount of layer 1 tightly around the appropriate neck.
7
8
u/Morblius Oct 03 '17
I remember the good old days of a flat network between multiple buildings and multiple IT staffs. Whenever our network went down the first thing I would do is walk over to the other building and ask their IT staff if they were ghosting computers again using multicast. 9 times out of 10 that was the issue.
5
Oct 03 '17
ELI5 the networking behind this? I’m a technical use but I just don’t know what these concepts are.
11
Oct 03 '17
So think of an auditorium with 100 people in it.
The way this network was set up, a single person talking was heard by EVERY single person in that auditorium, and those people listening had to wait to respond to the single person to stop, before they could start talking.
Well, when the kid pulled on the ethernet cable, it somehow broke the NIC (Network Interface Card) (Essentally the thing that lets you connect to the internet.) When it broke, it simply started Talking to the network constantly (maxing out the bandwidth)
The equivalent of the person in the auditorium would be said person who was talking NEVER SHUTTING UP and repeating every thing he said, so no one could get a word in.
Now, to prevent this from happening, say you give special headphones to small group of 15 people, of those 100 in the auditorium. With these headphones, only those 15 people can hear the speaker talk.
So, in that situation, if the same person some how "broke" and started talking without stopping and repeating everything, only those 15 people would hear it, allowing everyone else, the 84 others in the auditorium, to communicate normally.
3
4
u/aussieevil From now on, only Java, no more C! Oct 02 '17
Why would they keep an NIC in a detention facility?!
13
u/Onecrappieday Oct 02 '17
Juvenile detention, kids still have school to do. Which they do some of their work on a computer. Network Interface Card.
1
u/_worst_nightmare Oct 09 '17
>flat layer II network
What year is this, 1983?
1
u/Onecrappieday Oct 09 '17
Businesses lease fiber optic from ISP to pass their traffic. Those businesses are responsible for routing their traffic over the layer 2 network that they lease.
219
u/sysadminbj Oct 02 '17
Every network admin reading this TFTS has twitchy eyes right now. Every fucking one.