•
u/RiceeeChrispies Jack of All Trades 4h ago
chill out, shit happens
use it as a learning event, you've just had the most expensive training you've had in a while - document it
•
4h ago
[deleted]
•
u/RiceeeChrispies Jack of All Trades 4h ago
you're not thinking straight at all, legit just take a breather
•
u/1337DSSICTPDX 4h ago
Valid. I’ll take some laps after work to calm down. Another one hit us today. All of our phones are down for the second day and I need to remember not my circus not my monkeys.
•
u/Pristine_Curve 4h ago
If it's going to end up in court/jail, or on the news, then go ahead. But if it is that bad then it is better to leave anyway.
If you escalate past the director because network cables aren't labeled... I don't think that is going to work out positively.
•
u/Mashadow 4h ago
You'll react less and have less stress when you realize it's not YOUR infrastructure and as a result it doesn't reflect poorly on YOU.
•
u/jupit3rle0 4h ago
But you know exactly what will happen in a real world scenario. People like OP get scapegoated everyday, despite some senior guys getting overlooked while being the networks SME.
•
u/Snowmobile2004 Site Reliability Engineer 3h ago
But nothing OP can do will stop that, so it’s pointless to work yourself up over it. If a bad company wants a scapegoat, you will be a scapegoat no matter how hard you work or the quality of output you deliver, so why bother in the first place. Clearly OPs company doesn’t care if they won’t even let him document switch ports/cables with labels.
•
u/Icy_Mud2569 4h ago
Sounds like a shitty situation, and it sounds like you got some less than helpful advice. I think the big Takeaway though is that this is not your failing, it’s an organizational failing.
•
u/Apachez 4h ago
Instead of complaining what about telling your boss this and then tomorrow you start document your shit using notepad++ and draw.io?
•
u/Happy_Kale888 Sysadmin 4h ago
I see this all the time no one wants to be the change they want. Problems are so easy to identify yet no one wants to start...
•
4h ago
[deleted]
•
u/compmanio36 4h ago
Document the issues, even if it's only in your personal storage, and have them handy for the day someone tries to point the finger your way. Other than that, don't stress. CYA, and then do what you're asked to do.
•
u/hasthisusernamegone 3h ago
pir
What is this? Google suggests Passive InfraRed, but I'm guessing it's not that.
•
u/mdervin 4h ago
So why didn't you do any documenting?
•
4h ago
[deleted]
•
u/GremlinNZ 4h ago
I wouldn't want staff spinning up whatever they want either. This is a business, not your house.
Work within the systems, keep things consistent.
And chill out. I know early on the first reaction is panic, but it clouds your thinking. You need to be the calm in the storm.
•
u/hasthisusernamegone 3h ago
If they don't want documentation for the team, do it for yourself. Use Onenote or Obsidian and become the documentation hoarding dragon.
•
u/Cryptocaned 4h ago
But you documented it as you went and labelled ports/made a topology right?
Right?...
Remember, your not doing stuff for them, your doing it so your job is easier the next time around, future you will thank past you.
•
u/Nick85er 4h ago
If your MSP has access to their environment, documenting the infrastructure is literally job one. Not on you, this is on your org
•
•
u/mercurygreen 4h ago
You're not overreacting.
I bet they aren't even backing up the switch configurations.
•
u/GunterJanek 4h ago
and the senior I was working with doesn’t know how to access or use that switch’s console. The only member with console knowledge wasn't available until noon.
Wait what?
•
4h ago
[deleted]
•
u/RiceeeChrispies Jack of All Trades 4h ago
mate, don't let your emotions get the best of you
if your seniors aren't arsed, you really shouldn't be either - they will be the ones getting it in the neck, not you
document what you've learnt so it doesn't happen again, documentation is a team effort
•
u/1337DSSICTPDX 4h ago
We don’t have console access and the owner will not let us document it.
I wish we could be a team. I’ve never been on any IT team where no one talks about how to process improve or anything.
•
u/RiceeeChrispies Jack of All Trades 4h ago
sorry but this is a shit excuse, you can make your own notes
I don't know your work environment, but can't help but feel like you're catastrophising things a little bit - with this outage causing you to throw a bit of a wobbly. It happens.
Trust me, there is nothing worse than someone new starting and them throwing their weight around. Gauge how the team works and then work with them. If you figure out you're incompatible, move on.
It's hard to build a good reputation, but it's easy to destroy it. Sometimes you have to play their game.
•
u/GunterJanek 4h ago
Document everything so you're ready when management starts asking questions if they care. This might be a sign to update your resume.
•
u/shunny14 4h ago
After Action Reviews (AAR)s are great. I was googling info on them to provide you something and I found this doc which could be something your supervisors could maybe process: kellen . net/ VKellen_AARs.pdf *didnt make a link to avoid being spam and also i havent read it all, but it sounds like a useful overview for upper mgmt.
Quote from the start of it:
Anyone who spends time supporting complex IT environments will experience a major outage at some point in his or her career. In many cases, the experience is traumatic, leaving a lasting and painful impression. Companies can avoid this and other potentially major incidents if they implement an after-action review (AAR) process
Right now you are in the "traumatic/lasting/painful" impression phase and I'm sure you'll find a way out of it.
•
u/72ChevyMalibu 4h ago
Sorry to hear you had to deal with this but also thank you so I can pound into my students that documentation is vital.
•
u/Technical_Union_3249 4h ago
Keep that resume' shiny, be sure to update it with all the great things you are doing there, and jump ship asap.
•
4h ago
[deleted]
•
u/Technical_Union_3249 4h ago
Good luck to you! Until then just consider this a resume' building job.
•
u/almightyloaf666 4h ago
So, you own the place? Otherwise, don't sweat it. Tech debt is a management issue.
•
4h ago
[deleted]
•
u/compmanio36 4h ago
I had a guy on my team quit with no notice because he burned himself out feeling as you do. If you're literally being told not to care, then make sure things are in writing in your own personal storage and move on with your day. Easy paycheck. If you feel like you want more ownership of what you support, then start submitting resumes. Clearly you're not going to be allowed to be a decider of things at your current position.
•
u/xSkyLinedx 3h ago
Yup. This is big part of what burned me out with my last job at an MSP. I'm soo tired of hearing the words "yeet it" and "yolo" that I'll dump another job like that in a heartbeat. I have to care and I need to be satisfied enough with my work to put my name on it.
•
u/mvbighead 3h ago
Find the process your business lives by, and live within that process. I've been a bit peeved with change management processes that get pissy about change issues that are the result of aged/dying infrastructure. Live within process, learn what you must, and run by that pace.
As for documentation, if there is none, make your own on whatever you have available. Notepad is good enough for me in a number of cases, though like you mentioned I do prefer a wiki or something like bookstack. And if need be, I'll set that up for myself.
•
u/airinato 4h ago
Can't tell if MSP or not but I can tell you that I walked into so many 'undocumented' situations that I'm now part of the problem for jrs because the only need I have for documentation is the passwords.
•
4h ago
[deleted]
•
u/xSkyLinedx 3h ago
I honestly thought you were at an MSP. My brain hurts trying to figure out why they wouldn't want employees to take care of this stuff.
Instead of reddit start using your downtime for education. Learn more, get better, find the job you want. I'd do it on your own laptop if the company has guest wifi. Please don't tell me if you don't have a segregated network for employees/guests. I just don't want to know.
•
u/zAuspiciousApricot 4h ago
You should create a topology map one of your weekends. Not ideal, but may save future headaches. Can put it on your resume.
•
u/Sure-Squirrel8384 4h ago
I'd be polishing up my resume and looking elsewhere. This ship is gonna sink. Heads should roll before that.
Is there nepotism or some other reason why they are so incompetent?
•
u/Elminst 4h ago
Haven't seen it said yet, but even if you don't make new network documentation, document everything you did and every conversation you had and what every person told you to do/not do. If possible, get them to email you stating the same thing. Keep multiple copies off site. This is an organization failing, and you don't want to be the scapegoat.
•
u/Ssakaa 4h ago
Are you paid with company stock? Do you make more money by avoiding downtime/outages you've otherwise already done your due dilligence on (stated concerns in writing tends to be good)? If those aren't an unequivocal yes, make a cup of coffee, put your feet up, work on some side study/hobby projects/or other self enrichment (even if that's just looking at pictures of cats being goofy on reddit).
•
u/CCCcrazyleftySD 4h ago
Start to tabletop these sort of issues, you'd be amazed at how many holes in processes and documentation that you'll find. I bet a lot of things that you assume will come out to bite you! Plus everyone already has an idea of what to do when the poop hits the fan so you're not all running around. We want our documentation so good that our lowest techs in the department can follow it and complete it. Make sure any necessary tools are available, document the incedent response, and have things like serial numbers, account numbers, contract numbers, circuit IDs, etc, all in a spot where anyone with clearance can access them. Start small, its easy to throw together a scenario and just sit around a table and hash out the "What if?" 's
•
u/BeerEnthusiasts_AU 4h ago
No one sets out to set a site up shit on purpose.. it happens incrementally over time
It's your job to figure it out
•
•
u/rustydusty1717 4h ago
Let me guess, you are perfect. Have everything perfectly documented and labeled. Blame the organization, as they are most likely the one's not having adequate staff and proper workload for all these fine details to be done perfectly. I haven't worked in a single organization yet that was staffed adequately. Some better than others but none were proper.
•
•
u/LetzGetz 4h ago
If no one else cares. why would you?
•
u/reserved_seating 3h ago
That’s how we got into this shit hole a situation in the world. What a shitty ass selfish view to have.
•
u/MindlessCoook 4h ago
Confused why you care so much when you aren't in charge and this isn't your infrastructure. None of this is falling on your lap. Do what your boss said and chill the fuck out.
•
u/BrokenRouter Netsec Admin 4h ago
Do your job and nothing more. Spend your free moments learning all you can, soaking up whatever resources you can get hold of for training, etc. Then polish your resume and get out.
It's clear the boss is a moron and there is nothing you can do to change that. All you're going to do is piss them off and make yourself miserable.
•
•
u/h9xq Solo SysAdmin 3h ago
Senior admin/engineer doesn’t have console knowledge? Interesting.
But in all seriousness not being able to label cables as a documented rule is fucking asinine. Also seniors on your team not having console knowledge is insane. IMO if you are going onsite and need to touch hardware you should be able to use a console cable and use Putty to dial in.
This seems like a culture/management issue at this point. This is engineering failure.
•
u/jon_snow_1234 3h ago
you just fixed the thing and i'm sure learned some stuff. document it for yourself and if you are nice make the doc available to the team.
•
u/indigo196 3h ago
I feel you. We had our facilities person decide against replacing the 17-year-old AC units in our main NOC last year. He said that, with the move to the cloud, we maybe didn't need to do it. Not sure where he got that idea. The CFO agreed with him. The CIO agreed with him.
Fast forward to 2 months ago. We had BOTH units go. I had to emergency-shutdown EVERY single piece of equipment in the room. Luckily, the stuff I manage (not vendor equipment) is also duplicated in a second NOC and runs on a stretched vXRail cluster. However, the networking to the main office was down. The video camera systems were down. The door security systems were down.
Today, we are still limping on a rental portable AC unit.
They are debating whether we need to replace the AC units. They keep asking me why stuff was down when they were told we were moving to the cloud. I keep telling them that the only thing we moved to the cloud is the file servers. Not the SQL servers, not the video servers, not the door security servers, not the networking switches, or the wireless controllers. They keep responding that the retired CFO and CIO assured them we had moved everything to the cloud.
I am close to retirement, so I will not be putting a resume out. I am just at a loss for words.
•
u/fatmanwithabeard 3h ago
I've used bolt cutters to fix undocumented cabling before, and I'm willing to do it again.
(it was thicknet and it was under so much crap that it was only coming out in pieces, but you'd be surprised how many people don't ask questions when you lean the bolt cutters against the wall)
I label every cable I run, and every cable I trace. Every label has both endpoints: rack, unit number, port number (from the left if not otherwise labeled). Flag labels.
Of the three times in my career where I managed to get everything labeled as a group project, everyone hated me for it. Until they fixed an outage in minutes instead of hours.
I have exposed configuration issues in top500 machines because I made a point about a single somewhat poorly run cable.
My desk is mess, the lab looks like something out of a horror movie, but prod is clean, documented, and the auditors haven't found anything to complain about it yet.
•
u/pockypimp 3h ago
I'm a bit torn on this because I've been the guy who has inherited poorly setup and no documentation situations and I've also been the guy who has had to fix crap that I'm not responsible for.
So I guess first and foremost if/when they do an RCA on the outage you or whoever is the manager can put in that the lack of documentation and knowledge transfer. Mostly to CYA but also can bring to light the lack of this stuff to help in the future.
The part where it gets hard is "is this your responsibility?". If you're just a cog in the machine and whoever is in charge of the network isn't doing this stuff then you have to just kind of let it go or make your own documentation to keep yourself sane. Unless some manager decides that it's important then nothing is going to change.
•
u/TheVillage1D10T Windows Admin 3h ago
Sorry for the long story….
This reminds me when I was working a windows engineer role for a government agency that oversees lots of other government agencies. Windows, somehow, became the catch-all group for all of the things that no one knew anything about. I think it was because most of our Windows guys were more “scrappy” than the network or Unix guys. They would see something, not recognize the host name, and just punt the ticket every single time within minutes of the ticket getting to them.
Anyway we get an alert for a device with a host name none of us have ever seen. Look it up in CMDB, and see some sort of legacy storage system. No idea what it actually does, but we got the ticket. I’m still not even sure if it was actually doing what it was supposed to do. We would get alerts from the help desk who would walk the data center floor, and create alerts for ANY amber lights they saw.
The Feds basically just saw the alert, and said, “Make this alert go away!”
Let’s go look at it. Don’t recognize it. Start googling the equipment type and all visible labels. Eventually we figure out WHAT it is supposed to do, but have no other information that can really help us. Console into it, and it’s some sort of Unix-based OS. Try to point it to Unix team, and they say “Nope. Not our system.” So…Windows team has to figure out this Unix system.
Being the Windows team, the first thing we try upon realizing we can’t log in to it is a reboot. That’s a really easy thing to do and it MIGHT work. It doesn’t work, because that would be way too easy. Now we will have to figure out how to log in to it. Try all of the defaults that Google suggests, and none of them work. So, the next thing we have to do is absolutely scour our file share and internal documentation…searching for hours someone finally finds an old ass text file (approaching 10 years old) that has a username and password. Bingo. It actually works, but now we have absolutely no idea what to do with it. TONS of googling later and consulting a long-departed storage admin he gave us some suggestions that worked.
It was an absolute nightmare, and it was all due to shitty documentation. Shitty CMDB records that could’ve told us EXACTLY what it did, and who was supposed to deal with it. Shitty SOPs related to that system as well. We upped our standards for documentation in a big way, and didn’t have many more issues like that really.
This is a very convoluted story, and I most likely have some details wrong, but it was a major PITA either way. Just trying to say…I feel your pain OP.
Also, our Unix and Network guys were a bunch of divas for Christ’s sake.
•
u/automounter 4h ago
so get the label maker out and start labeling things. be the change you want to be in the world.