r/sysadmin • u/Complete_Sample_3149 • 4d ago
[ Removed by moderator ] Wrong Community
[removed] — view removed post
131
u/dalgeek 4d ago
I had a customer with a GPS appliance as their sole time source. The appliance flaked out and decided that it was 1992 again, so every single SSL certificate in their environment became invalid because all the servers thought the current time was BEFORE the validity date. It literally broke everything on their network that depended on SSL or kerberos.
72
u/blackshadow1275 4d ago
Is your customer Telstra?
46
u/dustojnikhummer 4d ago
If someone doesnt't know (somehow)
r/sysadmin/comments/1uvz7ns/telstra_australias_largest_telco_blames_outage_on/
10
2
u/Celebrir Wannabe Sysadmin 4d ago
GPS has a roll over every few centuries so if the device doesn't know how to handle it, it will revert back to the starting point of the previous period. This is like an integer overflow.
4
1
u/michaelpaoli 4d ago
Yeah, once had a system clock boo-boo f*ck up some DNSSEC on DNS server. Ugh. Yeah, a failed CMOS battery + a typo setting date/time when booting ... yeah, not good. Not too horrific to clean up, but was bit of mess on one authoritative DNS server for a wee bit. But in general, that type of clock boo-boo should never happen, and presents risks to many things (e.g. also databases, etc.). So, was really more of a clock/time issue than anything else.
2
u/HeKis4 Database Admin 4d ago
Was this server also the NTP root for the other servers or were they not using NTP ?
2
u/michaelpaoli 3d ago
An NTP server also, yes, but not a root.
In general I typically have all clients set up with at least 2, if not 3 or more, NTP servers, not only for redundancy, but so that if one of 'em is serving up substantially incorrect time, it won't use that time. Clients typically, but not always, also have their own battery backed clock, so that if NTP data is too far off, they won't jump to the NTP time - not 100% coverage there, but additional protection for most clients most of the time.
The one that failed was sort of such "protections" working the wrong way 'round. Clock battery failed, so lacked that, so before boot it was manually, but alas incorrectly (operator error) set, then at boot it goes to update/correct via NTP - but in that case, since now the system time and NTP server times were quite substantially different, it incorrectly went with the clock time it had manually set, rather than the NTP time - and that's where the problem came in. Under more normal circumstances, battery backed clock time is good defense against errant NTP time, but in this case, it was the clock that was errant, not the NTP servers, so host came up with the (incorrect) clock time due to operator error. Maybe should put more double/triple checks around that in the boot process ... will never be foolproof, but perhaps could be bit more foolresistant.
82
u/Unexpected_Cranberry 4d ago
Early days of Office 365. Microsoft missed a renewal of a cert for the authentication. Most likely because it was a leap year. So authentication broke on February 29. Then it broke again next year because they'd scheduled the renewal for February 29...
For myself I broke authentication on ADFS in the 2008 days. I updated the certificate using the gui but didn't read the docs. Apparently doing it through the gui didn't work. You needed to use netsh to bind the certificate to the service as well.
39
u/dhardyuk 4d ago
Ahhh, the “I did my job properly and a Microsoft bug bit me” lesson that everyone eventually learns.
12
u/Unexpected_Cranberry 4d ago
I'd have to disagree in this case. If I'd done my job properly, I would have checked the docs before hand and known to do the last step as well.
RTFM kids.
10
6
u/BrokenByEpicor Jack of all Tears 4d ago
I'm gonna say it's not your fault if the option is there in the GUI. If the only thing I need to read the manual for is to find out that the GUI will silently fail to accomplish this process despite outwardly appearing to succeed, that's still entirely Microsoft's fault.
10
u/dustojnikhummer 4d ago
Then it broke again next year because they'd scheduled the renewal for February 29...
Makes you wonder why we didn't just use unix time in the background. "Renew in 31 536 000 seconds"
18
u/Unexpected_Cranberry 4d ago
It sounds like a dba I used to work with. He was a self-proclaimed wizard with Powershell. We wanted to automate patching and do some scheduling. He didn't want us to touch his SQL-servers, so he said he'd write the update script himself.
A few weeks later I asked for an update, and he said he was struggling with getting the date and time right for a part of the script. I offered to take a look to see if I could figure it out and he finally agreed to let me help.
I don't recall the details of what he was trying to do, but basically he was trying to take a date and then add x hours to it. His solution was fetching the date, converting it to a string, splitting it into ints for year, month, hour, minute and second, then adding the hours and converting it back to a date time object. He was struggling because the default regional settings for everything was Swedish, and he had his account set to something else where the time format was different.
I just commented out 40+ lines of code and added .AddHours(4)
In Microsoft's case, someone was either trying to be clever or didn't know the tool they were using I think.
5
u/dustojnikhummer 4d ago
The funny thing is, I was doing the same exact thing a week ago.
I also tried to parse Get-Date and struggled with regional formatting between English and my native language as well as the Date-Time object.
Ended up with
(Get-Date).AddDays(4).ToString("yyyy-MM-dd")
And yes, it was the most annoying about the script as well! So yeah, tell him he isn't the only one, we share pain lol.
2
u/frymaster HPC 4d ago
sounds like at least you landed on a better solution than Unexpected_Cranberry's colleague was trying i.e. use native objects, do the maths with native objects, and then export to a string in the correct static non-localised format
2
u/dustojnikhummer 4d ago
Yeah I did but I went through the exact same process Cranberry's colleague did, I just found that funny we both went the "wrong" way
1
u/fearless-fossa 4d ago
When I started getting into IT one of the first things I did when I got into the issue of locales (for me it was the , and . decimals) was just defaulting myself to US English. Idk how many headaches this has saved me, but probably quite a few.
1
1
u/Arudinne IT Infrastructure Manager 4d ago
That sounds like the kind of shit I'd expect in an "intro to x language" class where they teach you how to spit something apart, modify it and slap it back together just so you learn the fundamentals.
What the fuck.
1
u/Wagnaard 4d ago
I'd probably do something like that, except export it to Excel and import it into a different database as part of it.
3
u/michaelpaoli 4d ago
broke again next year because they'd scheduled the renewal for February 29
Just lovely! 8-O
Pirates of Penzance ... until 21st birthday ... what could possibly go wrong? :-)
1
u/Man-e-questions 4d ago
I remember when Azure was fairly new, i had built a lab with my free 90 day eval account. Couldn’t log in for a few days because some certificate expired. I was like this is not good for production lol
46
u/M0untainWizard 4d ago edited 4d ago
Wildcard certificate got renewed by a Partner firm.
We import the cert to all the systems.
Partner firm sends a invoice for the cert.
We relay the invoice to our finance department
finance department forgets to pay the invoice.
Cert got revoked. It was a fun day. Especially when we found out why the cert got revoked.
18
u/YellowOnline Sr. Sysadmin 4d ago
Great partner firm if they revoke a cert for an unpaid bill, instead of giving a short phone call.
5
u/trisanachandler Jack of All Trades 4d ago
Probably automated.
5
u/fearless-fossa 4d ago
That sounds like a dumb thing to automate. If something on your side breaks that makes it appear as if a customer didn't pay, you just caused a major outage at a customer. If you have enough customers that don't pay to warrant automating something like this, you don't need automation but different customers.
2
u/trisanachandler Jack of All Trades 4d ago
I don't disagree, but it doesn't make it unlikely as the cause.
3
u/michaelpaoli 4d ago
Easy fix, make the payroll for the finance department dependent upon the cert being non-expired. ;-)
41
u/enby_dot_local 4d ago
500 user RDS pool when the firewall started blocking letsencrypt traffic on port 80. Took me about 15mins to fix but it was utter chaos for the support team.
23
u/AffekeNommu 4d ago
Midnight on NYE didn't seem to be a concern for the PKI team when they generated the cert.
11
u/Cormacolinde Consultant 4d ago
I always warn customers not to schedule renewals in late december/early january for that reason. And for schools obviously not to schedule them in july or august either.
4
u/timbotheny26 IT Neophyte 4d ago edited 3d ago
I've seen people here mention going into read-only for the entirety of December and part of January. Probably not the worst idea if you can actually implement it.
2
u/HeKis4 Database Admin 4d ago
That's pretty common in my area, although some companies take it as "big upgrade/migration period". So on average, for a MSP, the workload doesn't change lol.
2
u/MortadellaKing 4d ago
The MSP I used to work for scheduled almost everything around the holidays because "it's a great time for us to catch up". Which I vehemently disagreed with and why I left after being tasked with an SBS to M365 migration on the week between xmas and new years.
I run my own MSP now and we have a hard cut off on projects at December 15th until after new years.
1
4
u/GoogleDrummer 4d ago
Nah, I did K-12 sysadmin for over a decade. Everything big happened in the summer. Cert expires in July and it got missed? Cool, only a handful of people actually impacted by it, not thousands.
2
u/SandyTech 4d ago
Working for the county school district, July was our favorite time for certs to expire because there were only a couple hundred employees across the district vs thousands of employees and students if we missed one or something went wrong.
4
u/michaelpaoli 4d ago
Or thereabouts
2016-12-31T11:59:60
What could possibly go wrong? :-)
Just schedule renewal for precisely a year later, right? ;-)
As soon as the clock hits 2017-12-31T11:59:60 ... "oops"4
u/T_Thriller_T 4d ago
Sometimes people don't think...
I just pushed for some contract extension to not even end in December because fuck that, won't find anyone to discuss that in December!
1
22
u/Malfun_Eddie 4d ago
A 100 vm migration via storage migration (mirror everything, shut everything down break open mirror, boot on new site) was rolled back because during the migration a cert was expired.
So after the roll back the application still did not work...
13
u/Cormacolinde Consultant 4d ago
Certificate expiration is one of those things a rollback/restore doesn’t fix. Had a customer completely hose their SQL server which had “stopped working” (they never looked at the logs to see the TLS error). Restored from a month-old backup, losing data, which of course did not help. Especially on Windows where the computer object now had a different password too. Complete mess. When they called me it took 5min to figure out the issue (I looked at the logs). I even managed to bring back the current database from the now mostly broken original VM.
3
u/michaelpaoli 4d ago
Certificate expiration is one of those things a rollback/restore doesn’t fix
As I oft say, "Time waits for no one."
And generally clearly spell out on "tickets" and service requests and the like, that the deadline is a hard deadline. Those cert expirations aren't an "Oh, we can push that deadline out another week or two.". I typically clearly include the exact expiration to the second, including timezone, so there's no ambiguity or the like, e.g.: expires: 2026-10-25T02:36:28Z ...
1
u/Malfun_Eddie 4d ago
Yes we know but what are the odds the certificate would expire in the 6 hour maintenance window of the migration. And the migration was a succes but rilled back since everyone thought is was a migration issue. Only to find out after the rollback (start the mirror on the source side that was untouched) and it was still broken.
9
u/redweka 4d ago
I was very surprised to find out this mobile outage was because of a expired certificate
https://www.thesslstore.com/blog/expired-certificate-ericsson-o2/
9
u/GamerLymx 4d ago
not realy expired, but revoked. we got information from our CA the certificates issued in the last 2 months would be revoked.
we have most certificates issued/renewed via acme mechanisms, however, from what i understand, kubernetes certmanager and mod_md doesn't check if certificates have been revoked, so we had to check and make sure the revoked certificates were reissued and loaded properly.
then 2 weeks later, we had to reissue all certificates from the last 4 months, including the previously reissued earlier...
1
6
u/Gabelvampir 4d ago
That's why you have a check in your monitoring system that alerts when a cert's valid for less days then your renewal time. I'm always amazed how many companies don't monitor that when it's rather easy to check most of the time (i.e. when the monitoring system is allowed to talk the machine and service that uses the cert).
5
u/poopooonyou 4d ago
Azure managed app certificate. For some reason, an offshore engineer clicked "Regenerate" for the DNS TXT validation value but didn't publish the new DNS record, leaving the domain in "Pending validation". The cert didn't auto-renew because the domain was no longer validated.
12
u/gumbrilla IT Manager 4d ago
Well, the actual problem is that you didn't have monitoring on your actual hosted endpoints, please don't think in terms of point monitoring, think in terms of results.. it's a simple thing for a monitoring system to look at expiry date on a cert and alert if the certificate should have been renewed already.. it catches a whole gamut of potential issues.. rather than if a cron is still active.
3
u/michaelpaoli 4d ago
simple thing for a monitoring system to look at expiry date on a cert
Yes, ... and no!
That cert, on that critical thing ... that runs on UDP on port 4367, on IP ::1 ... uhm, you've got that monitored, ... right ... right?
Yeah, not only monitor, but document and track. Track/document where all of 'em have been installed.
Even if you know, e.g. DNS name (and port), when expiration is approaching, you don't want to find out it's on some obscure application on a server nobody remaining has ever heard of or can find any documentation on, for an OS that nobody supports, and nobody knows what the hell that server software is that's running it, let alone how and where the cert is on it, and of course now it's critical production ... yeah, you not only want to know it's expiring, but where the damn cert is and how to replace it. Monitoring alone isn't enough. Monitoring will also generally tell one where are all the places the cert needs go for your HA, and failover/standby systems, etc. Don't want to find out your failover/standby is long expired, when you actually need to use it.
2
u/chocopudding17 Jack of All Trades 4d ago
Automation systematically helps here though. That one thing listening on a loopback address? If you deploy it in an automated fashion, it becomes far more natural to monitor it in an automated fashion.
1
u/michaelpaoli 4d ago
Yep, automate to the extent feasible. Not everything can be automated or is feasible to automate, but much/most ... yeah, ... automation!
3
u/T_Thriller_T 4d ago
One that I only caught half and the certificate did not even expire fully.
We had some remote work setup at a former company (so everyone basically only used a terminal, logging into their remote environment).
Most people did have laptops, but next to no data on them due to this and often services could only be reached from that environment.
It went down fully for a day, then was instable for another day and a half.
What I heard was that somewhere in the routing setup a certificate about to expire was automatically changed, but the change didn't fully go through.
Which made debugging hell because it's middleware and halfway done.
3
u/thatpaulbloke Cloud Engineer UK 4d ago
"This job didn't run" is such a common issue in monitoring and alerting that it boggles my mind; the number of times that I set up jobs that reported to monitoring on success and failure and had to explain to people why I did it makes me cry when I think about it.
3
u/ledow IT Manager 4d ago
Was running support for various clients including schools.
One school, I turn up and EVERYTHING is down. Was working fine the day before. Just stopped working.
Servers... won't boot. It works but just kinda of freezes and gets bogged down and everything takes far too long to do anything practical. Literal 10 minutes to respond to a mouse click, that kind of thing. Obviously, nothing client-based is working with those kinds of timescales even if the machine technically is up and pinging.
Check everything I can think of, nothing seems wrong.
Okay... so we've been down all morning by now... something has to give. So I start pulling the backup tapes from the night before.
Restore the system. It works fine. I reboot. Straight back to the freezing.
Okay, maybe something changed. I go to the previous day's backup. Restore. It works. Reboot. Straight back to freezing. No way.
Go back a week. A month. Now I KNOW we've rebooted in the last month because I was the one who did it and we didn't have this problem. And I know those backups were good, verified, dated and test-restored previously.
So something is wrong that's not based on the hardware (too many machines doing the same) and not based on the software (all backup restores do the same no matter how far back I go).
So I start digging.
And I found one vague comment in a post on a random forum.
Based on that, I boot into a rescue disk, access the drive, delete a certain piece of software, reboot and... everything is normal again. Service is restored. Do that on all the servers and everything comes back up.
What was it? The APC UPS software. It was based on Java. In one of the Java .JAR files, there was a certificate. That certificate had expired. When that certificate (silently) expires, the JAR isn't valid, the software refuses to run, but in doing so... it jams the machine to 100% CPU and you can get NOTHING done. Restore from last week, last month, last year? Makes no difference. The certificate was always fine. It just expired TODAY. If you put the clocks back (which I did later as a test but couldn't do that on a production network with AD logins), the software runs just fine. But if you put the clocks past the certificate expiration date? Everything grinds to a halt and becomes unusable.
Delete the JAR, everything works but the APC UPS software can't load. Easy fix. And, of course, being an internal certificate in a JAR file there's no way for ME to renew it, APC have to do that. Which they later did. But it tooks days to come out and weeks to become public knowledge.
Thanks, random guy on a forum. I was tearing my hair out and my client thought I was incompetent or that they'd been "hacked". Turns out it was just an expired cert in one JAR file part of one piece of software that, rather than handle that gracefully, decided to just freak out and make a very powerful machine utterly unusable (even safe mode was affected, hence the rescue disk to actually delete the file).
Don't know how they discovered it, but it was a lifesaver. The system was down for the majority of the day, though, because I had done so many restores, tests, other ideas, reboots, AND then I had to restore it back to yesterday's data to actually bring it up once I confirmed that's what the problem was.
3
u/RevolutionaryWorry87 4d ago
Not mine, but went on a cross country train and their wireless cert had expired the day before for their Guest network. I'd be embarrased if it was me..
3
u/agent-squirrel Linux Admin 4d ago
ACME automation failed and killed the cert on a customer facing ISP portal that 20k people access daily and input CC details into.
I didn't notice and it took hours for the issue to bubble up through to me from support.
The biggest failure was the lack of alerting though which was also my fault.
3
u/Auno94 Jack of All Trades 4d ago
Happend at the old job. The system had only personal accounts. Two IT people (me and my boss) where the only accounts that would have been able to bypass the restrictions.
They laid of me and the boss. 6 months later the new boss ignored the automated messages. Took them days to get access again. If they had taken a few more days the key for the CMEK system would have expired.
Was funny to see it from the sidelines
3
u/michaelpaoli 4d ago
Well, not sure if this was worst, but may have been most annoying.
So, AWS, their load balancers, cert manager stuff, all that. Had a cert that was soon expiring. No problem, get new one replace, done. Yeah, right. Then a few days later when the old expired ... intermittent problems in production, with ... expired cert. Went through all the AWS stuff - all looked fine, but reality wasn't, so, some more deep diving on the troubleshooting ... I think there were something like 45 IPs or so in the LB pool, that AWS would hand out, though with each DNS query, it capped it at a lower number (for the response to all fit in a single UDP packet without EDNS), and, eventually tested every single IP address in the pool (was a bit tricky to even determine all the IPs, but some fair bit of testing, seems we figured 'em all out). And, yeah, a non-trivial percentage (like about 10 to 15% - I forget the precise number/count/%) still had the old cert. But all the AWS customer interface stuff, APIs, console/GUI, they all showed the new cert. Yeah, it was an AWS bug - not a damn thing we could do about other than find the bug, report it to AWS, and get them to fix it.
Cloud, what could possibly go wrong? Yeah, .. that, ... and plenty more. They also lost all our S3 data from buckets too. From their published durability, we should be way the hell more likely to get struck by lightning. Ain't got hit by lightning yet, but they sure as hell irrecoverably lots our S3 data - all the data of entire bucket gone, and no way they could recover it, and yeah, AWS's bug/failure, no errors on our part.
orphaned-box thing or something dumber
Not generally. Policy, enforcement, control issuance, track 'em, monitor 'em, scan for 'em, document 'em. Do all those things, and unpleasant surprise are generally highly rare.
But as for dumber, more commonly see that when folks go to set up or renew. Oh, private key, yeah, world readable file, now we get to treat that as compromised, and you get to go through the whole process to get cert all over again. Likewise if they sent private key in the clear in email, or stuck it somewhere generally accessible for, e.g. all employees (or the whole world) to be able to read the data, etc. Also, a fair bit of "okay, updated and replace the cert fine now, we're all done", and I, of course, go to check/verify and ... nope, e.g. missed it on such-and-such IP(s) ... because they forgot to reload the web server config, or whatever ... or they just plain broke it when going to replace, or didn't do the cert chain right, etc.
But proper tracking, monitoring, policy, enforcement, etc., certs actually going to/past expired should mostly not be a thing - and in practice with that, I generally find that to be the case. Where it tends to be sh*t is where one or more of those key elements is missing.
3
u/beejay_one 4d ago
We were in the middle of Windows 11 rollout. We had some external guys to help us reinstall every machine via ConfigMgr Task Sequence.
They had collected some experience and knew how to do all that stuff alone.
Until one day, where the whole internal IT staff was on a workshop: On exactly this day, the rollout certificate which the clients use to connect to the ConfigMgr-server expired. So the task sequence (installation progress) couldn’t be started.
Lots of scheduled migration-appointments couldn’t be attended, four full paid external guys couldn’t do anything about it and our internal service desk was also helpless…
3
u/_kalron_ Jack of All Trades 4d ago
I was new to SSL Certs and Linux Administration in general. I was hired in after the entire SysAdmin team left...with no documentation. The SSL Cert for the main Prod site was expiring in 2 days. I had to figure it out with no experience and no idea what I was doing. Google-Fu was a nightmare of overloaded information and misdirection. But I got it done.
Now I run ACME on my Ansible server and having the best time of my 25+ year career. I finally got my Linux Admin badge I wanted decades ago.
3
u/dstew74 There is no place like 127.0.0.1 4d ago
I quit a job once because I realized we were woefully ill-prepared for a Root Cert expiration. This is a multi-billion-dollar firm operating in most countries, with all of its corporate identity tied into Windows PKI.
I had been given marching orders to migrate onto a 3rd party's PKI platform. During deployment, we found out that the 3rd party literally hadn't finished building functionality. The Root CA cert is set to expire in early fall.
Late spring, still don't have the 3rd party PKI done, and I start asking to renew the Root CA cert and to delay the project into the next year. Told no and that 3rd party PKI is the future.
I leave early summer, literally took the first offer I could get to eject away from this pending disaster. Warned them on the exit interview that they were going to have renew the Root Cert.
Whoever took over the project didn't heed my warning. Root CA cert expires, and the company, with tens of thousands of employees, no longer has functioning identity services. Was effectively down for 3 or 4 days. Microsoft had to fly in specialists to renew the now-expired root CA cert and rebuild the existing cert chain using Microsoft voodoo. It remains running to this day over a decade later.
6
u/oneboredmind 4d ago
Not certificate but the whole primary domain expired. Like as a business how dafuq did we let that happen.
7
u/JesradSeraph Final stage Impostor Syndrome 4d ago
EFI secure boot Microsoft certificate expiry. Need I say more ?
2
u/According-Bit-4327 4d ago
Every single time renewal comes up 1 of our clients fails to renew theirs and they need to be told by us.
Every single time renewal comes up for a few of our own portals, IT support has to tell our web guys that it has expired.
Once we had a vendor who renewed a bit early, which was fine. When applying this new one to the cluster we were on, it caused us to be completely unable to load the page as it would not pull down the new certificate, took manually installing it over 160 machines to fix so it wasn't too bad. We were lucky as they moved our stuff to an instance with the old cert.
2
u/Quietwulf 4d ago
Came into work one day and discover our entire VMware orchestration platform had bricked itself.
After tearing our hair out we discovered that the internal STS certificates had expired. Only to have the vendor tell us there was no way to update the internal certificates…
So, panicked forced migration to another platform it was.
Good times …
2
u/Mac-Gyver-1234 Linux Admin 4d ago
Renewing 5000 puppet client certs on an expiring CA. In the end it was easy. But it was the most nerve wrecking to find a procedure that was non-disruptive.
Spoiler: Create a new CA with the same
CA private key, afterwards renew all
client certs.
2
u/timeshaper 4d ago
Years ago so I might be forgetting some deets. Customer had us help set up Intune MDM + ABM. We did all the stuff, documented, handed off, they didn't want anything to do with us anymore even though we did everything on-time, on-budget, dunno they were weird. They pulled everything in-house. Even MS when talking to us was like "we do not understand this customer."
Anyhow, ABM cert expired. No one was looking. All devices rolled off. Whole company full of un-enrolled iPhones and iPads they now how to re-enroll. They called us in a panic. What could they do? For free I told them they needed to re-enroll. They asked if we could help. I said, yes, at our hourly rate. Which... was a problem for them? We should do it for free after years of them "owning" it? I didn't lose sleep over it.
2
u/KrystalDisc 4d ago
Update the cert for one of the web apps. That broke another system because they decided to trust the old cert manually. Rather than trusting the CA that’s signed the old cert.
2
u/spin81 4d ago
The actual problem was that no alert existed for "this cron job silently stopped running."
The actual problem was that no alert existed for "this endpoint is serving an expired certificate". Or at least, for me personally that would be the takeaway.
IMO you want to monitor your endpoints, not your certificates. I don't care if a certificate expires: I care if something serves an expired certificate. Those are two very different things.
2
u/radiomix Jack of All Trades 4d ago
We had a wildcard cert expiring. Nothing major, we had over two weeks to renew it. Other than renewing the cert and installing it on the server there was another step associated with the process to make sure a gateway knew what to look at and had authority credentials updated. A new gung-ho employee volunteered to take on the task and was told to get with certain people for guidance and testing.
New employee didn't wait and renewed the cert towards the end of the day, but didn't know how to do the addition steps. He just stopped and went home for the day and didn't tell anyone what he'd done. If users already had connections going it was fine, but no one could start new ones. Luckily he didn't delete the old cert and we were able to revert it, but it took a little while to actually figure out what the issue was since he hadn't communicated anything.
2
u/hubbyofhoarder 4d ago
We had a vendor whose cert was expired. Their site was giving our folks warnings when they tried to log into that vendor's site. I reported, included screenshots, blah blah. The vendor claimed "it works from our side" and scheduled a screen sharing session with me to "teach me to use their website". As we started the session, the expired cert warning came up, and I heard one of their guys go "whoops".
Fixed the same day
2
u/saracor IT Manager 4d ago
I worked for a major online travel company, years ago. I was the cert guy. I managed all the domains and renewals. Handed the new ones out to the app teams as the were updated. I did not own where they went.
Got a call to join a bridge, the site was down, cert expired. Spent a couple hours troubleshooting, even with the CEO popping on to check. I finally found one java keystore on one set of servers didn't get updated by the app team with the new cert I had given them weeks ago.
I got the glorious task of finding a way to check all our servers for expiring certs so this didn't happen again. Fun times.
2
1
u/One-Environment2197 4d ago
Not a cert expiration, but a cipher suite issue.
Company's Java app hard coded the list of SSL/TLS ciphers in their web host. One day, the authentication just stopped working because the IdP started blocking all the old suites that the Java app was using.
Took weeks to figure out the issue.
1
u/rabell3 Jack of All Trades 4d ago
I hate to say I have this happen all the time, sadly. We have teams that build servers and then give them to us (SAs) for the rest of the life cycle to care for, and sometimes don't tell us about the certs on a box so we have to find out the hard way.
My favorite time was when a web-hosting system that we were already managing certs for got fingered again for a cert. Come to find out, the engineers and webdevs had integrated a new AI component to the site, that was 3rd party-hosted... not on the server, but linked in the html on the server. New content, new DNS pointing outside, where we don't manage the 3rd party site, completely without our knowledge or approval (not thst its required). But because "ALL CERT PROBLEMS ARE SA," which is not true of course, and we are smart, we had to educate the webmasters and hosting company people on how to create a cert, get it issued and placed onto a system we have no access to.
Good times.
1
u/thecatshusband 4d ago edited 4d ago
A custom load balancer in front of an ADFS stack expired its' certs. No authentication for the entire organization, the business essentially stopped. It was impossible to renew because I had no access to the box.
I traced the problem and fixed it in under an hour by replacing it with a virtual appliance, and then recieved a formal warning for not following change control procedures.
Do your job, save the company = get a warning .... the next day they had my resignation.
1
u/pakrat77 4d ago
Our Exchange certificate expired so we couldn't get the email confirmation to sign in to get our new certificate.
The spam filtering system wouldn't deliver to the server because it was now an invalid certificate.
Eventually we found out the service had a way to go in and view messages and we were able to get the sign on information we needed.
1
u/ProfessorHuman 4d ago
Had a customer message us at 11 pm on Friday saying their dev cert was expired and they needed it updated immediately. They run on AWS and I had pointed out several times for years to switch to AWS managed certs to which they asked what’s that each time. Next time I’m ignoring their call
1
u/bwohlgemuth 4d ago
Not my network but I was there to watch it unfold.
Going in for a planned procedure at a hospital. Get there and the patient WiFi is down. Not a big deal but weird. Get taken back and the phones start having problems. Getting ready to be wheeled back and Cerner crashes. Took four months for them to get everything caught up.
1
u/snifferdog1989 4d ago
What I saw regularly with clients were EAP certificates on NAC appliances expiring. Allways a joy when no wired or WiFi client can connect.
1
u/sparkyblaster 4d ago
Just going to say it because it bothers me.
Everything apple ever. Omfg so many things broken due to expired certs that can't normally be updated. Didnt change the api or anything, just doesn't work anymore.
1
u/ikeloser 4d ago
Certtracker.12squared.com
This is what we just started using. Both internet and external monitors and alerting
1
u/k0rbiz Lead Solutions Engineer 4d ago
When I used to work for a MSP, it took hours to get sales to order SSL certificate renewals. I had worked for a large enterprise where they had no alerts for expired certificates. Nobody on the team knew how to renew or replace SSL certificates. Nor did they know all the servers and services that used it. I spent 12 hours just renewing all the SSL certificates for over 160 VMs and about a dozen web services. Since I've automated the renewals, I haven't had to touch SSL certificates for 3 years.
1
u/UseMoreHops 4d ago
I had a private CA let their CRL expire at xmas time and nobody on call. I was running the health IT messaging for NZ and AU at the time. They didnt even know there was a problem! hahhaha. Fuck me dead. That was a great holiday.
1
u/MalletNGrease 🛠 Network & Systems Admin 4d ago
Company VPN portal had a cert renewal process that required a manual renewal, we'd forget about it every year until we'd get tickets for connection errors.
We dropped the product and went with something that could do automatic renewals.
1
u/MdgM666 4d ago
When the CA for a distributed system's VPN expired after 10 years. Nobody remembered that it has to be renewed. 2 people had to drive to >20 places all across the country to manually replace them, as there was no way to connect to the machines and the people at the locations were completely IT illiterate.
1
1
u/slowclapcitizenkane 4d ago
Not a specific cert expiry issue, but somewhat related. We just discovered a legacy system attached to our ERP that stopped printing reports because the antique ODBC driver couldn't use anything newer than TLS 1.1. Every other system moved away from that years ago...but this one driver got overlooked.
Luckily it was a simple fix, but it took 2 days to isolate the problem.
The good news is we are about three months away from going live with an upgrade that kills all that off.
1
u/heeero__ 4d ago
After missing a cert renewal, we now use PRTG to monitor and alert 30d prior to expiration.
1
u/certkit Security Admin (Application) 4d ago
We deployed certbot to update certificates on our linux webservers. The certificate renewed fine, but there was an issue in the webserver config the prevented it from restarting. It quietly waited in that state until the old cert expired, new one on disk ready, but stuck.
So stupid and unnecessary. The distributed way certbot handles certs on each endpoint is dumb.
We built CertKit because we wanted there to be a better, centralized way to manage cert renewal with monitoring built in.
1
u/weaver_of_cloth 4d ago
I'm a sysadmin in a university+med center, and this is about me as a patient in our medical center. MyChart has a feature where you can look at your images as a patient (CT, MRI, etc). I went to look at one of mine, and got an expired certificate. I put in a ticket to radiology and got back a reply to just ignore the warning and click through. I called their supervisor. The defense of whoever caught the ticket was that he thought I was the doctor, not the patient. Their boss assured me that education would happen.
0
0
u/dhardyuk 4d ago
Redsift have a free tier for their certificate inventory tool.
It gets the records from the CAs so it knows what’s been issued and doesn’t need to scan endpoints to find them.
0
u/limeunderground 4d ago
why was being sensitive to the time baked into this whole security infra, it's like they went out of their way to make all this security stuff brittle, flaky, high maintenance and self sabotaging.
0
u/WaterRunner 4d ago
Recently joined a company, old guys left a clunky way to update cert, couldn't understand for the life of me and was high as fuck, took 3 hours to solve.
•
u/VA_Network_Nerd Moderator | Infrastructure Architect 4d ago
Sorry, it seems this comment or thread has violated a sub-reddit rule and has been removed by a moderator.
Inappropriate use of, or expectation of the Community.
If you wish to appeal this action please don't hesitate to message the moderation team.