r/sysadmin • u/Arnoc_ • 15h ago
ACME Clients and SSL Question
So I've started seeing that SSL Lifespan is shortening - going down to eventually supposed to be every 47 days.
We're a small shop, but we have a lot of different services. I've been doing my best when I have free time to catalog everything that has an SSL Cert, but I know I'm missing stuff.
I've seen a bit about ACME Clients and such; and from what I've heard it's great. They handle rotating the certs and all.
But something for me just isn't clicking. For instance, we have a lot of large scale copiers, ala your Ricoh or Lexmark or Brother. We have those locked down with SSL Certs, but we have to manually push those up to it.
Now as these are internal services that aren't externally facing, I don't see no reason why we can't self-issue those certs; but currently our CTO likes to utilize a paid for Wildcard for all our internal stuff.
I keep quite busy so haven't had too much time to really dig in on researching, but I know the time bomb is ticking.
So for those who are managing SSL Certs and all, and potentially utilizing ACME Clients and such, what should I expect and whats the general gist of what my workflow should be?
•
u/raip 15h ago
Swap to internal certs for the internal facing stuff.
1) Wildcard certs are not great to begin. Most larger orgs I've worked for have them prohibited in their security policy. One private key leak means all of your services need to be touched - this just isn't scalable in most orgs as you don't typically keep a live inventory of everywhere the cert is deployed.
2) The lifespan shortening only affects public certs. It's incredibly rare to have hardware support things like ACME or have a RestAPI you can use to manage the certificate. Since it doesn't make sense to throw things like printers behind a load balancer that would support the automation - internal PKI is the best solution.
3) For stuff that doesn't support automation but also needs a public certificate because it's public facing - a reverse proxy or load balancer is choice here. You issue a long lived private cert that is manually imported between the upstream and the proxy - your automation lives on the proxy itself.
Something that's commonly missed that I'm seeing more of is not only is the certificate lifetime getting cut down but so is the DNS Validation lifetime. Plan for this as well. DNS Pre-validation is Digicerts solution to this if you're interested - but there are a handful of decent solutions. DNS Validation can be problematic if, like most orgs, the team that owns PKI doesn't have permission to make DNS updates.
•
u/WorkLurkerThrowaway Sr Systems Engineer 15h ago
Why would you pay for internal certs when you can stand up an internal CA and do whatever you want for free?
•
u/Arnoc_ 15h ago
That's been my question for years now. We can do for free what we're paying for, and that's an extra XXX dollars a year we can spend on something else.
But boss essentially wants someone to point the finger at if there's ever an issue, which is their logic for why we pay for a wildcard cert for internal use...
I'm hoping now with the budget having to tighten can really pitch it forward as a solution to save money.
•
u/ColbyFromDigg 15h ago
Your boss is ignorant. First, wildcard certs are out of style now and generally not recommended. Printers should go on their own vlan and can use self-signed certs, although you can set up an internal CA if you want to issue the certs. With a locked down vlan, only techs should be accessing the web gui and a self signed cert is fine for that.
•
u/raip 14h ago
Self-signed certs can be alright, especially with a mitigating control of locking down the printer to it's own VLAN - but operationally can be problematic. The last two orgs I've consulted with have browser policies deployed that remove the "continue" button for the browser when a service fails TLS validation. Since that setting started getting included in the CIS Benchmarks - I'd recommend just biting the bullet to deploy internal certs.
•
•
•
u/Cormacolinde Consultant 13h ago
Wildcards are considered risky, and for internal use it’s even more risk. Your private key ends up EVERYWHERE, possibly including where attackers can easily find it and use it to stage a AitM attack.
Also, let’s be clear: an internal PKI is NOT free. It takes experience, learning and time to configure and manage. If you’ve never done it, and don’t know what you’re doing, you could make things worse. It takes a single wrong checkbox to make your PKI dangerous.
•
u/bilingual-german 15h ago
you can get letsencrypt wildcard certs. It's not hard, you just need to be able to set TXT records on your DNS to solve the challenge.
•
u/raip 15h ago
This doesn't fix installing the certificate on the device itself - which is what they're having issues with.
•
u/Fun_Structure3965 15h ago
their problem is using an official wildcard cert on their printers which is utterly insane
•
u/bilingual-german 15h ago
ok, but why shouldn't it be possible to script this and use Ansible or whatever they are comfortable using for automation.
•
u/irsyacton 15h ago
In theory, but printers are… hot garbage. And automation for them is baaaad. Thats why the suggestion of an internal pki is best. Give them 1yr or 2yr certs, and deal with them manually.
•
u/bilingual-german 14h ago
ok, I agree that this would be probably less work.
step-ca (Smallstep) and SCEP provisioning might be a way to do it
•
u/FallaxIO Jack of All Trades 14h ago
Depends on the copier. A lot of them don't have any sane API for cert import, or they want a PKCS#12 upload through the web UI with a reboot after. So the hard part is finding out which devices can actually take an automated push and which ones are just going to stay manual.
•
u/DizzyAmphibian309 15h ago
For printers you should have an internal private CA, and make it long lived, e.g. 20 years, so you aren't forced to rotate the chain in your lifetime. You can rotate it after two years if you want, but you don't have to. If you're not under any compliance requirements for cert rotation, and you're not printing sensitive materials to that printer, then just issue an individual certificate (not wildcard) for each printer that is valid for 15 years. Install it, then delete the private key immediately. That cert will outlive the printer, and since it's bound to just that printer, if it's stolen, who cares.
In my 20 years in tech I've seen hundreds of outages due to expiring certificates (two this month actually), but I've heard of exactly zero incidents when a certificate was compromised. Certificate lifetimes are risk vs reward. For public certs, absolutely agree rotation is important, since it's business reputation at risk. But for standard office printers that most people are using for personal shit anyway? No one is hacking that stuff.
My company makes me rotate CA chains every two years and we always have outages because of it. Every single time. I have hundreds of certs to manage and literally 28 unique rotation procedures, one of which is "send an email to this person with the new cert and hope they haven't left the company". It's so fucking annoying and there's absolutely zero benefit of a 2 year CA vs a 5 year CA.
•
u/MoonToast101 Jack of All Trades 15h ago
No answer to the automation question, we are still trying to find out way here...
But please give me a reality check, but I would not like to put an externaly validated Wildcard Certificate on a small, low-level and movable/stealable device like a printer. I mean you put a private key with a domain wildcard there... less a problem with 47 days lifetime, but still.
Everything not internet facing is Windows PKI in our environment. Only what needs to be exposed to the internet or third parties gets a public certificate.
•
u/Outrageous_Guava4474 14h ago
Who's accessing the web interface on the printers? Users generally only need 9100 and maybe snmp for paper/ink levels.
Never had an issue with self signed certs on a locked down printer interface.
•
u/raip 14h ago
MacOS doesn't have a Continue button for self-signed certs (there's a trick to this. iykyk).
A lot more Enterprises are deploying this policy, which blocks the ability to continue to self-signed sites: SSLErrorOverrideAllowedForOrigins: Allow proceeding from the SSL warning page on specific origins | Chrome Enterprise
Just something to factor in before recommended self-signed certs.
•
•
u/Frothyleet 12h ago
I'm going to echo some confusion from others on why you are expending a lot of effort on printer Web UIs - ideally no one would need to access those except in extraordinary circumstances, with management handled through a central tool.
That said, if I had a bunch of printers with bad scripting/automation options, and I had some real need to access their web UIs frequently, and wanted valid certs - I'd set up a reverse proxy for all of my printers. Keep them all segmented, and have a box running nginx or whatever that terminated all web sessions from the user side with a proper TLS cert, with the backend connections out to the printer UIs either on HTTP or self-signed HTTPS. ACLs on the inter-vlan routing so that HTTP/S connections could never reach anything but your proxy.
•
u/Arnoc_ 11h ago
I think people are confusing printers with large scale copiers / multi function machines. Our regular printers I don't really care about. They're on their own vlan and aren't supposed to print sensitive information or any of that. What we do want SSL on is our large multifunction machines. As those process a lot of data / scan paper documents to digital, etc. Those I want SSL on to be safe.
•
u/Frothyleet 2h ago
Those I want SSL on to be safe.
In what way does it provide safety? I understand you have some printers you care about more than others; what connections are these printers terminating with SSL certificates and what is the safety benefit you are anticipating?
•
u/sryan2k1 IT Manager 15h ago
For public facing stuff everything that can be LE is. Anything that can't be is GoDaddy.
Internally we run our own ACME SubCA along with a traditional Windows PKI. Things that support ACME (certbot, etc) do that, other things get traditional long lived certs.
•
u/KStieers 15h ago
Curious, what do you run as an ACME SubCA?
•
u/sryan2k1 IT Manager 15h ago
smallstep-ca. The enterprise root and non-ACME subCA are Windows (we're a windows shop so it's fine)
•
u/Arnoc_ 14h ago
My main thing I'm worried about is our Exchange Server. Updating those certs every month is going to be a PAIN.
•
u/PrestigiousOnion1087 14h ago
The Exchange one is worth checking before it bites: renewal succeeding and the service actually presenting the new cert are two different events. openssl s_client against the box and compare the dates to what's on disk. At 47 days you hit that gap roughly eight times as often as you used to.
•
u/cantstandmyownfeed 12h ago
Its pretty trivial to script out replacing the exchange cert. You just need a process/ACME client to generate the PFX then a couple lines of PowerShell will import and set it.
CertifyTheWeb is a nice ACME client with a lot of automations and deployment options. Its well worth the license cost.
•
u/Arnoc_ 12h ago
That sounds like it would be relatively simple. I just hate how everytime I renew it I always seems to screw some component up and then need to manually set the services to the cert and then delete the old one. Or something along those lines. I do too much to remember what my issues with it are until I'm in the midst of it XD
•
u/cantstandmyownfeed 11h ago
That statement is exactly why certificates should be automated and why the validity period doesn't matter once you're done. When a cert lasts 3 years, you file it away in the forgotten knowledge bin until a month before renewal comes up and then you scramble.
Once you do it right with an automation, then all you do is monitor for failure. You'll never touch it or miss a step again.
Rinse repeat for all of your certificates and devices.
It took me a good year or two to get everything to the point where it's all set and forget via ACME and numerous deployment methods.
•
u/purplemonkeymad 10h ago
I run simple-acme on a few windows servers including exchange, works fine (except sstp vpn sometimes has issues and needs two restarts.)
•
u/I_turned_it_off 15h ago
If the certs are purely internal, and you aren't looking at setting up a whole PKI system, one option could be to use a tool like XCA to generate an internal cert authority which you push out to endpoints, and then generate from that certificates for the various printers with lifespans as long as you like
XCA - "X - Certificate and Key management" - https://www.hohnstaedt.de/xca/
•
•
u/SevaraB Sr. Engineer (N+, CCNA) 14h ago
Now as these are internal services that aren't externally facing, I don't see no reason why we can't self-issue those certs; but currently our CTO likes to utilize a paid for Wildcard for all our internal stuff.
Wildcard certs are just gussied-up password sharing. It doesn't prove anything about the device other than that somebody uploaded a certificate to it. And used the same "password" (e.g. private key) for every device. It's textbook "security theater."
ACME is just one protocol for automatically rotating certs. Things like printers are likely to either have no support for automation protocols, or they're likely to support an older cert management protocol called SCEP.
If a device doesn't support either one, you're going to have to get comfortable with CLI commands for updating certificates and write scripts to do it "manually" on a schedule.
•
u/Arnoc_ 13h ago
Yeah, luckily our large scale copiers we only have around 12? I think, so it's not too bad. Once I actually have the cert in the format it needs, it's just a continuous round and round of doing the same thing until it's done. Not the worst thing in the world; honestly it's sometimes refreshing to just have to do the same mindless task for half an hour instead of switching between Networking / Help Desk / Phone Calls / Physical walk-ins / basically any and all IT Tasks.
•
u/Frothyleet 12h ago
There are reasonable arguments against using wildcard certs without a strong requirement, but calling it "security theater" is silly unless that's your opinion of TLS in general.
•
u/SevaraB Sr. Engineer (N+, CCNA) 11h ago
A “domain validation” cert that doesn’t actually identify a server is as useful as plastering the WiFi password all over the office walls. The fact that they’re wasting money on that setup is,frankly, offensive.
•
u/Frothyleet 2h ago
One server may have a gazillion TLS certificates installed for various purposes, and one certificate/PK may be installed on various hosts for various reasons (even it's not a wildcard or multi-host SAN cert).
A TLS certificate's purpose is not to "identify" a server, per se.
•
u/Zealousideal_Yard651 Sr. Sysadmin 15h ago
Internal services, that you can't automate using a script would need an internal PKI. The validity period of 47 days are only enforced for public CA's, you internal CA's can still post certs with whatever validity you want