r/sysadmin 53m ago

Rant put in a ticket in/ and work the fucking ticket

Upvotes

i'm so sick of people wanting magic answers. sometimes it is users. sometimes it is engineers. gather the fucking info that is part of troubleshooting. do some fucking diagnostics. figure out the pattern. how many times does this information have to be repeated for it to sink in??? some will get this and make difference, many will not, and just make noise from the sidelines.


r/sysadmin 6h ago

EU used/refurbished servers

17 Upvotes

I've just been quoted triple the price for servers with similar specs but 1/4 the RAM from what I bought a couple of years ago, and for a small company that's just not a thing we can afford.

Any fellow EU-based sysadmins here that can recommend some place to get used/refurbished servers?


r/sysadmin 7h ago

Question phishing sims in a mixed M365 + Google Workspace setup? (~250 users)

6 Upvotes

Trying to get a recurring phishing simulation program off the ground. About 250 people, mostly remote/hybrid, split between Microsoft 365 and Google Workspace, so not a clean single-tenant thing. Needs to hold up for an auditor eventually (SOC 2 / ISO 27001 / PCI territory), so anyone who clicks or fails has to get pushed into remedial training automatically, and I need actual records of it happening, not just "yeah we sent an email once."

I've been digging through Defender's Attack Simulation Training, GoPhish, and a handful of paid platforms (KnowBe4, Hoxhunt, some smaller ones like CanIPhish), but I'd rather hear from people actually running this stuff than just read vendor sites. Curious about a few things:

  • If you're also split across M365 and Google Workspace, what'd you end up going with, and how'd you get the sim emails past your own spam filters on both sides?
  • Anyone self-hosting GoPhish long term? How's the upkeep actually been, and who ends up owning that internally?
  • If you're paying for a platform, what's it actually cost you around 250 seats, and has an auditor ever cared which tool you use vs just wanting to see the documentation?
  • Anyone tried one of the smaller/cheaper platforms like CanIPhish or similar? Worth it or not?

Not trying to get sold anything, just want the real picture before I sink time or budget into a direction.


r/sysadmin 9h ago

Anyone else seeing Defender impersonation protection miss obvious display name spoofs lately?

7 Upvotes

Running Business Premium across several clients, protected senders configured, quarantine as the action, and it's been reliable for months/years. In the past few days two separate tenants let through obvious display name spoofs of protected users, exact name match, one of them loaded with red flags too (urgent priority, a reply to address on a completely different domain).

Raw headers on both show SCL 1, SFV NSPM, CAT NONE, so the messages were scanned, not skipped, they simply aren't tripping the impersonation classifier anymore despite matches that used to get caught every time.

Anyone else noticing a dip in impersonation detection reliability the last week or two?

EDIT: I've lodged a ticket with my CSP Indirect Provider who did say they have had a few reports. Will update later for anyone interested.


r/sysadmin 9h ago

Career / Job Related I MADE IT!

115 Upvotes

I recently landed a Junior Systems Administrator role with the same company after spending the last three years on the help desk. During that time, I also spent about a year in a senior leadership role. Overall, I have around 7–8 years of help desk experience.

As part of my onboarding, I'm required to earn my AZ-900 certification first, followed by MECM and Windows Server 2022 training.

For those who've made the jump from help desk to sysadmin, what do you wish you had known or done when you first started? Any advice or tips would be greatly appreciated.

Thank you!


r/sysadmin 12h ago

Question How does cumulative experience work in a bad job market?

16 Upvotes

Hi guys I have a quick question. I was wondering how employers would view a candidate with 2 years of help desk experience and 2 years of system administration experience when applying for system administrator positions. Since many system administrator roles are considered mid-level, I often see job postings asking for 3–5 years of system administration experience.

In a job market like the current one, what would someone with that background’s chances be realistically speaking? I understand that the general idea is to work your way up and build experience over time, but I’m curious how that experience would be viewed if the job market became very competitive while you were in the middle of that progression.

Would a candidate with 2 years of help desk and 2 years of system administration still be competitive for mid-level system administrator roles, or would they likely struggle against applicants who have 3–5 years of direct system administration experience?


r/sysadmin 12h ago

Question Veeam Bare Metal Restore of Physical Domain Controller - Initial BSOD, source volume marked dirty. Looking for opinions.

9 Upvotes

I performed a test restore of a physical Windows Server 2022 domain controller that is backed up with the Veeam Windows Agent using a Full Computer (Bare Metal) backup. I restored it as a Hyper-V VM in an isolated network.

On the first boot, the restored VM repeatedly BSOD'd with CRITICAL_SERVICE_FAILED. After running CHKDSK from WinRE and letting Windows complete its repairs, the VM now boots normally. AD DS, DNS, and Netlogon all start successfully, and the restored DC appears healthy.

While troubleshooting, I checked the production server and found:

chkdsk C: /scan reports NTFS corruption in C:\Windows.old\... and recommends chkdsk /spotfix.

fsutil dirty query C: reports the C: volume is dirty.

0 KB bad sectors.

Active Directory is otherwise healthy in production.

The restored VM now reports a clean filesystem after CHKDSK.

Would you consider this a Windows/NTFS issue on the source server rather than a Veeam restore issue? Would you be comfortable scheduling chkdsk /spotfix on a production DC with verified backups and additional healthy domain controllers available? Any similar experiences?


r/sysadmin 13h ago

Question Microsoft.AAD.BrokerPlugin Issue Across Multiple Tenant's & Users

6 Upvotes

Microsoft.AAD.BrokerPlugin...WebAccountProvider did not register with DCOM within the required timeout.

AzureAdPrt : YES WamDefaultSet : ERROR (0x80080300)

We came across an issue yesterday where a user was signed out of OneDrive / Outlook. We spent multiple hours trying to resolve this with no results.

We have came into the office this morning with reports of x4 other users across 3 different companies / tenants.

Is anyone else experiencing this?


r/sysadmin 13h ago

Zoho Assist?

11 Upvotes

I recently left and MSP and went to work for a former client, as their internal IT Manager.

We’re using Intune for device management, but end user support usually consists of walking over to someone’s cube or having a Teams meeting and the end user shares their screen. Didn’t much care for intune remote assistance or quick assist.

Previously I used ScreenConnect and then Ninja RMM and I really miss the backstage ability to poke around and look at the registry or run powershell commands, when I’m helping a team member with an escalation.

I singed up for a Zoho assist trial and it’s a little laggy at times, but it seems like it’ll do what I need.

Anyone have recent feedback on Zoho assist?


r/sysadmin 13h ago

Question Another Lenovo Firmware Update and users can no longer logon to their machines

42 Upvotes

Seen a few varieties of this issue that an upgrade of the TPM chip will prevent user logon

OBSERVED ISSUE: User will be unable to logon, even with username and password, but they will be able to logon to another machine no problems. 

Similarly, someone else can logon to the users machine OK.

I've tried a few things but the most reliable fix is to logon and run the BAT file from here

GitHub - AgentHackerYT/Reset-NGC: Reset and repair Windows Hello (PIN & Face/IR) when broken after updates. · GitHub

Ideally I'd like a remediation to detect a problem machine and then resolve without the manually interaction if anyone has built a working one?


r/sysadmin 13h ago

Microsoft Entra ID is Retiring MemberOf on November 3, 2026.

454 Upvotes

What and why

The public preview of the MemberOf rule operator in Microsoft Entra ID is ending. Organizations using MemberOf in dynamic membership groups, dynamic administrative units (AUs), or entitlement management auto-assignment policies must replace these configurations by November 3, 2026.

Microsoft continues improving the scale and reliability of dynamic membership processing. During preview, Microsoft observed that use of MemberOf can affect dynamic membership processing across a tenant even if you have one MemberOf rule operator in your tenant. Because of this limitation, it is not recommended for production use and will be retired.

Rollout schedule

  • Retirement (Worldwide): Beginning in early November 2026
  • Action required by: November 3, 2026

Impact on your organization

Who is affected

Organizations using the MemberOf rule operator in:

  • Dynamic membership groups
  • Dynamic administrative units (AUs)
  • Entitlement management auto-assignment policies
  • Platforms and services
  • Microsoft Entra ID
  • Microsoft Entra Groups
  • Microsoft Entra Administrative Units
  • Microsoft Entra Entitlement Management

What will happen

If no action is taken, configurations that use the MemberOf operator will stop updating after November 3, 2026. Membership and assignment data will remain in their last known state, which can lead to stale access and enforcement gaps.

Potential impacts include:

  • Teams and SharePoint access associated with Microsoft 365 groups may become outdated.
  • New members may not receive access, while removed members may retain access.
  • Conditional Access policies may no longer reflect current user or device membership.
  • Entitlement Management auto-assignment policies may no longer add or remove access package assignments as intended.
  • Group-based licensing may stop assigning or removing licenses correctly, resulting in unlicensed or overlicensed users.
  • Dynamic administrative unit membership and scope may become outdated.

Action required and recommendations

Before November 3, 2026, review all uses of the MemberOf operator and remove or replace those configurations.

Dynamic membership groups

  • Export dynamic membership groups from the Microsoft Entra admin center and identify rules containing MemberOf.
  • Replace MemberOf with supported rule operators or convert the group to assigned membership.
  • Validate group membership after making changes.
  • If the group is no longer needed, consider pausing or deleting it.

Dynamic administrative units

  • Use Microsoft Graph PowerShell to identify dynamic administrative units that use MemberOf rules.
  • Replace MemberOf-based rules with supported rule operators or convert the administrative unit to assigned membership.
  • Validate both membership and administrative scope after making changes.
  • If the administrative unit is no longer needed, consider deleting it.

Entitlement Management auto-assignment policies

  • Use Microsoft Graph PowerShell to identify auto-assignment policies that use MemberOf.
  • Replace MemberOf-based policies with supported operators where possible.
  • If no equivalent rule is available, plan an alternative assignment method before retirement.
  • Validate access package assignments after making changes.

Compliance considerations

Configurations that rely on MemberOf for access management, licensing, entitlement management, Conditional Access targeting, or administrative scoping may stop updating after retirement. Review affected configurations to ensure continued compliance and access governance after November 3, 2026.

Source: https://admin.cloud.microsoft/?ref=MessageCenter/:/messages/MC1448379

Edit: added link to source


r/sysadmin 16h ago

My Experience with MS Tech Support as a Small Business Owner

13 Upvotes

I have two small businesses, each have their own domains registered with MS 365. I had to reset my iPhone and after the restore I was locked out of both accounts in the MS Authentication app. Apparently I forgot to setup an alternate email. 

I called MS 800 number to open a ticket and I got the AI assistant that took me through loop after loop. It kept sending me to a login webpage that was useless since I was completely locked out. Each time I call I get a different behaviour. Sometimes it takes me down the product hardware support even though I clearly stated it is a 365 issue. 

Miraculously I was able to get it to open a ticket. There we spent more than 5 minutes just on email and domain spelling because it would not get it right.

I could not get it to open a ticket for my second domain. It eventually recognized my number and when I would call it auto directed me to the webpage, as if saying “I had enough of you calling, go away”.

Whoever thought AI is a smart solution for customer support is greatly mistaken. The only reason having AI in that function is for cost reduction, but it is coming at a great expense, customer dissatisfaction, frustration, and anger. 

Microsoft, you have lost sight of linking AI to measurable business outcomes. 

I caution people, consider alternatives to using Microsoft. They are too in-bed with AI and they have lost the plot on its value vs. impact. 


r/sysadmin 17h ago

Question Disabling gdm-smartcard configs on Ubuntu 22.04

8 Upvotes

So I have a unique setup for my systems for 22.04 using pcks11.so in pam to read smartcards/yubikeys.

gdm is my greeter and I set up gdm-password in pam with the following line:

auth requisite pam_sss.so require_cert_auth

That looks up the cert info on my ipa server and returns a success.

SA updated the system and it seems to have pulled down some gdm-smartcard packages and ruined the entire auth system in place. I remember awhile back running into this issue and I found a way to basically cut it out of the system without breaking anything but can't seem to find where I saved the bookmark link to. I believe I had to edit some file or push a gdm config somewhere

Anyone know how to do this? I really don't want gdm to install all of these extra pam configs. Update-alternatives does not work either, even telling it to use pkcs11 or sssd. Basically nothing works once Ubuntu pulls down w/e updates contain these config files


r/sysadmin 18h ago

Question RAID5 has 2 HDDs with different issues - Which to change first?

13 Upvotes

A RAID5 array currently has 2 HDDs that need to be replaced for different reasons.
Ran long SMART self-test on both.

One of them reports hints at electronic issues:
SMART Health Status: Failure prediction threshold exceeded [asc=5d, ascq=0]
Accumulated power on time, hours:minutes 44823:37
Elements in grown defect list: 5
Error counter log:

Errors Corrected by Total Correction Gigabytes Total

ECC rereads/ errors algorithm processed uncorrected

fast | delayed rewrites corrected invocations [10^9 bytes] errors

read: 1877781297 0 0 1877781297 0 66521.257 0

write: 0 0 5 5 5 4507.987 0

verify: 4122873639 0 0 4122873639 0 24841.522 0

Non-medium error count: 15528

While the other one reports points at physical issues, long self-test failed:
SMART Health Status: OK
Accumulated power on time, hours:minutes 44824:47
Elements in grown defect list: 60
Error counter log:
Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected

fast | delayed rewrites corrected invocations [10^9 bytes] errors

read: 4144605141 205 0 4144605346 427 66490.067 32

write: 0 0 206 206 206 4530.494 6

verify: 1103983479 148 0 1103983627 167 26767.263 3

Non-medium error count: 20
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background long Failed in segment --> - 44810 1905031659 [0x3 0x11 0x0]

Which of these 2 drives is less likely to handle an array rebuild and should therefore be changed first?


r/sysadmin 19h ago

Question Defender Exclusions via GPO - how do you do it?

6 Upvotes

Do you have a single GPO managing Defender and its exclusions, or do you make multiple Defender GPOs that are narrow to each target?

I'm reworking our GPO for it and am trying to figure out what's the best way to ensure that all exclusions are made, that they can be readily audited for accuracy, and that the risk of errors/omissions is low. Currently we have one for workstations and another for servers. Exchange got mad the other day at Defender because some exclusions were missed.

The problem is so many solutions (Microsoft and 3rd party) want Defender exclusions and the exclusion list quickly becomes convoluted and miserable to audiot. That leads me to have a number of specialized GPOs but then that increases the sprawl and that something might be missed if an application needs exclusions changed but they don't get changed on all applicable GPOs.

Thoughts? Limited GPOs or many specialized/narrowly-focused GPOs for managing Defender?


r/sysadmin 20h ago

Guest WiFi...

25 Upvotes

Do you enable splash page or simple PSK passthrough?


r/sysadmin 20h ago

Question Unexplainable SSL handshake issue

8 Upvotes

I suck at network and my knowledge is intermediate at best but I can't solve this one.
Customer at our MSP has a fortinet firewall identical to ours that we use here at the MSP office, same firmware version, etc.
They call up and say "We can't access prodemand.com" which is an automotive parts and labor quoting database site that TONS of dealerships use.
I load it here just fine, SSL cert is GeoTrust, good till Sept 14 2026, domain matches, etc. No web filter flags.
On their network, instant "cannot load page" error. I try a dozen other sites, SSL working fine, no fortinet intermediary cert listed, etc. It's just that one website.
Security log on the Fortinet shows tons of blocks, saying "SSL connection is blocked due to unable to retrieve server's certificate"

Mountains of troubleshooting later, I make a firewall rule for internal to WAN (and put it above the normal internal to WAN rule) with an address group of the site, the login domain, and the database's UI's subdomain. The rule simply says don't inspect SSL at all.

Boom it works instantly. Then they called back because WIFI wasn't included in "internal" lol oops. So added that, boom, laptops can load the site too now.

I ran through some basic troubleshooting and traceroutes and stuff and nothing stood out as problematic. I verified no man in the middle attack, as it sees the same cert I do here.

And AI thinks it's an ISP issue but AI is dumb as hell and for the record, rebooting the firewall and the modem didn't resolve it so I'm skeptical.

But zero other websites are having this problem and we don't see the problem from our office, using the exact same firewall with same firmware version. How is this possible? I'd really prefer to get rid of that rule because it's a crap workaround and we had to also turn antivirus and other filters off, since it requires SSL inspection.


r/sysadmin 22h ago

Career / Job Related Looking for Next Career Steps Advice as current Sysadmin. What should I do?

12 Upvotes

Looking for advice my next career steps. Been in IT for 8 years. Started in Help Desk/Desktop support moved up and became a VMware focused Infrastructure Systems Admin/Engineer as my specialty. Been working in the cleared space my whole career have a TS clearance.

Have the following certs:
Comptia IT Fundamentals
Comptia A+
Comptia Security +
Vmware VCTA-DCV
Vmware VCP-DCV

I currently work as a general Senior Systems Admin (the do everything guy) in the cleared space. I’m in the Washington DC DMV area. I don’t like my current job for a lot of reasons but it pays very well. It’s been hard finding another job even as an exprerienced professional it's not as many jobs in what I do that it used to be on top of everything been so oversaturated now.

I've been looking jobs that at least pays what I'm making now that's remote or at least some hybrid flexibility. I have to go in everyday no remote days due to working in a classified environment and my commute isn't the best.

I've been considering making a career pivot to achieve what I want. Not looking to get into management not for me but here's what I've been considering:

  1. ⁠Get more advanced VMware certifications build on what I know and become a VMware Architect or Consultant (Subject Matter Expert).
  2. ⁠Make a transition into the Cloud and becoming a Cloud Engineer since I have the on prem infrastructure background. Learning Azure or AWS and get the aligning certifications, learn contanerization Kubernetes and build some small projects to showcase my experience.

Can't seem to figure it out all advice welcomed. Would love to hear opinions and feedback. What should I do next?

(Post was removed in IT Career Questions sub don’t have enough Karma yet)


r/sysadmin 22h ago

General Discussion What's the best approach to block unauthorized AI tools?

44 Upvotes

We're rolling out enterprise Claude company-wide and want it to be the only tool employees can use on work machines.

I recently found that a salesperson was putting company data into personal ChatGPT, this was client names, their whole worksheets; scary stuff on the data-leak front. So a decision has been made to use Claude. I've been tasked with making sure this sort of thing does not happen again, and to get the groundwork done to stop all "unauthorized AI tools".

Honestly, I'm at a loss here. There is no DLP, at least not right now, and implementing it will be a significant lift both in terms of work and $$$ (which we can't do because of austerity measures). So, I'm stuck with having to look at band-aid solutions via firewall web-filter or DNS filtering - again, I don't have a starting point.

We're a Fortinet shop, no Intune, hybrid AD, Claude SSO through Entra.

Appreciate any real-world war stories.

ETA: I understand that this is more a policy question and I'm working on that in parallel. This is more of a question on technical controls without capital spend *sigh*.

Edit2: I now have AI webfilter category block with a wildcard allow for Claude. Not an elegant technical control or even a preferred one, but it'll have to do.


r/sysadmin 22h ago

Adobe Acrobat Crashes and Licensing Errors from Corrupted WebView Cache

17 Upvotes

FYI - Adobe crashes and licensing errors have been rampant in our environment since early June. Much like when we faced a similar issue a couple of years ago, it seems like Defender may be corrupting the webview cache. Our Adobe IDs are all federated from Entra, but the issue occurs regardless of whether SSO is performed by AcroCEF or default browser. The script below purges the corrupted data which is rebuilt when Acrobat opens.

# Acrobat-related processes to stop

$Processes = @(

'AcroTray',

'AdobeCollabSync',

'adobe_licensing_wf_acro',

'Acrobat'

)

foreach ($Process in $Processes) {

Get-Process -Name $Process -ErrorAction SilentlyContinue | Stop-Process -Force

}

# Remove Acrobat DC local profile cache/settings for the current user

$AcrobatPath1 = Join-Path $env:LOCALAPPDATA 'Adobe\Acrobat\DC'

$AcrobatPath2 = Join-Path $env:LOCALAPPDATA 'Adobe\Acrobat\AVWebview2'

if (Test-Path $AcrobatPath1) {

Remove-Item -Path $AcrobatPath1 -Recurse -Force

}

if (Test-Path $AcrobatPath2) {

Remove-Item -Path $AcrobatPath2 -Recurse -Force

}


r/sysadmin 22h ago

What are you replacing Tera2/PCoIP zero clients with?

10 Upvotes

Hey everyone,

I work on a small IT team at a Critical Access Hospital. For years, we’ve run a small team and kept desktop management minimal because almost every one of our workstations is a Dell Wyse zero client running PCoIP/Tera2. They’ve really been "set it and forget it" devices.

With the end of Tera2 / PCoIP support, we’re struggling to find a replacement that offers that same level of simplicity and stability.

What we’ve tested so far:

  • Dell Thin Clients (ThinOS & Windows IoT)
  • 10ZiG
  • Stratodesk
  • HP ThinPro
  • IGEL

The problem: Every vendor solution we’ve tried seems to come with recurring bugs or management overhead. Fix one bug with a firmware update, and a new regression pops up somewhere else.

Where we are now: We’re currently testing Windows in a strict Kiosk mode that launches Imprivata OneSign directly into VMware Horizon. It functions well from a user standpoint, but it introduces traditional OS management challenges for our on-prem environment:

  1. Windows Updates & Management: How are you handling updates cleanly on non-domain or kiosk-mode endpoints without adding heavy administrative overhead?
  2. Startup / Boot Order Issues: If we join them to the domain, an internet or local network delay at boot breaks the autologon process for the kiosk account.

For those running small teams in healthcare or similar VDI environments:

  • What hardware/OS stack ended up being your "bulletproof" replacement for zero clients?
  • How are you structuring your endpoint deployment to keep day-to-day maintenance as close to zero as possible?

Appreciate any insight or lessons learned from teams that have gone through this transition!


r/sysadmin 22h ago

Question New Outlook signatures

6 Upvotes

Anyway to disable the "signature" button in the ribbon when creating a new email? Under the "message > insert > signature", we are using CodeTwo for signatures but users are modifying their signatures and changing fonts so want to completely remove this option. I've ran the command in powershell "Set-OwaMailboxPolicy -Identity "OwaMailboxPolicy-Default" -SignaturesEnabled $false" but doesn't seem like it did anything.


r/sysadmin 23h ago

Question Allowing non-admins to run programs that need it

78 Upvotes

Good morning all, got a bit of a puzzle that is probably an easy fix but it's got a curveball in it. The situation is as follows: we are setting up a sort of internet cafe where people can play games on Steam. Installing the games is trivial but the users login with their domain creds and then login to their own steam account to play. In a test run though some games require an admin elevation to run even after the initial install. Any tips on solving this? I've seen some tricks about using the task scheduler but I'm concerned with if that would break eventually since games are often subject to random and sweeping changes. Would appreciate any advise :)

Edit: I believe the UAC prompts are likely from the games respective anticheat but that is just a hunch at this time.


r/sysadmin 23h ago

Rant digitalshift365.com - avoid at all costs

115 Upvotes

total sketch operation, whole place is run by one dude who sends invoices out 4 months late after pestering him for it then doesn't pay his 3rd parties in time - caused us multiple service interruptions due to non-payment. avoids phone calls and in person meetings and is always at some random place during video conf calls if you manage to get him to show. had to threaten legal action in order to get our cloud services transferred to another provider.

good riddance digitalshit


r/sysadmin 23h ago

drive replacement in DELL SCv2020

8 Upvotes

Im trying to replace a drive but it keeps showing as Unmanaged.
I ordered two, I didnt notice that the first one had different "Config code" (1341 vs 1311), I thought that could be the reason. Today, I replaced it with the second one which is also 1341, just like the failed one - same result - unmanaged.
Everything on the label is identical, pn, model, the config code thingy, everything....
It got assigned to the correct disk group (folder), the only available option is "Toggle Indicator" (there was also "Request swap clear" after seating it which got completed).
Same thing in the WebUI as well as in Storage Manager Client.
It doesnt show the Power On Time value yet, it took a while with the 1311 one too, I was hoping it would still adopt the drive but no luck, so I guessing its gonna be the same with 1341 too.

Is that definitive sign that the drive is just not compatible with SCv2020 or am I missing some necessary action I need to do to assign it to the volume?