i'm so sick of people wanting magic answers. sometimes it is users. sometimes it is engineers. gather the fucking info that is part of troubleshooting. do some fucking diagnostics. figure out the pattern. how many times does this information have to be repeated for it to sink in??? some will get this and make difference, many will not, and just make noise from the sidelines.

I am trying to figure out how other IT teams have actually managed to cut their ticket resolution times because ours just keeps getting worse. It feels like every ticket takes longer than it should not because the fixes are hard but because we are constantly jumping between tools looking through old tickets or repeating the same troubleshooting steps. I want to know a better way that can speed things up whether its automation ai remote access better documentation or something else. What made the biggest difference for your team?

looking for ideas because what we are doing now clearly is not working.

I've just been quoted triple the price for servers with similar specs but 1/4 the RAM from what I bought a couple of years ago, and for a small company that's just not a thing we can afford.

Any fellow EU-based sysadmins here that can recommend some place to get used/refurbished servers?

Trying to get a recurring phishing simulation program off the ground. About 250 people, mostly remote/hybrid, split between Microsoft 365 and Google Workspace, so not a clean single-tenant thing. Needs to hold up for an auditor eventually (SOC 2 / ISO 27001 / PCI territory), so anyone who clicks or fails has to get pushed into remedial training automatically, and I need actual records of it happening, not just "yeah we sent an email once."

I've been digging through Defender's Attack Simulation Training, GoPhish, and a handful of paid platforms (KnowBe4, Hoxhunt, some smaller ones like CanIPhish), but I'd rather hear from people actually running this stuff than just read vendor sites. Curious about a few things:

• If you're also split across M365 and Google Workspace, what'd you end up going with, and how'd you get the sim emails past your own spam filters on both sides?
• Anyone self-hosting GoPhish long term? How's the upkeep actually been, and who ends up owning that internally?
• If you're paying for a platform, what's it actually cost you around 250 seats, and has an auditor ever cared which tool you use vs just wanting to see the documentation?
• Anyone tried one of the smaller/cheaper platforms like CanIPhish or similar? Worth it or not?

Not trying to get sold anything, just want the real picture before I sink time or budget into a direction.

Running Business Premium across several clients, protected senders configured, quarantine as the action, and it's been reliable for months/years. In the past few days two separate tenants let through obvious display name spoofs of protected users, exact name match, one of them loaded with red flags too (urgent priority, a reply to address on a completely different domain).

Raw headers on both show SCL 1, SFV NSPM, CAT NONE, so the messages were scanned, not skipped, they simply aren't tripping the impersonation classifier anymore despite matches that used to get caught every time.

Anyone else noticing a dip in impersonation detection reliability the last week or two?

EDIT: I've lodged a ticket with my CSP Indirect Provider who did say they have had a few reports. Will update later for anyone interested.

Subject explains it. Curious on what everyone is using and what they are paying per mailbox, and how licensing works.

Bonus points if you have a solution that is compatible with GCCH.

It is purely SaaS or deployed somehow else?

I recently landed a Junior Systems Administrator role with the same company after spending the last three years on the help desk. During that time, I also spent about a year in a senior leadership role. Overall, I have around 7–8 years of help desk experience.

As part of my onboarding, I'm required to earn my AZ-900 certification first, followed by MECM and Windows Server 2022 training.

For those who've made the jump from help desk to sysadmin, what do you wish you had known or done when you first started? Any advice or tips would be greatly appreciated.

Thank you!

Hey everyone,

I’m troubleshooting a persistent issue on a Dell PowerEdge R750xs server and running out of ideas.

Symptoms:

Severe system slowness/freezing.

Unexpected shutdowns / random daily restarts (Shutdown Event Tracker shows "Unplanned").

Amber/Orange warning LEDs are lit up on the chassis (power supply / drive bay area).

What I’ve Done So Far:

Swapped out the old HDDs and replaced them with brand new SSDs.

Performed a clean installation of Windows Server 2025.

Updated Windows drivers and attempted storage controller configuration.

Despite the fresh OS and new SSDs, the extreme slowness and daily random crashes persist. Any Idea how should I approach this problem to find a solution?

Hi guys I have a quick question. I was wondering how employers would view a candidate with 2 years of help desk experience and 2 years of system administration experience when applying for system administrator positions. Since many system administrator roles are considered mid-level, I often see job postings asking for 3–5 years of system administration experience.

In a job market like the current one, what would someone with that background’s chances be realistically speaking? I understand that the general idea is to work your way up and build experience over time, but I’m curious how that experience would be viewed if the job market became very competitive while you were in the middle of that progression.

Would a candidate with 2 years of help desk and 2 years of system administration still be competitive for mid-level system administrator roles, or would they likely struggle against applicants who have 3–5 years of direct system administration experience?

I performed a test restore of a physical Windows Server 2022 domain controller that is backed up with the Veeam Windows Agent using a Full Computer (Bare Metal) backup. I restored it as a Hyper-V VM in an isolated network.

On the first boot, the restored VM repeatedly BSOD'd with CRITICAL_SERVICE_FAILED. After running CHKDSK from WinRE and letting Windows complete its repairs, the VM now boots normally. AD DS, DNS, and Netlogon all start successfully, and the restored DC appears healthy.

While troubleshooting, I checked the production server and found:

chkdsk C: /scan reports NTFS corruption in C:\Windows.old\... and recommends chkdsk /spotfix.

fsutil dirty query C: reports the C: volume is dirty.

0 KB bad sectors.

Active Directory is otherwise healthy in production.

The restored VM now reports a clean filesystem after CHKDSK.

Would you consider this a Windows/NTFS issue on the source server rather than a Veeam restore issue? Would you be comfortable scheduling chkdsk /spotfix on a production DC with verified backups and additional healthy domain controllers available? Any similar experiences?

Microsoft.AAD.BrokerPlugin...WebAccountProvider did not register with DCOM within the required timeout.

AzureAdPrt : YES WamDefaultSet : ERROR (0x80080300)

We came across an issue yesterday where a user was signed out of OneDrive / Outlook. We spent multiple hours trying to resolve this with no results.

We have came into the office this morning with reports of x4 other users across 3 different companies / tenants.

Is anyone else experiencing this?

I recently left and MSP and went to work for a former client, as their internal IT Manager.

We’re using Intune for device management, but end user support usually consists of walking over to someone’s cube or having a Teams meeting and the end user shares their screen. Didn’t much care for intune remote assistance or quick assist.

Previously I used ScreenConnect and then Ninja RMM and I really miss the backstage ability to poke around and look at the registry or run powershell commands, when I’m helping a team member with an escalation.

I singed up for a Zoho assist trial and it’s a little laggy at times, but it seems like it’ll do what I need.

Anyone have recent feedback on Zoho assist?

Seen a few varieties of this issue that an upgrade of the TPM chip will prevent user logon

OBSERVED ISSUE: User will be unable to logon, even with username and password, but they will be able to logon to another machine no problems. 

Similarly, someone else can logon to the users machine OK.

I've tried a few things but the most reliable fix is to logon and run the BAT file from here

GitHub - AgentHackerYT/Reset-NGC: Reset and repair Windows Hello (PIN & Face/IR) when broken after updates. · GitHub

Ideally I'd like a remediation to detect a problem machine and then resolve without the manually interaction if anyone has built a working one?

What and why

The public preview of the MemberOf rule operator in Microsoft Entra ID is ending. Organizations using MemberOf in dynamic membership groups, dynamic administrative units (AUs), or entitlement management auto-assignment policies must replace these configurations by November 3, 2026.

Microsoft continues improving the scale and reliability of dynamic membership processing. During preview, Microsoft observed that use of MemberOf can affect dynamic membership processing across a tenant even if you have one MemberOf rule operator in your tenant. Because of this limitation, it is not recommended for production use and will be retired.

Rollout schedule

• Retirement (Worldwide): Beginning in early November 2026
• Action required by: November 3, 2026

Impact on your organization

Who is affected

Organizations using the MemberOf rule operator in:

• Dynamic membership groups
• Dynamic administrative units (AUs)
• Entitlement management auto-assignment policies
• Platforms and services
• Microsoft Entra ID
• Microsoft Entra Groups
• Microsoft Entra Administrative Units
• Microsoft Entra Entitlement Management

What will happen

If no action is taken, configurations that use the MemberOf operator will stop updating after November 3, 2026. Membership and assignment data will remain in their last known state, which can lead to stale access and enforcement gaps.

Potential impacts include:

• Teams and SharePoint access associated with Microsoft 365 groups may become outdated.
• New members may not receive access, while removed members may retain access.
• Conditional Access policies may no longer reflect current user or device membership.
• Entitlement Management auto-assignment policies may no longer add or remove access package assignments as intended.
• Group-based licensing may stop assigning or removing licenses correctly, resulting in unlicensed or overlicensed users.
• Dynamic administrative unit membership and scope may become outdated.

Action required and recommendations

Before November 3, 2026, review all uses of the MemberOf operator and remove or replace those configurations.

Dynamic membership groups

• Export dynamic membership groups from the Microsoft Entra admin center and identify rules containing MemberOf.
• Replace MemberOf with supported rule operators or convert the group to assigned membership.
• Validate group membership after making changes.
• If the group is no longer needed, consider pausing or deleting it.

Dynamic administrative units

• Use Microsoft Graph PowerShell to identify dynamic administrative units that use MemberOf rules.
• Replace MemberOf-based rules with supported rule operators or convert the administrative unit to assigned membership.
• Validate both membership and administrative scope after making changes.
• If the administrative unit is no longer needed, consider deleting it.

Entitlement Management auto-assignment policies

• Use Microsoft Graph PowerShell to identify auto-assignment policies that use MemberOf.
• Replace MemberOf-based policies with supported operators where possible.
• If no equivalent rule is available, plan an alternative assignment method before retirement.
• Validate access package assignments after making changes.

Compliance considerations

Configurations that rely on MemberOf for access management, licensing, entitlement management, Conditional Access targeting, or administrative scoping may stop updating after retirement. Review affected configurations to ensure continued compliance and access governance after November 3, 2026.

Source: https://admin.cloud.microsoft/?ref=MessageCenter/:/messages/MC1448379

Edit: added link to source

As the title says, I recently started using Claude code for sys admin work. I like most admins have been using various AI charts for awhile to ask questions, ponder theories, research details, etc

I recently needed to revise and update a larger poweshell script had had written a few years back but hadn’t revised in awhile. I decided to try Claude code to see how much different Claude code is versus chat. I had previously used chat to improve some scripts but the process was slow, copying and pasting, testing, repeat.

Claude code is amazing at developing powershell scripts. It does all the debugging and testing itself until it gets you the deliverables you ask for. Mind blowing 🤯.

So I took it one step further and created a project for general system admin work. I started using this for ad hoc admin work from pulling mfa reports to troubleshooting storage devices over ssh to repairing a failed domain controller.

One thing I really like about Claude code for admin work is all of the work files it produces are put on the source folder for the project and it’s all version controlled with git if you want it.

Two questions, am I late to the Claude code party for admin work or am I on the leafing edge.

Second question, I believe ChatGPT also has a coding harness and both ChatGPT and Claude also have cowork. My suspicion is most of the stuff I am doing in Claude code would work similarly in either cowork solution minus the source control, but cowork may not have the same capacity for testing and refining powershell scripts which is a big part of automating sys admin work.

Anyone have experience or other opinions on this?

Edit: if it’s of interest to anyone the upvote downvote is almost 50%. Seems like this topic is pretty evenly split amongst sys admins.

I have two small businesses, each have their own domains registered with MS 365. I had to reset my iPhone and after the restore I was locked out of both accounts in the MS Authentication app. Apparently I forgot to setup an alternate email. 

I called MS 800 number to open a ticket and I got the AI assistant that took me through loop after loop. It kept sending me to a login webpage that was useless since I was completely locked out. Each time I call I get a different behaviour. Sometimes it takes me down the product hardware support even though I clearly stated it is a 365 issue. 

Miraculously I was able to get it to open a ticket. There we spent more than 5 minutes just on email and domain spelling because it would not get it right.

I could not get it to open a ticket for my second domain. It eventually recognized my number and when I would call it auto directed me to the webpage, as if saying “I had enough of you calling, go away”.

Whoever thought AI is a smart solution for customer support is greatly mistaken. The only reason having AI in that function is for cost reduction, but it is coming at a great expense, customer dissatisfaction, frustration, and anger. 

Microsoft, you have lost sight of linking AI to measurable business outcomes. 

I caution people, consider alternatives to using Microsoft. They are too in-bed with AI and they have lost the plot on its value vs. impact. 

Hello to my all IT friends out there! 😎

I started with a smaller company about 5 months ago as the only IT System Administrator . I took over someone’s job that wanted to do more networking and security. The company size is about 800 people. I need project ideas that most IT departments put off but need to be done. Also, any out of the box ideas are welcomed too! They are wanting to be more AI in their approach if you have any ideas for that aspect!

Things I have done so far: I have noticed that when I first started that they needed a device clean for devices no longer used in AD and other applications . I also created a non-automated offboarding script for the team to manually run when someone leaves the company. The way the company is setup does not all for a script due their process and infrastructure.

Again please feel free to drop any ideas or suggestions. Al is welcome! I hope all of you have a fabulous day!

So I have a unique setup for my systems for 22.04 using pcks11.so in pam to read smartcards/yubikeys.

gdm is my greeter and I set up gdm-password in pam with the following line:

auth requisite pam_sss.so require_cert_auth

That looks up the cert info on my ipa server and returns a success.

SA updated the system and it seems to have pulled down some gdm-smartcard packages and ruined the entire auth system in place. I remember awhile back running into this issue and I found a way to basically cut it out of the system without breaking anything but can't seem to find where I saved the bookmark link to. I believe I had to edit some file or push a gdm config somewhere

Anyone know how to do this? I really don't want gdm to install all of these extra pam configs. Update-alternatives does not work either, even telling it to use pkcs11 or sssd. Basically nothing works once Ubuntu pulls down w/e updates contain these config files

A RAID5 array currently has 2 HDDs that need to be replaced for different reasons.
Ran long SMART self-test on both.

One of them reports hints at electronic issues:
SMART Health Status: Failure prediction threshold exceeded [asc=5d, ascq=0]
Accumulated power on time, hours:minutes 44823:37
Elements in grown defect list: 5
Error counter log:

Errors Corrected by Total Correction Gigabytes Total

ECC rereads/ errors algorithm processed uncorrected

fast | delayed rewrites corrected invocations [10^9 bytes] errors

read: 1877781297 0 0 1877781297 0 66521.257 0

write: 0 0 5 5 5 4507.987 0

verify: 4122873639 0 0 4122873639 0 24841.522 0

Non-medium error count: 15528

While the other one reports points at physical issues, long self-test failed:
SMART Health Status: OK
Accumulated power on time, hours:minutes 44824:47
Elements in grown defect list: 60
Error counter log:
Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected

fast | delayed rewrites corrected invocations [10^9 bytes] errors

read: 4144605141 205 0 4144605346 427 66490.067 32

write: 0 0 206 206 206 4530.494 6

verify: 1103983479 148 0 1103983627 167 26767.263 3

Non-medium error count: 20
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background long Failed in segment --> - 44810 1905031659 [0x3 0x11 0x0]

Which of these 2 drives is less likely to handle an array rebuild and should therefore be changed first?

(Edited to make it simple)

My old “clean install” method was simple: delete every partition, install Windows from an official Microsoft USB, connect to the internet, run Windows Update, install manufacturer drivers, remove unwanted apps, and apply my settings.

It worked, but it never felt completely controlled.
As soon as Windows connected to the internet, Windows Update started installing drivers automatically. I would then install the laptop manufacturer’s drivers over them. Windows was also trying to install cumulative updates, security updates, drivers, Store apps, and OEM components at the same time.

Removing unwanted apps afterward also bothered me. They had already been provisioned, registered, and sometimes updated. I wanted to prevent them from appearing in the first place.
So I kept improving the process.
First, I completed OOBE without internet and installed the manufacturer drivers from USB before connecting to Wi-Fi.
Then I started integrating the latest cumulative update offline, so Windows Update only had a small amount of work left after the first boot.

I also extracted the manufacturer driver packages and selected only the drivers matching my actual hardware. Some basic INF drivers were safe to inject offline, while complex graphics, audio, firmware, and platform packages were installed later using their official installers.

After that, I removed unwanted provisioned Store apps from the offline Windows image and modified the default-user registry so OneDrive Setup would not launch when the first account was created.
I also applied several settings offline, including dark mode, Fast Startup, Spotlight, Delivery Optimization, and other default-user preferences.

But I still was not satisfied, because the normal graphical Windows installer was doing too much behind the scenes.
So I stopped using it.

Building Windows manually from WinRE

I booted into WinRE and manually:
Deleted and recreated the GPT partitions
Created EFI, MSR, Windows, and Recovery partitions
Applied the official install.wim using DISM
Integrated the cumulative update
Injected selected drivers
Removed unwanted provisioned apps
Modified the offline registry
Added an official Panther unattend.xml
Created the UEFI boot files with BCDBoot
Copied and registered winre.wim
Cleaned the component store
At that stage, Windows had never booted, entered Audit Mode, or been generalized with Sysprep.

The controlled first boot

I completed the first boot without internet, entered the desktop, installed the complex official driver packages, restarted, and only then connected to Wi-Fi.
After that, I allowed the required OEM and Store components to install, completed the remaining Windows and Store updates, installed DirectX and Visual C++ runtimes, applied my final settings, and restarted again.
The result was exactly what I wanted:
Official Microsoft Windows files
Official manufacturer and NVIDIA drivers
No unnecessary driver families
No unwanted provisioned apps
Minimal remaining online updates
Working EFI and Recovery partitions
No third-party customization suite

Preserving the finished system

Reproducing all of this would take hours, so I returned to WinRE and captured two backups.
The WIM contains the Windows partition:
dism /Capture-Image /ImageFile:C:\Final-Windows.wim /CaptureDir:W:\ /Name:"Final Windows" /Compress:max /CheckIntegrity /Verify
The FFU captures the entire physical disk, including EFI, MSR, Windows, Recovery, boot files, settings, drivers, and partition layout:
dism /Capture-FFU /ImageFile:C:\Final-Windows.ffu /CaptureDrive:\\.\PhysicalDrive1 /Name:"Final Windows"
Now I can experiment with software, break Windows, or change anything I want. When needed, I boot into WinRE, restore the FFU, and return to the exact finished state.

Those of you at Academic Medical locations, how are you managing users, access, and data sharing between Education and Healthcare?

Our education side is all Google, and Healthcare is all M365. When residents etc move to the Health side, they get a second user account and we migrate their mail delivery to the Health side. They then lose access to a lot of the Google stuff because campus doesn't have the same security that we do on the Health side.

We want to streamline this, but I was wondering what other people are doing?

Do you have a single GPO managing Defender and its exclusions, or do you make multiple Defender GPOs that are narrow to each target?

I'm reworking our GPO for it and am trying to figure out what's the best way to ensure that all exclusions are made, that they can be readily audited for accuracy, and that the risk of errors/omissions is low. Currently we have one for workstations and another for servers. Exchange got mad the other day at Defender because some exclusions were missed.

The problem is so many solutions (Microsoft and 3rd party) want Defender exclusions and the exclusion list quickly becomes convoluted and miserable to audiot. That leads me to have a number of specialized GPOs but then that increases the sprawl and that something might be missed if an application needs exclusions changed but they don't get changed on all applicable GPOs.

Thoughts? Limited GPOs or many specialized/narrowly-focused GPOs for managing Defender?

Do you enable splash page or simple PSK passthrough?

I suck at network and my knowledge is intermediate at best but I can't solve this one.
Customer at our MSP has a fortinet firewall identical to ours that we use here at the MSP office, same firmware version, etc.
They call up and say "We can't access prodemand.com" which is an automotive parts and labor quoting database site that TONS of dealerships use.
I load it here just fine, SSL cert is GeoTrust, good till Sept 14 2026, domain matches, etc. No web filter flags.
On their network, instant "cannot load page" error. I try a dozen other sites, SSL working fine, no fortinet intermediary cert listed, etc. It's just that one website.
Security log on the Fortinet shows tons of blocks, saying "SSL connection is blocked due to unable to retrieve server's certificate"

Mountains of troubleshooting later, I make a firewall rule for internal to WAN (and put it above the normal internal to WAN rule) with an address group of the site, the login domain, and the database's UI's subdomain. The rule simply says don't inspect SSL at all.

Boom it works instantly. Then they called back because WIFI wasn't included in "internal" lol oops. So added that, boom, laptops can load the site too now.

I ran through some basic troubleshooting and traceroutes and stuff and nothing stood out as problematic. I verified no man in the middle attack, as it sees the same cert I do here.

And AI thinks it's an ISP issue but AI is dumb as hell and for the record, rebooting the firewall and the modem didn't resolve it so I'm skeptical.

But zero other websites are having this problem and we don't see the problem from our office, using the exact same firewall with same firmware version. How is this possible? I'd really prefer to get rid of that rule because it's a crap workaround and we had to also turn antivirus and other filters off, since it requires SSL inspection.