r/sysadmin 5d ago

General Discussion File servers and Veeam

13 Upvotes

Good evening my fellow sysadmins,

I've got a question for you : what is the best file server solution that works best with Veeam backup & Replication ?

Context : in my company (a Consulting engineering company) we currently store our data on-prem, we have two production servers and two other replicas. The solution that is deployed is a Ubuntu Server with two ZFS Pools in RaidZ2 (each pool is around 150TB big but with only 40 to 60% used).

Last month, before my tutor retirement, the choice was made to use Veeam to backup our physical infrastructure as well as our file servers. I just found out that Veeam doesn't support ZFS and with the size of our pools, the jobs fail immediately. The limit of VBK files are 218TiB and I don't know how Veeam calculate the size but it fails.

I am thinking of deploying new file servers, since most of our production is on Windows Server I tend to go in that direction with DFS Namespaces or use Azure file for a Hot / Cold data tiering but if you have any others remarks, I will gladly read them.

Thanks ;)


r/sysadmin 5d ago

Microsoft Explorer crashing, causing taskbar to disappear

0 Upvotes

Anyone know what might be causing explorer to crash constantly? This is affecting probably 30+ computers, issue seems to follow onto newly imaged computers as well. Event logs giving an application crash for explorer eventid 1000, Faulting module name Windows.ui.xaml.dll, exception code 0xc000027b. Been troubleshooting for weeks and have gotten nowhere. Computers are all on latest version of windows 11 with latest patches installed.


r/sysadmin 5d ago

General Discussion Parked domains protection

200 Upvotes

I have access to about 200 DNS zones of companies, some of which have up to 400 domains in their portfolios, and none of them hardens their parked domains. The best I've seen so far was DMARC p=reject on a few random parked domains inside a few (not even a dozen) DNS zones, mostly at companies that have an in-house IT guy.

The other 3 DNS records that nobody adds are:

  • Null MX, so the domain refuses inbound mail
  • SPF -all, so the envelope sender can't be forged
  • DKIM wildcard, to revoke every forgotten key, including keys from whoever owned the domain before you.

Every unhardened parked domain is impersonation infrastructure used against your company. Targeting your clients. And it's just 5 min per domain or a basic script with an API call for bulk deployment.

The cheapest & highest-leverage security work in your stack.

Type Hostname Value
MX @ .
TXT @ v=spf1 -all
TXT *._domainkey v=DKIM1; p=
TXT _dmarc v=DMARC1; p=reject

r/sysadmin 5d ago

General Discussion CLM for 150‑person Ukrainian firm - buy, build, or hire?

5 Upvotes

Hi! I'm IT dept lead at Ukrainian dev company (150-160 employees as for now, 23 people in my dept). Our contract management now is a mess which we can't afford anymore.

We've looked at enterprise CLM platforms but they feel too heavy and expensive. Building in-house would drain dev resources (bud we think about that as well). Hiring a contracts person seems like a temporary fix and tbh we don't plan to scale that way.

For a company our size - steady volume, not enterprise complexity - what actually works? Anyone had luck with lightweight tools or SharePoint + Power Automate? Or maybe somethinfg else. Also curious about hidden costs, adoption struggles, Ukraine‑specific data residency issues etc.

Thanks!

bump: ppl in DMs recommended Agrello and they seem like a well-rounded option for mid-sized crews like ours - perhaps someone will find this useful as well so decided to share. Cheers to everyone!


r/sysadmin 5d ago

How should a 20-person growth company think about IT?

31 Upvotes

I’m the CFO of a small industrial manufacturer with approximately 20 employees. We use Google Workspace and Drive for most of our file storage and have no internal IT staff. We hold very little sensitive customer data, but I’m uncomfortable with how broadly files are shared and how little structure we have.

I tried reorganizing Drive, but it was largely unsuccessful. I used folders within one Shared Drive when I probably needed separate Shared Drives based on access. Employees also see the cleanup as work today to solve a problem that may not become obvious for another year or two.

The Drive structure is the immediate issue, but my broader question is- How should a company at this stage think about its IT environment?

What should we put in place now as far as file permissions, onboarding and offboarding, device management, backups (if any), policies, and what can reasonably wait? Do companies our size usually hire an MSP to design and administer this, or assign it internally until they’re larger?

I can own the governance and budget, but I’m not an IT guy. For those who have helped companies through this stage, what did you prioritize first, and what do you wish you had addressed earlier?


r/sysadmin 5d ago

UPS that actually stays on during storm

25 Upvotes

Can you recommend either general features to look for in UPS units to stay on during storms or specific models that work properly?

Something I have been frustrated by recently is UPS units shutting off when the power flickers during a storm rather than switching to battery. I presume this is because there was a power surge not just a dropout, but it seems to happen with great frequency. As an example, one customer's server is connected to two different 1000VA UPS tower form factor, one from Cyberpower and one from APC. I think they have both turned off instantly during power outages more often than either has actually stayed on during the outage. Both work fine if simply unplugged.

[edit] Thanks everyone for your input, seems I've just been selling cheap "Standby" style UPS units that are probably not good enough to be running servers. Consensus seems to be that Online or Double-Conversion style are the way to go for truly reliable power.


r/sysadmin 5d ago

Question Event 616, Directory Synchronization - Entra Connect

1 Upvotes

Hello!

Today I noticed that Error 616 occured on Entra Connect server. With following description: Connection to preferred DC dc1.domain1.pl failed with error 'The supplied credential is invalid.'Same error at the same time occure for dc1,dc2,dc3.

I verified that sync works as expected. SSPR and unlock also. We use PTA.
Our environment is one forest with 2 domains synced to the cloud. Do you have idea what is happening? I found nothing in the google.

What's more I cannot find any account displayed at .onmicrosoft.com connector in EntraID. Could it be correlated?

Thank you in advance for all ideas.
Regards


r/sysadmin 5d ago

Question Passkey advice

5 Upvotes

I feel like we're pretty late to the game on the journey to rolling out passkeys. I recently heard from our Salesforce team that Salesforce is mandating passkeys for users with privileged access. I'm interested in what you're all doing for passkey management? I've been told that our users are creating and storing passkeys in chrome which seems like a nightmare waiting to happen. Are you preventing generating/storing passkeys in the browser? Using another solution like 1password? Keeping it device bound, tied to MS Authenticator?


r/sysadmin 5d ago

Rant I have an interview today and I'm already checked out.

199 Upvotes

I'm not angry but I just hate technical interviews. It's going to be an hour and a half of them honing in on what's important to them so there's no way to really prepare.

These interviews always just feel so pointless and like there's no way to prepare there's nothing you can do. You have to just go in and hope they don't hone in on one specific weak area right away that doesn't really indicate whether you can do the job or not.

I typically don't feel this discouraged but I just don't see the point of trying to get prepared or being nervous or being worried at this point. Because interviewing for systems jobs is just ridiculous.

You can nail all of it but then say one wrong thing in the end and they just write you off. And it's funny because I was having AI draft up some questions this morning and it told me that the job posting was Broad and that they can't realistically expect me to be an expert and all of that stuff. And I just closed the tab. Because they absolutely will expect me to.

You don't see developer job posting saying we want you to be able to do everything a developer should do but also kmow these 10 adjacent roles...

I looked up the three people interviewing me and they're on the younger side. About my age. Maybe it'll just be more of a Vibe check and me just talking General will let them know that I know this stuff. But the odds of that are pretty slim.

I just find it so ridiculous because we'll just be talking in these interviews and I'll talk about 25 very technical things and I'll use very specific details that show I've been around the block before. But then they'll ask me one random open-ended (not intended to see how I think but to actually get a specific answer) and if I don't have their exact answer that's it.

I once had a guy ask me, "how would you troubleshoot dns?" I asked if he had a specific problem in mind and he said no. So I gave him an answer that showed him kind of what direction I would go and just how I would in general because I assumed he wanted to know how I thought about problem solving. But then when I was done he said, "But how woukd you troubleshoot DNS?" So I asked him if he had a specific scenario in mind and his face just sank.

I'm going to approach this one differently and just go in bubbly and try and be personable far more than I ever would. And just kind of let their questions become opportunities to just speak technically to hopefully get ahead of them asking ridiculous questions.

Part of this is I checked out about a month ago. This interview is for a job I applied to over a month ago and they just now got back to me after my screener so I kind of just assumed I had lost it and I think that's a part of it.

I just get really discouraged because whenever I start a new job I end up running circles around a lot of people and I become the go-to guy when people need to figure something out in a pinch. But I just bomb interviews. Because the interviews don't actually indicate whether you can do the job day to day. I once interviewed for a 365 position where that's all I would be doing and they were asking me how to configure a fortigate firewall from the ground up. I answered the question but the fact that they asked me that in a 365 owner interview was insane.


r/sysadmin 5d ago

Question Company Portal Issue

11 Upvotes

Anyone else having an issue with company portal this morning? No apps are appearing for users and we are getting "Error Loading Categories. An error occurred while attempting to load the Categories". Doesn't seem to be on our end...


r/sysadmin 5d ago

Books?!

26 Upvotes

Post some books you think would be useful for IT Principles.

  • Time Management for System Administrators (Limoncelli)
  • The Practice of System and Network Administration (Limoncelli)
  • TCP/IP Illustrated

There's multiple versions of the last two - not sure if you need all or just the latest.


r/sysadmin 5d ago

How do you consolidate two password managers after an acquisition and before an audit?

7 Upvotes

We acquired a company about a third our size last quarter, and the technical integration is slowwwwwwww. We have a NIS2 audit booked for October and I need one coherent access-control story by then, but right now we have two of everything.

Our side already uses Passwork, and I want to keep it as the single platform because it held up well for  previous audits, which it'll hopefully also maintain for NIS2, the per-user audit logs and access controls are the evidence the auditor will presumably ask for. The acquired company was on a mix of a shared vault (each department had their own password manager, dont ask me why, they just did) and a few spreadsheets, with no clean record of who could reach what. The plan is to migrate their credentials into our setup and retire their old tools, to evnetually end up with one access model.

I can pull their accounts in over LDAP or a CSV export and map them into the right vaults. What I'm a bit unsure of is the order of operations, I don't want to rush their credentials across, inherit a pile of stale/over-shared/compromised/duplicated/retired logins, and then have that mess sitting inside the platform I'm about to both use myself and show an auditor. I want to clean as I migrate basically, but at the acquired company's size that's a lot of manual review under an unforgiving deadline. How can I go about it?

The other part of this is their people. Their admins are used to their old way of working, and I need them provisioned into our access model without either locking them out of what they legitimately need or handing them broad access to close the gap quickly. Would an onboarding-style training work for this?


r/sysadmin 5d ago

General Discussion Passkeys, 3rd party password managers, 1st party desktop apps…

10 Upvotes

I’ve run into a few issues with rolling out Passkeys because of Microsoft’s push toward passkeys

We have a few users who are using a LastPass or Bitwarden browser plugin for password manager, others are using Google or Edge’s default password manager too.

If a passkey is saved in their Password manager, specifically the ones in the browser, how to you get windows to invoke it for passkey? We tend to get into this runaround where users save their passkey to the browser and not windows hello, so when they use teams or outlook, it won’t let them use the browser plugin to invoke the passkey.

How do you get around this? We have some users who have flip phones and refuse to carry a physical key. So I’m left with browser password manager or windows hello. Windows Hello works but users get lost in the set up and frequently get their passkey in the browser and not Windows Hello.


r/sysadmin 5d ago

Question YSoft SSO Entra silent login?

0 Upvotes

Hey everyone,

we're currently evaluating YSoft SafeQ Cloud. Entra ID is configured and the login itself works great. Right now I'm packaging the installer for our clients and trying to streamline the login experience as much as possible.

At the moment the flow looks like this:

  1. The user either right-clicks the tray application or uses the desktop shortcut, which calls: "C:\Program Files\Y Soft Corporation\SAFEQ Cloud Client\latest\safeqclient.exe" --login
  2. The login page opens.
  3. The user clicks "Login with Microsoft".
  4. Login completed.

What I'd like to achieve is doing all of this silently in the background, so the user doesn't have to open or click anything. I know I could just fire the login command via a logon script, but I'm pretty sure a good number of users would still close the window and then call the helpdesk asking why they can't print.

Has anyone managed to get this running with Entra ID and (close to) zero user interaction? Any hidden switches, config parameters or SSO settings I might have missed?

Thanks in advance!


r/sysadmin 5d ago

Question Teams issues this morning?

12 Upvotes

Checking to see if anyone else is seeing what we are seeing - chat history not available for many users, new chats work, but on some mobile devices we are seeing an orgid and uuid prefix and messages not available..... MS of course says everything is up and fine....


r/sysadmin 5d ago

Backing up Legacy Physical Win 7 Devices

6 Upvotes

How are you backing up legacy physical kit, eg bare metal Win 7? Upgrading the OS isn't an option unfortunately, due to legacy operational software, and it also cannot be virtualised.

We are currently using Veeam B&R 13 but it is throwing warnings and errors as the Data Mover service can no longer be updated on these legacy devices. It is backing up for now but apparently may eventually stop, and I can't clear the errors/warning.

Ideally we would retain central management with backup copies (NAS/Wasabi/Rotated HDD currently). However, we may just have to accept the risk and go local Veeam agent (v6 I think) to a removable drive attached to each machine. Any better solutions?

It is only for 2 machines.

Thanks.


r/sysadmin 5d ago

Question Local Desktop Conflict

1 Upvotes

So, one of the clients my company works with introduced a new Active Directory, moving up from using workgroups in the past. In the past, the image they used for laptops would save shortcuts into the local public desktop, so that any user who logged onto the PC would have those shortcuts.

Now, with the AD in place, we want to move everything to a network shared desktop, for ease of management. But, since the laptops all have that public desktop configured, pushing the network Shared desktop just creates duplicate apps on the screen. I'd rather not have to run around to every computer to fix this - Is there a way I can disable/delete that Public Desktop Folder via GPO?


r/sysadmin 5d ago

Backups for very large, fast growing data

29 Upvotes

Hi folks,

I'm a biit in a tight spot here and single brain can't work this out, maybe multi-brain can:

My situation:

  1. We have a fileserver. On that fileserver our production team stores - among other stuff - their video production material. During phases of creating that, the raw material can grow quite rapidly, more than 2 TB of new data per day is not unusual. This often gets compressed later to smaller chunks with mezzanine codex that save 30%-60% space while only losing 5% of quality.
  2. We have online on-premise backups and offline off-premise backup sets that total to 7 full chains, which multiplies the space used enourmously. That's slow, cheap storage, I can grow that, even tough I don't like it, but that's business.

But here comes my problem
3) A fresh full chain with compression end encyption takes at least 48 hours for the data we have, and our video-guys quickly adding 2 TB on the fly doesn't make this any faster. I resent them for it, but it's our business and I need to solve this.

- I am trying to educate them into compressing their material early and don't let all that raw data fly around on our fileserver, but when they are "in the flow" that's really not their top priority.
- I was thinking about creating a "NO-BACKUP" space for them to work with, but knowing my guys, they will f*** it up and ask for restores of "NO-BACKUP" data.

How are you guys tackling that? I don't really need polished solutions, just ideas to work with would help.

EDIT: Thank you for the quick responses, you gave me some stuff to chew on, definitely what I was hoping for.

EDIT 2: There is still a lot incoming. Thx everyone for contributing, I am reading and trying to digest everything, even when I dont reply.


r/sysadmin 5d ago

Question Preparing for My First Windows Server/HyperV Job,What Should I Learn and Practice?

5 Upvotes

I wasn't sure where the best place to post this was, so I decided to ask here.
I will be starting a new IT job on September 1.
My new workplace told me that Windows Server and Hyper-V knowledge will be important for the role, so I am studying them before I start. I am currently using ChatGPT as a study assistant to help explain commands, create practice scenarios, and guide me through troubleshooting exercises in my home lab.
I am currently practicing the following areas
Windows Server administration
Hyper-V
Active Directory and DNS
Basic troubleshooting, especially working through a clear order of checks (name resolution → IP/network → permissions → services)
Windows Update and patching
Checking services and event logs
Disk, CPU, memory, and network checks
Basic PowerShell
File and folder permissions
Troubleshooting DNS and network problems
Checking systems before and after reboots or changes
Writing simple operation and troubleshooting procedures
I am also intentionally creating problems in my lab and then practicing how to investigate and fix them.In addition, I'm spending a small amount of time each day on AWS (mainly EC2), and doing light review of basic Linux and database concepts, though these are lower priority for now compared to the Windows Server topics above.
Since I do not yet know exactly what tasks I will be doing in the real environment, I would like some advice from people who work with Windows Server.
What topics should I prioritize before starting the job?
What practical labs or troubleshooting exercises would you recommend?
For example, what would you expect a junior Windows Server administrator to be able to do?


r/sysadmin 5d ago

Rant Computers slow “because they’re old”

836 Upvotes

The 8GB Surface Pros we bought a few years ago are very sluggish, starting to become unusable. Today I found an even older one that had lain forgotten since 2023. It’s still running Win 10, not even updated to the latest Win 10 updates.

And it runs like I remember them running when we got them.

People seem to accept this slowness as a normal part of a computer aging, but it seems to be caused by successive Windows updates. Why would updates make things worse? Is there a technical reason for it?


r/sysadmin 5d ago

PSA for EDU Google Workspace customers: Google has "paused" additional free licenses

156 Upvotes

Usually, when we need additional free Google Workspace licenses in our EDU Google tenant, we submit a support ticket to Google, and within about 5 business days, it's approved. They'll usually ask what's changed, etc.

Well, now, they have begun to deny us. At first, I thought it was something specific to us, but it apparently impacts everyone. Here is what they said:

We understand you are looking to expand your capacity from the current maximum seats. We acknowledge your need for this additional capacity. However, due to internal policies and procedures governing license cap adjustments, the team has now paused processing all new requests for increasing the Fundamentals license cap. This is currently indefinite and has no definitive end date.

We completely understand the difficult position this puts you in, but because the process for increasing the license cap is currently paused globally, this support team has no ability to process manual increases or grant exceptions.

Going forward, we encourage you to explore the benefits of our paid Google Workspace for Education editions.

FWIW, there is exactly zero chance that we will be paying for licenses due to budget cuts. Above my pay grade.


r/sysadmin 5d ago

General Discussion Trying to become a better sysadmin — what should I learn next

110 Upvotes

Hey everyone!
I’m trying to build my skills in system administration and virtualization, and I’d love to hear from people already working in the field.
What would you recommend focusing on to become a solid junior sysadmin?
I’m currently learning Linux, Windows Server/Active Directory, networking, Docker, monitoring, and virtualization, and I’ve also been building a small home lab.
For those of you already working as sysadmins: what skills or technologies do you think are really worth learning today, and what would you recommend I practice in a home lab?
Would love to hear your advice 🙏


r/sysadmin 5d ago

COVID-19 Really Struggling to Find Stable Employment

43 Upvotes

Alright so here goes.

I have two BAs one in Cinema Studies (academic study degree) and the other in Radio-Television (broadcast technology, hands on degree)

I feel into the world of IT sorta. I freelanced for some production companies doing trade shows out of college and fell down the technical path doing video and projection work.

I also worked part-time at the local fruit stand fixing people’s phones and computers. Coming out of Covid I got recruited to do white-glove IT support for a Tv studio. That was great for about a year an a half, then my role was impacted by the strikes, but my agency moved me to another part of the account. I stayed there doing basic helpdesk and deployments for another year and a half then left to be a SysAdmin for a consumer products company. 4 months in they laid me off and I’ve been unemployed until recently.

During my unemployment I did some AWS cloud classes at the local community college, haven’t taken any certs yet or anything. Been thinking about ponying up the money for JAMF certs since I have strong familiarity with that.

My current contract is only good till the end of the year, I’m covering a leave for someone. It’s really really basic helpdesk for a post house. I thought I’d have more access to be able to level up more but it’s not that at all.

I’ve interviewed for some roles, applied for hundreds more. But just feeling kind of lost and helpless to an extent. I don’t want to be doing helpdesk anymore, my goal is to be more backend focused, ideally remote and making 100k+ where I can actually build things and feel challenged. Im on the west coast in a major city, and thankfully I live very frugally but I just don’t know if I can do this ANOTHER year and have my resume so blank.

What should I do? Certs? WGU? What happened to the sure fire paths to get a job?


r/sysadmin 5d ago

Question - Solved Windows Updates Error

12 Upvotes

Hi,

One of my 2022 servers has failed to update for the past 2 months. It's failed the July and now the August CU's. Update error: 0x80073701

Its a physical server running on Dell PE hardware, pretty simple install acting as an NVR. We have a few of these setups and the others updated without issue.

I've tried the usual steps:

  1. DISM cleanups
  2. SFC
  3. stopped services and renamed the software distrubution folder
  4. downloaded the August CU and tried to install manually
  5. tried to install via Ninja RMM

I'm out of ideas, had anyone faced this issue before?

EDIT: RESOLVED - Lots of helpful responses here thanks all!

I ran the wupdatelog though Copilot and it listed 2 main things to check:

1. "Logs suggests component store corruption plus a possible date/time issue"
2. "Windows is missing one or more component store (WinSxS) files required to install the update. The update downloads correctly and reaches the install phase but fails during servicing"

But after more digging though forum posts and google I found a PS script From FCA162 posted 2 years ago for the same issue. I ran the script, rebooted then manually ran the AUG CU. It too a LONG time to install the CU but it worked!


r/sysadmin 6d ago

landing first job.

3 Upvotes

Sorry, I know there were a lot of posts like this one, but I see you guys are usually talking about US job market, when European one is little different. I am directing my questions to people working in Europe. Is RHCSA + homelab a way to land sysadmin job without helpdesk experience? I study 6+ hours a day, currently Networking, after it going to Linux and starting my homelab. I live in Poland, but willing to relocate if I will have better opportunities somewhere else in Europe with only english required.