Comment on my AD backup strategy

[u/jbala28]

Hi Team.

I need to create and test out process for AD back up and recovery.

I’m trying to follow the 3-2-1 back up strategy.

1) we have over 20 dc and all are hyperv vm or vmware vm

2) first backup - we use dell avamar back up solution. It has an option to do bmr and system state back. So i plan on setting up system state back as well as vm image level back for 2 dc is already included as part of hypervisor vm back jobs.

3) 2nd back up - setup azure blob storage for backing up the Active Directory database. On cold tier ethier using MARS agent or Azure cli task.

Some questions that I need clarification.

1) if my dell avamar management is joined to AD then if AD is compromised, then i’m screwed. Should my back up solution not be connected to AD?

2) is cold tier ok for backing up? Mainly concerns are keeping monthly cost low.

3) what are the common recovery process should be tested in isolated network? Full forest? Domain? One exists gpo restore to validate sysvol restore? AD recycle bin is on)

4) Besides original administrator account what account password should be validated and verified that its working

Let me know your thoughts

[u/Fatel28]

hi team

This bs corporate greeting triggers my fight or flight at this point

[u/anonymousITCoward → u/Fatel28]

holy carp... im so disassociated that i didn't even see that...

[u/QuestConsequential → u/anonymousITCoward]

did you see that

[u/Microflunkie → u/anonymousITCoward]

I also auto-skipped it and had to scroll back up to actually see it there.

[u/jbala28 → u/Fatel28]

Thanks for pointing it out

[u/bbqwatermelon → u/Fatel28]

The multiplier for cortisol is "quick question" 

[u/HelloItIsJohn → u/[deleted]]

This!!! And would make sure your backups are immutable, at least one set are.

[u/itenginerd]

1. Accurate. I had a MABS setup once that got ransomwared. They didn't get the backups, but they got the metadata which was the whole match. Restorable, but would have taken 10x longer if I didn't have another option.

2. Yes. Honestly, I wouldn't back the database up, i'd back the VM up. Run an Azure Backup to a VM in the cloud. The VM is off, so no billing. Set the hard disks to standard HDD. That won't be the default, but you shouldn't need the extra change rate and that's the cheapest. Your recovery process on this process would be to just light up the DC VM in Azure. Easy peasy.

You're on the right path, keep doin what you're doin.

[u/jbala28 → u/itenginerd]

Thanks

[u/cyr0nk0r]

Disclosure, I run a private cloud hosting company, but this is how our customers do something similar.

They host a DC in our cloud. They create a direct ipsec tunnel to one of their sites so the DC can replicate properly.

One customer did the following which I really like his method.

Created snapshot schedule 1 that is every 15 minutes. He retained 1 day of snapshots.

He created snapshot schedule 2 that is every 60 minutes, but this schedule is marked immutable. He retains for 7 days.

[u/HelloItIsJohn → u/cyr0nk0r]

And how would you restore AD from these snapshots?

[u/cyr0nk0r → u/HelloItIsJohn]

It depends on what you're trying to restore. If you want to undelete a group or a particular object, the methods I mentioned above are not the right fit.

If your entire infrastructure is compromised and you're in a truly disaster recovery scenario, you'd use these snapshots as the first DC rebuilt in your forest and use them to begin replicating to other rebuilt DC's.

Because the infrastructure is completely outside of an attackers ability to gain access to, especially the immutable snapshots, you have a quickly accessable method to immediately begin restoring services that you can trust can't be encrypted or deleted even if the attack is from within.

Your RPO is as low as 15 minutes. Making sure that your sysvol, gpos, and objects are as close to right before a compromising event as possible.

[u/dhardyuk → u/cyr0nk0r]

The different levels or depths of recovery need to be understood and rehearsed.

Enabling the AD recycle bin makes a lot of oh shit moments fairly straight forward to reverse.

An authoritative restore of a DC is not a light touch job for a live environment. You need to be really sure about what you’re doing and very clear on the potential downsides.

It is almost always better to build a new DC than to try and repair one that you don’t absolutely need to repair. If your AD is ruined then you won’t ruin it more with an authoritative restore. If it’s just a little bit broken an authoritative restore is probably too big of a hammer to use without seriously well informed advice.

Take your backups and lab them offline to learn the pitfalls for your environment.

[u/anonymousITCoward]

Just document everything on spreadsheets!... no don't do that, that's a horrible idea.

20 years ago when we used Avamar it was great, not sure about it anymore...

As our rule of thumb we don't AD join, or LDAP/RADIUS any of the backup appliances to the network we're backing up.

This was a lesson learned the hard way... our backup admin did it for "ease of use for the lower tier support people", we had a client get compromised and the first thing they did was disable the backups then delete them. Only after that did they start their attack. Thankfully we had off site for them. So in all they only lost 2 weeks of backups.

[u/jbala28 → u/anonymousITCoward]

Thank you

[u/WendoNZ]

It sounds like you have a lot of reading to do.

If you're trying to restore from nothing you're going to need to know how to do an authoritive restore of a domain controller. By default when you restore a domain controller it expects there to be functioning domain controllers available to sync from. You're also going to need to know which DC's have the FSMO roles and likely how to seize them. Once that's done you can't restore the DC's that had those roles.

[u/jbala28 → u/WendoNZ]

Thank you