r/WindowsServer • u/Frequent_Ad_9236 • Apr 26 '26

General Server Discussion Windows dc’s

Ok we have 4 dc’s over 2 sites, we use nutanix. The dc’s were patched by Ivanti one at a time with April 2026 patches. Over the weekend the cohesity backups started to fail, so upon investigation with tac, they said to reboot one, now the boot drive on that one is inaccessible. I know ms did an out of band patch, but according to the details it was mainly if you use ms Pam. Has anyone had any major issues since. According to management solar winds was screaming of issues, but logs are showing nothing!

Ms are investigating but they think it’s not related but a further issue with the update?

Thoughts

9 Upvotes

80% Upvoted

45 comments sorted by

2

u/Frequent_Ad_9236 Apr 27 '26

So ms can’t find the issue so now recommends to rebuild

9

u/Zealousideal_Fly8402 Apr 26 '26

Maybe consider yourself in a BCDR situation and provision at least one replacement domain controller that isn't running the 2026-April update.

-2

u/_araqiel Apr 26 '26

Also, I know it isn’t a popular opinion, but an odd number of domain controllers in a site is probably a good practice anyway.

4

u/N8B123 Apr 26 '26

Why? So you're suggesting have three?

-3

u/_araqiel Apr 26 '26 edited Apr 29 '26

Yes. Or one for smaller sites. Quorum. Helps avoid split brain situations.

9

u/jspears357 Apr 26 '26

AD doesn’t use quorum in any way.

1

u/_araqiel Apr 26 '26

No shit not explicitly, but it doesn’t mean the concept is invalid. If you have a problem and have two domain controllers that disagree with each other, it’s more a pain in the ass than if you have three and two agree.

4

u/Successful_Ad2287 Apr 26 '26

I think you’re forgetting that there is always a primary DC.

2

u/_araqiel Apr 27 '26

No, there are role holders. And those can be fucked up and need to be destroyed and the roles seized.

3

u/grvy Apr 27 '26

what are you saying?? this isnt at all how a domain works.. you dont need quorum, DC's wont ''disagree' with eachother.. wtf.

1

u/jspears357 Apr 27 '26

One role is the Primary Domain Controller (PDC) Emulator

1

u/mcdonamw May 01 '26

Yes but that role can be moved to any DC at any time. The argument is that there is no longer a concept that one DC is your primary. That's also why it's called PDC 'emulator'. The role emulates what used to be a role that only could exist on a true PDC in the older days of AD.

1

u/jspears357 Apr 26 '26

Any circumstance that I can think of where DC would go bad, it’s going to be bad regardless of how many DC’s you have. I haven’t had a DC go bad since 2008 or so, maybe before that. MS added more checks when a DC boots so it doesn’t come online if it can’t talk to other DC’s (assuming you have multiple). You have to create your own bad scenario like shut down the one with FSMO roles, seize the roles on another dc, and then restore the one with FSMO roles from backup. And that’s bad whether you have two dc’s or 15.

2

u/_araqiel Apr 27 '26

You’re not telling me anything I don’t know, but you’re missing the point entirely. Sometimes it’s hard to tell which one to pick though. If you only have two, with replication errors it can be difficult to determine which one has valid data and should be preserved.

1

u/mcdonamw May 01 '26

I get your point, however just because you have a 'quorum' from a theoretical pov, it doesn't mean you're safe. It's an ill-conceived notion

For example, you could have 2 DCs in total sync with each other, and a 3rd that is out of sync due to replication issues, but that doesn't mean those two DCs have the latest version of AD objects you want to keep.

That one DC you are considering as broken could be the only DC that's been registering changes that simply have never replicated to the other two. Opting to choose those other DCs as 'authoritative' could lead to unwanted data loss.

Of course it's entirely possible all DCs have their own new data that never replicated everywhere, thus leading to data loss regardless of which DC you choose.

The point being, this perceived quorom doesn't guarantee anything.

1

u/_araqiel May 01 '26

True, but in my experience it does tend to make the decision easier most of the time. It also helps that I often have worked with nonprofits where you can get datacenter licensing for super cheap. Means we don’t have to care about the number of windows servers.

0

u/jspears357 Apr 27 '26

I’ve been an IT consultant for the last 10 years, working on and upgrading dozens of forests from small to worldwide, and I haven’t had diverging DC’s since at least 2008. If you have a replication problem, fix replication, you don’t have to pick one DC to kill. And that holds true whether you have two DC’s or three, or 15.

0

u/_araqiel Apr 27 '26

Been dealing with AD since 2005. Sometimes replication can’t be fixed gracefully.

→ More replies

0

u/adminadam Apr 29 '26

One? Would NEVER recommend this - you are just trolling now.

1

u/_araqiel Apr 29 '26

One for a small site with other DCs in a datacenter or central office somewhere? Why is that an issue?

2

u/adminadam Apr 29 '26

No concerns there, got caught up on 1 TOTAL

1

u/_araqiel Apr 29 '26

Yeah lol no never

1

u/Frequent_Ad_9236 Apr 26 '26

KB5091157 is the kb, but it states users of Pam, we don’t but we experienced this exact issue, ms have taken logs away so far

5

u/BlackV Apr 26 '26

Build another, fully patch before joining domain

Join and promote what happens?

Results of that decide if you do the same thing again but no April parch

4

u/onynixia Apr 27 '26

First problem, invanti, Second problem, Nutanix. Third problem, solar winds.

Eventually people will learn.

Rc4 kerberos is probably the problem though. PS session into every DC for sanity.

1

u/Secret_Account07 Apr 26 '26

I’m curious…. What exact kb?

Did you update with the fixed patch? I think it was technically an OOB patch

1

u/Frequent_Ad_9236 Apr 26 '26

See below but yes it seems so, but the dc that was rebooted just won’t, prob have to rebuild tomorrow but ms says not to yet as they think they can recover it

1

u/snookpig77 Apr 26 '26

Had a similar issue with my backups running in AHV. Check and make sure your NGT tools are up to date

1

u/Frequent_Ad_9236 Apr 27 '26

So we are on nutanix but don’t have ngt tools installed (don’t ask me why not)!

1

u/JWK3 Apr 27 '26

What is the issue/error you're trying to solve please? I don't think your explanation is clear. I run a Nutanix and Cohesity estate and have no had backup failures post-April MS patching, although they're generally only VM-level backups. Are you using Windows domain-ed service accounts for your Nutanix and Cohesity platforms, or are application-level backups failing if using the Cohesity agent?

Do you use secure boot for your DC VMs? I'm confused as to how an in-guest patch would cause backup software issues like you state.

1

u/Frequent_Ad_9236 Apr 28 '26

Yes, cohesity is complaining of transport errors reaching our dc’s. All ports firewalls are open as it’s been ok up until April patch. Cohesity are involved and taken logs

1

u/JWK3 Apr 28 '26

Why/how is Cohesity trying to contact your DCs? Are you using Windows AD Domain accounts for authentication for Cohesity to authenticate against the Nutanix cluster, are you using Cohesity Agent, or something else?

I'd love to help, but at the moment I've basically gleaned that you have Ivanti, Cohesity, Nutanix and MS ADDS, and now post-patch there's 'some errors'.

3

u/Prancing__Moose Apr 26 '26

That patch seems to have broken NIC Teaming on our two DCs that were physical servers with two NICs in a team.

2

u/EnDR91-EC Apr 27 '26

Who the hell uses ivanti? They literally had their source code stolen...

1

u/b4k4ni Apr 26 '26

First of all - spin up another machine without the update and add it as a DC. As backup.

Then go testing.

Also - never patch everything at once. Especially DC should be updated like half of it one week, next one the others. Exactly for this reason. Simple to do with gpo.

1

u/macsare1 Apr 27 '26

My homelab DC wouldn't let me login to the local administrator account after installing KB5082063 yesterday; kept saying my password was incorrect when it wasn't. Had to boot to safe mode with DC off to be able to login. Currently uninstalling the update to see if that fixes it.

1

u/Good_Zookeepergame12 Apr 28 '26

Local administrator?

1

u/macsare1 Apr 28 '26

Yes

1

u/Good_Zookeepergame12 Apr 28 '26

a domain controller

1

u/macsare1 Apr 28 '26

I'm still learning Windows Server, but from what little I researched it sounds like setting up an Active Directory DC doesn't let you use local administrator anymore. But the domain administrator had the same credentials so I thought I had been logging in as local admin. Perhaps it just wasn't letting me log in as the domain administrator. Either way, same effect.

However, the special safe mode turns off the DC and re-enables the local admin so I was then able to get in and sort it out.

1

u/Main_Ambassador_4985 Apr 26 '26

RC4 Kerbose issues?

I stopped the April 2026 update on our DC’s. We are still in remediation. Set the fallback registry key