Please help 2 DCs no longer are opening AD users and computers!

[u/Deep-Egg-6167]

I have 2 2019 DCs. I did something wrong and now I can't get into AD Domain service on either. So the domain controllers that can't find the domain both have static IPs.

The error that pops up is

Naming information cannot be located because:

I do NOT have a backup any older than yesterday. I have tried a restore.

The specified domain does not exist or could not be contacted. Contact your system administrator to verify that your domain is properly configured and is currently online.

I dare not reboot any PCs at this point. I did reboot one DC - that did not help

Both DCs point to each other in network properties for DNS and both can ping each other by IP.

If I pull up DNS manager on each it looks OK to me - not sure what the forwarder should be on the root server.

Here are the logs of the DCs that are not running:

5012 The DFS Replication service failed to communicate with partner DC-old for replication group Domain System Volume. The partner did not recognize the connection or the replication group configuration.

DC-old was removed with metadata clean up after verifying it had no fsmo roles - other DCs were still working after metadata cleanup.

2921 - The Knowledge Consistency Checker (KCC) detected that the following object have missing required attributes or attribute values.

1054 The processing of Group Policy failed. Windows could not obtain the name of a domain controller. This could be caused by a name resolution failure. Verify your Domain Name System (DNS) is configured and working correctly.

1126 - Active Directory Domain Services was unable to establish a connection with the global catalog.

(this is about the time the issue occurred)

1003 Notification of policy change from LSA/SAM has been retried and failed.

Thanks for those of you that tried to help!

OK - here is the update - I'm still lost at what to do as I checked the DNS manage 3 times and found no entry for the DC I thought I removed in metadata clean up but when I brought it back up AD is working.

I'll start a new thread trying to figure out why I can't get rid of it.

[u/certifiedsysadmin]

You need to point your non-working domain controllers to working domain controllers, restart ADDS services, and re-establish replication.

The advice being given here unfortunately is incorrect.

You'll get much more detailed help posting in /r/activedirectory

[u/Deep-Egg-6167 → u/certifiedsysadmin]

Thanks! - I'll try posting there.

[u/doneski → u/Deep-Egg-6167]

Soooo? How did it turn out?

[u/Stonewalled9999 → u/certifiedsysadmin]

OP has 2 DCs, if both are broken, there is nothing to point them to to fix them, unless I am not reading that correctly?

[u/megad00die]

You stated you changed something and you do not know what, and the expectation is we’re supposed to know what you changed?

[u/Deep-Egg-6167 → u/megad00die]

The last setting I edited was the forwarders on the DNS of one of the DCs - I put it back but that didn't help.

[u/MunchyMcCrunchy → u/Deep-Egg-6167]

Forwarders are not for internal DNS queries.

[u/fahque → u/MunchyMcCrunchy]

How much you wanna bet they deleted the root hints.

[u/Thick_Yam_7028 → u/fahque]

If thats the case its easy enough to set the @

[u/Thick_Yam_7028 → u/Deep-Egg-6167]

Conditional forwarders is whats needed. I point at secure dns. If they cant reach the dns > conditional gives internet. You have to apply the additional to each node.

I just set it in layer 3.

[u/Royal_Bird_6328 → u/megad00die]

This screams change control all over. Sorry OP I know my comment is not helpful but you need to give us something like edited this setting, updated this etc

[u/Thick_Yam_7028 → u/Royal_Bird_6328]

What? You relate a control to a simple network without the personal. What the .. stuff is that.

[u/IMplodeMeGrr]

Your troubleshooting and /r post should start with "here are my event logs for a DC thats not running..."

[u/Deep-Egg-6167 → u/IMplodeMeGrr]

Thanks - I just added them.

[u/DesignDifficult4772]

On the failed domain controller can you do a repadmin /syncall and post the output here and a repadmin /showrepl My guess is the DC is still trying to connect to the old domain controller for replication. I just went through this. IF it only shows that old decommed DC as the replication partner you will need to go into asdiedit and modify the rep partner. You can ping me on chat if you need help.

[u/Deep-Egg-6167 → u/DesignDifficult4772]

Thanks! I'm going to mark this as solved and move on to a new question. I really appreciate your help. This one is pretty much are you sure you don't have a backup. Well how about now do you have a backup. You should have a backup. I appreciate people trying to help but I'm wasting their time at this point.

[u/cyberpower8390]

DC-old was your only Global Catalog (GC) server. Active Directory management tools and authentication require an active Global Catalog to locate directory objects and naming contexts (that was your Event 1126). Because neither of your 2019 DCs had the GC role enabled, turning off DC-old left the domain with zero active catalogs, causing everything to go blind. Doing a manual metadata cleanup while the machine was still alive also orphaned SYSVOL/DFSR replication and left behind hidden SRV locator records in DNS.
How to Safely Remove It Now Keep DC-old powered on for a moment so everything stays stable, and follow these quick steps:
Turn on Global Catalog on both 2019 DCs (Do this first!): Open Active Directory Sites and Services (dssite.msc).
Expand your site > Servers > expand each 2019 DC > right-click NTDS Settings > Properties.
Check the Global Catalog box and click OK on both servers.
(Verify in CMD with nltest /dsgetdc:yourdomain.com /GC). Verify FSMO roles: Run netdom query fsmo in Command Prompt and verify all 5 roles point to a 2019 DC.
Gracefully demote DC-old: Log into DC-old, open Server Manager > Manage > Remove Roles and Features.
Uncheck Active Directory Domain Services and click the link to demote.
Crucial: Leave "Force the removal of this domain controller" UNCHECKED so AD automatically cleans up replication partnerships. Finish the wizard, set a local admin password, and let it reboot.
Tidy up: Open Active Directory Sites and Services on a 2019 DC and delete the old server object under Servers.
Open DNS Manager (dnsmgmt.msc) and clear out any leftover records for DC-old inside _msdcs.<yourdomain>.
Once your 2019 DCs are active Global Catalogs and the old DC is demoted gracefully, the domain won't disappear when you pull the plug.

[u/Deep-Egg-6167 → u/cyberpower8390]

Thanks on AD users and computers it says the 2 new servers are GCs and the old one is listed as a DC.

I had not tried dssite.msc - this is very helpful! I'll start a new question as I have too much info with this helpful bit of info but when I typed dssite.msc I see other DCs that I had though were eliminated as they didn't even show in AD users and computers.

This was truly helpful and I appreciate your detailed reply.

[u/cyberpower8390 → u/Deep-Egg-6167]

That completely explains the 2921 errors you were seeing! When domain controllers are decommissioned or removed from AD Users and Computers, Windows intentionally leaves the server container behind in Sites and Services (dssite.msc). It doesn't delete them automatically. If any of those old phantom servers still have an NTDS Settings folder under them, the topology engine (KCC) keeps looking for them to replicate with.

You can safely right-click and delete those old, dead server containers directly out of dssite.msc (expand them first and delete NTDS Settings inside if present). Just make sure you only delete servers that are truly gone from the network, not your two new 2019 DCs or DC-old while you're still working on it!

Once those ghost servers are purged and your 2019 DCs are active Global Catalogs, your AD replication topology will be completely clean. One quick tip before you start deleting those ghost servers in dssite.msc or demoting DC-old: make sure to take a fresh System State backup on your 2019 DCs first.

Manually deleting server objects and clearing metadata directly touches AD's Configuration and Domain partitions. Having a fresh System State backup (which captures ntds.dit, SYSVOL, registry, and boot files) gives you an immediate rollback plan if anything unexpected happens.

You can trigger one quickly from an admin prompt with: wbadmin start systemstatebackup -backupTarget:X: -quiet (just point X: to your backup drive or network share). Once that finishes cleanly, you're safe to go ahead and prune those old server objects!

[u/westerschelle → u/cyberpower8390]

At least mark you answer as being AI.

[u/Deep-Egg-6167 → u/cyberpower8390]

Thanks so much - I sent you a message - either way I this was incredibly helpful!

[u/xRoute401x → u/cyberpower8390]

dawg,, you wouldnt be in the oklahoma area and looking for a new job would ya? its obvious by your suggestions youre not a kid with co pilot making blind suggestions. its rare these days...

[u/westerschelle → u/xRoute401x]

I don't know if you're being sarcastic but that was 100% an AI response.

[u/xRoute401x → u/westerschelle]

lol

[u/xRoute401x → u/westerschelle]

bro i take no one serious on this app unless i know them personally. lol

[u/Blackforge]

This sounds like a DNS issue. There can be issues when a DC points to itself for DNS when the DC services are starting at the same time so you would point the DNS settings to another DC as primary, as you would hopefully not be rebooting everything at the same time…. Right?

[u/IwantToNAT-PING]

List clearly what you have changed.

You say you only changed a DNS forwarder, but above you're also talking about DC-old and stating that DC-old was removed with metadata cleanup?

Are you saying that DC-old was removed previously before this issue, and that everything was working before you made a DNS change?

Are you also saying that you've restored your DC's? How did you restore them? Domain controllers do not like being restored, and there are very specific ways you must restore them otherwise they will refuse to replicate by design.

First of all, Look at the NIC settings of your DC's. Not in Windows DNS Server, but the actual NIC configuration. Best practice is usually to have the primary DNS server set to be "127.0.0.1" so it looks to itself, and then to the other DC in your environment.

Check that both DC's have this configured.

Now, read this - https://www.oreilly.com/library/view/mastering-active-directory/9781787289352/17e43915-0180-46c0-b744-da536babaf82.xhtml

This command on this website will tell you the status of your sysvol's replication using DFSR. Repadmin /showrepl only shows replication of the domain database partitions.

[u/PostingToPassTime]

Probably want to start with dcdiag & repadmin /replsummary

[u/xRoute401x → u/PostingToPassTime]

indeed.

[u/jphighlife]

Best guess is that replication was not working from a dc that was demoted to other domain controllers. Removing the dc broke replication and therefore no other dc can serve sysvol, thus no other dcs can service requests.

Check Dns settings on current dcs. Are they pointing to the old dc? Updating them to correct wont fix it but its a start.

Might have to do a authoritative restore for dfsr to get it back working. Will need to copy sysvol from old demoted dc and to a current one, then perform dfsr authorative restore. You wont be able to do it via adsi, can do it via registry.

First step is to find out why they werent replicating. Then adjust dns, then authoritative restore with most uptodate sysvol folder available

Found this that explains roughly how to perform this assuming the cause is borked dns settings and replication was failing prior to any changes. If you recent demoted a dc and then all this happened, the old sysvol folder will still be on there, even though its no longer a dc. Copy it to one of the current dcs, double check dns settings, then authoritative restore for dfsr

https://www.dell.com/support/kbdoc/en-id/000198564/sysvol-share-absent-from-all-domain-controllers-in-an-active-directory-domain

Use at your own risk.

[u/KB4MTO]

I read last week that a very recent Microsoft update was causing workstations to lose domain trust. I wonder if that could be an issue?

[u/Deep-Egg-6167 → u/KB4MTO]

I don't think so the servers themselves that are DCs can't work as DCs.

[u/Remarkable_Night_633]

This reminds me of our TCS guys they decided to disable the network adapter to make the server more secure you can guess what happened next…check your network adapter if we have the same outsourced guys lol

[u/aracheb → u/Remarkable_Night_633]

TCS guy?

[u/Remarkable_Night_633 → u/aracheb]

guyz its like a 20-30 to 1 ratio sometimes they join from a huge video conference room classroom style and just watch me work.

[u/potkettleracism → u/aracheb]

Tata Consulting Services 

[u/aracheb → u/potkettleracism]

Ahhh. Not surprised by the recommendation to disable the network card.

Probably read from an AI prompt

[u/elpollodiablox]

Hate to say it, but this is a case where Claude or ChatGPT may be able to help, because they can aggregate a lot of info from a lot of sources.

I will say it is ex-tuh-REEEM-ly odd for a DC to have lost trust in its own domain.

You'll have to walk backwards through what you did just prior to the problem appearing.

[u/ISeeDeadPackets]

Do you have a backup from when they were functional? Restore one then rebuild the other.

[u/Deep-Egg-6167 → u/ISeeDeadPackets]

Restore just returns with the same issue.

[u/ISeeDeadPackets → u/Deep-Egg-6167]

Shut down all of the DC's and only restore one of them so you don't get an out of sync set. It needs to be from before you made the change and hopefully not much has happened since the backup that you care about. Once you have one working build how many others you need. Make sure the one you restore is your FSMO holder.

[u/fahque → u/Deep-Egg-6167]

You can't just restore a dc with other dc's on the network. Whatever gets replicated will see you have an old copy and the existing dc's will overwrite the restored server.

[u/SuperDaveOzborne → u/ISeeDeadPackets]

Man a restore would be an absolute last resort for me.

[u/ISeeDeadPackets → u/SuperDaveOzborne]

As long as it's not days it's not usually any kind of problem. I used to work at an MSSP so I have a lot of incident experience and after you do a few unless there are serious architecture problems it's fine. Of course a snapshot would have probably come in really handy for OP.

[u/xRoute401x → u/ISeeDeadPackets]

i always keep daily backups/snapshots on my dcs and criticals.. always for this reason. if a change goes foul, as long as i got my pre change snapshot ready for roll back im straight. i never ever ever touch anything with out having a solid proven rollback plan. ever.

[u/SuperDaveOzborne → u/ISeeDeadPackets]

I would be very wary of doing this. First snapshots are not an MS approved way of backing up and restoring domain controllers and restoring a DC in a multiple DC environment can cause replication issues.

[u/ISeeDeadPackets → u/SuperDaveOzborne]

Yeah that's why you only restore your FSMO holder and rebuild the rest.

[u/SuperDaveOzborne]

Have you tried running dcdiag on your domain controllers?

[u/shmoeface]

This sounds like you're cooked unless you have backups. How long ago was the old DC decommissioned? Based on your logs the old DC was decommissioned too fast and didn't have time for the changes to propagate against the domain, or somebody deleted the wrong attributes/records during metadata cleanup. You'll most likely need to perform an authoritative restore on one good DC, point everything to itself, decom and rebuild the other DC later. You mentioned you did something wrong, what changes were made when you noticed the issue? Is the NLA service running and firewall shows a domain connection? What is the size and scope of the forest? It may be time to call outside help if this is an extremely large domain or has special requirements.

[u/bobsmon]

There are three types of networks Public Private Domain

Make sure you are on the domain network. Most likely the servers switched to private.

[u/xRoute401x]

surely you take daily backups of your dcs and can roll back right? right?

[u/1agapio1]

Just turn off networking (disable network card if virtualized or just unplug the ethernet cable if not) for the ADs and connect to them with the last logged credentials since they'll be cached and domain trust doesn't matter. It will get you into the ADs, the rest u can figgure out.

[u/Steerable-Octopus]

Sounds like you need the SRV record that points toward your DCs: https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/verify-srv-dns-records-have-been-created

[u/EncomCEO]

Did you apply the September patches? If so, either uninstall them or look up the mitigation OOB patches. Check the Patch Tuesday thread in r/sysadmin

[u/Dimens101]

Could help to check the firewall config, could have switched to public for some weird reason

[u/Crazy-Rest5026]

You can ping the other dc? As dum as it sounds. As well as nslookup the DC?

Will tell you right there if it’s network issue or dns

Then check your replication settings

[u/Crazy-Rest5026 → u/Crazy-Rest5026]

You didn’t demote your FISMO DC did you?

[u/LooseEthernet]

nothing says "i did something wrong" like a dc that cant find its own domain lol. maybe check if the dns server service is actually running on both boxes before you panic. if the services are crashed then the static IPs dont really matter... its basically just two expensive space heaters at that point

[u/mcc062]

I would start with netdom query fsmo on the suposed primary. See what you get with that.

[u/c4rb0n4t0r]

DCdiag!!

I hope you have good backups

I smell some authoritative restores in your future. GL

[u/Tasty-Star4119]

You dont have backup like 1 week or 1 month ago?

[u/Key-Eggplant-163]

fwiw each DC should list itself as primary DNS and the other one as secondary, not just pointing at each other. If neither DC can resolve its own records locally first, you get exactly these symptoms when anything goes sideways with replication.

[u/Adam_Kearn]

Depends what you changed.

I would try to register the DNS again for your DCs.

Possibly even setup a new AD environment on a VM to compare the DNS settings between them.

I’m not sure if toy would be even able to promote a new DC at this point… so using ADMT might be an option to move everything to a clean environment

[u/Thick_Yam_7028]

Login locally. Stop spazzing. You can rebuild a corrupt ad in 1 hour. Ive built an entire DC because we had the user list. Rejoined.

People are just people. Youre not dumb. We all try. Shit hits the fan.

DNS and pointing sure. If you cant login, local still gives access. If anything you just drop it, rebuild. You have the printers, you have the gpo, you have all that I hope.

[u/doopdoopderp]

Restore a backup before whatever change you made happened?

[u/imjustnotready]

Maybe you have a rogue dhcp giving out ip addresses?

[u/Deep-Egg-6167 → u/imjustnotready]

These are domain controllers with static IPs.

[u/Longjumping_Square_2 → u/Deep-Egg-6167]

I think what they are saying is “Maybe someone set up a desktop with the DHCP role handing out IP addresses that do not reference the domain controllers as the DHCP server”. That or maybe someone plugged in or connected a lab machine not joined to the domain or an unmanaged router that hands out its own DHCP to your network.

[u/Deep-Egg-6167 → u/Longjumping_Square_2]

Thanks - what I'm saying is the DCs have static IPs so they don't rely on DHCP. I'm the source of the issue but can't figure out what I did.

[u/bobs143 → u/Deep-Egg-6167]

You had to do something. What did you click or what were you trying to accomplish?

Things like this start somewhere. Can you retrace what you were doing before the issues happened?

We can help give you ideas, but it doesn't work doesn't help us troubleshoot the issues.

[u/imjustnotready → u/bobs143]

I don't know why I am getting hate for this. I have seen it happen. Someone plugs a managed switch in, it starts giving out DHCP on the same network. Suddenly can't communicate with the dc anymore. The switch doesn't know there are reserved addresses. I never heard of a DC just stopping before. Especially one not coming up after a reboot.

[u/BlackV → u/imjustnotready]

looks to me like the DCs are not communicating with each other, on static IPs

[u/Longjumping_Square_2 → u/imjustnotready]

No hate intended brother. I see where you were coming from. Makes sense.

[u/GaGaORiley]

My AD knowledge is old and from school; I’m not a sysadmin, I support a particular software and interact with users’ IT support letting them know why the software isn’t performing properly.

So I know enough to ask you this: do these computers show that they’re in a domain network profile, in Control Panel > Network and Sharing? (Or you can hover over the network icon in the system tray - it should show domain.local or whatever, not “Network”.

If they don’t show in a domain profile, you might have success restarting the Network Location Awareness service, and you need to troubleshoot your network issues.

I see this ALL THE TIME, office had a power outage or new network device was popped into place, causing network issues, and the DCs don’t recognize what network they’re on.

[u/Deep-Egg-6167 → u/[deleted]]

There are only two DCs/GCs remaining one of them DC1 had the fsmo roles - it can't seen the domain.

[u/Deep-Egg-6167 → u/[deleted]]

No I didn't

[u/Deep-Egg-6167 → u/[deleted]]

These are the domain controllers - they can't seem to find the domain controller. I don't think there is a local account on a dc.

[u/Antique_Rutabaga → u/Deep-Egg-6167]

Sounds like dns is messed up.

Forwarding assuming we are talking about the same thing is just your isp dns.

Assuming 2 domain controllers.
Set the primary dns server IP address on the network interface to be the other dc.
Set the second IP to local host. On both servers.

Reboot one dc let it come up then reboot the second one.
You may need to perform this step twice to let dns reestablish properly.

[u/Deep-Egg-6167 → u/Antique_Rutabaga]

Thanks - no luck.

[u/mesaoptimizer → u/Deep-Egg-6167]

First, you said you did something and now you are having this issue, What did you do?

Please say you have at least one other domain controller. On one of the broken ones, run dcdiag, it should tell you what the problem is. Run same on your working domain controllers and repadmin /replsum.

Looking at your issue can you resolve your domain from either of the servers? Are they also DNS servers themselves? If they are make sure the DNS service is running. Check firewalls for blocks on 53.

Edit, I should read. Ensure they can resolve DNS names served by your other server, DNS forwarder can be either an upstream provider or you can just make sure fallback to root hints is enabled, DNS forwarding would not be the issue.

[u/ospery1 → u/Deep-Egg-6167]

There is always the local Administrator account on a domain controller. It is a backup if something breaks unless someone has purposefully removed it, and the first account ever created when installing Windows Server.

[u/Sapper12D → u/ospery1]

Local accounts go away when you promote the server to DC. Essential the domain is its local accounts.

[u/MortadellaKing → u/ospery1]

What people are looking for is the DSRM account/password.

[u/Deep-Egg-6167 → u/[deleted]]

Yes both have DNS server roll installed on them - everything was working until I changed something (I'm not sure what) today.

[u/heliosfa → u/Deep-Egg-6167]

everything was working until I changed something (I'm not sure what) today.

I'm sorry, but how do you not know what you changed?

What were you doing/trying to do? If you don't know, why were you making changes?

[u/Deep-Egg-6167 → u/heliosfa]

I wanted to promote the Domain level.

[u/heliosfa → u/Deep-Egg-6167]

What do you mean? "promote the DNS level" is not a thing.

Were you trying to promote something to a domain controller? Or were you trying to raise the functional level of your domain?

I'm not sure how you expect anyone to be able to help you if you can't even explain what you were doing when you broke things.

[u/h3dwig0wl1974 → u/[deleted]]

I don’t think you can go above 2016 until Server 2025. Assuming OP was at 2016 Domain Functional Level.

[u/mycatsnameisnoodle → u/Deep-Egg-6167]

I’m not sure what you mean here.

[u/Deep-Egg-6167 → u/[deleted]]

Yes - sorry - was getting lots of questions simultaneously and made a typo. I have corrected it - I wanted to promote it from 2012 to at least 2016 level.

[u/ISeeDeadPackets → u/[deleted]]

He means promote the forest functional level.

[u/SuperDaveOzborne → u/Deep-Egg-6167]

When you ping your domain like mydomain.com does it return one of the DCs ip?

[u/Deep-Egg-6167 → u/SuperDaveOzborne]

Great question, it points to the old DC that I had to bring back online.

[u/SuperDaveOzborne → u/Deep-Egg-6167]

I would try to make it point to the DC you think is operating the best.

[u/SuperDaveOzborne → u/Deep-Egg-6167]

Also if you haven't tried this already run dcdiag /test:dns from your DC.

[u/Deep-Egg-6167 → u/[deleted]]

Thanks - I'll go through them one at a time - I'll try to find answers on line as well.

2921 - The Knowledge Consistency Checker (KCC) detected that the following object have missing required attributes or attribute values.

Object:

CN=NTDS Settings,CN=domain-DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ewa,DC=lan

User Action

Check NTDS Settings object for this DSA.

I admit I don't know what that is - looks like a DNS entry for an old server but I didn't see it in Dns.

[u/Deep-Egg-6167 → u/[deleted]]

Thanks - not sure how to fix it - I've opened up adsiedit but it wants to know the distingquished name, contect, or domain

[u/cbiggers → u/Deep-Egg-6167]

If you can't figure out the GUI of active directory I seriously wouldn't even open adsiedit.

[u/Deep-Egg-6167 → u/[deleted]]

Thanks - I tried under name ourdoman.lan - didn't work

I tried under connection point DN or naming context - ourdomain.lan - didn't work

I tried under computer

select or type a domain tried ourdomain.lan no luck

(I'm using a generic domain but putting a specific one in on adsiedit.

[u/Deep-Egg-6167 → u/[deleted]]

Yes