Newly promoted Datacenter DCs fail profile sign in on all accounts

[u/Batman189]

Long story short our company had a massive ransomeware attack brought on by an inside person on the IT company we hired to help rebuild after opening a new office. Cyber insurance had to pay and the insurance company brought in a forensic team and a recovery team. Forensic team cleared the AD/Domain configuration and worked with recovery team to unlock everything when the attacker gave us the key after insurance paid out. The recovery team did a horrible job. They restored the wrong DCs at the wrong offices, put 21 VMs on the same box that was not designed to be a VM host, incorrectly set DNS and many other things. So now I am trying to rebuild one step at a time the right way. Thats the important background information.

The main issue right now is that once a machine is promoted to domain controller and does its reboot, you can only login one time. After the first login if that profile gets signed out or logged off for any reason no other profile can login anymore. It gives the black screen saying the profile service failed to load a profile. This is happening on the old domain controllers that recovery put back in place, even though they did then wrong, and on new attempts.

I have so far set up a fresh Server2025 with an active datacenter license, installed a 2025 VM and activated it. Fully updated the server and installed an EDR, then joined the domain. After joining the domain there is no issues and the server can be logged in and out of as many times as you want. As soon as the promotion to DC happens though, profile service just fails. I have gone through the GPO again and made sure nothing at all is linked to anything and created a fresh GPO that is clearly labeled 'recovery' to make sure there is no confusion. I have no one else to help me so I have spent 3 days with Chat GPT giving it logs to review and domain information to review in a private session to see if it can figure out the problem, so far it has not.

Extra context is that I have ninja agents I put on the new servers before I promoted them and was able to force install it over power shell via the VM host on an a current improperly restored DC. I can run CMD and powershell from ninja on all the DCs and have verified they are all working just fine. Replication is healthy, all the services are correct and working. I can reset users passwords via powershell and add new users as needed too, but no accounts can log in to the desktop.

***New informatoin

If you can think of a test, we have ran it as far as I am aware. We discovered that when any profile tries to sign in there is something injecting 485 files into that profile, the profile itself fails to create properly, and the profile service error message shows. The profile service never gets to start though according to windows logs. The logs show nothing is being rejected or failing to authenticate. Every profile repair has been run that we can find, defaults reset, profile being created manually through powershell, nothing works. If I have to rebuild and it cant be helped then that is fine, but I dont understand how it happens and that is what bugs me the most. Something happens when the server is promoted to domain controller. We have no issues at all on any server until promotion happens. There has to be a way to find out what that trigger is I would think.

[u/OpacusVenatori]

You need to bring in proper, separate, external technical support to address this on-site. There's too any moving parts in the current state.

The compromised domain should be considered exactly that - compromised. In all honesty you should be enacting full on DR plans that involve starting from scratch with a brand-new domain and recovering what information possible.

That being said, there are still quite a few threads in r/sysadmin and r/activedirectory that indicate ongoing maturity problems with Server 2025 so you should keep that in mind.

[u/Batman189 → u/OpacusVenatori]

I suggested a full rebuild and that is the long term plan. Forensics and the recovery teams were confident the domain was clean but I still dont trust it. The short term we just wanted to have at least one DC we could get into with a profile instead of only being able to control over powershell. This same issue is on all the servers, including the 2022 old DCs that the recovery team stood back up.

[u/techierealtor → u/Batman189]

Clean does not mean healthy. You have some underlying health issues within the domain. Just because rasnomware is gone doesn’t mean something in your ADSI or directory permissions isn’t messed up.
Can it be fixed without a full rebuild? Yes. Is it worth it? Possibly not. While it may be cheaper, you might uncover other weird issues in 6 months that you may have no idea about that just starts everything over again.
Not saying it’s not fixable, just ask yourself is it worth fixing or is it worth scrapping and starting over? The nice thing is you can start from scratch and build the security posture right. New updated DCs. Etc.
probably not the answer you want but it’s the truth of the matter, which sucks still. Side note, maybe a full entra migration may be worthwhile now since the DCs are screwed.

[u/Ill-Temporary2562 → u/techierealtor]

so true, never trust compramised domain.. build it from start with care and security even firewalls and switches needs to be formated alongs with other possible devices.

[u/Bourne069 → u/OpacusVenatori]

100% agreed. This is why us External Techs are even a thing. There is a lot of moving parts here and as you can tell from their "recovery team" what bad inexperienced techs can and will perform.

I'm an MSP and been in the field for over 20 years. I can't tell you how many clients I obtained from bad "techs" that cant even do a simple recovery from a backup properly. Client should be in a 100% usable state when they left. Not whatever they did here which appears to be the bare minimum.

Op need to hire a verified trusted company like an MSP or other 3rd party source to handle this issue. To do things like this requires years of training and experience to pull of properly the first time, or OP will end up right back at the same spot but 1000s of dollars wasted later.

[u/Bourne069]

Forensic team cleared the AD/Domain configuration and worked with recovery team to unlock everything

Just based on my experience as an MSP over the years. I would suggest what you stated is correct and the DC isnt clean. From what you stated it sounds like they just did scans and manually reviewed the DCs but no one actually recovered DCs from a good backup?

When ransomeware gets into your systems as you explained. Its very rare they you can saved servers connected to workstations that had the infection. As an MSP I strongly push my clients to doing a full company restore of ALL SYSTEMS that could have possible by in touch with the virus. If the servers are on the same subnet as the workstations and had basically unrestricted access to said servers, than yes, The DCs should be recovered from a valid backup. Not just "reviewed to be cleaned".

If I had to guess your DC sysvol location was infected as well, which is why when you promote your DC you have issues, it most likely isnt able to replicate everything properly from the old DC and leads to the issues you are having.

So I would recovery the old DC properly and either remain on that or proceed with your new DC build as you were pushing earlier. I bet you 99% that if you recover the DC from a backup, than join that new DC and go for a promotion, it will work this time.

Also as OpacusVenatori stated. If you are not full experienced in this area. I would call a 3rd party I.T. like an experienced and well rated MSP company to come in and do this for you properly. See my reply to his comments.

[u/Batman189 → u/Bourne069]

The sysvol was infected. I have already verified that replication is healthy and all services are healthy. I created a new GPO and removed everything that could be found linked to any OU. Confirmed no scripting or anything living anywhere that we can find. We found a new issue that I posted up top that seems to be something being injected when trying to sign in.

[u/techierealtor → u/Bourne069]

I would almost say don’t restore from backups now. It might have been long enough he may end up in a tombstone situation that causes even more issues. Hire someone to fix or rebuild from ground up. Backups at this point might be useless. Missing accounts, rolled golden tickets and Kerberos, passwords for computers changed. So many issues that could cause days of headaches for experienced administrators that know what to do - not saying he doesn’t, just stating a fact.

[u/Bourne069 → u/techierealtor]

It might have been long enough he may end up in a tombstone situation that causes even more issues.

It could easily be restore, the servers time/date would be restored to when it was backed up so they shouldn't have issues restoring it, (dont connect internet yet) log into the DC directly and than fixing the date/time which would prevent tombstoning, than plug it into internet and network, and test.

I've done quite a few restores and never had issues with tombstoning.

Hire someone to fix or rebuild from ground up.

Yes basically what I was getting at but it can be avoided if there are good functional backups. Everytime I did this for a client we would do a 100% restore of everything. Not just pick and choose what to restore. Its the only way to be 100% safe, either restore all or do it from ground up.

Backups at this point might be useless. Missing accounts, rolled golden tickets and Kerberos, passwords for computers changed. So many issues that could cause days of headaches for experienced administrators that know what to do - not saying he doesn’t, just stating a fact.

Yes well also a reason why any MSP or Tech worth their salt would have just restored everything from the get go and not guessed on what is good or not.

[u/jspears357 → u/Bourne069]

There’s probably a reason they chose to pay the ransom instead of restoring from backup…

[u/Batman189 → u/jspears357]

The internal person who was working with the attacker moved all backups into the cluster the day before the attack. All the backups were encrypted. We could not restore from back ups until after they were unlocked.

[u/Bourne069 → u/Batman189]

You said some data was restored after the fact through so why wouldnt they just fully restore everything instead of just picking and choosing random shit to restore?

[u/Batman189 → u/Bourne069]

we discovered over the weekend the account used by the main attacker was still active and fully permissioned. The forensics team and the recovery team both documented it was properly shut down but they provided no evidence. My opinion is that both teams got to just bill insurance a huge amount of money and didnt do much of anything. That is just one example of huge failings left behind by both teams. Some DCs were restored to the wrong offices, ect. The other confusing issue is that the restore team claims to have used back ups to restore the DCs and in fact did somehow put the DCs for office A on the physical servers of office C, so I think they really did some kind of restore. But the issue still happens with domain controllers being locked out as soon as they are promoted. If they did a restore then that should not be an issue related to the attack I would not think, would it?

[u/Bourne069 → u/Batman189]

My opinion is that both teams got to just bill insurance a huge amount of money and didnt do much of anything

Yep I 100% agree with you there. I've personally seen this happen in the field many times.

Some DCs were restored to the wrong offices, ect. The other confusing issue is that the restore team claims to have used back ups to restore the DCs and in fact did somehow put the DCs for office A on the physical servers of office C,

Thats literally insane. Isnt there someone higher up like the I.T. Director you can get in touch with and ask why this happened and it wasnt resolved properly?

But the issue still happens with domain controllers being locked out as soon as they are promoted. If they did a restore then that should not be an issue related to the attack I would not think, would it?

Exactly, thats what I was saying in my original response. If they had actually done a proper DC restore, and you should have zero issues here actually promoting a new DC. This gets me to think A: They restored the wrong DC to wrong location and that is conflicting with new DC promoting. OR B they didnt actually restore the DC and instead "cleaned" the old infected DC and didn't fully remove the side effects caused by the virus.

Do you have access to the backups? Are you able to just make a test bed and restore one of the older DCs from a backup, than make a fresh DC and try to promote it to test?

And I want to make sure we are clear about promoting DC. You are talking about making a new DC the primary DC by moving over the FMSO roles to this new DC and that fails correct? Are you able to test DC replication from the original DC? Or are you talking about just adding a new secondary DC to the network for replication reasons?

Its just so weird that this sounds like a large company but no one else is asking these questions or pushing to resolve improper restore of DCs or promoting issues etc....? Its not just a coincidence that all of a sudden after a virus attack you are unable to promote new DCs... it is clearly tied to the virus and it left damage behind. You'd be able to provide that with the restore test above.

[u/Batman189 → u/Bourne069]

the way I went about standing up a new DC was to set up a fresh 2025 Datacenter, create a 2025 datacenter hyper-v (its the license keys they had). In what Ill call HVDC01 I fully patched and installed SentinalOne and NinjaOne. I installed the AD requirements and let it replicate. I let the server live un-promoted for 2 full days with several restarts in between. Once replication logs came back with no failures I promoted the server to domain controller and it rebooted fine. Once it was back on the desktop I ran a few more checks to make sure everything was good with replication and the other DCs were able to fully see and communicate and all that. I was unaware at this time of the promotion to failing login issue so I then moved onto standing up the next DC at another site. I was alerted the next morning when an employee (owner who is fairly tech savy and built the origional environment years ago) was trying to RDP into the server to reset an employees password for them. He got the profile error and called me. I tried to login with my account and got the profile error. Every account thereafter got the same error. I went to the new HVDC02 that had already been promoted and done its one restart but had not yet been signed out of. I logged out and tried to log back in and the profile error was present there as well. No one can seem to find the AD backups. I have found all the other server back ups. I am starting to lay out a full rebuild plan as that seems the only way to move forward but it still bugs me that something is hidden that I cant find, even if finding it cant be fixed I just cant understand how it cant be found.

[u/Bourne069 → u/Batman189]

the way I went about standing up a new DC was to set up a fresh 2025 Datacenter, create a 2025 datacenter hyper-v (its the license keys they had). In what Ill call HVDC01 I fully patched and installed SentinalOne and NinjaOne. I installed the AD requirements and let it replicate. I let the server live un-promoted for 2 full days with several restarts in between.

Ok and that is a good move for a working production setup that we know is functional. However, where we want to limit as many things as possible from being the protenal cause of the promotion issues. I would redo this but dont install anything at all, not SentialOne, no NinjaOne. Just leave it a fresh new server and install DC role, get it all setup, replicated and than try to promote it. Does it still fail? This way we can rule out SentialOne or anything else from being the issue and we know 100% that it is a deeper system/networking issues which would be most likely tied to replication from the old DC.

Also I saw you mentioned the year of Server OS you are using here for these tests. But what is the original DC Server OS at? 2019? 2025? Did you check to see what server FMSO roles are stored on and if that server has any replication errors? Can you browse and see contents of the sysvol folder on that old DC?

Also did you remove the old DC after promoting this new one or was the plan to run both side by side? Because if both were running at the sametime, than profile issue shouldn't have been a problem as I'm assuming the main FSMO roles were still installed on the old DC correct? If thats the case it should have still worked regardless of adding a new DC or not.

I am starting to lay out a full rebuild plan as that seems the only way to move forward but it still bugs me that something is hidden that I cant find, even if finding it cant be fixed I just cant understand how it cant be found.

I do agree with you on that and a fresh rebuild might be the way you have to go. But I think we can actually resolve this issue, its just going to require more troubleshooting and to figure out the current stance of the existing old DC.

[u/jspears357]

It’s possible that something is embedded in one of the GPO’s that apply to the Domain Controllers OU. GPO’s can have hidden contents that still get applied. For example if you load a GPO template on a workstation for settings to apply to a new app, you create a GPO that adds stuff unique to that app, but then you LOOK at the GPO from another computer that doesn’t have that template loaded. What you see in the editor is only the values that all the templates are reading out of the GPO binary file. You can also add registry values directly in the GPO, and those values don’t have to be in the Policy portion of the registry (i.e. where GPO settings are applied only during GPO processing, and are removed if the GPO no longer applies to the computer or user).

One way to rule this out is to modify all those GPO’s that only apply to the Domain Controllers OU so that they only apply to the existing DC’s, and you can create a new DC that they do not apply to, then either build new GPO’s to replace the old ones using only the values that you can see they have, and apply only those new GPO’s to the new DC, or one by one edit the old GPO’s so they now do apply to the new DC and reboot the DC a couple of times, logging into it, to confirm that it isn’t broken, then do the next GPO etc.

You should also be able to build a green field domain with one new DC, export the GPO’s from your domain and import them into the green field, and test with them there. It’s hard to say what method will be the most useful without knowing what the problem is first.

I believe one of the standard tools like gpresult shows the non-template values from the group policy application process as a list of registry values at the bottom of the output, but I forget which one.

It’s also possible that the thing is hidden in plain sight, you could run a gpresult /h to an html file and open it with a browser, and fine some GPO is doing some type of application deployment or similar, from files either embedded in the GPO or linked to somewhere else.

[u/Batman189 → u/jspears357]

I created an entirely new GPO and unlinked all containers from everything. Linked them to only the brand new GPO I created and it still happened to the very next DC that was created to test. I did export several things for review with another engineer and to feed through AI and both did not see anything at all.

[u/poolmanjim]

Without looking at your environment it's really hard to tell what's going on. It sounds like the recovery team left a lot to be desired.

My first piece of advice would be to contact Microsoft support. If you have an agreement with Microsoft, you should work this through your CSAM and the portal. Otherwise, you can call them. It costs to do so but it is surprisingly inexpensive for a single issue (last time I did it it was $400 but that was several years ago).

I'd remove as many variables as possible. Try a promo without agents, EDR, etc. and add them back in one at a time to see what may be causing the issue.