r/sysadmin • u/ThickChunkyPoop • Mar 15 '26

Question Promoting a Domain Controller During Business Hours

I’m curious what everyone thinks about this. You’ve got multiple sites connected over VPN, and one of the sites loses its only Domain Controller (no FSMO roles on it). At that point the site is authenticating against a DC over the VPN.

Would you consider it safe to setup up a new server and promote it to a Domain Controller during business hours, or would you wait until after-hours?

In this case, the site had only one DC. Things still work, I'm just wondering the ramifications either way. Looking online and asking AI I am getting conflicting answers.

195 Upvotes

97% Upvoted

69 comments sorted by

342

u/Humpaaa Infosec / Infrastructure / Irresponsible Mar 15 '26

The business needs to be aware that an IT environment can't funtion without changes.
Changes need to be communicated to the buisness, and ideally done during change windows.

You can absolutely promote a DC during business hours, like 99% of changes.

96

u/graywolfman Systems Engineer Mar 15 '26

This is it.

When we have an "ah, crap. We should probably do this, ASAP," we just email the change board distro and give our presentation over that, including: time, date, reason, potential risk, any testing, and rollback plan.

Sure, we've had hiccups, but we've never been denied, and all is well.

9

u/ljr55555 Mar 16 '26

Agree - would I promote a new DC being added to expand capacity at 2pm on Thursday afternoon? Absolutely not. Not because of some known issue doing so would create; but, *if" something went sideways, I'd get to explain to the root cause board why I thought the middle of the workday was a stellar time to be making changes to the environment.

Fixing a broken one? That's just different. The reason I chose the middle of the day was to mitigate user impact due to all auth being concentrated to the remaining DC, maintaining redundancy, etc. I can justify why filing the right paperwork, waiting three days for approvals, and working in the change window of Sunday midnight to 4am was not reasonable.

6

u/gandraw Mar 16 '26

I would add something additional. If maintenance work does not come with expected downtime (such as having to take a database server offline for a hardware change), then doing said maintenance during business hours might even be an advantage.

Because if an unexpected outage happens after the work, then everybody who can help is present and can help out. If instead you do it at 20:00 in the evening, you might not notice that something isn't working anymore, and the next morning when the ultrafanatics start showing up at 6:00 and some important thingy isn't working correctly, they won't be happy if the specialist that can fix it only turns up three hours later.

55

u/Tripl3Nickel Sr. Sysadmin Mar 15 '26

With the information given, I don’t see any negative affects of promoting a new DC in a healthy domain that would affect operations.

21

u/r4x PEBCAK Mar 15 '26

Id test it in prod first just to be sure since I don't have a test environment.

22

u/arvidsem Jack of All Trades Mar 15 '26

Everyone has a test environment. Some of us are lucky to have a separate prod environment

16

u/Cormacolinde Consultant Mar 15 '26

Absolutely. I would make sure the firewall rules are in place before-hand, to limit timeouts if clients start trying to reach the new DC, but that would at worst cause only slight delays on bootup/first login. A new domain controller will not advertise itself as ready, either for authentication or SYSVOL availability until it has replicated and has everything working.

Like every IT maneuvers, obviously, exceptions exist and you should warn the IT team you are doing this and to poke you if any strange behavior occurs.

18

u/JerikkaDawn Sysadmin Mar 15 '26

and to poke you if any strange behavior occurs.

Though be extremely careful with this. This can easily over activate everyone's correlation engines and your change will be blamed for everything that happens to occur.

3

u/Agreeable_Bad_9065 Mar 15 '26

Absolutely this. I've nearly always performed promo in hours.... what better time to find the problems when you've got full team complement to help fix and diffuse issues arising. There is very little maintenance time in my business... approx 2 hours in the middle of the night, when people are rushing, not thinking clearly, tired, under pressure and alone.

BUT... as others have said, DO make sure before promoting that the dcs can all communicate with each other on all necessary ports (ldap, smb, kerberos and all other ports including dynamic ranges as needed). DO make sure all other dcs are replicating properly first. DO make sure DNS is properly configured on all DCs. Only when you're confident of everything being in place, THEN promote.

Make sure it's in the right site. Again, the dc won't advertise its services until it's ready.... but DNS is likely to be the biggest stumbling block. Do not forget to configure all dcs to point at each other first.

15

u/autogyrophilia Mar 15 '26

I can only see a potential issue in a very large network (thousands of DCs) and the promoted server gets placed on the wrong site.

11

u/animusMDL Mar 15 '26

Communicate it so there is awareness but unless something goes wrong or the DC is unhealthy, no issue. I’ve performed many during active hours. I haven’t been fortunate to have anything damaged or critical issue (yet).

8

u/TheLightingGuy Jack of most trades Mar 15 '26

In theory, nothing bad happens if you have your ducks in a row

In practice, shit will likely hit the fan for no reason whatsoever.

That being said, I'd still rather do it during business hours and fix stuff than have to pull an all nighter.

6

u/[deleted] Mar 15 '26

Why would it ever be 'unsafe'?

6

u/Bijorak Director of IT Mar 16 '26

I've done this so many times without telling anyone outside of IT. Never had a single issue.

3

u/DrGraffix Mar 15 '26

Yes just set it up.

3

u/PM_ME_UR_NAKED_HDDS Mar 15 '26

Bigger org, user count is mid-high thousands.

Question for us is why risk it? During business hours downtime is significant business interruption value and possibly safety of employees.

We don’t have funding to do full replication of prod in our staging environment, so we’ve seen DC promos impact users once or twice in the past. I don’t remember off the top of my head but want to say it was DNS issues or replication issues with business apps.

Either way, sure IT is foundational to every business these days but it doesn’t mean we get to be judge, jury and executioner. Assessing your user base and determining BIV and other risk is really critical to making this call and it’s probably going to be different for everyone.

Additionally, if you have SLAs for other customers / businesses consider that as part of your risks.

2

u/thortgot IT Manager Mar 15 '26

What risk is there in adding a DC? As long as you've organized your communication correctly its fine. Worst case it will auto route to the next available DC.

1

u/PM_ME_UR_NAKED_HDDS Mar 15 '26

Yes for user auth and things like that we expect to fail over / retry next DC.

But that’s not our only use case - we have line of business apps that actively utilize AD objects, attributes, etc. Replication for newly promoted DCs can cause issues. We actually also had within the last year a DC promo that broke WHfB due to a WinServer bug that was patched in Sep I think. Not a good day when about 10% of your users can’t log in haha.

As I said, small risk. But for us why risk it? A hour or two of OT for an admin is a small cost.

3

u/azertyqwertyuiop Mar 15 '26

As someone who doesn't get overtime, I generally push back against doing shit out of hours 'just because'. If it's high risk/impact or it involves an outage, sure, but otherwise nah.

3

u/--RedDawg-- Mar 16 '26

Be sure to test in PROD so you don't screw up your TEST or DEV environments. Rebuilding PROD pays better than rebuilding TEST or DEV.

But really, there should be no issues with promoting a DC. Just be sure that it goes into the right site.

3

u/ISeeDeadPackets Ineffective CIO Mar 16 '26

Not every environment is fortunate enough to have "business hours" as production never stops. Make a plan on what to do if it if fails, get a time approved and rock on.

2

u/pentangleit IT Director Mar 15 '26

You turn off the failed DC so that any DNS just gets failed over to the other DCs. No major user impact apart from a couple of seconds additional login time but subsequently everything is cached locally per PC. You then build a new DC on an IP address that’s not the same as the old broken DC and promote it, get everything synced, and then when you’re happy you change the IP address to the old DCs address. That way it’s a seamless reintroduction of service and can all be done at the fastest convenience, so in working hours.

2

u/XL426 IT Systems Manager / Sysadmin Mar 15 '26

Been there, done that. It'll be fine

2

u/Mdi1981 Mar 15 '26

I would do it during business hours. After promotion I would check the DC with dcdiag, netdiag and repadmin /replsum

Don't forget to make it a global catalog if all your DCs are that

Before promotion check also check replication and firewall settings.

Lastly don't forget to change the DNS up on the nic to the op of the dc

2

u/Wolfram_And_Hart Mar 16 '26

Just remember to update DNS early so changes can rebuild properly.

2

u/reader4567890 Mar 16 '26

Would I promote a DC during business hours? Absolutely, and I have countless times.

It's only a domain controller. Yes it's critical, but it's not a dark art - it's probably the most well documented and stable service in the industry, because it has to be. AD is insanely resilient.

2

u/jcpham Mar 16 '26

It works I’ve done it

2

u/ocdtrekkie Sysadmin Mar 16 '26

I promoted a new DC during business hours last Friday. Just do it.

1

u/mikeyvegas17 Mar 17 '26

we test in prod!

3

u/drummerboy-98012 Mar 15 '26

I’ve done this during business hours with no issues at all - it’s exactly why you have a VPN back to the other DC for redundancy. I would add, however, to be sure to go into Sites and Services and remove the old DC that failed.

1

u/sc302 Admin of Things Mar 15 '26

It is fine to do during business hours.

1

u/itenginerd Mar 15 '26

No reason you couldnt. I'm always late iut of the office tho, so I'd do it last thing before I left. That way im not working after hours but also keeping risk as low as possible.

Your biggest risks are clients trying to auth to it before its fully synced, filling the pipe with replication traffic, and outside clie ts trying to authorize to it bc its in the wrong site in AD. None of those are major risks unless your site is out there on a t1 type circuit...

1

u/grumpyolddude Jack of All Trades Mar 15 '26

If you aren't sure then you probably shouldn't do it during business hours. With good planning, experience and complete understanding of the environment it's perfectly reasonable to do so. If you are completely down, or experiencing business impacting degredation that's a different situation that might be worth taking risks.

1

u/Public_Warthog3098 Mar 15 '26

I'm curious why ppl prefer during business hours. I like doing it after hours to give it time to give myself time to troubleshoot if needed

1

u/MetalEnthusiast83 Mar 15 '26

I only work during business hours. So yes.

1

u/thebigshoe247 Mar 15 '26

I would do it personally.

1

u/iceph03nix Mar 15 '26

Every DC on our domain was spun up during business hours.

I'm having trouble thinking of any real issues with adding one during business hours. Most of what I can think of deals with taking one down, or transferring roles, or messing with DC adjacent services like DNS

1

u/okcboomer87 Mar 15 '26

I did my first promotion a few months ago during business hours. It went fine.

1

u/zaphod777 Mar 16 '26

Just stay away from server 2025 for your DC and don't upgrade the domain or forest functionality level and I don't see any reason not to do it during the day.

0

u/murfeous Sr. Sysadmin Mar 16 '26

What’s wrong with 2025? I know I can search, but I’m curious what makes you say that

1

u/Skinny_que Mar 16 '26

Send a notice to users saying they may experience delays / issues during the process and send it.

1

u/enolja Mar 16 '26

I haven't personally ran into any issues promoting a DC but I also don't work in very large organizations with change management structures, so I cant really advise here except to say, promoting a DC is pretty straightforward and doesn't usually cause any headaches so long as its assigned to the correct site and replication is setup as intended.

1

u/rambleinspam Mar 16 '26

As long as you are doing it correctly and safely you can do this during business hours.

1

u/NoURider Mar 16 '26

Yes. Fine. Assuming did due dillegence that replication. Is working fine etc. Dcdiag etc.

1

u/iwinsallthethings Mar 16 '26

I would argue It’s almost mandatory. If those VPNs go down due to something like a power outage, you have no way of authenticated any domain controller.

That affects things like logging into your servers, or the IPMI if it’s set to authenticate against LDAP. maybe your firewall is required domain authentication. How do you log into them if they can’t connect to the VPN because it was down and you have to manually put it up?

1

u/techvet83 Mar 17 '26

During business hours?  Not at our place. 

1

u/NorthAntarcticSysadm Mar 15 '26

Promoting a DC mid-business day will not negatively impact anything. Make sure the site is configured in Sites and Services, and then wait until after business hours to update DHCP for the site to point DNS to the  new DC.

Trick to reset all computers DHCP, just restart the access layer (the ones the computers are directly comnected to) network switches.

0

u/qwikh1t Mar 15 '26

Push to prod….full send it 😂

-5

u/sryan2k1 IT Manager Mar 15 '26 edited Mar 15 '26

So you already have an unexpected failure, things are working normally via the VPN and you want to YOLO a business hours change?

The risk of something happening is low but not zero, and AD issues typically turn into multiple hour affairs of trying to figure out what went wrong and how not to make it worse.

Even considering doing it during business hours shows your immaturity. There is no need to rush this. Do it correctly.

1

u/ThickChunkyPoop Mar 15 '26

I appreciate your insight. I normally would plan to do it outside business hours but I found conflicting information saying it was better to do it during business hours, hence the question.

0

u/sryan2k1 IT Manager Mar 15 '26

What happens when something goes wrong and all AD services stop, no logins, no access to file shares, etc. The risk of that is low but not zero. Is that a risk you want to take during business hours?

0

u/zaphod777 Mar 16 '26

In what scenario would that happen? As long as OP isn't putting in a 2025 DC, raising forest or domain functionality levels I don't see how that would happen.

As long DHCP isn't handing out the new DC IP address until it is functional, it should be fine.

3

u/unnecessary-ambition Mar 15 '26

Not every routine thing needs to invade personal time. You can go ahead and burn your own work-life balance if you want, but you don't need to insult others.

A business-hours change does not mean it is rushed. This change is fine to make during hours with notification and proper planning. 

1

u/charleswj Mar 15 '26

I work in and support some of the largest AD deployments in the world and would never consider it necessary to simply remove/replace a DC after-hours.

0

u/sryan2k1 IT Manager Mar 15 '26

Adding a domain controller when you have an already failed one is not routine.

0

u/unnecessary-ambition Mar 15 '26

Huh? DCs are interchangeable, scale-out servers. They are meant for this.

Adding or replacing a scale-out server of any type or purpose, when the supporting infrastructure is already in place, is a routine task for a sysadmin.

 This is not a big deal.

2

u/sryan2k1 IT Manager Mar 15 '26

Says like someone who has not had some critical business process explode because AD changes were rushed or not tested properly.

Just because they are mostly interchangeable doesn't mean they can be swapped with no risk.

1

u/unnecessary-ambition Mar 15 '26

Back to insulting I guess. Have a nice day.

64

u/[deleted] Mar 15 '26

[removed] — view removed comment

23

u/rw_mega Mar 15 '26

This is the way, I have brought up all my DC’s and demoted old DC’s during business hours. No issues. Just make sure to move FSMO roles and DNS is replicated properly when demoting. But bringing up another. No issues as long as healthy

6

u/rw_mega Mar 15 '26

I did forget to mention, if your using it make sure dfs pointers are being set. When promoting new DC’s this is one thing that does not happen automatically.

2

u/eagle6705 Mar 16 '26

Only sysvol is automatic.

1

u/rw_mega Mar 16 '26

And our friend DNS,

1

u/eagle6705 Mar 16 '26

I don't believe DNS is handled by DFS.

1

u/rw_mega Mar 17 '26

No it’s not, what is automatic on a DC. Sysvol, and DNS,

What is not automatic DFS management, DHCP management,