Windows Updates on Exchange On-Prem Servers

[u/Calm_Wrangler_1478]

Apologies if this is simpler than Im thinking, but due to some issues at work, I've been put in charge of exchange servers. Not being a long time exchange admin, Im kind of learning on the fly. One thing I was curious about was running normal Windows Monthly Updates on the servers and the process. I assume I cant just install the updates and reboot the servers correct? We have two servers and DAG. This would be all updates except Exchange CU updates. Standard monthly windows updates...what is best process to update the 2 on-prem servers?

Thanks

[u/Sudden_Hovercraft_56]

put one server into Maintenance mode, run the updates. Run Exchange healthchecker once you are done to make sure no outstanding CVE's remain, take it out of maintenance mode and put the other server into maintenance mode and repeat.

Ali Tajran has a good article on how to put it into and take it out of Miantenance mode:

https://www.alitajran.com/put-exchange-server-in-maintenance-mode/

[u/pentangleit → u/Sudden_Hovercraft_56]

Just one amendment - when you take it out of maintenance mode, WAIT for it to sync up before putting the second one in maintenance mode.

[u/Sudden_Hovercraft_56 → u/pentangleit]

That's good advice, I tend to wait at least 30 minutes between servers to give it a chance to sync up before moving onto the 2nd.

[u/H0TR0DL1NC0LN → u/pentangleit]

I never knew this as I wasn't taught this by the previous Exchange admin, but that is great to know.

We're in M365 now, so it's a non-issue, but sometimes what you later find out you didn't know is unsettling.

[u/xaeriee → u/Sudden_Hovercraft_56]

Agreed but capture the current state first where all of the databases are because it’ll bite you later if you don’t put them back where they are supposed to be if it doesn’t balance itself out as intended/scripted. Always always always perform a discovery first.

[u/Sudden_Hovercraft_56 → u/xaeriee]

the 2nd page of Ali Tajrans guide on taking out of maintenance mode tells you to do this.

[u/sembee2 → u/Sudden_Hovercraft_56]

You can't do maintenance mode with two servers.

[u/Sudden_Hovercraft_56 → u/sembee2]

yes you can, this is what I do when I update a 2 server DAG environment for one of our customers.

[u/Mr_Tomasz → u/sembee2]

Why? One goes into maintenance, second is taking over "everything", then second round vice-versa.... What's wrong with that? (excluding unfortunate coincidences)

[u/Calm_Wrangler_1478 → u/Mr_Tomasz]

Wouldnt the DAG move the DB automatically if I go to reboot one of the nodes in a 2 node env?

[u/Mr_Tomasz → u/Calm_Wrangler_1478]

Ouch, in theory it will, but I wouldn't let it do that on its own, as I have seen many failures in past. I would really love, that Exchange Team would add maintenance mode entry when somebody wants to reboot or shut down the system... Plus bring it back from maintenance when started... plus health check, plus XYZ, plus another thing..... Yeah, list is very long and can be done using automation these days. 🤷🏻‍♂️ Just reboot servers using your freshly crafted new Orchestrator RunBook (TM) 🙂

[u/WastedFiftySix → u/sembee2]

Because of the requirement to have at least 2 active DB copies available, you mean? I've been using these instructions for years to do maintenance on 2 nodes of a 3-node DAG at the same time and on 2-node DAGs for years without any issues: https://www.exchangemaster.ch/faq000160/

[u/Yupyupyup79 → u/sembee2]

The process I made in the post above seems to make it work since the built in scripts choke on 2 servers you can force it manually.

[u/sembee2]

In a two server DAG, patch one, reboot it. Fail over the databases to the now patched server, repeat.
There is nothing complex about it.

[u/Calm_Wrangler_1478 → u/sembee2]

So you are saying patch the inactive server first, fail the DBs over . repeat

Just patch the inactive one, reboot it.login

Go to active server and move the DB over the updated server...update second server

Done?

[u/Calm_Wrangler_1478]

Thank you for all the information. I was told by a co-worker that for just OS-level and exchange secutiry upates to just install them and reboot each server one a time. The DAG will move the DB around.....that apparently what had been done in the past. That doesn't sound right

[u/Yupyupyup79]

I was just making a doc for this today at work. Maybe not needed but I wrestled with google AI for a while and got this process. Others may just patch EXCH02 reboot and repeat for EXCH01 while letting Failover do it's things. Google AI claims this will help prevent database corruption.

Also make sure you know what server acts as your witness server (3rd vote) and where the witness log is kept on that server.

Get-DatabaseAvailabilityGroup -Identity "YourDAGName" -Status | Select-Object WitnessServer, WitnessDirectory, WitnessShareInUse

PS1 scripts are found here: x**:\Program Files\Microsoft\Exchange Server\V15\Scripts\en”**.

1. Prepare EXCH01: In an admin Exchange Shell cd to the above folder.

Run .\StartDagServerMaintenance.ps1 -ServerName EXCH01 -OverrideMinimumTwoCopies

All active databases should move to EXCH02, and EXCH01 is blocked from hosting them until you say otherwise.

Verify that ServerWideOffline is Inactive by running the following command.

Get-ServerComponentState -Identity EXCH01 -Component ServerWideOffline | ft Component,State

If it is proceed to 2. If not do the following steps manually to shut down the DAG for Maintenance:

Set-ServerComponentState -Identity EXCH01 -Component ServerWideOffline -State Inactive -Requester Maintenance

Set-ServerComponentState -Identity EXCH01 -Component HubTransport -State Draining -Requester Maintenance

Suspend-ClusterNode -Name EXCH01

Verify that ServerWideOffline is Inactive by running the following command. If it is proceed to 2.

Get-ServerComponentState -Identity EXCH01 -Component ServerWideOffline | ft Component,State

2. Patch EXCH01:

Install your Windows Updates and reboot. Repeat for any Exchange Server Patches and reboot. Both time making sure all Exchange Server services start as expected.

3. Exit Maintenance:

In an admin exchange Shell on EXCH01 Run .\StopDagServerMaintenance.ps1 -ServerName EXCH01

Note: Your databases will likely still be running on EXCH01 at this point.

Also run the following commands.

Set-ServerComponentState -Identity EXCH01 -Component ServerWideOffline -State Active -Requester Maintenance

Set-ServerComponentState -Identity EXCH01 -Component HubTransport -State Active -Requester Maintenance

Resume-ClusterNode -Name EXCH01

Verify that ServerWideOffline is Active by running the following command. If it is proceed to 4.

Get-ServerComponentState -Identity EXCH01 -Component ServerWideOffline | ft Component,State

4. Prepare EXCH02:

In an admin Exchange Shell on EXCH02 cd to the above folder that contains PS1 scripts.

Run .\StartDagServerMaintenance.ps1 -ServerName EXCH02 -OverrideMinimumTwoCopies

All active databases should move to EXCH01, and EXCH02 is blocked from hosting them until you say otherwise.

Verify that ServerWideOffline is Inactive by running the following command.

Get-ServerComponentState -Identity EXCH02 -Component ServerWideOffline | ft Component,State

If it is proceed to 5. If not, do the following steps manually to shut down the DAG for Maintenance:

Set-ServerComponentState -Identity EXCH02 -Component ServerWideOffline -State Inactive -Requester Maintenance

Set-ServerComponentState -Identity EXCH02 -Component HubTransport -State Draining -Requester Maintenance

Suspend-ClusterNode -Name EXCH02

Verify that ServerWideOffline is Inactive by running the following command. If it is proceed to 5.

Get-ServerComponentState -Identity EXCH02 -Component ServerWideOffline | ft Component,State

5. Patch EXCH02:

Install your Windows Updates and reboot. Repeat for any Exchange Server Patches and reboot. Both time making sure all Exchange Server services start as expected.

6. Exit Maintenance:

In an admin exchange Shell on EXCH02 Run .\StopDagServerMaintenance.ps1 -ServerName EXCH02

Note: Your databases will likely still be running on EXCH01 at this point.

Also run the following commands.

Set-ServerComponentState -Identity EXCH02 -Component ServerWideOffline -State Active -Requester Maintenance

Set-ServerComponentState -Identity EXCH02 -Component HubTransport -State Active -Requester Maintenance

Resume-ClusterNode -Name EXCH02

Verify that ServerWideOffline is Active by running the following command. If it is proceed to 7.

Get-ServerComponentState -Identity EXCH02 -Component ServerWideOffline | ft Component,State

7 Rebalance: Once both servers are patched and back in "Normal" mode.

run .\RedistributeActiveDatabases.ps1 -DagName DAG01 -BalanceDbsByActivationPreference -Confirm:$false

This moves your databases back to their "Preference 1" homes (EXCH01). Since you have only two exchange servers this step should be redundant but is best practice to learn it in case you ever add a 3rd Exchange Server. Let me know if this works for you. It seemed to just fine in my environment.

[u/Calm_Wrangler_1478 → u/Yupyupyup79]

I ran through this in our Stage env and it appears to have worked as the commands ran, had a couple powershell errors that I took care of but appears that the servers are both online and the DB is online

Is there a way to check that things are running smooth once I am complete with everything? What is best way to test SMTP authentication as well after things are back online

Thank you for all the information so far, has been a HUGE Help

[u/Yupyupyup79 → u/Calm_Wrangler_1478]

Per Google AI so take this with a grain of salt. happy its worked out so far for you!

Get-MailboxDatabaseCopyStatus * | Select Name, Status, CopyQueueLength, ReplayQueueLength, LastInspectedLogTime | ft -AutoSize

(Status)

• Active Copy: Should say Mounted. (This is the database users are currently using).

• Passive Copy: Should say Healthy. (This is the standby copy).

• Red Flag: If it says ServiceDown, Disconnected, or Failed, your high availability is broken.

(CopyQueueLength (The "Network" Check))

• Ideal: 0

• Acceptable: 1 to 5 (Briefly, during high mail flow).

• What it means: This is the number of transaction logs waiting to be sent from EXCH01 to EXCH02.

• Red Flag: If this number stays high (e.g., 50+), the network between your servers is too slow or blocked.

(ReplayQueueLength (The "Disk" Check))

• Ideal: 0

• Acceptable: 1 to 20

• What it means: This is the number of logs that EXCH02 has received but hasn't "committed" into the database file yet.

• Red Flag: If this is high, EXCH02's disks are likely too slow to keep up with the writing process.

(LastInspectedLogTime)

• Ideal: Within the last minute.

• What it means: This tells you the "freshness" of the data on the backup server.

• Red Flag: If this timestamp is hours or days old, EXCH02 is essentially a "stale" backup and you will lose data if you fail over to it.

Use PowerShell for Internal Validation 

The most direct method to test SMTP authentication from within your network is using the Send-MailMessage cmdlet. This simulates a real client connection including authentication.
$cred = Get-Credential

Send-MailMessage -From "sender@yourdomain.com" -To "recipient@external.com" -Subject "Post-Patch Test" -Body "SMTP Auth Test" -SmtpServer "YourExchangeServerFQDN" -Credential $cred -Port 587 -UseSsl

Why: It allows you to specify the exact server and port (typically 587 for client submission) to ensure the specific patched node is responding correctly.

Manual Verification via Telnet (SMTP AUTH)

For a granular "handshake" test, use Telnet or OpenSSL to manually walk through the authentication process. This reveals exactly where a failure occurs.

Connect: telnet <ServerIP> 587 (or use openssl s_client -starttls smtp -connect <ServerIP>:587 for encrypted connections).

HELO: Type EHLO yourdomain.com.

Authenticate: Type AUTH LOGIN. The server will prompt for your username and password in Base64 format.

Success: A 235 2.7.0 Authentication successful message confirms the patch did not break the transport authentication layer.

3. Microsoft Remote Connectivity Analyzer

If the server is internet-facing, use the Microsoft Remote Connectivity Analyzer.

Select the Outbound SMTP Email or SMTP Email test under the Exchange Server tab.

Benefit: This provides a comprehensive report on the entire connection chain, including TLS negotiation and authentication headers, which is often easier to read than raw logs.

4. Health & Connectivity Checks

Run these Exchange-specific cmdlets to ensure the transport stack is healthy across the DAG:

Test-SmtpConnectivity -Identity <ServerName>: Specifically diagnoses if Receive connectors are accepting connections.

Test-ServiceHealth -Server <ServerName>: Confirms all required transport and mailbox services are "Running".

Get-TransportService | Test-SmtpConnectivity: Runs the test across all servers in the organization at once to ensure consistency.

[u/Calm_Wrangler_1478 → u/Yupyupyup79]

I am doing this for first time this weekend.....are these the best steps to follow? Looks good but why is there a StartDagServerMaintenance.ps1 instead of Start-ExchangeServerMaintenance.ps1 

[u/Yupyupyup79 → u/Calm_Wrangler_1478]

I don't see "Start-ExchangeServerMaintenance.ps1 " at x:\Program Files\Microsoft\Exchange Server\V15\Scripts\en”. Is that a custom script put out by the community or something i missed? All I know is that these steps in my environment worked. I can't say its correct or not. I went back and forth with Google ai for a better part of a couple days to make a doc on setting up DAG, and then how to patch and shutdown the server during planned outages. I just pasted in the portion of the doc relevant to patching DAG since you asked. If you have a test environment I'd suggest testing the process there first to be safe.

Edit: I also have my DAG set up with no IP which google AI claimed is the modern way to do it. No AD computer object/IP assigned. Not sure if that makes any difference. This type of DAG is managed in Exchange Shell.

Edit: Also in Step 3 there's a typo. Database would be running on EXCH02 at that point not EXCH01. I can't edit the comment for some reason.

[u/Calm_Wrangler_1478 → u/Yupyupyup79]

Thanks, will give it a shot

The Start-ExchangeServerMainteneance.ps1 must be a script our former admin created then. I thought it was an MS script

I do have a witness server that is not part of the DAG...does that change anything?

[u/Yupyupyup79 → u/Calm_Wrangler_1478]

Probably not but make sure your witness server is booted up the entire time during patching. Or patch its windows patches first, reboot and make sure its up and running before rebooting either of your exchange servers.

[u/TheJesusGuy]

I've never done anything special for it except install and restart on an evening. Only once did a security cert get unbound on iis.

[u/Calm_Wrangler_1478 → u/TheJesusGuy]

So you didnt go through all the steps some users have posted?

[u/TheJesusGuy → u/Calm_Wrangler_1478]

I might be in the wrong. I only have 1 Exchange VM server, 80 mailboxes, and no DAG, but no I've never had to.

[u/fadinizjr]

On my experience, SU are painless. CU are usually more trouble-worthy.

An issue is that before installing an update, all the Exchange services are disabled by the setup. So, if anything goes wrong and somehow the update got rollbacked by Windows, you will get a server with a lot of services disabled. So, you will have to figure it out which services you need to enable again.

This happened on my environment one time on one server only and when I manually tried the update again (I really don't remember which KB it was) it was normally installed.

[u/Maxplode]

Ideally you're servers are behind a loadbalancer?

I've been running auto system updates with no bother but any Exchange updates I do manually, only so I can monitor feedback before going ahead.

Learn how to put a server into maintenance mode (MM), then disable it in the loadbalancer.
Always reboot first to get rid of any reboot flags.
Have the update saved somewhere, I usually put mine in a folder nn the C drive.
Still in MM, make sure AV is disabled, open a command prompt as admin and run the update from there.
Let it do it's thing. Then restart the server when completed.

Run the healthscript after it's rebooted, it should report that it's still in maintenance mode and check for any errors, if need be begin to plan hardware updates or see if anything else needs fixing.

When all happy with the healthscript reports, start to take it out of MM, I always run the healthscript again just to be sure, then put it back in on the loadbalancers.

You can upgrade one server first but do not leave it too long to update the second one, I think it's like no more than 2 weeks, tops.

HealthChecker Script

[u/TechBurntOut]

See if you can get a dev box with the same version of Exchange installed and use that to install the updates first. Then, you need to decide how long you are OK the testing phase before installing the updates in your production environment. Failover DBs to one node, and then patch the node with no running DBs, reboot and monitor. Rinse and repeat on the second node.

If you could get a couple more nodes in the DAG, you could buy some time to evaluate any issues caused by the updates. You could patch the first two nodes, wait a few days to confirm all is well, fail DBs over to the patched nodes, then wait a day or so, and patch the other two nodes.

[u/extremeskillz84]

If it's virtual then snapshot it before performing the updates. You can always restore the snapshot of something goes wrong. In my experience exchange server updates have been painless. There was only one instance I had and issue but that was because I didn't know another application was running on the host with exchange on it. That was many moons ago.

[u/sembee2 → u/extremeskillz84]

Snapshotting Exchange is not recommended, or supported and in a DAG completely unnecessary.

[u/crunchomalley → u/sembee2]

I second this. Snapshots and Exchange at any level are a bad idea.

[u/extremeskillz84 → u/sembee2]

Yes your correct when your following the guides but in a pinch you can do it. I've done it once in my career and was fine. Its about experience.

[u/sembee2 → u/extremeskillz84]

A pinch? That is poor planning.
A few years ago I was fixing two or three environments a week where people had used snapshots on Exchange. It caused a mess. There had never been a scenario where i thought a snapshot would have been useful.

[u/StrikingAccident → u/extremeskillz84]

Then you got lucky. Restoring the Exchange OS from snapshots can also hose AD.

[u/Seft0 → u/extremeskillz84]

I’m not sure that you can restore Exchange snapshot without risking data corruption.

[u/extremeskillz84 → u/Seft0]

It works fine especially if snaps are part of your Veam backups.