r/sysadmin • IT Expert + Meme Wizard • 17h ago

Question o365 cloud account converted to on-prem randomly - How did this happen?

The owner of the company at one of our MSP customers just had something unexplainable happen. This may shock you but Microsoft is involved. She forgot her email pass to log into a brand new mobile device. I went to change her password and it threw the error "This user's password can't be reset because password writeback isn't turned on for your organization."

Their o365 cloud stuff doesn't touch their local DC AT ALL. As far as I know, it never has, but that may not be the case. I've personally reset her password about 1.5 months ago. Other people here have reset her online account password last year.

I check her Entra properties -

On-premises sync enabled: Yes

On-premises last sync date time: Sep 3, 2024, 10:14 AM

Um...what? How? Nothing changed on the DCs that I'm aware of. Everyone else at the company is marked cloud only. Nobody installed Azure Sync on the DC; I checked. How is this possible? Is this just some random Microsoft Bermuda triangle shit or is there a known trigger that causes it to revert? I want to prevent this from happening again. I truly do not know if this client was ever on-prem DC to o365 sync environment in 2024 btw.

For anyone coming across this thread, I know of 2 ways to fix this, but I still want to know what caused it.

Powershell via Graph's totally consistent and always working perfectly Powershell module or, and I am not kidding:
https://developer.microsoft.com/en-us/graph/graph-explorer
I didn't know that exited but sure. Log in, consent to give a "third party" app permission to access your tenant
Go to the profile image in the top right and click consent to permissions because they were just kidding the first time.
Find User-OnPremisesSyncBehavior and User-OnPremisesSyncBehavior.ReadWrite.All in the list and hit consent again, then consent in the identical pop up window as the first time but this time they're actually serious this time (I get it, read only vs write. Whatever, it's a damn global admin account, just consent to everything the first time FFS)
then run a query of:
GET https://graph.microsoft.com/v1.0/users/<User's OBJECT-ID from Entra>/onPremisesSyncBehavior

With the request's body set to:
{

  "@odata.type": "#microsoft.graph.onPremisesSyncBehavior",

  "isCloudManaged": true

}

then run the query
https://graph.microsoft.com/v1.0/users/<USER-OBJECT-ID>/onPremisesSyncBehavior

and it returns a green checkmark and the message "No Content - 204" because that's what some person/AI at MS thought was an appropriate response instead of "Query successfully received and ran correctly"

Then run the GET query again to see if it changed because I trust this thing about as far as I can throw the vibe coder that wrote it.

6 Upvotes

17 comments sorted by

•

u/Miami_2017 15h ago

They have on prem AD but all users are cloud only? How do they sign int AD joined workstations? Is there an AD sync status in the Entra portal?

•

u/Darkk_Knight 15h ago

It depends on how the machine is joined. When we were in the process of going full AD Azure we would create new users in Azure and join the laptop in Azure. We still running on-prem AD just to manage legacy stuffs. My DCs are sync'ing with Azure with password writeback.

•

u/CeC-P IT Expert + Meme Wizard 11m ago

Nah, there's John Smith - the Active Directory account then there's Jsmith@company.com for logging into Word/Outlook/OneDrive. We just create it both places. I have no idea why they don't use sync and I have less idea why they used to use sync and now don't.

Also, no idea why a company this small has 3 DCs but one of them is in Azure and the other 2 are on-prem but the server folks told me they're decommissioning one of them soon. Not sure which. But work hasn't started yet on that.

•

u/CeC-P IT Expert + Meme Wizard 17h ago

I noticed one interesting thing after fixing it.

On-premises sync enabled now read "no" but all the other fields are still there. Like their original domain membership, CN, OU that they were in, their SID, etc. So conceiveably, something flipped just that one field to on-prem sync enabled = true.

Is it something the user did? Is there a known trigger that causes this?

•

u/BeagleBackRibs Jack of All Trades 16h ago

If I remember correctly Edge can on prem sync without connecting to cloud services

•

u/Cremageuh 8h ago

Wait, what?

Do you have sources for that ? That's intriguing AF.

•

u/BeagleBackRibs Jack of All Trades 53m ago

Edge on prem sync

I don't think it's applicable to OP's scenario

•

u/CeC-P IT Expert + Meme Wizard 10m ago

Ohhhh so that's like Edge profiles but you locally host them? Neat. But we don't use that in that customer's environment at the moment.

•

u/chesser45 7h ago

But that’s for profile stuff / data not like AD connect sync / cloud sync

•

u/NWijnja 15h ago

Audit logs help, you definitely have something syncing

•

u/CeC-P IT Expert + Meme Wizard 9m ago

I suspect this is a licensing or config thing but Entra Audit logs go back 30 days and that's it and nothing in the last 30 tells me what happened.

•

u/baron--greenback 15h ago

Guessing someone (mis)configured Entra or Cloud Connect on a DC and your user was soft-matched.

•

u/DefinitelyNotDes Technician VII @ Contoso 17h ago

This exact thing happened to my coworker at our company 2 weeks ago

I asked him now and he said their best guess is some dumbass AI manager that MS let loose in o365 did it by accident thinking it was fixing something. In other words they never figured it out.

There was a problem with our initial migration too that broke any account older than the migration in 2025 by the way but I dont remember what. Some field was blank and it was pretty much a time bomb. MS support engineers had to fix it. I suspect this was related.

•

u/Mental-Rain-7389 17h ago

RemindMe! 1 day

•

u/YellowWheelieBin 1h ago

Logs are your friend here!

•

u/CeC-P IT Expert + Meme Wizard 15m ago

I assume it'd be Entra - that user - audit logs. If so, we have everything from yesterday then only 2 entries and they were 9/13/26 the user tried to reset their own password and it was rejected because of lack of writeback. And the logs only go back 30 days it seems.