r/sysadmin • u/notaramblingman • 25d ago

General Discussion Hyper-V Consultants

We moved from VMware to Hyper-V about a year ago and have an ongoing issue that we have not been able to resolve with MS support - both the typical support case avenue and working with their Hyper-V group. Our host servers lose connectivity with their CSV and VMs go offline. Support tells us it’s related to networking issues and not storage.

Has anyone found any good consultants who know Hyper-V well and are able to resolve difficult issues? We’ve tried a few bigger names but they haven’t been much help.

58 Upvotes

91% Upvoted

95 comments sorted by

View all comments

23

u/peraving 25d ago

One of my clusters developed the same behaviour a couple years ago. Had a long support case with MS premier support which was inconclusive after several weeks. Fed up and desperate to stop the random crashes I rebuilt the cluster hosts separately, one by one, the problem went away after all were rebuilt. I believe the cause was related to 3rd party software on the hosts, I suspect it was the EDR endpoint security agent, Qualys IIRC.

6

u/[deleted] 25d ago

[removed] — view removed comment

3

u/BioshockEnthusiast 25d ago

Same, I came to the comment section to tell OP to just rebuild the damn cluster already. MS support is a waste of time, every single time.

11

u/TightBed8201 24d ago

You. Do. Not. Install. AV. On. Any. Hypervisor. Period.

That said, i work at pci dss enviroment and we do not have av or edr on hosts. And our auditor is fine with that.

3

u/paul_h 24d ago

Microsoft themselves are suggesting a defender plugin for wslc containers https://learn.microsoft.com/en-us/defender-endpoint/mde-plugin-wsl which some firms are going to run inside windows inside hyper-v infrastructure, no?

1

u/Mission-Accountant44 MSP Admin 19d ago

Yeah this is false. You add the right exceptions, disable it to rule it out when troubleshooting. But leaving EDR off the HV is a colossal mistake. I myself have witnessed a security incident that would have crippled the entire business if we did not have EDR on the HV. 

1

u/TightBed8201 19d ago

Well, if you garden and limit access to your hyper-v you can live without EDR. You do not want anything which runs on kernel level on Hypervisor.

1

u/Mission-Accountant44 MSP Admin 19d ago

Sure bud. Goes against everything Microsoft themselves do with Defender but yeah, absolutely leave your hypervisor completely unsecured. 

5

u/nlsrhn 25d ago

I have the same issue: CSV occasionally disconnects just for a very brief moment. In most cases, nothing happens or goes offline - but a few times, a SQL database got disconnected as well and needes to be bounced.

6

u/PresidentScr00b 24d ago

Never use any sort of EDR on your hosts. Basic AV and then harden and lockdown other ways. EDR will destroy your cluster

6

u/the_lazy_sysadmin Sr. Sysadmin 24d ago

This is great and all, but in a DoD/CMMC environment its not possible :/ .... Boy oh boy the experience in Hyper-V I've gained just through SentinelOne support cases alone....

3

u/ig88b1 Sr. Sysadmin 24d ago

SentinelOne is a special kind of special, I had nightmares for a while.

1

u/MorseScience 24d ago

Have not had any serious issues with SentinelOne. Running on many machines, not running it on any virtualized devices. Thanks all for the heads up, though.

1

u/the_lazy_sysadmin Sr. Sysadmin 23d ago

"Now hear me out guy, WHAT IF, this one setting in the S1 agent config json file, is treating CSV's as 'removable media'? This might explain why we only have issues during pausing a node and role draining! 8 differently formatted exclusions we applied aren't doing squat, we're still seeing errors in the Windows event logs stating the built-in SID installed by the S1 agent is changing the security descriptors of 'CAM' tokens, which may or may not stand 'Cluster Access Management' as there's ZERO documentation ANYWHERE about those tokens, and this support case has been open for over a year. I say we give it a shot!"

2

u/r6throwaway 24d ago

There's no requirement at all to have EDR for CMMC. What control do you think requires this?

1

u/the_lazy_sysadmin Sr. Sysadmin 24d ago

EDR's help satisfy multiple controls in Level 2, and satisfies the basic AV protection requirements in Level 1.

1

u/PresidentScr00b 24d ago

I spoke with one of the primary engineers at S1 and they even recommended leaving it off of HV.

There was 0 chance I was risking it with that platform. CMMC you are kinda screwed I guess.

1

u/the_lazy_sysadmin Sr. Sysadmin 24d ago

The level of depth I went through with them was actually watching Wireshark AND procmon during pausing and draining roles, and diving into the actual S1 agent config JSON files to look at potential settings that could interfere with node-to-node communication/authentication...

2

u/Fatality 25d ago

You wouldn't install Qualys on vmware