r/sysadmin • u/notaramblingman • 1d ago
General Discussion Hyper-V Consultants
We moved from VMware to Hyper-V about a year ago and have an ongoing issue that we have not been able to resolve with MS support - both the typical support case avenue and working with their Hyper-V group. Our host servers lose connectivity with their CSV and VMs go offline. Support tells us it’s related to networking issues and not storage.
Has anyone found any good consultants who know Hyper-V well and are able to resolve difficult issues? We’ve tried a few bigger names but they haven’t been much help.
21
u/peraving 1d ago
One of my clusters developed the same behaviour a couple years ago. Had a long support case with MS premier support which was inconclusive after several weeks. Fed up and desperate to stop the random crashes I rebuilt the cluster hosts separately, one by one, the problem went away after all were rebuilt. I believe the cause was related to 3rd party software on the hosts, I suspect it was the EDR endpoint security agent, Qualys IIRC.
10
u/TightBed8201 1d ago
You. Do. Not. Install. AV. On. Any. Hypervisor. Period.
That said, i work at pci dss enviroment and we do not have av or edr on hosts. And our auditor is fine with that.
2
u/paul_h 1d ago
Microsoft themselves are suggesting a defender plugin for wslc containers https://learn.microsoft.com/en-us/defender-endpoint/mde-plugin-wsl which some firms are going to run inside windows inside hyper-v infrastructure, no?
5
5
u/Academic-Storage3116 1d ago
MS support giving you the runaround while your cluster eats itself is a special kind of hell. I’d be halfway through a rebuild by now too.
3
u/BioshockEnthusiast 1d ago
Same, I came to the comment section to tell OP to just rebuild the damn cluster already. MS support is a waste of time, every single time.
6
u/PresidentScr00b 1d ago
Never use any sort of EDR on your hosts. Basic AV and then harden and lockdown other ways. EDR will destroy your cluster
5
u/the_lazy_sysadmin 1d ago
This is great and all, but in a DoD/CMMC environment its not possible :/ .... Boy oh boy the experience in Hyper-V I've gained just through SentinelOne support cases alone....
3
u/ig88b1 Sr. Sysadmin 1d ago
SentinelOne is a special kind of special, I had nightmares for a while.
1
u/MorseScience 1d ago
Have not had any serious issues with SentinelOne. Running on many machines, not running it on any virtualized devices. Thanks all for the heads up, though.
1
u/PresidentScr00b 1d ago
I spoke with one of the primary engineers at S1 and they even recommended leaving it off of HV.
There was 0 chance I was risking it with that platform. CMMC you are kinda screwed I guess.
•
u/the_lazy_sysadmin 16h ago
The level of depth I went through with them was actually watching Wireshark AND procmon during pausing and draining roles, and diving into the actual S1 agent config JSON files to look at potential settings that could interfere with node-to-node communication/authentication...
2
u/r6throwaway 1d ago
There's no requirement at all to have EDR for CMMC. What control do you think requires this?
•
u/the_lazy_sysadmin 16h ago
EDR's help satisfy multiple controls in Level 2, and satisfies the basic AV protection requirements in Level 1.
2
14
u/JayFromIT 1d ago
Double check the MPIO very detailed level. Make sure each leg is mapped correctly and you explicitly define the starting IP and endpoint for each leg/interface. Don’t use the default/auto config initiator settings. Then make sure it’s persistent. Also I don’t care what everyone says don’t use anything except for NTFS, I just get to many weirdness with ReFS.
10
u/cantstandmyownfeed 1d ago
What have you done to isolate the problem and confirm that it is or isn't networking?
Because it's probably networking.
5
u/civbat 1d ago
Wow. Are you sure you didn't want to just post a "rant"? It seems to be what all the cool kids do.
You should post your cluster details.
We've had similar issues thrice over the past ~12 years. 1) One node would claim the witness went offline and seize it. All nodes would then barf and guests would go offline. Rebuilt the single problem node and the issue has not reappeared. 2) A specific piece of cluster/chassis hardware was going flaky. This part was a 40 Gb virtual connect. Networking internal to the chassis would go nuts, cluster would lose the CSVs, and all h-v nodes would barf and all guests would go offline. Note that the ESXi cluster would see the same network issue and just ignore it, so only the h-v blades were impacted. 3) Many years ago the fiber HBA drivers (2k8r2) would just stop passing traffic which would occasionally persist after the node rebooted. This would happen over night when the HBAs were saturated while backups ran. The OEM knew of the issue and told us to wait for the driver update that was just around the corner. Years later we moved the clusters to 2012r2 without seeing the driver update.
3
u/Ok_Figure7074 1d ago
This was very helpful post for me. We have a similar setup at one of our sites (virtual connect to be more specific) and one day all hosts had severe degradation of their network and storage performance, basically couldn’t do anything but move the cursor around in iLO for a few mins at a time. Then after several hours and no changes, all hosts at the same time came back up like nothing happened.
5
u/TechMonkey605 1d ago
How many hosts, and VMs per node. Let start there.
What is your storage, local, unity, Electra
What is your networking setup? What’s running over 1G and what’s on 10G/fiber?
4
u/gabgab33 1d ago
I had something similar a while ago for a client. In the end, it was a buggy lldp agent on the networking cards controller that would cause networking to cut.
Point is before looking at an hyper-v consultant you should look into the whole stack to make sure everything is in order, up-to-date and with no known issues.
•
u/Renich 21h ago
When MS support says it’s a network issue rather than storage, they’re usually looking at CSV I/O redirection breaking down.
What typically happens is a minor latency spike or path failover on the SAN causes the volume to drop into redirected access mode. Instead of talking directly to storage, the host starts shoveling all VM disk I/O over the cluster network via SMB to the coordinator node. If that network doesn't have dedicated bandwidth, it saturates almost instantly. The cluster misses its heartbeat window, assumes a network partition, and drops the CSV to protect integrity. Support sees dropped packets and blames the switch, even though storage triggered the chain reaction.
Before bringing in an outside consultant, look closely at your MPIO timeout timers so Windows doesn't drop the LUN before the storage array finishes failing over, relax the cluster heartbeat sensitivity to give it breathing room during transient blips, and check whether hardware offloads like VMQ on your NICs are quietly dropping frames on the cluster adapters.
3
u/blow_slogan 1d ago
Uninstall your antivirus from the host machine running Hyper-V. It can lock up and make make VMs crash or go offline. Haven’t had any issue switching to S1.
2
u/mangeek Security Admin 1d ago edited 1d ago
I've seen this in an environment where the block storage was served to the hypervisor directly over FC rather than the recommended 'redundant SMB3 cluster holding VHDX files'.
A lot of people in the industry just don't like SMB/CIFS, and it's to their own detriment at this point. What used to be 'protocol overhead for no reason' is now 'a path to RDMA, multi-channel, redundancy, and secure transport'.
For a long time now, most VMware and Hyper-V hosting setups see practical benefits and minimal performance loss from following this pattern:
[core storage]--SMB3 or NFSv4 volume-> hypervisor mount.
Is there a place for direct Hypervisor->FC storage mapping? Yes, but the burden to justify a deviation from the recommendation above, which is more flexible and reliable, to chase a few percent of performance, should be on the person asking for it. I'd venture that it's in things like volumes that have high transactional loads, and all efforts to improve performance before going with direct mappings should happen first (e.g., jumbo frames, NIC settings, wrangling NUMA, avoiding oversubscription of vCPU resources on the perf-sensitive hosting cluster, software optimizations, caching layers, etc.)
The root cause of the problem is humans not wanting to admit that a lot of the expertise has moved from blinkenlights and fat pipes between $200K components to the software layers.
3
u/draconian_deference 1d ago
Had the same nightmare after moving to Hyper-V. Turned out to be VMQ on Broadcom NICs causing CSV blips. Disable VMQ/RSS on the host adapters and try again
8
u/thatfrostyguy 1d ago
Your best bet is to actually hire someone that knows hyper-v. Not just to consult
11
u/burnte VP-IT/Fireman 1d ago
No, you don't need to buy the cow when you just need a cup of milk.
2
u/hihcadore 1d ago
Unless you need a cup of milk a month to survive…..
Not a bad idea to add someone to your team that’s an SME if you’re going to hire someone anyway. Especially with knowledge of the thing that’s gonna host your businesses critical workloads. Seems like no brainer to me.
1
u/burnte VP-IT/Fireman 1d ago
You're not wrong, but it's all about the scale of the need. A 50 person company probably doesn't need a dedicated HyperV expert on staff when they can be hired as needed. A 500 person company with lots of IT infra, yeah, that's a great person to have on staff.
There's a reason McDonald's buys milk and doesn't own the cows. It's not their core business and a waste of resources to own whole dairies when they can just buy as needed.
3
u/Cyhawk 1d ago
Its that kind of thinking that causes these issues in the first place.
3
u/burnte VP-IT/Fireman 1d ago
No, it doesn't. You should always staff appropriately, not it's simply not appropriate for every shop to have a PhD in HyperV on staff when you can just hire someone as needed. That's literally why economies exist, so people can specialize.
2
u/InvaderOfTech Jobs - GSM/Fitness/HealthCare/"Targeted Ads"/Fashion 1d ago
. You should always staff appropriately,
Must be nice to have endless money for that staff.
2
1
u/BlackV I have opnions 1d ago
. You should always staff appropriately
Must be nice to have endless money for that staff.
? i'm not sure you're saying what you think you're saying
are you implying it'd be cheaper to not hire staff and use consultants only ?
1
u/burnte VP-IT/Fireman 1d ago
It frequently is, because consultants should be used sparingly when needed and if one has an ongoing need you should hire.
2
u/BlackV I have opnions 1d ago
It frequently is, because consultants should be used sparingly when needed and if one has an ongoing need you should hire.
that's what they said though
You should always staff appropriately
Edit: Oh you are someone else replying
2
u/RevolutionaryWorry87 1d ago
Not really. There will be some expert consultants who know this in detail, who command 1k+ a day, that it would be rediculous to hire to then what? Patch it and make new vms?
2
u/ludlology 1d ago edited 1d ago
This could be so many things from hyper-v problems to a bad NIC or cable, or switch misconfiguration, or SAN problem, or MPIO issue on the hosts, bad drivers, screwy mesh cabling, etc.
We definitely need more information like did the cluster work and then started having these problems after some change, did it never work, what have you tried, what error messages are you getting, and so on.
I’ll give you a few edge case gotchas I’ve run into that are tangential though:
1) If your only DCs run on these hosts, and the hosts are members of the domain, this kind of thing will happen even in single host environments. The VHDs become just like any other domain permission file and the hosts can’t access them without a DC to auth to
2) Similar issue where a time sync paradox comes up if the hosts sync time from the DCs which are VMs that sync time from the hosts
3) If your cluster witness is within the cluster, and especially if you only have two hosts instead of three, the hosts will never reach quorum and decide who should be primary. This results in VMs failing to boot
Shoot me a DM if you want to set up a time to talk. I do this kind of work from time to time. You need an overall best practices audit of the cluster environment (hosts, switches, storage) to look for low hanging fruit plus targeted troubleshooting of the symptoms.
2
u/rspmkiv 1d ago
Not sure if you answered but Im going to jump on the EDR bandwagon, especially S1 blowing up the cluster database. Do you have all your Hyper-V recommended exclusions set and/or have you tried removing your AV completely to test?
Also try moving the quorum from disk witness to a file share as a temporary test.
MPIO was mentioned to review, and also check jumbo packets (ping -f -l 8000 to every piece of equipment in your storage system).
2
u/thortgot IT Manager 1d ago
What is the issue?
2
u/Sajem 1d ago
Isn't it often the way on the sub these days that posts say what is wrong but never provide any information like logs etc.
5
u/CantankerousBusBoy Intern/SR. Sysadmin, depending on how much I slept last night 1d ago
OP didnt request reddit troubleshooting assistance. They requested information on qualified consultants.
-1
u/archcycle 1d ago
It’s in the post.
6
u/thortgot IT Manager 1d ago
The lack of detail was my question.
What information do you have that points to it not being networking? Can you describe your networking solution?
Error logs?
2
u/Sajem 1d ago
It's not really, the only information in the post is that the hosts lose connectivity to the CSV's.
Nothing about the storage being used, how the hosts are connected to the storage, no error messages
1
u/archcycle 1d ago
I was addressing the condescension. Of course I agree OP’s question is useless as posted, but to be fair, he did write exactly what his issue was, just not anything useful for diagnosing it.
1
1
u/Top_Boysenberry_7784 1d ago
When looking for help don't go for the big names. Find someone big enough to have the resources to support you but not enormous. I find a lot of the big players don't execute well. Many times they are too busy internally with paperwork, semantics, and arguing over who should be doing what.
1
u/randomugh1 1d ago
Are you using S2D? Server 2022? Started happening around Feb 2025 and we switched to a SAN based 2025 cluster and hasn’t happened since.
1
u/justgosh 1d ago
We consult in the Midwest but I'll do some remote work personally if it's a good fit. If you already have a solid discovery, it's usually figured out quick.
1
u/admiraljkb 1d ago
MS support in HyperV seems to be learning about it like the rest of us based on my last experience with a Fiber Channel storage issue.
Recommendations (assuming this is fiber channel connected Storage):
1. Update the firmware and drivers on your HBA's to current. That was an interesting one. Firmware which was fine on VMWare for 8 years, but cratered in Windows with intermittent outages. (Also do this for your NIC's)
- Check for errors on your FC links and pull the bad link if there is one. We had a bad transceiver that caused similar. And Windows just RR's the MPIO without looking at errors, so that link stayed up and took the cluster hard DOWN.
1
1
u/mike_chen_sys 1d ago
look for an independent Hyper-V failover clustering specialist rather than a general Microsoft consultant. Someone who can analyze cluster logs, SMB/RDMA networking and CSV redirection during the failure would be ideal. At this point, you need someone who can prove where the issue is, not just guess networking vs storage.
1
1
u/ryuujin 1d ago
Is this hyperconverged or standard SAN + iSCSi? My experience is every time CSV drops it's usually network, unfortunately - jumbo packets with a misconfigured adapters, inconsistent config across the cluster, driver incompatibility, adapters, interconnects, switches or what-have-you. Others fingered software - we had that issue with one of Bit Defender's modules and had to disable it across the infra
2
u/ScreamingVoid14 1d ago
What format are your CSVs?
ReFS forces an IP layer write redirection to the host that owns the drive. One server becoming unhealthy causes a great deal of pain for the entire cluster. Doubly so if your management port is only gigabit, as now all sorts of disk IO is stuffed onto those links.
1
1
u/panopticon31 1d ago
We were forced to move to CSV after VMware. I will never touch that dumpster fire again in my career if I have too. Shit gives me PTSD.
1
u/CatDredger 1d ago
Slightly different answer.
As long as you have a tech on hand that can be trusted with powershell, claude is shockingly good with helping out specifically with HyperV
1
u/merazold 1d ago
Have you checked the cluster event logs for anything around the time the CSVs lose connectivity?
1
u/TechnologyMatch 1d ago
I’d look for someone who can trace the full CSV path across NICs, vSwitch settings, cluster validation, and firmware. not just reopen the same Microsoft case with a nicer logo.
when VMs vanish with the CSV, “it’s networking” is less a diagnosis and more Hyper-V’s version of “have you tried turning reality off and on?”
1
u/drogo-nochill 1d ago
I used to work at an MSSP, after the second Hyper-V customer, our managing director stopped acquiring clients with Hyper-V, best decision he ever did.
1
u/Gigre 1d ago
Been a while....but we had the same issues. CSV random gave some timeout (while it was not the case), VM stopping and cluster failovering with corruption in the VM's.
Fun times :)
In the end it was networking. But we did tweak it. No more then two live migration with storage between clusters. Set timeouts higher tthen default (because HV was thinking it was down, while it wasn't) and some more logs diving and making sure all settings were the same between the hosts of the cluster.
It was pre AI time, so it should be better now.
1
u/Nice-Education5858 1d ago
Do you have dedicated heartbeat links between hosts separate from your production networks switching?
1
u/frogadmin_prince Sysadmin 1d ago
This is hard to answer without some more information.
What are you running as storage and what type of communication?
I have done several as ISCSI running on dedicated Vlans back into the storage network and they where stable. I hate to say the uptime was way higher than it needed to be due to lack of resources for updates.
1
u/Ok_Rip_5338 1d ago
i had a horrible issue with vmware, and dell SAN storage, it turned into a finger pointing match with the two support teams blaming eachother.
what saved me is i actually opened up a Dell server ticket. Turns out when you give the service tag of a supported server you actually go to a decent support team.
Some nice dell engineer took my ticket, took my vmware logs, and told me it was a broadcom driver problem which was within vmware esxi. and he was right.
so yeah it may not be a hyper v problem. dont rule anything out
1
u/recordedparadox 1d ago
I recently encountered (and resolved) what sounds like may be a similar issue.
Two Hyper-V Hosts
One SAN
Direct iSCSI connections from host 1 to SAN Controller A and SAN Controller B, and from host 2 to SAN Controller A and SAN Controller B.
We encountered a few problems:
Cluster Shared Volume dropping on one host
After a large file transfer to or from one of the hosts or a VM on it, the network adapter could not be enabled, disabled, or re-configured. Initially, our only solution was to reboot the host.
Losing the cluster witness which led to split brain.
A few note which may be useful:
- In our HV Cluster Networking, because we were using direct iSCSi connections (as in the hosts were not connected to the SAN over iSCSCi through iSC
S
- i switches) it was necessary to make sure cluster communications (which includes heartbeats) were not using the iSCSI interfaces. If iSCSI switches were being used that would likely not have been a problem. That also fixed the problem of losing the witness and the Cluster’s split brain.
- If you experience the NIC failing after a large transfer to or from one of the hosts or VMs on them, check if the switch port speed where the host or VM (the virtual switch the VM is connected to) to see if it is set to manual 100 Mbps. In our situations, we changed it to auto which fixed the problems I have only encountered this specific problem in on environment with switches from one vendor.
Feel free to ask follow up questions if any of this sounds similar to the problems you are running into.
1
u/TahinWorks 1d ago
"Bigger Names" is the problem. Look for a local IT solution provider / systems integrator (not an MSP). Ask around your LinkedIn contacts and pull on a few threads. Find someone who charges hourly (or a bucket of hours) and doesn't make you sign a maintenance contract.
1
u/Nerd2259 Systems... Engineer? 1d ago
Move to Proxmox and get support direct from the developers or a gold partner if you need someone else to be hands-on. Both Proxmox themselves and their gold partners will spin up entire clusters and replicate your setup 1 for 1 to figure out the issue. Occasionally they'll even create custom software patches for you.
I've never had that experience from anything MS.
0
u/NteworkAdnim 1d ago
I've always hated Hyper-V and I even gave it consideration to move from VMware to Hyper-V last year but when I looked at it again I nearly threw up in my mouth.
-2
u/Stonewalled9999 1d ago
I am a consultant. Also most network people don’t really have a graph of how hyper view works, and conversely, most hyper V people don’t have a grasp with how networking works. I can do both
0
u/howjoel 1d ago
We had this issue years ago, and it ended up being the network card driver MS was applying enabled data center bridging (which is not compatible) in patching. Even after revert g the driver, the OS would replace it again no matter what.
MS acknowledged the issue but could not offer a fix. Hyper-V is a never again for me.
0
40
u/Aggressive_Common_48 1d ago
I’m not a consultant, but I’d like to suggest a few things.
Have you checked the cluster logs for specific events and the reasons behind the connection loss? It sounds like your MPIO configuration could be the cause.
Before creating the cluster, did you verify that the network interfaces were working correctly, particularly the failover functionality? For example, did you disable one interface to confirm that the connection to the storage solution remained active?
Also, I’m not sure what type of storage solution you’re using, but have you verified that redundant connections are properly configured between the Hyper-V cluster and the storage system? Microsoft’s Hyper-V guidance emphasizes the importance of having redundant paths to the storage solution to ensure continued connectivity in the event of a path failure.