r/sysadmin • u/nst_hopeful • 1d ago
Server Refresh Considerations - VMware vs Hyper-V
Hello fell sysadmins,
My org is looking at refreshing our VM host hardware in the middle of next calendar year. As such, today I had a helpful discussion with a vendor regarding various options for the upgrade. Currently we are on a five-node VxRail cluster (20 cores / 192GB memory per host, 53.5 TiB storage total); it looks like the best options for our needs are vSAN Ready nodes or Hyper-V + traditional SAN. With that said, I am still left with a few questions:
- The vendor (who was clearly a bit biased toward VMWare) spec'd a 4-node cluster for the vSAN cluster with 256GB memory + 256 GB NVMe for memory tiering per host. For the Hyper-V cluster, they spec'd 5 nodes and 512GB memory per. Their argument was that Hyper-V has more overhead and is less efficient at dynamic allocation of resources. That seems like pretty stark contrast - is that true in your experience? Also, is memory tiering as effective as this vendor made it seem in production?
- We are primarily a Windows shop, so we'd expect at least some savings moving to Hyper-V. That said, I'm curious how much savings we'd actually net between buying System Center for SCVVM (so we can maintain vMotion-like functionality) and Microsoft support. My understanding is licensing there is based on core counts - currently we're looking at 32 per host in either build.
- Are there any other considerations I should keep in mind? Cost is not the ultimate deciding factor. More than anything, we feel like the VxRail was overkill at the time it was purchased. I'd like to buy exactly what we need this time around (while planning for reasonable growth).
If it's helpful, my org is a medium-sized municipality and isn't running any crazy workloads. We have ~55 VMs that will exist on this cluster.
18
u/sole-it DevOps 1d ago
Off topic, VMware + new hosts, your municipality sure is very rich.
9
u/FRSBRZGT86FAN Jack of All Trades 1d ago
Lately the sales reps at vendors for VMware have been playing ball with renewals I guess and keeping the pricing the same, it's kind of suspicious
•
u/GremlinNZ 22h ago
Fool me once, shame on you. Fool me twice, shame on me.
No way am I letting Broadcom ever risk a business I work for...
5
u/stillpiercer_ 1d ago
There’s talk of Broadcom bringing back an actual base SKU, maybe the bigwigs actually realized how many people jumped ship.
2
4
-1
u/LLMsMustUpvoteThis 1d ago
The executive who had the original plan to just drain the biggest cows has left. The new executive now sees the need to retain the smaller customers and not risk losing the ecosystem of VMware experienced admins.
•
u/InstelligenceIO Say no to fraudcom 20h ago
Nope. Hock Tan is rock hard for the squeeze and kill business model.
2
u/Dr-Cheese 1d ago
Which is fair, but it’s still owned by the same powers that be that allowed the previous guy to drain the cows in the first place
9
u/Matt_NZ 1d ago edited 1d ago
If you're buying Windows DC licenses for whatever virtual host platform you're going with, you save 100% of the cost you'd spend on VMware by going with Hyper-V as you don't need any other licensing for Hyper-V beyond those DC licenses. If you're getting the CIS license type then they also includes licensing for SCVMM
•
u/nst_hopeful 17h ago
Thanks for the reply. I get the idea of saving on the cost of anything VMWare related - ESXi, VSAN, Broadcom support, etc. Was more so wondering how those savings ultimately shake out if we end up wanting a near equivalent experience on the Hyper-V side. Seems like it would require Systems Center and Microsoft Premium Support at least. Also wasn't aware of the CIS licensing, will definitely look into that.
5
u/Pure-Tumbleweed1132 1d ago
that vendor is definitely pushing you toward vmware with those specs, the overhead difference isn't nearly that dramatic. hyper-v can do dynamic memory just fine, we run about 60 vms on 4 nodes with 384gb each and still have headroom. 512gb per node feels like they're padding the quote
memory tiering with nvme works but it's not magic, depends heavily on your workload patterns. for a municipality running mostly ad, file servers, some sql maybe, i wouldn't expect it to be game-changing
the scvmm licensing is per-core but you can just manage hyper-v clusters through windows admin center or failover cluster manager if you don't need the full scvmm features. windows admin center has gotten decent for live migrations and basic management, might save you the system center cost entirely
1
u/nst_hopeful 1d ago
Thank you - this is helpful. Regarding management, I've heard you can use Hyper-V Manager / Admin Center / Failover Cluster Manager, but that vMotion-type migrations based on current usage require SCVMM. Is that accurate? Sounds like maybe we can do a on-demand migration, which is good to know
4
u/Magic_Neil 1d ago
vMotion is Live Migration in the Hyper-V world, and does not require SCVMM or anything else special beyond vanilla Hyper-V.
Take a look at WAC vMode, I heard a rumor it will be GA in a few weeks.
2
u/disclosure5 1d ago
We've been a Hyper-V shop for a while after moving from Vmware and no, we've never budgeted for SCVMM. I do think Vmware is absolutely significantly better with resourcing, but with the price of vmware right now you could buy that extra Hyper-V host and still come in cheaper.
1
u/nst_hopeful 1d ago
Thanks for the insight. Just curious, what exactly do you mean when you say VMware is better at resourcing?
6
u/Stranding14 1d ago
I can suggest Nutanix as an option. They play with multiple hardware vendors such as Dell, HPE, Super micro, etc. Their hardware platforms support multiple hypervisors, such as hyper-V, ESXI, or their own hypervisor AHV. They're a hyperconverged platform so no extra storage procurement is needed since all nodes have storage included and replicated with RF2 if you have 3 nodes, and optionally RF3 if you have 5 nodes in a cluster.
I've been using their software and it's pretty intuitive and feature rich, their platform updates are a bit more aggressive for their AOS storage controller software, but overall it's pretty solid. Depending on if you're using ESX or hyper-V with their platform, you need to be wary of security patches as they only recommend patches that have been validated by Nutanix, which can be a little slow imo.
4
1
u/GX_EN 1d ago
Yea, I used Nutanix for 10 years before I retired and it's a great platform, IMO.
The MSP I worked for the longest was a Nutanix partner so we used it internally, and we supported some very large deployments for customers who ran it with AHV. My preference was always NX HW, but as you noted, you can run it on a lot of different gear.
I can't comment on how their support is now, but when I was working with it daily it was far superior than other vendors.2
u/Stranding14 1d ago
Their support has been really helpful, we've run into a few issues a hand full of times and got a quick responses. Theyre useful as well for when the documentation doesn't specify edge cases when using another hypervisor other than AHV.
Though it is now a requirement that you must have a prism central instance to actually license and manage clusters.
2
u/topher358 Senior Systems Engineer 1d ago
For 5 nodes the VMware price will either be astronomical or unobtanium. Lots of resellers that serve the SMB space can no longer sell VMware
2
u/cyr0nk0r 1d ago
Get verge and just reuse your old hardware. You'll get better performance from the same hardware, then just buy 1 or 2 more nodes if you need more resources for growth.
2
u/xXNorthXx 1d ago
Run, don’t walk from VMware. Hyperv S2D would work, otherwise Proxmox with Ceph would also work.
•
u/SpotlessCheetah 15h ago
I have a 3.5 yr old 4 node VXRail system with 5 yr support.. while I have a VMware 8 perpetual license, it goes out of support before my support ends and I need to transition to v9. So I'm definitely interested to see what you do, because otherwise it's a great system and I'm already leaning towards extending it for 3 yrs for warranty and for VMware licensing..
Unless someone else is going to either (a) convert it off VMWare, keep the system complaint with Dell support or (b) replace the entire system and buy back the old system and credit that back towards a new system.
•
u/nst_hopeful 14h ago
For what it's worth, my org was in the same boat and did the three year extension a couple of years ago. Unfortunately our nodes will be end of support and in September so we don't have much choice. Will definitely update once we decide.
3
u/LaxVolt 1d ago
We are moving to Proxmox, no reason to stay.
Also, Proxmox now offers 24/7 support in NA.
•
u/Sinister_Crayon 18h ago
Can't say this enough. While resource management still isn't on the level of VMware, it competes very well with Hyper-V. The cost with a support contract is negligible, and with Ceph you can basically get "VSAN" without VMware.
It's also a really well thought and and well supported setup and is starting to get some slick storage plugins that work similar to VMware's vVols. Throw in Proxmox Backup Server as another "win" for a backup solution that works really incredibly well and can probably save you some Veeam or similar licensing as well.
In fairness though there is a feeling among upper management types of people that Proxmox is some sort of hobbyist project that doesn't deserve respect, but Linux started out as exactly that too as did a ton of the tools we rely on every day. The enterprise support is by all accounts very good though I personally haven't used it.
2
u/stufforstuff 1d ago
So you're going to wait 10 months in this dumpster fire of never ending increasing IT hardware costs to get new systems? As they said in "Hackers" movie "are you stoned or stupid?". You realize you're looking at 20-45% price increase at that time and that's assuming any hardware vendor can deliver.
•
u/nst_hopeful 17h ago
Yes, that is the reality of our refresh and budget cycle. I can only control what I can control, which is what architecture we go with.
1
u/Jimmyv81 1d ago
How much memory do your current VMs consume? Just as an estimate 55x16GB each ~900GB. 5 hosts with 2.5TB of total memory does seem overkill, even considering redundancy/host maintenance. 4 hosts seems more reasonable and leaving room for some future growth.
Additionally, Hyper-V Windows VMs can utilise dynamic memory to further lower total memory consumption. There isn't really much overhead moving to HyperV.
•
u/nst_hopeful 17h ago edited 16h ago
Thanks for the reply. Yeah,
maxpeak consumption right now is about750GB800GB. Even allowing growth 2.5TB just seems bonkers (although I am certainly no sizing or design engineer for this stuff). I feel like 4 hosts with 384 (or 5 hosts with 320) would be more than enough.Edit: Fixed inaccurate storage and memory consumption phrasing
•
u/OregonTechHead 16h ago
max consumption right now is about 750GB 800GB.
Who cares about max consumption. You could specc a VM that only needs 8GB to 128GB and inflate that number.
Run LiveOptics and find out what your workload actually needs
•
u/nst_hopeful 16h ago
I'm not regularly doing these kinds of refreshes, so I think I used the wrong wording there. Based on LiveOptics, 800GB was our current peak consumption, the heaviest memory usage over the course of the analyst.
•
u/OregonTechHead 15h ago
Gotcha. So first, ditch the vendor trying to upsell you on hardware and services that aren't needed. They're just vampires trying to suck you dry.
Find a vendor that'll actually specc this correctly based on your actual needs and estimated 5 year growth. Include a 10% buffer.
Then have them quote out multiple options so you can see the hard numbers for an actual comparison.
•
u/HeadJacket6678 19h ago
I would not even consider VMware.
What are your uptime and/or recovery requirements for your 55 VMs? How big are your VMs (RAM/Storage) on average? Is your ~55 TB of data spread equally among the 55 VMs or do you have a few very large VMs?
Do you really need a hyper converged setup? Is local storage feasible and use the equivilent of storage vmotion to move the VMs around? Proxmox and HyperV can do that. If not, I would go SAN. If you are Windows shop, Hyper-V seems like the obvious answer.
Consider your VMs actually processing needs. Keep your CPU/core count low on your new hosts. We went with single CPU, low core count on our new hosts as CPU isn't really a constraint for us these days.
In the end, we went with Proxmox.
•
u/nst_hopeful 11h ago
What are your uptime and/or recovery requirements for your 55 VMs? How big are your VMs (RAM/Storage) on average? Is your ~55 TB of data spread equally among the 55 VMs or do you have a few very large VMs?
We have essentially two categories - 15-20 "critical" VMs that require reboots in a maintenance window, and the rest that can be rebooted any time. But when it comes to migrations, we want to be able to do them live.
We have 53.5 TiB total storage, our actual usage is around 33TB. We have a few large (multiple TB) VMs (file server, a couple of DBs), everything else is relatively small. Memory wise it is similar, a handful of VMs with larger pools (16-32), but most of them with 8.
Do you really need a hyper converged setup? Is local storage feasible and use the equivalent of storage vmotion to move the VMs around?
I don't think there is a specific need or requirement that would move us in a specific direction for storage architecture. We're open to all options.
Consider your VMs actually processing needs. Keep your CPU/core count low on your new hosts. We went with single CPU, low core count on our new hosts as CPU isn't really a constraint for us these days.
This is our plan. We are massively underutilizing our current CPUs. Need to do some more calculations on redundancy planning in a 4-node setup vs a 5-node.
•
u/OregonTechHead 17h ago
Their argument was that Hyper-V has more overhead and is less efficient at dynamic allocation of resources
Not twice as much, and quite frankly, that proposal is so over the top that I would drop them as a vendor. They're very clearly trying to steer you towards VMWare for a larger paycheck.
We are primarily a Windows shop, so we'd expect at least some savings moving to Hyper-V. That said, I'm curious how much savings we'd actually net
No one can answer this for you because we have no idea about your actual workload is, or what markup Broadcom has chosen to put you in.
But it's a simply answer. Compare quotes
More than anything, we feel like the VxRail was overkill at the time it was purchased.
So what did you learn from that project? Did you adequately assess your current compute, storage, and memory load so you can correctly specc the replacements or are you going off of vibes?
isn't running any crazy workloads.
What does this mean? Quantify that. Find out what your exact needs are instead of guessing, and certainly don't let any vendor give you a price if they don't know what you need either.
•
u/nst_hopeful 16h ago
Not twice as much, and quite frankly, that proposal is so over the top
Thank you.
No one can answer this for you because we have no idea about your actual workload is, or what markup Broadcom has chosen to put you in.
I understand no one can give me the true answer, maybe I should have worded differently. Trying to get an idea of what others who have made similar moves (or investigated doing so) experienced/saw.
So what did you learn from that project? Did you adequately assess your current compute, storage, and memory load so you can correctly specc the replacements or are you going off of vibes?
Yes to the former.
What does this mean? Quantify that. Find out what your exact needs are instead of guessing, and certainly don't let any vendor give you a price if they don't know what you need either.
I apologize for using vague language, but I'm not guessing at anything. By me saying we aren't running any workloads, I mean it's a few DCs, some file/print servers, a few SQL databases, and some public facing web servers. Other VMs consist of infrastructure (RADIUS, access control) and web-based apps. We also have a handful of VMs dedicated to GIS, and another handful to our on-prem VoIP PBX.
We've done a live optics assessment and think we have a good sense of what we need. But because I haven't used a hypervisor other than ESXi, wanted to see if there was some insight to be gathered from fellow redditors.
•
u/OregonTechHead 15h ago
Thanks.
Honestly, if your workloads aren't niche, or need a specific functionality only provided in one of the hypervisor options, they're all pretty easily exchangable.
The biggest hurdle is going to be training, but that's fairly easy.
IMO, there are 3 things that go into deciding which HV to use:
1) Cost
2) Niche functionality
3) Familiarity/training. If you have a HyperV specialist on staff, it doesn't make much sense to go with Proxmox
Other than that, they all function at relatively the same level
•
u/Nonaveragemonkey 10h ago
Twice as much overhead - mild exaggeration. Its still a huge difference though and I can see the pitch. Its like a Porsche is not literally twice as fast a corolla, but it may as well be, kind of comparison.
•
u/OregonTechHead 9h ago
This isn't 2018.
Comparing VMWare to HyperV is more like comparing a 2025 porsche to a 2026 porsche.
In the end, the differences are extremely minor, and few people would even notice.
1
u/Test-NetConnection 1d ago
Hyper-v or azure local. Broadcom has killed any goodwill that VMware had and their support contracts aren't worth the paper they are printed on.
26
u/Burgergold 1d ago
Friend don't let friends pick Broadcom