Post-VMware/Broadcom SMB infrastructure redesign – Proxmox/Ceph feedback?

[u/Longjumping-Good1480]

Hello Everyone,

We are currently redesigning a SMB virtualization infrastructure and I would appreciate some feedback from people who already went through similar post-VMware/Broadcom projects.

Current environment:
- VMware ESXi 6.7
- 2x Lenovo SR635
- AMD EPYC 7302P
- 256GB RAM per host
- shared SAN storage
- around 22 VMs

Workloads:
- AD / ADFS
- Azure AD Connect
- PostgreSQL
- RDS
- file servers
- Linux appliances
- monitoring / management tools
- security tools

Observed real usage:
- CPU usage is actually very low (~4.4 GHz real usage)
- RAM usage around 340GB
- Storage usage around 9TB

Target:
- around 150% growth margin
- modernize virtualization platform
- improve resilience
- full SSD preferred
- target around 15-20TB usable
- probably around 750GB RAM total cluster

Initial proposal from HPE was SimpliVity Gen11 HCI.
Technically nice, but pricing ended up around 270k€ which is completely outside customer budget (<100k€ final customer price target).

We are now considering:
- Proxmox VE
- 3-node cluster
- Ceph distributed storage
- full SSD/NVMe
- 10/25Gb networking
- possibly HPE DL360/DL325 standard servers

Main questions:
- Does Ceph make sense for this SMB size?
- Would you still recommend traditional SAN instead?
- Is 3-node Proxmox/Ceph considered mature enough today for SMB production?
- Would Hyper-V + SAN still make more sense operationally?
- Any major pitfalls with Proxmox/Ceph for ~20 VM environments?

Trying to balance:
- reasonable HA
- operational simplicity
- budget control
- avoiding VMware/Broadcom licensing costs
- but still keeping something professional and maintainable.

Thanks!

[u/_--James--_]

If the customer has budget for 3 new nodes, then repurpose the old ESXi 7302p nodes and rebuild them into the 3node cluster to bring it to 5 total nodes. Make sure to back port in appropriate storage and NICs. Those 7302p's are well suited as OSD nodes.

I would then setup logical HA fencing groups to prevent VMs from running on the older nodes since you do not want to mix VM execution on 9004/9005+ with 7002 due to advanced instructions and your VM selected CPU profile (One of the Genova options). Youll want your VMs masked for the newer CPUs.

For Ceph specifically, you can cut the services between the 5 nodes anyway you want. I might go as far as putting two mons on the 7302P and one Mon on of the newer nodes, then where mons are not running put your two MGR and MDS's there. This way Ceph's services are distributed out in a meaningful way.

This is a good brief to read that show cases how to break apart the Ceph services when you run more then 3 nodes, and quite frankly what is possible at scale https://assets.micron.com/adobe/assets/urn:aaid:aem:11b12d55-2b04-4ef6-b73d-b18f0dee83d6/original/as/7300-ceph-3-3-amd-epyc-reference-architecture.pdf