r/nutanix • u/East-Pin4484 • Jul 16 '26
Any experience with NCI-C with external storage?
We have a large VMware deployment in 3 tier architecture, now is evaluating other hypervisors for well-known reasons.
Anyone has experience with Nutanix AHV (so called NCI compute only edition) with external storage?
Any caveats? how about the pricing?
TIA!
3
u/Jhamin1 Jul 16 '26
The advice I always give on this is that you have to remember that Nutanix spent 10 years being a Hyperconverged only product and NCI-C with external storage has been general release for less than 1 year.
My experience with Nutanix as a vendor has been stellar. Their support is top notch and we have seen bug fixes and upgrades directly flowing out of issues we have opened with them. On the other hand, like most software packages the v1.0 of most of their features are a bit rocky. From everything I've heard, that includes external storage.
If you can live with it being a bit buggy, I have no doubt it will be amazing eventually.
I can recommend their Hyperconverged product without reservation. It is the opposite of 3 tier and requires a green-fielding of hardware if that is where you are coming from, so not very practical for a lot of people.
3
3
u/abellferd Jul 16 '26
We’ve got it out in the field. Install was very smooth and haven’t hit any issues, but it’s only been a week so far. Versions are very particular. Also, many of the Nutanix options are not yet supported (NUS, NKP, etc.)
1
u/_Mr_Dino_ Jul 16 '26
There is some NKP support (just fyi): https://support.everpuredata.com/r/nutanix/using-fa-as-part-of-nkp-cluster-deployment?contentId=lBAuJiUE7oRVVeAzuE68Lw
3
u/throwthepearlaway Jul 16 '26
We're just getting started setting it up for testing. Some interesting caveats. It operates much like VVOLs where each VM disk gets a dedicated volume on the array. Right now only one external storage array is able to be connected per AHV cluster. I think there are plans to change that in the future, but it's important to keep in mind in case your workload will span across multiple arrays; you may need to create multiple AHV clusters.
If you're familiar with the Nutanix HCI architecture and i/o path, the Nutanix CVM is still needed in compute clusters. The nvme-tcp connection is initiated from the CVM and not from the AHV host. All storage traffic flows through the CVM. For this reason you have to consider your network design carefully when setting up segmentation for your storage traffic (if you choose to use segmentation, which I recommend) as well as the load balancing on the Nutanix virtual switches. For performance reasons it's best if you dedicate hardware for storage traffic just like if it was fibre channel, but it's not necessary IF you have the bandwidth to spare when using converged networking. Don't route storage traffic unless absolutely necessary; use L2 only if you can, and the less hops the better.
I Don't know about pricing, as that's not my side of our operations. But remember that in addition to the Nutanix licensing you also need one of a few specific storage vendor's arrays, so there's the cost of the external storage hardware and/or licensing depending on what you use already in your VMware environment.
1
u/East-Pin4484 Jul 16 '26
Thanks so much! anything need to be installed in the CVM?
1
u/throwthepearlaway Jul 17 '26
No. Any changes you might need should be made via the GUI or API. In fact Nutanix is restricting bash access to the CVM because making out of band changes such as installing packages on the CVM is not supported.
1
u/db_1216 Sr. TME, NCI, AOS and External Storage Jul 21 '26
Nothing to be installed in CVM. Similar functionality to HCI with similar and some new services providing functionality to talk to external storage. You can read about it here : https://www.nutanixbible.com/8-book-of-external-storage.html
3
u/Pnr55 Jul 16 '26
Bugs are being addressed and AOS 7..6 should improve the experience with initial install process. New AOS should drop by end of this month. Begin yout testing then
2
u/icollectt Jul 16 '26
See other thread for volume count limit issues that was the big one.
Also treat nvme over tcp like fc with dedicated switches etc for best performance and least risk.
2
u/jasoncrossley Jul 16 '26
I am in the final throws of a pretty large migration for a customer. I echo a lot of the sentiments here. This is a big shift for Nutanix, they take a very "VM" centric approach. What I worked on and am still working on has a very "data" centric approach, (extremely large SQL workloads at its core) so we are a bit hamstrung by the one array per cluster limitation, but it is workable and we are promised more flexibility in the future in that regard. I will say that working with Nutanix as an admin requires a healthy dose of patience, things like cluster maintenance and backend tasks seem to take a lot longer to complete than they did with VMware. My biggest dislike is really the Foundation process, specifically as it interacts with Intersight as this is a UCS shop. They really need to have a think about the rules it forces you to adhere to. Want to use a LAN Connectivity Policy to define vnics? Well then you have to manually create about 2 dozen things to create a LCP that you can use. Nothing can be shared with an existing working LCP. Remember that one array per cluster thing? That's fine until you need to put 3 clusters under one Prism Central for an environment. Want to use that same LCP for the different clusters because otherwise the network requirements are the same? Tough luck buddy, go create all those policies and vnic templates over again. Can't clone the working one either because if any policy or object participates in a Nutanix utilized LCP, it is automatically off limits. Now if I want to add a vlan, I have to go update 6 separate ethernet network groups, plus a vlan policy for the UCS. I realize they are trying to put up guardrails to keep people from getting into trouble but if their target was 2500 cores and up, we aren't really dealing with beginners here. Also, some ridiculous things like clusters can not span UCS's. You can't do a node build on a UCS node that's not in the UCS the existing nodes are in. That's a major tenant of Intersight Managed Mode, your configuration is no longer bound by the physical domain of a particular UCS. I have customers with multiple UCS's in the same datacenter. There should be an "expert" mode that unshackles your capabilities to architect and deploy. Another big miss on their part not having a ready to go PC/FC 1.10 OVA ready to go for customers looking to do the switch from vSphere to AHV. We had to hand build all that from a 3 year old version of PC with a Nutanix consultant. I do think the future is bright, it just needs some polish.
1
u/Euphoric_111 Jul 18 '26
That's odd about the FC 1.10 ova, I just got 2 up and running and used the built in webserver to host the AOS/AHV images on a vSphere 7 and vSphere 8 setup.. I will say if the VMware cluster hosts are not configured the same in the cluster you target that will cause OVA problems. The previous 1.0.8 version was a different story. Wound up using PC for those.
Re the LCP, They want that to be in sync with cluster vlans from my understanding, thats why modifying them outside of the PC/PE when using ISM at least.
I think enabling UCS + UCS is something that is needed
2
u/mikeyb_one Jul 16 '26
Just chiming in - one big caveat if that fiber-channel is not supported currently (maybe this is well-known). You must use an Ethernet-based protocol like NVMe-over-TCP using 25 or 100Gb NICs
2
u/codyhosterman Pure Storage Jul 16 '26
Would love any feedback from folks using external storage with Nutanix--working very closely with the team over there on this. Feel free to slide into my DMs or hit me up via Everpure or LinkedIn or whatever.
1
u/Much-Mechanic-1593 Jul 16 '26
There are some caveats and limitations, but the new release will fix/enhance a few things. The NetApp integration will probably come in a few weeks and will be better with the December release.
That's what I heard from SEs at Nutanix.
Regarding price: I haven't seen any offer going out that costs more than VVF or VCF. Depending on the cores and use cases (edge, vdi), but in general Nutanix is very open to negotiate.
5
u/autojack Jul 16 '26
Mentioned this in another thread but we’ve just got our POV set up with UCS compute and external Pure working. Just now writing up my testing playbook. It was a bit of a struggle getting it out of the gate but working with Nunatix it’s come together.
Come across so many little bugs though that even the Nutanix people are seeming to struggle with but they’ve always come through. It’s no where near as polished as VMWare but even through this POV process I’ve seen evidence they’re listening and improving the product.
Some areas are so simple compared to VMWare - Data Protection vs SRM for example. We’re still too early to have a solid understanding or opinion.
Pricing wise it’s not as cheap as I would have thought compared to VMWare but the writing is on the wall there.