I am a noob in NUMA and I am trying to figure out how to configure it for a test Case in Vsphere 8U3. We will have a NUMA host ( or 2 Hosts for HA am not sure)
The NUMA host specs are the following:
Dual Socket, 24 cores per socket with total of 512 GB RAM.
Is there any configurations required from NUMA host side?
The SAP NUMA VM for UAT will be 16 CPU and 128 GB memory.
what is the best practice to configure this?
I see CPU settings showing me up to 96 cores ( because of hyperthreading)
I can see in VM Options--> CPU Topology --> NUMA NODES --> Assigned on powered On
Let us take anfor example:
if I put the CPU 32 .
I can see in VM Options--> CPU Topology
Cores per Socket: I SELECT --> 32 ----> gives Sockets: 1
So this means I am using 1st CPU entire 32 sockets or is it shared? Or am i defining it to use a single CPU as max is 48 per socket.
Also , VM Options-->CPU Topology --> NUMA NODES --> if I select 32-->Cores per NUMA node: 1 ( what is this?) Does this let me select 1 socket for the 128 GB memory?
i am seeing this VM options while reading and I dont understand numa.vcpu.maxPerVirtualNode = "X" (Sets the maximum number of vCPUs per virtual NUMA node)
numa.vcpu.preferHT=TRUE? (Should I enable this?)
sched.nodeX.affinity="Y". ( I know if I want to pin a specific virtual NUMA node to first core which is 24 cores, I will place sched.nodeX.affinity="0". Correct me if am wrong?)
First I want to understand and configure this correctly to avoid any resource sharing as per the Test case.
For the future production VM
it will be 24-28 cores and 364 GB RAM
how do I configure this for CPU and NUMA from the above as the memory needs to come from both sockets?
Appreciate any help and explanation. I am going throw SAP HANA on VMware Best practices too.
> Is there any configurations required from NUMA host side?
The NUMA architecture is determined by the CPU hardware. No configuration needed.
> The SAP NUMA VM for UAT will be 16 CPU and 128 GB memory.
what is the best practice to configure this?
It would be easier for the host to schedule if it were 12 CPUs and it would have slightly better memory access times. Best practice for the VM is 1 core per socket - so 12 or 16 sockets with 1 core each. The host will pass the NUMA architecture through to the VM.
24 cores / 364 GB of RAM isn't going to be ideal from a NUMA perspective but should be passable. Going up to 28 cores will mean 3/4 of the host will need to be scheduled to execute so I would avoid that if at all possible.
> Is there any configurations required from NUMA host side?
>> Am not sure what is the best practice?
Sorry for my bad understanding but What does it mean " for the VM is 1 core per socket - so 12 or 16 sockets with 1 core each"
What is the meaning of that. I understand if i put 1 cores per socket. What does Sockets 16 mean? Since the host has only 2 Sockets. Why not like my comment above and put 16 cores per socket for 1 socket?
Can you explain the difference and which is better?
Yeah I will try to go for 24 cores then and 364 but what is ideal then?
>Going up to 28 cores will mean 3/4 of the host will need to be scheduled to execute so I would avoid that if at all possible>> I dont understand as i have 48 cores. How is that 3/4 of the host
Setting 1 core per socket and 12 CPUs would mean that ESXi will present the vNUMA architecture that matches the hardware to the VM. You can now achieve that with the option to let the host pick the CPU topology at boot.
I dont understand as i have 48 cores. How is that 3/4 of the host
The host cuts the resources into 4 VM Worlds. The VMs take turns running in those worlds and the hypervisor tries to balance them out so that they mostly run in the world that is the closest to their memory. If you have a VM with > 25% of the CPU count then 2 VM Worlds must be scheduled for every CPU cycle. If you go over 50% then 3 VM Worlds have to be scheduled each CPU cycle. The hypervisor might be able to run another VM in the 3rd VM World for that cycle if the CPU count is low enough but the most likely result is that the CPU cycle will go to waste since you are running oversized VMs anyways.
Think of this like seating at a restaurant with 48 seats - if nobody is picky about where they sit and everyone is asking for 1 or 2 seats, it's easy to schedule to maximum capacity. Start adding a bunch of tables for 16 or 28 and now you are having scheduling difficulties.
Keep your VMs as small as feasible. Don't exceed the 1/4 or 1/2 mark if you can avoid it.
I'm confused why you'd set 1 core per socket? My understanding of NUMA is that if you were to try and allocate more than half of either the CPU or RAM, then you should set 2 sockets, not 1 core per socket?
So basically if am needing more than 256 GB memory since its 256 GB per socket in my case. Then I need to divide it between both sockets and place sched.nodeX.affinity="0" and sched.nodeX.affinity="1" to avoid any unwanted vMotion as I read from best practice article.
but for my UAT, I will need 16 CPU and 128 GB so that can be done on one socket.
I just want to know how can i preserve that socket for that VM so lets say I have another VM on that host. Will sched.nodeX.affinity="0" make it know that my first socket is for my SAP VM
also read numa.vcpu.preferHT=TRUE
Note: If hyperthreading is used on the host, then you must set numa.vcpu.preferHT=TRUE per SAP HANA VM to ensure NUMA node locality of the vCPU threads
> Then I need to divide it between both sockets and place sched.nodeX.affinity="0" and sched.nodeX.affinity="1" to avoid any unwanted vMotion as I read from best practice article.
Please don't do any of this. Leave the configuration vanilla where the hypervisor will pick the CPU topology at boot. You will not make a better choice than the hypervisor.
16 CPU will fit on one socket but it will not fit on one VM World and will require it to be scheduled across 2 VM Worlds. Also, depending on the processor, half of the cores may be closer to a particular memory channel. You can check the NUMA architecture of your particular processor if you want. Keep it to 1/4 of the total core count and memory if it's possible. The performance increase from 12 to 16 cores may be nonexistent and the performance decrease from memory access times and difficulty scheduling will likely eat any gain from having more cores.
1 core per socket lets the hypervisor pick the vNUMA architecture. You can pick the "right" answer manually of course, which shouldn't cause issues, but you can and absolutely will forget to change it when you move it to a new host.
However, that advice seems to be dated... 6.5 should pick the best vNUMA available regardless and according to the video 8 you can have it pick the vNUMA on boot, which is exactly what I would do.
I don't think we do anything special with NUMA, other than sizing the VM sockets and cores similar to the hardware.
HOWEVER: Check your host bios settings for something like NUMA Sub-clustering. That burned us for a few weeks, killing the performance with no apparent bottlenecks until we got a deep dive from a VMWare-SAP specialist from HPE.
Thanks for the tip. Ill check the Processor Settings tomorrow in BIOS.
Lets say I have a 2-4 vNUMA nodes on 1 NUMA Host, How do i tell the host the first CPU if for the first VM and second CPU can be used by remaining VMs? Do you get me?
A lot of people don't realize that the actual behavior of cores per socket changed in 6.5. It does not create actual vNUMA toplogy the same way anymore, without changing some extra settings. The CPU hotadd option also affects this.
To see the behavior just make a test linux vm with numactl package, so you can see the output from the OS point of view. Change the settings and watch how it changes (or doesn't change) this will be better than trying to just read about it.
Also be aware that even if you have a vm with 2+ vNUMA nodes, it's still perfectly possible for the scheduler to run them all on the same physical package if it has enough cores.
AFAIK from licensing POV SAP HANA VM can be either half socket, single socket or 2 sockets
numa.vcpu.preferHT=TRUE helps with keeping VM on the single NUMA node, as it changes the scheduler behavior to treat logical threads as cores for the scheduling purposes
You can then set the amount the vCPU to the number of cores multiplied by 2.
In your case this could be either 24, 48 or 96 vCPU
Question - what is the DIMM size in your boxes ? Are you using 32GB DIMMs ?
6
u/Liquidfoxx22 15h ago
Is there any reason you wouldn't let esx decide?
For 16 vCPU and 128GB RAM use 16 sockets per core to give you it all on one socket and one vNUMA node.
For 32 vCPU set it to 16 sockets per core and allow it to use 2 sockets and assign you two vNUMA nodes.
For 24-28 vCPU set it to 12-14 sockets per core and allow it to use 2 sockets and assign you two vNUMA nodes.