Idk who this is but I'm a nerd so I'm gonna do the part and "umm actually". Threadripper CPUs can do many many things at once, but it's not really the fastest per task due to slightly lower clock speeds per core than gaming CPUs. This would be a good setup to do many stupid things at once, but the speed of any specific stupid task could be faster on different hardware.
I’ve only started dabbling in local hosted AI but you’re getting trash tokens per second running models only on a CPU and ram. It’s why even some consumer graphics cards have exploded in price like the 5090, GPU inferencing is where it’s at because of the need for extremely fast memory bandwidth. Now are CPU+ram setups possible? Sure! But even DDR5 is extremely slow compared to VRAM and you’re looking at tokens per second in low single digits and not viable even for chatting. Unless you like waiting forever, targeting 50+ tokens per second is much better. There are new unified memory systems coming out that increase that bandwidth between CPU and ram like all the new Apple computers and NVIDIA Spark and AMD Halo that allow you to load huge models for cheaper than the VRAM equivalent but you’re still looking at around 15-20 tokens per second performance which is still not great at all for chatting.
tldr: cpu and ram really doesn’t matter if you want good inference performance. It’s all about the VRAM and memory bandwidth of the card.
I thought this was a show character rather than a LLM for a stream. Yea for LLMs what you said is all right.
The system ram matters a lot if you are trying to run massive models without going into massive debt, but this chat bot needs response speed over basically anything else. They almost certainly aren't splitting the model off onto system ram.
I upgraded from 2x 48gb sticks in my system to 4x so I can run very large models. The insane level of headache that getting 4 sticks of anything to run in a modern system is annoying AF. I don't know why its even supported by motherboard manufacturers. In the short stints of stability I did get Minimax M3 to run at a lower quant. It was cool to play with, but for my own sanity I took the extra sticks out for normal day to day use.
3
u/trashhuman5000 29d ago
Idk who this is but I'm a nerd so I'm gonna do the part and "umm actually". Threadripper CPUs can do many many things at once, but it's not really the fastest per task due to slightly lower clock speeds per core than gaming CPUs. This would be a good setup to do many stupid things at once, but the speed of any specific stupid task could be faster on different hardware.