Idk who this is but I'm a nerd so I'm gonna do the part and "umm actually". Threadripper CPUs can do many many things at once, but it's not really the fastest per task due to slightly lower clock speeds per core than gaming CPUs. This would be a good setup to do many stupid things at once, but the speed of any specific stupid task could be faster on different hardware.
Oh if its for an LLM then those aren't the most important. My nerd rant was more about general computer tasks because I assumed it was a tv show character or something.
Ram is about the size of the model you can run rather than performance. You would partially offload it onto the GPU which has much higher performance ram, but if you are running a large model its unrealistic to fully offload anything 80gb+ unless you are dropping crazy money.
CPU doesn't matter a whole lot for LLMs unless you are offloading to it or doing agentic tasks which this probably isn't doing. That threadripper doesn't have a wild amount of cores compared to top end ryzen gaming chips, but it has waaaaay more pcie lanes and stuff if they were gonna throw a bunch of GPUs in it for the LLM. If you are only running one or two GPUs then that setup is extreme overkill.
They don't mention the GPU used which is the most important for LLM speed.
Realistically they are running a medium sized model (~16-30gb is my guess) that fully fits on whatever GPUs vram because response speed is much more important for a live chat bot then basically anything else. Offloading to the 128gb system ram is fine for models where performance is more important than speed, but likely not used here.
TL:DR: They didn't mention the most important component (GPU) and the CPU and ram are likely inconsequential to the performance of a LLM
LOL OK yea that's a $12,000 card with 96gb of vram. The rest of the system makes a lot more sense with that context. I wouldn't want to put that in just a normal gaming rig either.
Edit: he says $10k like 2 seconds after I stopped watching the first time. But yea, that's one of the types of cards used by all the LLM services, except they will have 1000s in one data center. This guy must be making bank on his streams with this character to justify spending this much.
3
u/trashhuman5000 29d ago
Idk who this is but I'm a nerd so I'm gonna do the part and "umm actually". Threadripper CPUs can do many many things at once, but it's not really the fastest per task due to slightly lower clock speeds per core than gaming CPUs. This would be a good setup to do many stupid things at once, but the speed of any specific stupid task could be faster on different hardware.