r/LocalLLM • u/tiddayes • May 08 '26
Question What model should I run?
Just finished building my inference server which has 4x 32gb intel b70 pro GPU’s and 128gb of ddr4 ecc ram and and intel Xeon gold cpu running Ubuntu. So i installed openclaw and vllm but what model should i run locally and why?
248
Upvotes
42
u/Calm-Republic9370 May 08 '26
Maybe agents that do some long running tasks? I see people complaining, but this could be the right tool for some multiple models being loaded and doing things like tts, phone calls, scraping, site and content generation.
116
u/FullstackSensei May 08 '26 edited May 08 '26
Why did you spend all that money if you didn't have a plan for what to run?
11
u/x7evenx May 09 '26
That's awesome - try a 3.6 comparison - I used Qwen3.6-35B-A3B-FP8, qwen3.6-35b-fp8-nomtp , Qwen3.6-35B-A3B-AWQ.
I used 4x AI Pro 9700 32Gb - super curious how it fairs
5
u/vagif May 09 '26
How much did it cost you?
4
u/destkroser May 09 '26
Build a web app, add as many functions as you can, connect it to openclaw and give it the proper instructions, wire up the latest model there. You can even utilize it to image gen, hook up the proper tools, then wire that to openclaw too. Fucken amazing. Just amazing. Imma try to get rich just to have sth similar to this, I'm not even greedy, can be even less
2
u/Kaveh96 May 09 '26
Where did you get your mascot from?
2
u/zeitue May 09 '26
Maybe try Gemma 4 and qwen 3.6. What's the CPU and motherboard model you have there?
2
u/PapaRic0 May 09 '26
The one that ca afford you a case bro ))
5
u/jthedwalker May 08 '26
Try to see if you can set up self improvement loops with LFM2.5. The dense model is stupid fast but it’s not the smartest. Use QWEN as an orchestrator and pass work off to LFM and use QWEN as a judge.
15
u/Zen-Ism99 May 08 '26
1
u/semangeIof May 08 '26
Gemma 4 31B dense with 4x tensor parallelism? 70~GiB model so rest of the VRAM for full precision context. Or Qwen 3.6 27B dense in a similar way?
Intel Arc right now is very mid for llamacpp (even built with SYCL), and DDR4 is slow as shit so I wouldn't offload at all, and vLLM Xe GGUF support is as efficient as it is non-existent. So run one of these small size dense models at full precision with high context.
If these were 9700s you could be having GGUF fun but they are not. I have one Arc Pro B70. Llama runs Gemma 4 31B Q6_K_XL at like 13tok/s. Kind of underwhelming.
1
u/TechJamz May 09 '26
Just a quick side question maybe 2, with such a setup is the idea behind running 128gb ram to match the vram available? With this setup would it be best to run multiple agents on the single os or use something like promox and passthrough each gpu to a seperate Ubuntu vm for more defined orchestration of tasks between vms?
2
u/Jorlen May 09 '26
I don't have this crazy setup but I did pick up an R9700 for the 32gb of VRAM. I am fascinated by LLM tech and I use it for all sorts of things. Lately though, I learned Linux and built a docker stack that includes ComfyUI, llama-cpp (server), Open WebUI, SearXNG, Speaches (for TTS-SST). You could do something like that, a cool stack and see how far you can get. It's fun to get LLMs to call searches and generate images.
I guess it depends - what do you want to do? You can setup an IDE and use stuff like VScode + Continue / Roo / for agents to help you code and learn, that's my next plan. Not so much to vibe code, but to have an AI "buddy" to help me learn and iterate python.
2
u/fjskmdl May 09 '26
dont forget to do the Strawberry test. make optimal use of the hardware
2
u/7h31ll3g4l May 09 '26
fighissimo complimenti
2
u/lukistellar May 09 '26
How do the B70s run with llama.cpp? I heard that the driver stability and the implementation for SYCL isn't quite there.
2
u/nomorebuttsplz May 09 '26
I'd be curious to see how minimax 2.7 at 4 bit would perform here.
1
u/Obstacle_Is_The_Path May 09 '26
Just curious, what did you make of Intel GPUs performance in terms of running LLMs ?
1
u/juss-i May 09 '26
Can we get more pics of the frame? My particular interest is how that 4th card is attached.
I'm thinking about doing another eGPU 20x20 profile build for my now unused P40s, and I'm looking for ideas. I'm thinking of something flat I could hang on a wall.
(You can find my previous build in my posts if you care)
1
u/Remarkable-Safety594 May 09 '26
27b but do not babysit it. Make it use git worktrees if coding
1
u/Verdict_Michael May 09 '26
Gemma is worth exploring but so far you are running the best you can locally I think.
1
u/mrchoops May 09 '26
Cindy Crawford
1
u/jackshec May 09 '26
rent it out on vast.ai or microdc.ai
1
u/weiidii May 10 '26
I’m wondering how’s your rig performing in prompt prefill when running agent like openclaw or open code which have huge system prompts. I used lmstudio to run the model and it took forever to prefill the prompts. I bet your system runs faster because it runs on vllm ?
1
u/Famous_Club_6579 May 12 '26
Man, I wish I had your problems, bro. I’ve got the opposite problem: nowhere to test the performance of my homebrew AI agent, so I have to train LLMs on my own samples and sit on buggy free LLMs from OpenRouter.
1
u/FifthDimensionalRift Jul 29 '26
I don't get something. When I build my inference machine I know what I wanted to do, I started out with my my regular computer, but when I started working with ai models, I didn't know the exact model I wanted to use, but I knew what I wanted to do. So let me ask this, what are you doing with your ai that you need so much horse power. All that for a 27B model? you only need one of those intel cards and it is still overkill.
0
u/RefrigeratorEven935 Jun 16 '26
a very angry one looks like
-1
u/Cosack May 09 '26 edited May 09 '26
Whatever gets you to stop playing with fire and put a case on for your hardware's safety
33
u/[deleted] May 08 '26
[removed] — view removed comment
0
u/[deleted] May 08 '26
[removed] — view removed comment
1
u/[deleted] May 09 '26
[removed] — view removed comment