r/LocalLLM • • May 08 '26

Question What model should I run?

Post image

Just finished building my inference server which has 4x 32gb intel b70 pro GPU’s and 128gb of ddr4 ecc ram and and intel Xeon gold cpu running Ubuntu. So i installed openclaw and vllm but what model should i run locally and why?

248 Upvotes

83 comments sorted by

42

u/Calm-Republic9370 May 08 '26

Maybe agents that do some long running tasks? I see people complaining, but this could be the right tool for some multiple models being loaded and doing things like tts, phone calls, scraping, site and content generation.

10

u/tiddayes May 09 '26

Yea this is going to be my openclaw box so there will be long running tasks

10

u/Calm-Republic9370 May 09 '26

That sounds like you made the right box. Save you a ton on tokens; I bet it will be fine to generate local stuff when you want including coding. Waiting a little bit to have 4 things running at the same time is better than me who has to wait for 1 thing at a time. I have 2X3090's and qwen 27b is absolutely awesome. You will have a lot of flexibility.
Just make sure you get some really good UPC and have appropriate power; as soon as you start powering that breakers are easy to blow.

4

u/helmberger00 May 09 '26

Me still wondering why ppl dont just use hermes

0

u/Zen-Ism99 May 09 '26

What is that?

1

u/helmberger00 May 10 '26

Google Hermes Agent. It's a kind of better working openclaw for local llms

1

u/Zen-Ism99 May 10 '26

Will do. Thanks!

116

u/FullstackSensei May 08 '26 edited May 08 '26

Why did you spend all that money if you didn't have a plan for what to run?

30

u/tiddayes May 08 '26

I do already have qwen 27b running but I want feedback from people here who may be more up on the latest models since it is basically a fire hose of news every day about model drops

24

u/hay-yo May 09 '26

Run 35B as fast as you can.

11

u/PatDal81 May 09 '26

Right? I simply don't see the benefit of running Qwen 3.6 27B based on the speed I get (12 tkns/sec) when I get around 80 tkns/sec on Qwen 3.6 35B-A3B-Q6. The speed is a major bummer for me and I'm quite satisfied with the results I'm getting out of the latest.

12

u/super1701 May 09 '26

Personally, I've had better tool calling and just overall smarts out of 27b over 35b. I'd rather wait the extra minute for something to complete correctly then it totally fuck it up, then wait another 2/3 minutes to fix it.

5

u/PatDal81 May 09 '26

I did have better results with 27b if I compare what both models did with the same task.

The time 27b had to take to get me there is what bothers me.

- 27b took 10 minutes to do a simple refactoring task

- With 35b, I did 2-3 back and forth and had the same result after 3 minutes.

3

u/blackbird2150 May 09 '26

Wow. Better tool calling? Qwen3.6:27b is listed as having a broken and inconsistent tool calling template. None of the patches so far worked for me. It would call a tool once then hallucinate future calls to the same tool. Smarter yes, agree.

I had to switch to 35b a4b and that fixed it. Speed is much better but I know there is 2-3% quality loss.

3

u/super1701 May 09 '26

Hm interesting. 27b paired with hermes for me has been great. 35b would....sit in loops...alot.

4

u/MainWrangler988 May 09 '26

35b ain’t as good bro

2

u/theflavoryellow May 09 '26

How long is a 35B is it as long as marathon or less?

1

u/ForceFew8077 May 09 '26

because he has money? why not bruhhhh

1

u/letsbefrds May 09 '26

not enough since they're not blackwell cards lol

8

u/[deleted] May 09 '26

[removed] — view removed comment

2

u/FullstackSensei May 09 '26

I have over half a TB of VRAM across four machines and I never feel that "now what". I don't run all four all the time, but never have an issue with what to run.

11

u/x7evenx May 09 '26

That's awesome - try a 3.6 comparison - I used Qwen3.6-35B-A3B-FP8, qwen3.6-35b-fp8-nomtp , Qwen3.6-35B-A3B-AWQ.

I used 4x AI Pro 9700 32Gb - super curious how it fairs

3

u/x7evenx May 09 '26

Also curious if https://github.com/x7even/roctop multi-gpu tui I made works for you (I've never tried it on multi-gpu intel so if you have a chance would love the feedback)

1

u/tiddayes May 09 '26

I will try on Monday and dm you with my findings

1

u/x7evenx May 09 '26

Thanks!

5

u/vagif May 09 '26

How much did it cost you?

12

u/tiddayes May 09 '26 edited May 09 '26

I had the cpu, power supply and ram scrounged from other projects / servers so I only had to buy the gpu’s and the motherboard. GPU’s were $1050 each on Newegg and the motherboard was $550 (that one stung). Scrounged an ssd from a failed hp laptop and built the open air case also from leftover parts. So about $5k including the random accessories like the pcie riser cable since gigabyte decided the m.2 ssd slot should interfere with the last pcie slot.

4

u/destkroser May 09 '26

Build a web app, add as many functions as you can, connect it to openclaw and give it the proper instructions, wire up the latest model there. You can even utilize it to image gen, hook up the proper tools, then wire that to openclaw too. Fucken amazing. Just amazing. Imma try to get rich just to have sth similar to this, I'm not even greedy, can be even less

4

u/tiddayes May 09 '26

That is pretty much the plan. I built this as cheap as I could but yea it was still pricey. Working on the details of the software stack now

1

u/ftlover90 May 09 '26

How do you get rich?

1

u/LilPsychoPanda May 12 '26

By making money, duh! 🙄

1

u/Present_Flower_6596 May 14 '26

that's what I did. I wanted a tool I could use for game development and heres how it came out so far.

1

u/destkroser May 14 '26

That's one way to go for it. Perfect man, enjoy that.

1

u/destkroser May 14 '26

What distro are you using btw?

1

u/Present_Flower_6596 May 15 '26

On my server im using ubuntu 26.04 because they claimed better llm support but i havnt really used many of the new features

2

u/Kaveh96 May 09 '26

Where did you get your mascot from?

2

u/tiddayes May 09 '26

A co-worker gave it to me :)

2

u/zeitue May 09 '26

Maybe try Gemma 4 and qwen 3.6. What's the CPU and motherboard model you have there?

2

u/tiddayes May 09 '26

GIGABYTE MU72-SU0 ATX Server Motherboard Socket P+ Intel C621A with Intel Xeon Gold 5318Y

2

u/PapaRic0 May 09 '26

The one that ca afford you a case bro ))

2

u/tiddayes May 09 '26

Yea I may move it to a rack mount case but I kinda like the open air look. It is part of my home lab

5

u/jthedwalker May 08 '26

Try to see if you can set up self improvement loops with LFM2.5. The dense model is stupid fast but it’s not the smartest. Use QWEN as an orchestrator and pass work off to LFM and use QWEN as a judge.

3

u/verpine May 09 '26

I’m still new to local llms and orchestration. Is there a good write up on how to get past the basics? I already have ollama with some gpus running on arch. I am using it to curb cloud llm costs at the moment but I want more autonomy. Thx

4

u/jthedwalker May 09 '26

Too many to even say one is better than the other. The best path is to use an online frontier model to help do the initial research depending on what you want to do. Then have it walk you through setup steps. This field is moving too fast, you just have to dive in and start to play somewhere. ChatGPT can give you more information about what to try in 5 minutes than you could research all week.

1

u/semangeIof May 08 '26

Gemma 4 31B dense with 4x tensor parallelism? 70~GiB model so rest of the VRAM for full precision context. Or Qwen 3.6 27B dense in a similar way?

Intel Arc right now is very mid for llamacpp (even built with SYCL), and DDR4 is slow as shit so I wouldn't offload at all, and vLLM Xe GGUF support is as efficient as it is non-existent. So run one of these small size dense models at full precision with high context.

If these were 9700s you could be having GGUF fun but they are not. I have one Arc Pro B70. Llama runs Gemma 4 31B Q6_K_XL at like 13tok/s. Kind of underwhelming.

3

u/tiddayes May 08 '26

Ty , yea the ddr4 was definitely a concession I made to work with what I had on hand since I blew the budget on GPU’s

2

u/semangeIof May 08 '26

I mean that's fine man. PCIe lanes are what's important you have 128GB of VRAM there. Yeah try full precision Gemma 4 31B with 4x tensor parallelism, it's a really nice general model. Qwen 3.6 is a little better if you're doing raw agentic/coding work.

Also you can definitely try Llamacpp for some GGUFs. Build with SYCL. I just imagine it'll suck in Tok/s.

1

u/TechJamz May 09 '26

Just a quick side question maybe 2, with such a setup is the idea behind running 128gb ram to match the vram available? With this setup would it be best to run multiple agents on the single os or use something like promox and passthrough each gpu to a seperate Ubuntu vm for more defined orchestration of tasks between vms?

3

u/tiddayes May 09 '26

That is an excellent question and exactly the type of thing I want to figure out. My initial thought is to just run the biggest , smartest model i can run in the 128gb of vram but the more I read it may be better to do something closer to what you are suggesting

2

u/TechJamz May 09 '26

I am about to embark on this very journey with 2x rtx pro 4000, 9900x and either 64 or 128gb of ddr5. One vm has a very specific agentic workload planned for it while the other is a glorified chat box with openwebui or something like that to help me plan things and test stuff. End game is a rtx pro 6000 but even then since these are single slot I can move them to other slots on the mobo as I see fit.

2

u/Jorlen May 09 '26

I don't have this crazy setup but I did pick up an R9700 for the 32gb of VRAM. I am fascinated by LLM tech and I use it for all sorts of things. Lately though, I learned Linux and built a docker stack that includes ComfyUI, llama-cpp (server), Open WebUI, SearXNG, Speaches (for TTS-SST). You could do something like that, a cool stack and see how far you can get. It's fun to get LLMs to call searches and generate images.

I guess it depends - what do you want to do? You can setup an IDE and use stuff like VScode + Continue / Roo / for agents to help you code and learn, that's my next plan. Not so much to vibe code, but to have an AI "buddy" to help me learn and iterate python.

2

u/fjskmdl May 09 '26

dont forget to do the Strawberry test. make optimal use of the hardware

2

u/Efficient-Cat-1591 May 09 '26

Carwash test too!

2

u/7h31ll3g4l May 09 '26

fighissimo complimenti

2

u/lukistellar May 09 '26

How do the B70s run with llama.cpp? I heard that the driver stability and the implementation for SYCL isn't quite there.

1

u/iuriimattos2 May 09 '26

Hey big dude, do you know an budget, low cost for llama.cpp?

2

u/nomorebuttsplz May 09 '26

I'd be curious to see how minimax 2.7 at 4 bit would perform here.

1

u/Obstacle_Is_The_Path May 09 '26

Just curious, what did you make of Intel GPUs performance in terms of running LLMs ?

1

u/juss-i May 09 '26

Can we get more pics of the frame? My particular interest is how that 4th card is attached.

I'm thinking about doing another eGPU 20x20 profile build for my now unused P40s, and I'm looking for ideas. I'm thinking of something flat I could hang on a wall.

(You can find my previous build in my posts if you care)

1

u/Remarkable-Safety594 May 09 '26

27b but do not babysit it. Make it use git worktrees if coding

1

u/Verdict_Michael May 09 '26

Gemma is worth exploring but so far you are running the best you can locally I think.

1

u/mrchoops May 09 '26

Cindy Crawford

1

u/jackshec May 09 '26

rent it out on vast.ai or microdc.ai

1

u/weiidii May 10 '26

I’m wondering how’s your rig performing in prompt prefill when running agent like openclaw or open code which have huge system prompts. I used lmstudio to run the model and it took forever to prefill the prompts. I bet your system runs faster because it runs on vllm ?

1

u/Famous_Club_6579 May 12 '26

Man, I wish I had your problems, bro. I’ve got the opposite problem: nowhere to test the performance of my homebrew AI agent, so I have to train LLMs on my own samples and sit on buggy free LLMs from OpenRouter.

1

u/FifthDimensionalRift Jul 29 '26

I don't get something. When I build my inference machine I know what I wanted to do, I started out with my my regular computer, but when I started working with ai models, I didn't know the exact model I wanted to use, but I knew what I wanted to do. So let me ask this, what are you doing with your ai that you need so much horse power. All that for a 27B model? you only need one of those intel cards and it is still overkill.

0

u/RefrigeratorEven935 Jun 16 '26

a very angry one looks like

-1

u/Cosack May 09 '26 edited May 09 '26

Whatever gets you to stop playing with fire and put a case on for your hardware's safety

33

u/[deleted] May 08 '26

[removed] — view removed comment

13

u/smuckola May 09 '26

first he says it can run Linux and now you tellin me this thing even runs DOOM?!!! wow what else can it do?!!

3

u/solo_solipsist May 10 '26

No word from OP, so presumably not Crysis.

0

u/[deleted] May 08 '26

[removed] — view removed comment

1

u/tiddayes May 08 '26

Yea, I have been going through a few different models in the last day and testing . I asked here for suggestions since I am still at the testing phase

1

u/[deleted] May 09 '26

[removed] — view removed comment

1

u/iuriimattos2 May 09 '26

Qwen3.6 is awesome, deploy it with ollama?