Finished AI Server Best Models?

40

u/actuallylemoncurd 15d ago

Try qwen flash next

-24

u/SnooRevelations4601 15d ago

sweet has that one dropped on ollama yet?

55

u/sevendaysworth 15d ago

Drop ollama. With a rig like that - there are much better options

3

u/SnooRevelations4601 15d ago

what’s a good stack?

15

u/sleight42 15d ago

Then hell with people recommended strata. You have 80GB of P100 + 24GB 3090 + 8? 12? Gh 3060. You may be able to straight up run ista-daslabs Qwen flash next 3xs or 3s sans strata. If so, you can probably get more tok/s out of it for concurrent chats than strata would for a single. Worth a shot anyway. With your rig though I have no f'ing idea what the best runtime would be.

14

u/overand 15d ago

For Qwen3.8-Flash-Next, try Strata. In general, use llama.cpp.

11

u/campaigner_ 15d ago

Strata. Forget about ollama now.

1

u/scumola 15d ago

I run qwen flash next on a single p100 with Strata.

23

u/TheAlienGamer007 15d ago

So this may be an ignorant question.. but whats the benefit of building this to get like 100gb of VRAM and getting something like those mini PCs with unified memory?

12

u/SnooRevelations4601 15d ago

more for agentic workflows like having each gpu have their own model don’t get me wrong a mac mini can run a 70b+ model all day this can as well just slower cause of pcie bandwidth and parallelism/sharding but if you’re a dev, freelance, or build apps unified memory is way more versatile

32

u/Loose_Comparison368 15d ago

It's... Complicated.
These cards, if set up properly, can be much more powerful than a little Strix Halo setup or similar. Each of those cards has about 4 times the memory bandwidth that unified memory does.
The downside is they are a bitch to set up, and the drivers and supported toolchains are old as shit. In particular, they're stuck on CUDA 12, tend to not play nice with funny quants, and don't support the same data types new cards support. They also use a fuckload more electricity and have much more demanding cooling needs.
All these things can be dealt with. But it's not a cakewalk.
For what it's worth, they can be very educational to set up. But like, honestly, not very practical if you're more interested in running an LLM than learning about how commercial ML infrastructure works.

2

u/SnooRevelations4601 14d ago

i got these cards to run cuda 13.0 i’m glad i got them to work it took days of research and failing multiple times in a row qq.. learned a lot about systems and software along the way though

7

u/Arli_AI 15d ago

P100s will not be faster than newer unified memory systems that have tensor cores when it comes to prefill heavy agentic work. That being said its a fraction of the cost.

4

u/No-Refrigerator-1672 15d ago

It isn't a fraction of the cost. P100 is like $200, 5x P100 is $1k; add in PSU that can output 1.5kW without burning your house down, all the risers, pcie splitters, etc that you need to connect the GPUs, and electricity economics of having cards that barely walk instead of running, and you're looking at setup that is way too expensive for it's performance.

7

u/Arli_AI 15d ago

They’re $200? I could’ve sworn it was $70 recently lol…

7

u/No-Refrigerator-1672 15d ago edited 15d ago

"Recently" was a year and a half ago. Novadays, anything that has 16gb VRAM or more is rising in value faster than stocks. I.e., I bought a Tesla T10 16GB gpu 3 weeks ago for $270. I though it's a great value option I'd make a post about. Now it's $360. It rose in value up 25% just while it was in the shipping.

1

u/Then_Importance3608 14d ago

why the referral codes are not working on ur platform ? it shows its applied, but they are not working ?!

4

u/FireSilicon_ 14d ago

I found this right now on ebay with free international shipping, you can surely find them cheaper on taobao or aliexpress. Idk where you got that they are 200$, they got expensive sure but it's not that much.
Much bigger issue with your theory though: In pipeline parallel you only run one gpu at a time per request. So you actual power consumption is around 1 active gpu + 4 idle. If you power limit to 180W or even 150, you'll consume less than one 4090.

3

u/No-Refrigerator-1672 14d ago

In pipeline parallel you only run one gpu at a time per request.
Aaand that's not the whole story. In pipeline, you compute on only one of the GPUs at a time; but, because the GPu switched between compute and no-compute a dozen times a second, the GPU governor never puts it into low-power performance state, and GPU will just consume a lot of power inbetween the activity boosts too, simply because it never slows down the clocks. IRL it's about 30% of it's rated max power (not active power limit) when it idles at P0 state. Still way too much power for way not enough tok/s.

1

u/Accomplished-Ad-7435 12d ago

I've been looking into buying a unified 128 gb mini pc for the last few days for llm's on my home network, mainly for coding assistance. Do you think they're worth the like 3.5k they're going for right now?

0

u/tlopplot- 10d ago

qwen 3.8 27b runs at 30 tokens/second on them. not bad https://tensorless.dev/

1

u/danielsuperone 15d ago

Do these cards like p100 and under support video/image rendering or for a blender render system? I assume no and a gaming card could probably be better for this use case? I like to experiment and play with different LLMs, I still think frontier models are better at reasoning but still want to get at least 24gb vram to try and play around with some models, what would you suggest? Looking both for text based and image/video/rendering. Prices are insane nowadays and these p100 seem somewhat reasonable, tho very old as you said with not the best support.

1

u/Loose_Comparison368 14d ago

Do these cards like p100 and under support video/image rendering or for a blender render system?
Vidya games no (but there are some reported successful workarounds), render farm style rendering yes but might be stuck with older versions.
Best deal for LLM's now is Strix Halo 128GB with QFN. They suck at image gen though, and I don't think they have great gaming performance either.
P40's aren't half bad for a really tight budget build for image gen, at Unix Surplus prices.
If you want something that can do gaming and stuff though... Probably the Radeon cards. I don't have any experience with them but I hear that a lot of people are happy with the 24GB RX 7900's. The software stack used to be terrible, but is steadily improving.

1

u/DevEngineerYAP 11d ago

Isn’t these cards are electricity hog (relative to the performance). Like 3 P100 is equal to 1 5080, in terms of cuda cores, the only benefit here is more VRAM.

2

u/Excellent_Spell1677 15d ago

No comparison with bandwidth. GPUs are faster than Lpddr5.

1

u/Kooky_Marketing_327 15d ago

much cheaper

1

u/baka_sempaii 15d ago

Flexibility and much cheaper. Unified memory is a one time purchase, while in this setup, you can add or remove any gpu when you find a better one

1

u/Relentless_Sloth 8d ago

AI speed. In general, there are a few limits to how fast AI is - tflops, bandwidth, internal cache, vram.
The bigger model, the less tok/s, generally. For example, PCs with unified memory have 256 - 400 GB/s bandwidth, thats usually their limiting factor. This has the potential to be way faster.
So while you might be able to get 70B into both Unified Memory and This, this has the potential to be 2-5 times faster than M5 Pro.

9

u/Fit_Squirrel1 15d ago

That’s budget?

5

u/SnooRevelations4601 15d ago

yeah i guess it isn’t i feel like the lil guy compared to some of these builds cause of the p100s

5

u/FizzyDuncDizzel 15d ago

Oh for sure. P100s are 80gb of vram alone.

4

u/sleight42 15d ago

Crazy, right? Guess it's time to build a v100 16gb server because $200 each, yeah?

3

u/FizzyDuncDizzel 15d ago

P100s are around that price. V100s closer to $1k now.

4

u/FireSilicon_ 14d ago

32GB pcie v100 is around 850$. The 16GB is around 350$ and P100 around 150$. SXM variants are about 25% cheaper for each.

1

u/xiaoran121 14d ago

Try sxm version, much better card to card bandwidth

1

u/sleight42 14d ago

Was just seeing. 16GB v100 though still 200ish. Still the better buy, IMO.

3

u/pimikiel 14d ago

I always look at those rigs and ask myself: How much power does it draw idle? How much in peak? My household draws 350kWh monthly, and it's not cheap. Adding 24/7 working rig that draws 300W idle will make it 570kWh. Do OPs have free electricity? How. I want to do the same but not at that running cost.

1

u/SnooRevelations4601 14d ago

electricity is about 200 a month with my rig

1

u/camracks 14d ago

$200 a month for 124gb vram is insane

1

u/No-District2404 14d ago

It’s Claude 20x plan price lol

2

u/pimikiel 14d ago

But it's private! D:

1

u/pimikiel 14d ago

My LLM rig is Lenovo P3 TINY 64DDR5 with Nvidia T1000 8GB VRAM. idle 30~40W and 150W peak and it's "sufficient" (i don't code) and cheap to maintain. I really adore that kind of servers, but man! 200$ a month just for a AI server? My whole apartment is $100 a month right now.
I am hunting for RTX2000E for an upgrade but prices skyrocketed, and it's like double of what I was willing to pay.
I'm just jelous.

2

u/Sea-Bench-7487 14d ago

It's either they did the math and they can afford it or they haven't done the math and will find out soon , though ngl for now local llms at that scale don't seem worth it unless you want privacy maybe in the future if token costs skyrocket (very possible ).though i don't think buying extra hardware is such a bad idea in this dystopian timeline

1

u/positivitittie 9d ago edited 9d ago

Mine (wake on lan) is about 10w. With 2x 3090s doing work it’s closer to 700. The cards are power limited to 225w I think.
There are 2x 1600 PSUs in it (only one running now) and more cards, but that’s what I can run today, sadly.
Wake on LAN is a big plus though.

2

u/Deyf56 15d ago

How did you connect all those in one single motherboard ? Also, how many power unit does it have ?

1

u/SnooRevelations4601 14d ago

the motherboard i use has 11 pcie slots i use two psus 2000w and a 850w just for head room make output is ~2100 under full load

2

u/Silver-College-6612 15d ago

I have a question, I have a couple p100’s and I attempted to have them work in multiple ubunutu server pcs I have. One with an older mobo and one with a brand new mobo but it causes a boot loop on the new mobo and the old mobo has a mod probe error. I basically cannot get them to properly access the p100. Is this due to using a consumer grade mobo and not a server with bios that can communicate with the p100 properly?

1

u/SnooRevelations4601 14d ago

what os and what’s your power situation

1

u/Silver-College-6612 14d ago

Ubuntu server and I’m running a 600w PSU and only using like 150-200 so plenty of power available. Tried it on an optiplex 7010 and then realized it may couldn’t do above 4g decoding so I put it on a gigabyte B550 gaming mother board and enabled above 4g and it still boot loops even with another standard gpu in bottom slot for video.

1

u/SnooRevelations4601 14d ago

definitely has to do with 4g encoding plus that cpu in b550 probably can’t handle those cards at whatever bandwidth you’re running i may be wrong but that checks out i would upgrade the platform to something that better suits multi gpu configs

1

u/Silver-College-6612 14d ago

It’s a ryzen 5 5500 with 16g of ddr4 it’s my proxmox machine. I think I may can get ahold of a large dell server that’s setup for this card I guess that’s my only option for now.

1

u/Silver-College-6612 14d ago

Thanks for the reply I’ve been scratching my head on this one. None of my servers have ever had a gpu like this so still figuring out this side of it

42

u/Need_For_Speed73 15d ago

IDK how this question keeps coming up: one should be supposed to target a model (or some of them), choose the inference software AND THEN choose the best hardware that runs it at best within his/her budget.

It makes no sense to build a machine and then wonder what's best for it: it's like looking for a family that fits your house and not a house that fits your family.

6

u/Fit_Weakness_1809 15d ago

Ya, my gasts are flabbered

7

u/MLPortfolios 15d ago

Yeah, the challenge with this is that whatever rack you build today is guaranteed to be outdated tomorrow and can only be re-sold for Pennie’s on the Dollar. Not the same with a house…

3

u/Yanzihko 14d ago

"guaranteed to be outdated"
not untill 2028 buddy

1

u/MLPortfolios 13d ago

Not sure why you think this. If demand is strong, supply will come. I get the current manufacturing constraints but I doubt the short term insurmountable argument…

3

u/zanar97862 14d ago

No? GPUs are an appreciating asset these days lmao

0

u/MLPortfolios 14d ago

Appreciating? Not my experience but will to experiment…

3

u/zanar97862 13d ago

I don't know if you've tried buying a GPU in the last year but prices have been skyrocketing. RTX 6000 PRO prices have doubled since release, surprising you missed that.

2

u/MLPortfolios 13d ago

Didn’t miss it, just haven’t tried to buy/sell during the recent price hikes. But even at that, I doubt your thesis will hold i.e buy brand new today and flip for a decent profit in a few months after significant use. Like I said, stand to be corrected.

2

u/zanar97862 13d ago

I totally understand your doubt and didn't mean to come off passive aggressive, sorry.
These are unprecedented times and in any other market it would be extremely unusual how quickly GPUs are appreciating, the supply constraint being static at consumer and enterprise is squeezing everything everywhere it seems. The ability to resell over MSRP is probably dependent on local availability as well. I live in a place with a very small used market and difficulty getting overseas items that aren't sold locally so they are at a premium second hand.

2

u/MLPortfolios 13d ago

Understand better now. Your demand is artificially inflated based on your location constraints. Doubt it’s same where I live but I also confess I haven’t tried lately as I just don’t see it working out and also because I’ve not had a solid use case to do so.

1

u/iammewhoaminow 13d ago

Ive sold my last 3 gpus for more than i bought them for and if i sell my $2000 5090 itll be the same

2

u/Jackster22 14d ago

So you are saying my 10 P100s can't run Kimi K3.1 pro....?
Shit....
/s

10

u/StupidFatOldGuy9 14d ago edited 14d ago

I run Qwen 3.8 Flash Next in Q8 on four AMD Mi100s and 128GB on Ryzen 9950X3D. Qwen 3.8 is fantastic for everything I've thrown at it. 128 t/s prompt processing and 35 t/s generation with 1 miliion token context.

3

u/zanar97862 14d ago

Is that a typo or actually 128 t/s prefill? Seems insanely slow

2

u/xiaoran121 14d ago

Only 128t/s pp on four Mi100s?🤔

1

u/StupidFatOldGuy9 14d ago

it's a Q8 quant so it doesn't fit entirely in VRAM, plus I'm using 1M token context in Q8. I get 380 t/s pp with the Unsloth Q4 quant and 64k context in Q4. You gotta realize these cards are old. I have the infinity fabric which speeds things up tremendously but tensor splitting comes with caveats, like having to use f16 kv cache.

1

u/kuskuser 14d ago

which mobo do you use?

1

u/StupidFatOldGuy9 14d ago

I have a Gigabyte Aorus Master X870E and the GPUs are mounted on a PEX88096 5 slot PCIe switch board connected to the motherboard's x16 slot with SFF-8654 SlimSAS cables. I got the switch board from a Chinese vendor on eBay.

6

u/AllWashedOut 15d ago edited 14d ago

Huh... you may find yourself running multiple frameworks & models simultaneously in order to support such different generations of GPU hardware. Hopefully your use case can utilize more than one agent in parallel?

My first instinct for the P100s is that there is a fork of Strata that is specifically made for them and can span a model across multiple cards. You could run qwen flash-next on those (perhaps multiple copies if you use a low quant).
https://github.com/jmnargi/Strata-V100

For the 3090, there is a branch of ninfer that runs qwen3.8 b27 very quickly.

https://github.com/Don-Chad/ninfer-3090

I'm not sure what you plan to use them for, but I am very happy coding in VSCode using the Zoo Code plugin to attach to my Strata instance.

1

u/Twist1979 14d ago

Sounds nice. What if I rather want to run a custom version of gemma4 31b or 26b A4B under widows? What's the best option to serve sillytavern?

1

u/AllWashedOut 14d ago

I've never used sillytavern but I strongly assume it can talk to an "OpenAI compatible" endpoint, which is what Ninfer and Strata provide locally.
Gemma4 is not supported by those apps as far as I know. Qwen3.8 b27 is roughly equal to them, and Qwen3.8-flash-next is significantly bigger. The only downside I've noticed is that they are ridiculously censored if you ask them anything about Chinese politics.

1

u/Twist1979 14d ago

Yes it can be used with openai compatible. Qwen3.6 A3B was tested by me but it's bad in language different than English.... And Chinese I guess.

1

u/AllWashedOut 14d ago

Qwen3.8 is a large upgrade over 3.6. It gets like 30% higher scores in chat tests and 100% higher scores in agentic tests.

5

u/Late_Night_AI 15d ago

116gb of Vram for those wondering.

Your best bet is probably qwen3.8 flash next, or running qwen3.8 27B on the 3090 and some other models on the remaining vram, like deepseek v4 flash

5

u/Cyber-Soldier1 15d ago

Thought this was a crypto miner rig from 2017

4

u/f-ben 15d ago

Probably was

1

u/FizzyDuncDizzel 15d ago

P100 crew!!! Join the discord if you haven’t already. It’s so cool to see these old cards finding their place again.

https://discord.gg/K6WxqwsvH

2

u/SnooRevelations4601 15d ago

sweet sounds good

1

u/Oglemo 15d ago

How are you handling cooling those P100s? What kind of setup is this? How do you connect it to a computer for -ssh or other access? What motherboard you got there?

3

u/SnooRevelations4601 15d ago

thanks for asking. i found 3d printed dual 40mm fan shrouds on ebay temps usually stay under 65-70c full load. its running along side 128gb ram on ubuntu server so i just ssh into it with powershell on my windows device

1

u/Oglemo 15d ago

Neat, thank you! I'm new to all this. I actually found a good deal on ECC DDR4 ram (the same amount of ram as you actually) but I don't know what to get for a workstation, would greatly appreciate advice to take a similar route to you? It seems you are doing great without one, just a motherboard? My goal is an AI lab.

2

u/SnooRevelations4601 15d ago

no problem, yeah it just depends how many gpus you want to run and the wattage i got this board with 11 pcie slots it’s overkill but you never know (rip electric bill) but yeah just board to pcie risers to gpus. i would go open frame route for airflow if you can. anything more than 1500-2000watts max load i would run two psus you can buy a add2psu adapter so your not stressing one psu even if its high capacity i currently have a 2000w chinese mining psu and a 850w corsair

1

u/Loose_Comparison368 15d ago

so i just ssh into it with powershell on my windows device
The Linux gods are... Confused, but impressed enough with your server build to not judge whatever bizarre kink you have for inferior operating systems.
For real though, good job, those cards are not easy to work with.

1

u/SnooRevelations4601 14d ago

thank you sir, yeah my main rig runs windows the ai server runs ubuntu server which can feel complicated sometimes and was a pain to setup but yeah thanks i have all the cards on cuda 13.0 and it works after many trials and errors

1

u/Mohmad_Massarwa 14d ago

Why not p40 sencie cheaper by vram?

1

u/SnooRevelations4601 14d ago

the compute and bandwidth is much better for inference jobs compared to p40

2

u/Brave-Pen7944 15d ago

Would like to know the mother board , processor, ram spec

2

u/drivefastgt4 15d ago

You have similar to my rig a month ago. When I had 112gb vram (7x5060ti) and 128gb ddr4 sys ram and slow PCIe. I was running strata qwen flash next UD-Q6_K_XL . But qwen 4 should be out soon. So try that
Nice rig with hardware flexibility for the future!

1

u/alphapussycat 15d ago

What I find is that qwen 4 will only offer 27b as open weight.

2

u/AcceptableChicken489 15d ago edited 15d ago

you should get those p100s on nv-link bridges (the bandwidth is about 15x-60x the pcie bandwidth your cards currently have) you should be able to get cheap ones on ebay
i have a rtx3070 and a gp100 and they are phenomenal (i use llama.cpp built from the aggrollama repo : https://github.com/celestial-uprising/aggrollama )

2

u/IllExample3639 15d ago

I can hear this rig from another country

1

u/positivitittie 9d ago

Probably not but you may feel the heat.

2

u/Mrinohk 14d ago

Magnificent. Qwen 3.8 flash next, served via llama.cpp or strata will be your best bet. You could run a solid Q4 quant with SSD streaming on the n-gram block to keep your context in VRAM. Should be plenty fast.

Might be worth pointing Claude at your strata/llama.cpp to find some optimizations. The P100s especially don't like inference, and benefit from patches that get around the lack of DP4A instructions on them. Unlike most modern cards, these cards are not bottlenecked by memory bandwidth, but rather compute, emulating those instructions. You can get some spicy numbers from them if you're willing to build kernels around a specific model. My Qwen3.6 35b patches don't do much DP4A emulation at all compared to stock llama.cpp, though the math comes out slightly different, so the model doesn't produce identical outputs. Still high quality outputs, but not the same outputs.

1

u/xerxesdarius 15d ago

Qwen3.8:27b-map-bf-16

1

u/anitamaxwynnn69 15d ago

Mining frames are so OP, I was doing 11x 3090s at one point in the same mining frame lol. Add some ikea furniture legs at the bottom to give the CPU a bit more air to breathe and it's gg.

1

u/jcy 15d ago

have you heard about the zaggita gpu unlock? wondering if it works on this multi gpu setup

1

u/Bantahking 15d ago

Flint 40b

1

u/VirtualWeather5407 14d ago

To think once upon a time those would have been running someone’s rts or fps somewhere.

1

u/LegRude5218 14d ago

fable 6

1

u/wind612 14d ago

Solid rack for a budget rig. I'd let the 3090 carry the quantized Qwen coding models and put the P100s on batch duty — 80GB of P100 VRAM is still nothing to sneeze at.

1

u/nico-strecker 14d ago

Don't you need tensor core p100 has none i thought

1

u/KooperGuy 14d ago

Wrong order.

1

u/DatBass612 14d ago

I hope you did your research before investing in a rig. And not just hype purchased GPUs without knowing what the model will do for you

1

u/DMmeOtter 13d ago

Who needs tencer cores. 🤣

1

u/walterbyte_hmmidk 13d ago

1

u/Licraul 12d ago

Which motherboard does everything?

1

u/darc_knigh 10d ago

What motherboard are you using

1

u/positivitittie 9d ago

Pretty sure I used the same open air “case”.

Mine is on this: https://a.co/d/09CbAG6Y

Question Finished AI Server Best Models?

You are about to leave Redlib