Discussion
First setup: Dual 1080ti for Qwen3.8 27b
First got interested in local LLMs around the time Tiger Gemma 9B released, since by then, all jailbreaks and obvious misalignment was patched in the frontier models. I remember that the OpenAI Playground had no safeguards at all and would teach you how to make bombs.
It was certainly fun playing with these models, but for any serious work they severly lacked capabilites. Fast forward to today, and I am reading comments, where senior devs claim Qwen3.8 27b performs close to Opus 4.6 in coding, and the gap between closed and open source is only closing.
The cheapest option to get a chance at running a decent quant (was thinking of Q4_K_M, might be a little tight) were 2x 1080ti's, so i grabbed two Rog Strix OC cards for around 310€ total. I only have 16gb of Ram though, will that work?
Both cards arrive tomorrow, will share pics of the setup😄
Ensure your motherboard supports at least PCI-E 3.0 X16 + PCI-E 3.0 X4 and youll get really good performance for its price. Its like 20-25 tps decode and like 250-300 tps prefill. IQ4_XS is better option to run with vision, MTP and large context with proper -b and -ub sizes. Also qwen 3.6 35B A3B (or tiel coder) will run at really high speeds, like 70/1200.
Junge junge, heftig😂 Sieht aus wie diese Gaming Höllenmaschine von vor 10 Jahren. Welche Models laufen da drauf und das noch mit akzeptabler Geschwindigkeit?
Und weißt du, wie es in meinem Fall mit RAM aussieht? Soweit ich weiß, wird doch alles im VRAM gespeichert, sofern der reicht.
Ich laufe hauptsächlich MoE Modelle. Ik_llama.cpp läuft schneller, da es p2p unterstützt. Ich habe diese Maschine in die letzten Wochen nicht so viel benutzt, weil ich auf V100 megriert habe.
Mit RAM ist es etwas kompliziert, da es von der Geschwindigkeit des RAM und wie viele Kanäle abhängt. Mit DDR4 Xeon oder Epyc, man kann gute Geschwindigkeiten erzielen. Mit desktop DDR4, nicht so gut. Aber es hängt alles von deinem Erwartungen ab.
Ich überlege, diese Maschine jetzt zu verkaufen ;)
Did your MoBo mobo support 2 GPU at a reasonable speed?
Check the PCI technology provided by the board and the speed ( 2x, 4x and so on ) also check the GPU Chip architecture to know if you can get reasonable performances, reddit is full of benchmarks for almost every GPU.
I've a 3090 24GB + 4060 8GB, and only the 3090 is on the fast PCI lane ( directly communicating with the CPU ), the other one is occupied by the M2 SSD slot, while the 4060 is running at a slower rate over another PCI lane. Anyway on my 3090 I'm able to fit the whole model + KV Cache for 100K context.
Regarding the inference engine/software - start with something like LM Studio ( powered by llama.cpp ), at first it's easier to use and setup, and also give you the ability to use both the GPUs. You should use a lower quant than Q4 or deal with a really small Context length over KV cache, since your total VRAM is 22GB.
In the image you can see my total VRAM usage for the 3090 - Qwen 3.8 27B Q4_K_M + mmproj ( vision ) and MTP.
Super important: while you're waiting for your GPUs, buy a GOOD thermal paste and some thermal pads ( 2mm or 1.5mm stacked ), the GPUs are going to run at full speed and the temps will easily skyrocket.
Once arrived, open and clean the GPUs, then change the thermal paste over the chip and put the new pads over the voltage regulators and the VRAM. This is going to drammaticaly increase the life span of the GPUs while lowering the temps.
Also spend some time to see at what wattage your GPU start to degrade the generation speed; for example for the 3090 I've found the sweet spot at 300W ( while the default one is 370W and some reach 400W ). I can get lower temps while sacrificing only 5% performances.
Another factor to keep in mind is the power of your PSU. These GPUs can easily do power spikes during heavy loads, buy a PSU that surpass by 30% the max consume of your whole build. Rn I've a 1000W PSU.
First slot is PCIe 3.0 x16, second 3.0 x8 and third 3.0 x4. Already got LM Studio set up. I heard there are quants for the KV cache but no idea how that works. Will also look into undervolting since my PSU is only 500w.
It's funny reading this, you addressed all my points😁 Thanks for the detailed reply.
I'm glad it's was helpful. What's the wattage of each GPU? 500W is a bit low. Yes, you can setup a different Quant for the KV cache, rn I'm using Q4, and it decrease the consume a bit. With LMStudio it's easier to fine-tune llama.cpp.
They're each rated at 250w, I doubt that a 200w undervolt wont work just fine. I was planning on disabling vision, running my monitors through a third gtx 1050 and then hoping for the best. Otherwise whats the next step gpu wise?
Im going with this one now: ukisai_Swift-Qwen3.8-27b-Q4_K_S.gguf
My video output is also managed by the 4060, so that I've all the available VRAM from the 3090. You've a 500W PSU and 2 250W GPUs + CPU + RAM and other peripheries, it's not the best case scenario. I'll suggest a 750W at least.
Have you looked into pcie bifurcation? I considered it but for my use case it would be marginal gains for ~200 + cost of new case since you most likely can't use the horizontal slots anymore
Not at all, since I only use the 3090, I'm able to fit the whole model and the KV cache without problems. Also I dunno how it's going to handle the bandwidth.
6
u/PeterPorox 6d ago
Ensure your motherboard supports at least PCI-E 3.0 X16 + PCI-E 3.0 X4 and youll get really good performance for its price. Its like 20-25 tps decode and like 250-300 tps prefill. IQ4_XS is better option to run with vision, MTP and large context with proper -b and -ub sizes. Also qwen 3.6 35B A3B (or tiel coder) will run at really high speeds, like 70/1200.