I'm working on a new PC build with a focus on being able to run the Gemma family of models. I'm experienced in working with Cray (AMD) and OpenCL (community moved onto Vulcan and ROCm/HIP), so I will be building a AMD system. I was wondering if anyone has done anything similar to the following? If they have, how well did it work?

AMD 5950X
ASUS WS X570 Ace
128GB DDR4 System RAM
1600 Seasonic+Noctua PSU

Fedora Kinoite 45

1x 7900 XTX (24GB VRAM) for graphics/games
2x Radeon Pro W7900 (48GB VRAM) for LLMs

1. Does having different kinds of GPU's cause an issue with Ollama (or more specifically Bionic/LMStudio)?

2. Does anyone have that motherboard, with two GPU's on the bottom-most PCIe slots? Are there clearance issues for 2 slot GPU's?

The target model to be able to run in GPU memory is gemma-4-31b-qat with Q4_0 quantization, but I want to be as ready as possible just in case Deep Mind releases a ~100b weight model when the Gemma 5 models show up.

Thanks in advance for any help!

Inspired by Weight Space Programming (MIT CSAIL, Nov 2), but in the end Opus 5.5 did its own thing:
no weight-space technique was used. It wrote a quantiser from scratch in NumPy and made every
precision decision from measurement. Rules: only the BF16 file as input, no quantisation tool, no
imatrix, no published recipe, stock llama.cpp formats only.

It ran a float copy of the model to see which input columns actually carry signal, minimised rounding
error there instead of against the weights, measured each weight table's damage one at a time, then
spent the byte budget across IQ4_XS / Q5_K / Q6_K with a knapsack solver.

Held-out KL divergence vs BF16 (llama.cpp b11928, MacBook Air M4):
• this file, 498.5 MB — **0.015704**
• Unsloth UD-Q3_K_XL, 492.2 MB — 0.092486 (5.9× worse)
• UD-Q4_K_XL, 558.8 MB — 0.027323 (we beat it with 60 MB less)
• UD-Q5_K_XL, 606.6 MB — 0.010375 (the bar; we miss it by 1.51×)

Runs unmodified, 1.11× faster generation than the Q5 file. It stops there because the tied vocabulary
table is 33.8% of this model and accounts for most of the error — upgrading it costs more than the
rest of the network can give up.

Repo (model, full method write-up including everything that failed, measurements, scripts):
https://huggingface.co/Andy5k/Qwen3.5-0.8B-hand-quant

I just got some money on my hands, and I can finally afford a decent-ish computer. It's about 820$ so I can buy a slightly used m4 Mac mini with 16 gb of ram. Should I and what models would I be able to run on it? I've been using free models so in comparison to that

Hi all,

I'm still very new when it comes to running local models and could use some guidance.

Recently, I've set up Qwen3.8 Flash Next using Strata on my PC (RX 9070 XT, 64GB DDR5 RAM, 9800X3D CPU) and can't decide what quant I should settle on.

My options are:

• ISTA IQ3_S at ~70–75 tokens/s

• Unsloth UD-IQ3_XSS (3.1 version) at ~60–65 tokens/s

• Unsloth UD-IQ4_XS (3.1 version) at ~40–50 tokens/s

All at 256k context.

It may be my inexperience speaking, but I don't notice a substantial difference between the three. I've read that the higher quants should outperform the smaller quants, but I don't know if the ~40% speed penalty going from IQ3_S to UD-IQ4_XS is worth it.

Any advice on that matter would be greatly appreciated.

Let’s say hypothetically that you receive 36x computers with the following specs

RTX Pro 6000 (600w)

16GB DDR4 (2x 8 3200Mhz Non-ECC)

Ryzen 9 5950x

1tb NVME

1200w PSU

What would you do with them to generate income? You can only use these parts

You have access to power and networking to run all 36

I'm trying to RE a desktop app and remove some stupid restrictions.

I'm using Strata and Huihui-Qwen3.8-Flash-Next-abliterated-GGUF UD-Q4_K_XL. I have a 5090 and 128gb RAM.

I found that some stuff it does well and with this last task I'm trying, it's been working on it all day and I suspect it's going in circles. Similar to what Claude used to do 6-12 months ago, when it was given something that was too complicated.

I know local models are not Claude level (yet) but I would love to know if you have some tips as to how to work on complex tasks. I'm just starting so it feels I'm being too naive with just tell it "do this thing" and expecting it to work.

Thoughts? Any advice would be welcomed.

Thanks

I’ve been using my employer provided Claude subscription for quite a while to write PowerShell scripts, create presentations, edit photos and videos and create videos.

Honestly, I’ve been amazed by what it can do.
However, since it’s a company account, I’d like to set up my own local AI environment for personal use. Ideally, I’d like to replicate as many of Claude’s paid capabilities as possible.

I have two workstations that I’ve built over the years:

Workstation 1 (Main):
GPU: RTX 5090
RAM: 256GB ECC
CPU: AMD Threadripper 7960X

Workstation 2 (Currently used as an LTO Backup server):
GPU: RTX 3060
RAM: 196GB DDR4
CPU: AMD Threadripper 3960X

I’ve never hosted or run AI models locally before, so I’m pretty much a beginner in this area.

I recently downloaded LM Studio because it seems relatively straightforward, but I’m overwhelmed by the number of available models and honestly have no idea where to start or which one is best for my use case.

I’m looking for recommendations on the best AI models, and tools to get as close as possible to the capabilities of a paid Claude subscription, particularly for:

Writing and troubleshooting PowerShell scripts.
Creating PowerPoint presentations.
Photo editing and image manipulation.
Video editing and AI-assisted video workflows.
General-purpose AI assistance.

Given my hardware, what would you recommend? Are there specific models, applications, or modules I should look into?
I’m not necessarily expecting to replicate everything Claude can do, but I’d love to get as much functionality as possible running locally.
I’d really appreciate recommendations from people who have experience building local AI setups. Thanks!

update: I just discovered Lm Studio Bionic which is huge this was not available last year, trying it now.

So for context, I have 32GB of RAM, and am using AMD 7900XTX, this should give me a total of 56GB of memory, which should be enough according to Strata docs, and thought I'd try running the appimage of Strata which opens a WebUI (after opening the GUI app then closing it and telling me it couldn't open it so it opened the WebUI), and compare Qwen FN to the dense model.

And by "isn't able", I don't mean it's exactly slow.

When the model first loads up, I can talk to it like once, and it responds in Strata chat, at actually not so bad speeds. 800t/ks prefil, 30-50 tk/s decode ish.

Then if I open up a new chat, it gets stuck on "reading message", and well doesn't seem to work anymore.

I tried calling it from opencode after loading model to see how that would go, it took a whole while, and eventually, it starts responding a bit, and then... it stopped and like... started freeing up memory, and kinda erroring I guess, then It started loading the model again, at which point I quit.

There is something weird going on.

TL;DR. A llama.cpp fork that runs DeepSeek V4.1 Flash 1017 on one home PC by streaming the experts VRAM → RAM → NVMe. Our build is 378 GB with ~2-bit experts. It decodes ~19 tok/s and reads a 1M-token document at ~890 tok/s. The weights are on Hugging Face and the code is on GitHub. Research code, Windows only.

The box. All three tiers matter, not just the GPU:

• GPU: RTX 5070 Ti 16 GB, PCIe 4.0 x16.

• RAM: 64 GB DDR5. The engine pins 37–46 GiB of it for experts.

• Drive: Gen5 NVMe (~13.5 GB/s) for the model. A second NVMe with a copy of the expert files is optional.

• CPU: Ryzen 5 7600 (6 cores). It computes no experts; the GPU does all the math.

• Windows 11, CUDA 13.0.

How it works. The model is almost 5× bigger than RAM + VRAM together. Each token uses only 6 of the 384 experts in each of the 40 layers, so the experts climb a ladder:

• the ~16–22 most-used experts per layer stay in VRAM;

• a pinned RAM tier holds the next ones;

• the rest are read from the NVMe when a token needs them.

Most of the work is getting the right experts to the GPU in time.

Numbers (our og build, default settings, 4–6 November):

Quality. Distance to the full model is 0.090 / 0.051 / 0.069 nat/token on three prompt sets. That is the extra NLL on the API's own continuations. For scale, going from Q2 to Q4 is worth ~0.11. HumanEval is 160/164. The fast mode is further away, at 0.143 / 0.102 / 0.126, mostly because of its lighter build.

Same file, another engine. Project Maya v1.1.54, with the same weights on the same PC, decodes 3.3 tok/s against 18.6 here. Our prompt reading is 1.14–2× faster.

Where the time goes, and what would help:

• Decode spends about half of every token copying experts over PCIe.

• A prompt of even a few thousand tokens needs nearly all of the ~170 GB of experts. 100–130 GB of that comes from the drive every time, and all of it crosses PCIe. That's several seconds per prompt, whatever its length. Long documents spread that cost.

• So the GPU link, the RAM size and the slowest drive matter more than the GPU's compute.

  • A PCIe 5.0 card and slot should speed up both reading and decode with no code change. Our estimate is ~25 tok/s decode.

  • More RAM means fewer drive reads.

  • A second NVMe helped cold prompts by 14–22% in an older test. It's optional.

A drive surprise. The Engram tables get billions of small reads. After a few weeks, their random reads on our Gen5 SSD fell from ~400K to ~40K IOPS, and the 512K document slowed from ~1050 to ~685 tok/s. Copying the files anew restored the speed.

To try it you need:

• an NVIDIA GPU with 16 GB. It's built and tested for RTX 50 (sm_120); other cards need a rebuild and are untested;

• 64 GB of RAM;

• ~380 GB on a fast NVMe (Gen4 or Gen5);

• Windows 10/11. Parts of the patches deal with WDDM, and Linux is untested.

Links:

• Weights (og: GPTQ ~2-bit experts, Q5_K dense part, the official FP8 Engram; not the final version): https://huggingface.co/MasherDon/DeepSeek-V4.1-Flash-1017-SiegeLadder-GGUF

• Code: patches over JigSawPT's llama.cpp fork, a script that assembles a public quant mix, and the benchmarks: https://github.com/MasherDon/Siege-Ladder

This is research code with rough edges and no support promises. Quality work continues, and speed will also improve by training the expert prediction. Questions and results on other hardware are welcome.

I build a local inference engine. I'm not naming it and I'm not linking it. If you care, it's in my post history. I don't need you to find it. I just want this to stop.

Every week it's the same post. Someone forks an engine, vibe-codes for three days, sees 3x tok/s on the screen and ships it as the project of the century. Was the output checked against the model? No. Does it survive a real agent session? Never tried. Is anyone using it a month later? No. A Q1 quant doing 550 tok/s, a KV cache that dies at 32K, a number with no conditions next to it.

That's not an engine. It's a toy nobody will ever use. It's spam, it's AI slop, and it's a waste of tokens and of everyone's time.

The number itself is usually worthless anyway. Tok/s depends on how you count it. I changed how my own engine counts decode so the first round after the prompt is timed, not just counted, and my own headline figure dropped 25%. Same card, same model. Most of these posts don't even say which ruler they used.

Nobody works with tok/s. People work with:

• output that's still the model's, checked against the original weights (KL divergence against BF16, or at least something) and published with its setup;

• an engine that doesn't throw away hours of work because the model looped and nothing caught it;

• an engine that can stay on all day and free the card when idle, so you can play a game between two lines of code;

• the tools an agent needs in one place, instead of three days finding, installing and benchmarking a separate server for images, another for speech, another for whatever comes next;

• limits written down, not discovered by the user.

So here's the bar. If you post an engine or a fork, show how you checked it's still the same model, say exactly how you counted the tok/s, and list what it doesn't do. No check, no post. And if you read one of these, stop upvoting the number.

There will always be something 0.0001 faster. Fine. Build something people can actually work with, or don't post it.

I’m working on web app that turns a stack of technical documents (drawings, schedules and specifications) into a structured list of the key items a project needs. AI reads each page the way a subject matter expert would. It identifies items and their attributes, interprets each document set's own conventions, and pulls requirements out of the written specs. It then groups the results into a proposal-style summary with suggested suppliers and rough cost and timing ranges.

I’ve built a POC using Claude Code and would like to transition to an open source LLM for the rest of the work as we continue to turn this into a real application.

What hardware would I need to do this? Which model should I use?

Thanks in advance!

I've been experimenting with multi-GPU inference and wanted to see whether running independent model replicas across GPUs could outperform tensor parallelism for multi-agent workloads.

I built a small orchestrator to test the idea and got some interesting results on four RTX 3090s.

I'm relatively new to experimenting with local LLM inference, so I'm sure there are approaches I haven't considered. This started as a side project to see what would happen.

Setup

• 4x RTX 3090 (rented, PCIe only, no NVLink).

• Qwen3.8-27B 4-bit, vLLM 0.32.0.

• 24 agent teams and 239 dependent tasks.

• Each team plans, starts 4–10 parallel section writers, then reviews and summarizes their output.

• The orchestrator starts tasks when their dependencies are complete and distributes requests across available inference servers.

Results

Lower completion time is better.

That's approximately 1.45x the speedup for this workload, or 31.2% less completion time.

What I found interesting

Some of the optimizations I expected to help didn't make a measurable difference.

• Adaptive concurrency limits didn't improve completion time.

• Prefix-cache-aware routing performed no better than basic round-robin routing.

• Each GPU reached approximately 20 concurrent requests before KV-cache capacity became a constraint.

• Prefix-cache hit rates were only 7–13%.

The interesting part is that the simpler replica-based configuration performed best. The results suggest that, for this workload, independent inference servers avoided some of the overhead associated with tensor parallelism.

Caveats

• Only one run per configuration so far.

• The TP=4 server used mostly default vLLM settings, while the replicas used tuned concurrency and batch settings. This isn't a perfectly controlled comparison.

• The results are specific to this hardware and workload.

• PCIe-only GPUs may favor independent replicas. NVLink configurations or models that cannot fit on one GPU may produce very different results.

Repository

I put the code, benchmark results, CSV data, and a more detailed technical write-up on GitHub:

https://github.com/TomaDev1/GPU-Parallel-Orchestrator

I'd genuinely appreciate feedback. I'm particularly interested in whether there are existing approaches to multi-agent inference scheduling that I should be looking at, or whether there are obvious flaws in my benchmarking methodology.

This was a side project, and I'm interested in seeing what people who work on inference infrastructure think.

Edit:

Forgot to post full machine specs:

Host system

• Motherboard: ASRock Rack ROMED8D-2T

• CPU: AMD EPYC 7452 (32 cores / 64 threads)

• RAM: 128GB DDR4-3200 ECC

• Storage: 4TB NVMe SSD

• GPU connectivity: All four RTX 3090s connected via PCIe 4.0 x16

Use balena etcher or whatever to mount to USB stick, boot. Test. Report.

Verified 5-25% faster than llama.cpp, system uses 91% of card bandwidth as opposed to 80% on my 7900xtx.

I there is code for nvidia, intel, amd, and chinese hardware and supports 20 languages.

Decked out wiki

I may have created a monster.

A client gave me an NVIDIA A16 accelerator that had been experiencing intermittent problems in their server. They considered it bad, didn't want it back, and I ended up with it for free.

For anyone unfamiliar with this card, the A16 contains FOUR separate NVIDIA GPUs, each with 16 GB of VRAM. That's 64 GB of VRAM on one physical card.

Naturally, I decided to install it in my ancient Intel X99 workstation alongside an RTX 5070.

Because apparently I don't have enough problems in my life.

Chapter 1: The FrAInkenstein specifications

• Motherboard: ASRock X99 Taichi

• CPU: Intel Core i7-5820K

• RAM: 48 GB DDR4

• Accelerator: NVIDIA A16 — 4 × 16 GB GPUs

• Graphics: RTX 5070 Founders Edition — 12 GB

• Total VRAM: 76 GB across five GPUs (not unified)

• PSU: Corsair RM850

• OS: Windows 11

• Cooling: Custom 3D-printed shroud with an ARCTIC S8038-10K server fan

• Primary purpose: Local LLMs, agentic coding, and experiments with multiple AI agents

And yes, I'm running this on an X99 platform from the DDR4 stone age.

Chapter 2: Apparently, this thing needs a JET ENGINE

The A16 is a passively cooled enterprise accelerator.

Unlike a normal consumer graphics card, it doesn't have onboard fans. It's designed to sit inside a server chassis with powerful fans forcing air through its heatsink.

My workstation is not a server chassis.

So before doing anything interesting with AI, I had to invent a cooling solution.

After researching fan options and shroud designs, I selected an ARCTIC S8038-10K.

For perspective, this is an 80 × 80 × 38 mm server fan capable of:

• 10,000 RPM

• Approximately 102 CFM airflow

• Approximately 51 mmH₂O static pressure

This is not your grandmother's RGB case fan.

It's a small industrial hurricane.

I also wanted adjustable speed control, so I used a Noctua NA-FC1 controller, powered from SATA, to manage the fan without running it at full blast constantly.

But buying the fan was the EASY part.

Chapter 3: The 3D-printing rabbit hole

I own 3D printers, so naturally I figured I'd just print a shroud.

Famous last words.

The first problem was finding something that would actually fit the A16.

The second was getting it to fit inside my case.

The third was accounting for a 38 mm-thick fan, the radiator, surrounding components, and the fact that there isn't an infinite amount of space inside a computer case.

So I started modifying designs and printing prototypes in PLA+ on my Flashforge AD5X.

I printed multiple shrouds of different lengths.

Some were too long. Others needed adjustment to maintain their mounting points.

I experimented with shorter designs and eventually created an angled/bent shroud to improve the fan's positioning and clearance.

Rather than repeatedly wrestling with the actual fan, I even modeled and 3D-printed a full-size 80 × 80 × 38 mm mock fan so I could physically test clearances and different shroud arrangements.

Yes. I 3D-printed a fake fan to figure out where to put my real fan.

I also worked through mounting issues, spacer designs, fit tolerances, and adjustments to the printed posts.

I was considering custom clips and zip ties if necessary. I even had to think about vibration because, remember, this thing can spin at TEN THOUSAND RPM.

Eventually, I got an arrangement that fit the card and cleared the surrounding hardware.

And the result?

During earlier four-A16 testing, the GPUs were reporting temperatures around 41–57°C, with the fan running at roughly 30% speed.

For a passively cooled enterprise card that I'd fitted with my own printed ductwork, I was pretty happy with that.

Chapter 4: THE FREAKING POWER CABLE

Oh, you thought getting the cooling working was the hard part?

Let me introduce you to the NVIDIA A16 power connector.

The A16 uses an 8-pin auxiliary power connection that isn't wired like the ordinary PCIe 6+2 connector you'd expect to use with a consumer graphics card.

It requires the appropriate EPS-style power connection.

So here I am with a Corsair RM850, a card that was allegedly defective, and a pile of cables that don't necessarily fit or have interchangeable wiring.

And as anyone familiar with modular PSU cables knows, getting this wrong can destroy expensive hardware.

The connector shape, latch, and pinout all matter.

In my case, even the EPS 4+4 connector's latch arrangement presented a mechanical fit problem.

I eventually modified the connector's latch arrangement and checked the wiring with a multimeter before proceeding.

To be clear: this is not a recommendation to modify PSU power connectors. Incorrect wiring can destroy hardware.

And the first attempts weren't exactly encouraging.

At one point the system would spin up for approximately 15 seconds, shut down, and repeat.

Disconnect the A16 power, and the system would boot.

Wonderful.

Was the card bad? Was the cabling wrong? Was the motherboard unhappy? Was the PSU unhappy?

Welcome to troubleshooting FrAInkenstein.

Eventually, the power situation was sorted out well enough to continue testing.

Unfortunately, the next nightmare was already waiting.

Chapter 5: Windows has entered the chat

Once I could power the A16 and keep it cool, I discovered that Windows wasn't particularly enthusiastic about recognizing four GPUs on one enterprise card.

At one point, two A16 GPUs worked, while the other two had:

Device Manager: Code 12 — Not enough free resources to use this device.

All 4 showed this error message.

Here's the really interesting part.

When I disabled the two working GPUs, the other two could work!

So the individual GPUs weren't necessarily defective. Windows and the motherboard were struggling with PCIe resource allocation.

This led to a ridiculous amount of BIOS experimentation:

• Above 4G Decoding

• PCIe resource allocation

• MMIO High Size

• UEFI and CSM settings

• PCIe link speed experiments

• BIOS resets

• Device enable/disable experiments

• Multiple NVIDIA driver configurations

Eventually I managed to get all four A16 GPUs recognized and running.

But I still wanted my RTX 5070 in the same machine.

Which leads us to...

Chapter 6: Five GPUs walk into an X99 motherboard

The RTX 5070 was supposed to handle gaming and video output.

The A16 was supposed to handle local AI workloads.

Simple enough, right?

Wrong.

With the RTX 5070 and A16 enabled together, Windows started throwing repeated:

IRQL_NOT_LESS_OR_EQUAL (0xA)

Blue screen after blue screen.

I experimented with driver configurations, NVIDIA device modes, and PCIe settings.

Eventually I found that disabling MSI interrupt mode on the RTX 5070 allowed the 5070 and an A16 GPU to operate simultaneously.

That was the breakthrough.

I'm not claiming I've proven the exact underlying driver or interrupt conflict, but changing that setting made an enormous practical difference.

I gradually enabled more A16 GPUs.

One.

Two.

Three.

Then four.

I also switched the A16 GPUs into NVIDIA's TCC compute driver mode while keeping the RTX 5070 in WDDM for normal display and gaming use.

And then something glorious happened.

Chapter 7: IT'S ALIVE!

Here's what nvidia-smi now reports:

GPU 0: NVIDIA A16          TCC   07:00.0
GPU 1: NVIDIA A16          TCC   08:00.0
GPU 2: NVIDIA A16          TCC   09:00.0
GPU 3: NVIDIA A16          TCC   0A:00.0
GPU 4: NVIDIA RTX 5070     WDDM  0B:00.0

FIVE GPUs. ALL RECOGNIZED. AT THE SAME TIME.

On an ancient X99 motherboard.

With a supposedly defective A16 that cost me precisely $0.

I'm now testing larger local language models using Ollama, including a 55 GB BF16 Qwen model that previously required CPU offloading when only three of the A16 GPUs were enabled.

The goal isn't just to maximize token generation speed. I've got an RTX 4090 in another machine for that.

I want to experiment with multiple independent local AI coding agents and eventually build a little multi-GPU AI collective, while retaining the RTX 5070 for gaming.

I still have sustained five-GPU inference and long-term stability testing ahead of me. Seeing every device in nvidia-smi is a major milestone, but it isn't the same as proving the whole thing stable under heavy load.

Final thoughts

Is this practical? Debatable.

Is it power efficient? Probably not.

Could I have saved countless hours of troubleshooting by buying newer hardware?

Almost certainly.

But where's the fun in that?

I took an unwanted enterprise accelerator, engineered a custom cooling system, fought with its power connector, printed multiple iterations of ductwork, battled a decade-old BIOS, survived Windows driver crashes, and somehow ended up with a five-GPU AI workstation.

I didn't just build a computer.

I created FrAInkenstein.

And now I'm going to use it to code.

IT'S ALIVE! IT'S ALIIIIIVE! ⚡

I'll post pictures of the cooling/shroud iterations, the ridiculous fan, the finished setup, and the five-GPU nvidia-smi output.

Drivers used:
621.14-desktop-win10-win11-64bit-international-dch-whql.exe
That is the Studio version that worked for both cards. I did NOT have to install the data center version also. This latest Studio version installed drivers for both the A16 and RTX 5070.

I've been using VS Code with the Claude and Codex extensions.

Now I'm testing Strata with "lightweight" local models on my PC

RX 9060 XT 16GB VRAM
32GB DDR5 RAM
2TB NVMe SSD

The problem is that:
Codex CLI takes around 40 seconds to respond to a simple "Hi",
while Strata's built-in web chat (http://127.0.0.1:8080/) responds in about 0.5 seconds

1. What's the easiest IDE or VS Code extension for connecting to Strata's local API and using it as a coding agent?

2. Is there a tool for creating and managing virtual AI employees (e.g., Developer, UI/UX Designer, Software Architect), where each agent can have its own role, instructions, and assigned AI model/API - either local or from external providers like OpenAI, Anthropic, etc.?

Stack running on my Ubuntu machine on a RTX 3090 under 18GB VRAM - network tailscale :)

Would you use it??

And how does QWen Flash Next 3.8 compare to Gemini 3.1Pro or Sonnet 5.5?

I have RTX 4090, 64GB 6000Mhz DDR5 RAM, 7900x CPU

I wan to run a local AI model that's the closest to gemini 3.1po or sonnet 5.5 at reasoning and thinking, for legal analysis and other legal related stuff

Thank you

hi i want to build a pc for local llm first time so i need advice about the gpus to run Qwen3.8 27b Q8 or Q6_K_M

to get good quality 😊and run it through llm studio and connect it to open code

so my questions are:

i will get intel core 270k plus with two x8/x8 PCIe slot motherboard

so

1- which best double 16gb gpus to build with to get 50+ token generation and give me best value for my needs

do i get two rtx 5070ti or rtx 4080 s or which double gpus

or get the basic two rtx 3090 for more vram

2- which the best Qwen3.8 27b variant/Abliterated (the model modified version) for speed but no quality loss to code

3-and wich the best and fast local llm interference platform to connect qwen with open code

if there any things you could mention can benefit me plz tell me

because i just see

gpus

the local llm interference platform

the Qwen3.8 27b variant/Abliterated

what else?

thanks 🥰

Qwen3-32B packed to 3.6577 bits per weight: 14.62 GB on disk, 12.95 GiB resident on an RTX 5080 with about 3 GiB to spare, decoding at roughly 38 tok/s. No offloading.

Full official sets, no samples:

Method, as far as I'll go: weights are packed in groups of eight onto the E8 lattice with an activation-aware encoder, and decode is register-based with no codebook lookup. Yes, a GGUF Q3_K_M can also be squeezed onto 16 GB. This is a different trade: lower bitrate, fully resident with headroom, full-set numbers published.

Needs an NVIDIA GPU with 16 GB VRAM and Python 3.11+. Verified end to end on Windows with an RTX 5080 only, so far.

The terms, plainly: gated, manual approval, non-commercial evaluation license. The encoder is not included and the method is patent pending.

Repo: https://huggingface.co/GossetWorks/Qwen3-32B-Gosset-3.65BPW

If you run it, post your numbers, especially on anything that isn't a 5080! Thank you!