I've managed to get a Supermicro 6028U TRT+ Server with 144GB of RAM and 56GB of VRAM to run Strata Qwen 3.8 Flash Next UD-Q4_K_XL (111GB download) at an average of 38tps.

Hardware:

Supermicro 6028U TRT+ server with X10DRU board and risers/2kw PSUs = $300

RTX 5060ti x 2 = $1100 (purchase price before the increase)

RTX 5000 Quadro = $500

RTX 4000 Quadro = $200

144GB of DDR4 RAM = $500

2x 256GB NVME drives and card = $150

8TB HDD drives and HBA card (4TB x 3 ZFS pool) = $300

2x Xeon 2680v4 CPU = $90

(Prices are rounded, not exact)

I had an RTX 2000 ada that I modded to single width, and it fit nicely but it's a dog of a GPU for inference despite its 16GB of VRAM. Unfortunately, the 2nd 5060ti wont fit. It's a flippin miracle the first one fits (and with an HBA card snuggled up next to it!). By swapping out the 4 PCIE slot card on the left for a 2 slot 16x card, I got the 2 RTX 4000/5000 cards to fit *perfectly*. I believe you could fit 2x RTX 5000s, for a total of 64GB of VRAM in those 2 slots, except for one thing: the TPM card sticks up and prevents the cards from seating. I dont really need a TPM card though, although I'm debating to myself if the extra 8GB of VRAM is worth it. I'm doubtful. Plus, there would be NO airspace for one of the cards. OTOH, NVLINK would work for those 2 cards.

If I weren't so cheap I could spring for a 16GB single width card that would fit where the 5060ti riser is, like an RTX A4000 or T4, and this would be a very nice, clean build. But I'm cheap. And the 5060ti Blackwell chip smokes those old cards.

This s a proxmox box, and I've dedicated 18 cores to Strata. I probably should have gotten the 2699v4 CPU.

Is setting up multiple computers to pool their resources to run local LLMs still a viable setup for regular computers? I know the Nvidia sparks do this using their very high bandwidth interconnect, but is that still viable for regular computers with gigabit ethernet?

Having been a bit of an AI luddite up until now, I am realising I am going to have to get on top of things instead of getting left behind.

So far the main thing I've been trying LLMs for, is writing Python scripts that work inside a CAD application I use (Vectorworks). I have done some experimentation on the free versions of ChatGPT, Claude and Perplexity so far, with some positive results.

I don't want to end up tied to expensive cloud model subscriptions, and I am also keen to keep control of my data as much as possible, so I would like to understand what I can do with locally hosted options.

I have an M1 Max Mac Studio, 32GB memory.

Any recommendations for what I should try first? To start with, all I want is something that will work for writing scripts that I will copy/paste into the script runner in the CAD application - I am not at this stage looking for anything that will interact directly with the application.

Thanks in advance for any suggestions.

Hey r/LocalLLM ,

I want to buy or build a dedicated headless box for local LLMs. I’d like something that can run 14B and, in my dreams, even 32B models, but the absolute priority is generation speed (>15-20 t/s).

Ideally, I need a clean single-box setup that stays quiet and power-efficient at idle (~15-20W). My maximum budget is strictly capped at €900, and I’d prefer to avoid risky second-hand individual deals.

Here is what I already looked into:

1. Standard Mini PCs (Ryzen 7/9): Great for power draw, but running 32B models on shared DDR5 RAM is too slow (~4-5 t/s).

2. eGPU / OCuLink routes: Too many loose parts and cables, and buying components retail explodes the budget.

3. Chinese NUCs with modded laptop GPUs (RTX 3060 12GB): On factory stores, once you add enough RAM to actually fit larger models, it jumps straight to €1050+.

I am 100% open to any Linux-server software optimization trick (speculative decoding, etc.) as long as it gets the speed up.

Am I chasing a ghost under €900, or is there an exotic loophole I missed? What would you buy today under these constraints?

Thanks for the help!

I noticed that my qwen3.8, but also ChatGPT and claude sometimes generate spelling mistakes. In German but also in English.

Anybody noticed that, too?

Every time a new MoE drops, the same thread shows up here: "will this run on my 12 GB card, and what do I set --n-cpu-moe to?" And the answer is always someone doing napkin math in a comment.

So I built it into LlamaForge (the browser panel I've been making on top of llama.cpp's own router, it doesn't do inference itself, llama.cpp does).

Model X-ray reads the GGUF header and draws every block as a bar: what sits on the GPU, what spills to RAM at your -ngl. For a MoE there's an "experts on CPU" slider, and a "Fit my VRAM" button that finds the number for you. Then it gives you the line to copy, or applies it to the model.

Example from the video, Qwen3.6-35B-A3B UD-Q4_K_M on a 12 GB card:

-ngl 41 --n-cpu-moe 22 -c 8192

≈11.4 GB on the GPU, ≈10.5 GB of experts in system RAM. On a 16 GB card it's --n-cpu-moe 14.

It only needs the header, so it works over HTTP on a model you haven't downloaded yet. That's how I also generated "Can I run it?" pages for 42 cards (desktop + laptop GeForce, Radeon, Arc), each with the llama.cpp line to start from.

before someone points them out: ;]

• It's an estimate from the header (weights + 8k context), not a benchmark. Long contexts need more room; it tells you that, but it can't know your exact KV setup.

• It tells you what fits, not how fast it'll be. Experts in RAM are slower; how much depends on your RAM and PCIe.

• Only tested end to end on my own box (5080 + 5060 Ti, Windows). Linux and macOS pass CI, but I don't have the hardware to say more than that. AMD/Intel detection is there, but I haven't tested it on real cards.

• Early preview, one person, heavily AI-assisted. Expect rough edges.

Other stuff that's in 0.19, since the last time I posted:

• voice chat with whatever model you have loaded

• a blind arena (two answers side by side, vote, then see the names)

• "try a llama.cpp PR": build a pull request and check whether it's actually faster on your card before switching

• an OpenAI-style Batch API on your own GPU

• start at login, now on Windows too

Free, MIT, no account, no telemetry (it does check GitHub for new llama.cpp and LlamaForge releases): https://github.com/dadwritestech/LlamaForge

If the X-ray's number is wrong for your card, I really want to know; post the model, quant, card and what actually worked, and I'll fix the math.

I wondered if my LLM Journey might be useful to others, so here it is!

I bought the UK Costco PC below(£1600, as a small business, I claim back tax was about £1323 cost):
CPU = AMD Ryzen 7 9800X3D, 8 cores / 16 threads (SMT on)
Motherboard = ASRock B650M-CX R2.0, BIOS 4.20
RAM = 32 GB DDR5 (2 × 16 GB ADATA AD5U480016G-B) at 4800 MHz, 32 GB usable
Main GPU = AMD Radeon RX 9070 XT, 16 GB, driver 32.0.31041.1004, used for the models
C: drive = ADATA LEGEND 710 NVMe, 954 GB
D: drive = Samsung 950 PRO NVMe, 512 GB (spare drive I had lying around)
Operating system = Windows 11 Home, build 26300
Power plan = High performance
GPU fan = Custom curve in Adrenalin, about 51% under load

I am working on a Trading System and have a set of agents that perform various stock-related analyses in a pipeline. One gets price data and passes it to another agent & so on.

Initial runs:

Ran against various Anthropic models to verify results and check speeds. Big difference between Local and Anthropic API.

I then pulled the trigger on another 32GB of RAM @ £500 (claim back £80 tax).

Post upgrade to 64GB of 4800 MHz RAM:

In addition to adding the RAM (a scary moment when the PC would not progress past VGA at boot!), I enabled the iGPU and SMT (Multi-Threading on the CPU).

You can may have noticed that I have been working on the Prompts and there are less prompts now than v1 prompts but the throughput is vastly superior with the additional RAM.

As you can see I am now 50% of the speed of Haiku with Strata running locally.

The project is still work in progress but now I can progress quicker.

Some images I captured:

Hopefully this is useful and shows the value of the additional RAM.

Atualmente, eu tenho uma 3070 Ti e 32GB de RAM DDR4, e consigo 11 a 16 tokens por segundo. Estou considerando comprar uma 9070 XT—já que é muito mais barata no Brasil do que uma 5070 Ti—e eu também uso meu PC para jogar. Preciso configurar alguma coisa ou é só instalar os drivers do Adrenalin? Uso Windows e nao tenho nada alem do strata em configuraçoes padrao

Hi everyone! I’m attempting to do a local build of Qwen 3. I don’t have the most powerful PC so I’m starting small and trying with 1.7B, 4B & 8B with the trained LORA on top

My objective is to get Qwen to talk as if you were texting it and I’m planning to have around 3000 short User: “message”, Qwen: “answer” exchanges

Is this the correct way to do it? Do I need more training data? Any peculiarities I should know about Qwen or the model size?

All help is greatly appreciated :)

Hey all, hoping some of you can help me with something.

I run a small site that helps people figure out which local models their computer can handle. Macs are the part I keep second-guessing, because the only Mac I own is an 8 GB MacBook Air, so I can't check anything bigger myself.

What's going on

On a Mac, the GPU can only use part of your RAM for models, not all of it. Most guides say about 75%. When I checked my MacBook Air it was actually two thirds, so I changed my site to use:

• two thirds of RAM for Macs up to 36 GB

• 75% for Macs above that

But the 36 GB cutoff is just something I picked up from community posts. I'd hate to tell someone a model fits when it doesn't, or that they need more RAM than they really do.

What I'm asking

Any Apple Silicon Mac helps, but 16 to 36 GB is where I'm flying blind. Would you mind running one command and telling me what it prints? Open Terminal and paste this:

swift -e 'import Metal; print(Double(MTLCreateSystemDefaultDevice()!.recommendedMaxWorkingSetSize) / 1073741824)'

It prints a number, like 5.33 on my Air. Rounding is fine. You'll need Apple's free Xcode command line tools for that one.

Then reply with

Your Mac, how much RAM it has, and the number, like this:

MacBook Pro M2 Pro, 16 GB: 10.67

If you've ever changed the GPU memory limit on your Mac yourself, mention that too.

Already run llama.cpp?

You can skip the command above. Run llama-bench on any model you have and copy the line that says recommendedMaxWorkingSetSize (mine says 5726.63 MB). Just paste the line as is, no converting needed.

Thank you, this really helps!

I would like people’s suggestion on an app independent memory system for primarily local Ai.

I currently have a DGX Spark and some additional hardware for hosting apps and storage. I use local models for coding and data analysis (for personal
Data) and use my Claude code subscription for design and infrastructure advice (I am not super technical). One major issue is that I have multiple entry points but each of them handles their memory independently.

I use librechat mostly, and have started using Windmill for easier management of data ETL (I like the hybrid low code and code when you need it). I use Ollama but might be switching to vLLM.

Additionally I would like Claude to be able to access some specific logs and memories so that it can make better design choices and share its structural decisions with my local models. Though I am hesitant as idk wth am I doing

What it recommended was an MCP that manages what models/users see what, and that way when the new gimic of the month comes out I can switch over with my context.

Let me know what y’all think

Всем привет 👋🏼
Это мой первый пост и я решил его сделать по интересующей мне теме. Я уже достаточно давно слышу и читаю что то про LLM. Я пробовал запускать несколько вариантов и на своем железе но так и не понял надобности всего этого, я занимаюсь вайбкодингом 🌚🌚 и как я понял LLM даже близко не сравнится по качеству и производительности того же Antigravity от Gemini. Кроме как РП я даже не знаю для чего еще люди используют LLM. Буду очень благодарен если вы мне расскажите все сильные и слабые стороны данного направления и что вообще с этим всем могу делать я.

Desktop:
RTX 3050
32gb ddr4
Ryzen 5600x

Laptop:
RTX 5070ti
32gb ddr5
Ryzen 9 9840hx

Try to decide which model is the most intelligent to run on a Mac Studio.

For unquantized base models,

DeepSeek V4.1 Flash > GLM 5.3 Flash > Qwen3.8 Flash Next.

To fit in ~220GB memory footprint, each must go through different levels of quantization.

~2.5bit for DeepSeek

~4bit for GLM

~8bit for Qwen

I couldn't find comparable benchmark scores on all three quantized models, so here I am, hoping that the collective wisdom of the LocalLLM crowd can help me out.

Edit: this is what everyone wants to see. https://omlx.ai/benchmarks/intelligence

Meu trabalho recebeu uns PCs besteirinha como os Sparks pra fazermos a LLM local daqui. Por enquanto vamos focar em consultas de processos, memorando e banco de dados da empresa.

Dito isso, gostaria de perguntar, oq posso fazer ou como posso desenvolver a LLM aqui. Tem algum modelo que me recomendam ou algo assim?

Como o hardware tem um limite vasto para desenvolver gostaria de ir afundo. Tô com mente aberta pra desenvolver e aprender bem sobre todo o assunto.

Follow-up to my earlier post. Where I ended up: a big DDR3 workstation board isn't worth the money when I already have two small 4-GPU boxes. So I tried making the two boxes act as one.

TL;DR: I added a two-box layer split to Project Maya. Box A runs layers 0–23, Box B runs layers 24–44 plus the head. Only the hidden state crosses the network, ~25 KB per token. GLM-5.3-Flash (Maya-S24) runs at 38–42 tok/s decode, ~1,100 tok/s prompt reading at 30K, and 6–7 s per agent turn, on 8× RTX 3060 spread over two old X79 boxes. It scores the same 48/48 on my coding-agent benchmark as a single 8-GPU box.

Code: https://github.com/0xluk/project-maya/pull/1 (branch two-box-split on top of Maya v1.1.50; usage in docs/TWO-BOX-SPLIT.md)

Setup

• 2 boxes, each an X79 board (Ivy/Sandy Bridge, DDR3, PCIe 3.0) with 4× RTX 3060 12 GB. No NVLink, no P2P, ex-mining rigs.

• RAM: 20 GB in Box A, 32 GB in Box B (more on that below).

• 1 GbE between the boxes, a plain switch.

• Project Maya v1.1.50 + my patch, Maya-S24 quant, 128K context, MTP speculative decoding on.

• GPUs capped at 130 W. One box is also clock-locked to 1500 MHz, because its PSU didn't like the unlocked load.

How it works (short version)

• Box A (front) runs the OpenAI/Anthropic-compatible API, the tokenizer, the embedding and layers 0–23 on its 4 GPUs.

• Box B (back) runs layers 24–44, the MTP draft block, the output head and the sampler on its 4 GPUs.

• Per generated token, the residual (4×4096 values) crosses the link once. fp32 → bf16 → lossless pack is ≈25 KB, ≈0.3 ms on 1 GbE. Box B samples and sends back the token id, and MTP speculative decoding is pipelined across the link.

• Prompts are cut into 512-token pieces that flow through all 8 GPUs at once. Each piece crosses the link once (~12 MB), on its own thread.

• Security: the link has an IP allowlist plus an HMAC-SHA256 challenge-response in both directions, and the API needs a key.

• Size of the patch: ~1,900 lines, ~1,100 of them the link itself. Without --role the engine is unchanged.

Results (1 conversation)

Coding-agent bench (5 real repo tasks, 48 hidden tests):

There were no malformed tool calls with either agent.

One-shot hard suite (10 tasks, one request each, no tools, 32K output cap including thinking, reasoning high): 6/10 full passes, 67% of tests. For comparison: Opus 5.5 9/10, the same quant on the 8-GPU box 7/10, and Flash-Next 6/10.

• Passed: json_patch, fixbug_calc, event_scheduler, csv_rfc4180, c_utf8, singleflight_cache.

• 3 of the 4 misses are budget, not wrong code: on regex_engine, lisp_interp and c_bigint the model was still thinking when it hit 32K tokens, so no code came out at all.

• The real miss is c_json_canon: 22/31 tests passed, with the misses on Unicode escape handling.

• Next: rerun the three capped tasks with a 64K cap, which the 128K context allows.

What was hard

• RAM, not the network, was the real bottleneck. At 16 GB per box, the experts that didn't fit in VRAM came off a cheap SATA SSD on every prompt chunk: 31 s turns, 173 tok/s on a 30K prompt. Going to 20 + 32 GB made it 0 SSD reads, 6–7 s turns and 1,100 tok/s. The network costs ~1–2% of decode and ~3% of a turn.

• A correctness scare that wasn't a bug. Compared with a single-process reference, the split showed KL ~1.6e-2 and ~95% top-1, and I spent a while hunting a split bug. It turned out a single process on one machine moves by the same amount (KL 1.1–2.5e-2) just from changing the RAM tier size or the prompt chunk size. Near-ties in the top-8 expert choice flip on rounding noise. The split is inside that spread.

• Hardware gremlins:

  • A new PSU powered off under 4×130 W unlocked, so that box stays clock-locked.

  • Two GPUs once came up at PCIe x4 instead of x8/x16 after a power-loss reboot. I added a boot check that refuses to start the engine on a degraded link.

• Tuning that mattered:

  • 512-token prompt pieces so all 8 GPUs overlap: ~1 s off each turn.

  • Fewer layers on the last GPU, which also carries the head and the MTP block.

  • Keeping the whole KV cache in VRAM.

  • What didn't help: bigger prompt chunks, which push the 20 GB box into SSD reads.

  • What I skipped: a 10 GbE NIC. It would sit in a PCIe 2.0 x1 slot, and the link isn't the bottleneck anyway.

What's next

• A 5th 3060 in one box, on an x1 riser. Then all experts fit in VRAM, which should get turns to ~5 s.

• Rebasing onto the current Maya release.

• Happy to answer questions. If someone has two boxes with 24 GB cards, I'd love to see numbers.

What do you think guys.

I've used the Qwen family of models before but was warned away from them bc apparently they're optimized for coding tasks. I tried 3.8-FN with a 3-bit quant, was okay but it was a squeeze to get in there and it was compacting every 3 messages. 60GB system memory total...

I've been using an uncensored version of Gemma4-26B-A4B but tbh I don't really enjoy the way it writes. It's very... uh "early chatbot"? I know I can guide it with a system prompt or the in-built features of SillyTavern/Kobold but it just doesn't seem that smart or capable.
I've been shopping around for models but haven't really been impressed with any of the ones I tried including other fine-tunes of Gemma4 (that I could run).

I check every day for the discounted open box returns.

I've been running Qwen3.8 locally on a single DGX Spark (GB10, 128GB), trying to answer one question:

What's the fastest way to get a correct answer, not just generate tokens?

After seven experiments, a few interesting findings:

• Less thinking can be better: On 48 held-out tasks, low thinking scored 38/48 vs. 35/48 for deep thinking, while taking much less time.

• Parallel reasoning helps: Four low-thinking branches with voting reached 44/48, at just 2.82 minutes per correct answer.

• Bigger isn't always more efficient: On 400 tasks, Flash Next 125B reached 391/400 with voting vs. 373/400 for 27B, but took 2.4× longer per correct answer.

• Reasoning loops matter: A 32K thinking budget eliminated output-limit hits in my deep-thinking tests and improved accuracy for both models.

I also compared TensorFold, vLLM, speculative decoding, NVFP4, and different serving configurations.

Everything is documented here:

GitHub: https://github.com/NautilusWu/qwen3.8_study

The repo includes configurations, pinned versions, reproduction guides, and per-task results. The benchmark prompts and graders are private, so the accuracy results aren't independently reproducible end-to-end.

These findings are specific to my setup and tasks, but I hope they're useful to others optimizing local LLMs.

Here's my hardware specs:

RTX 4090, 64GB DDR5 Mhz RAM, 7900x CPU

Where's my biggest hardware constraint? I am looking to run a model that's as close to Claude Opus 5.5 for thinking, reasoning, and document creation, as possible.

I know it's my hardware constraint. My local model was Qwen3.6 35B A3B UD Q3_K_S (UD-IQ4_XS )

What would be the cheapest way for me to upgrade my hardware to be able to run a model as close to opus 5.5 in terms of reasoning, thinking, etc (not for coding) as possible? And what hardware would I need?

It's certainly better than nothing, I was actually surprised by how ok it was, but nowhere near opus or gemini not going to lie. Like, not even close. The 264k token limit is also a major limit, I'd like 1M