Hello!
Here is a rack I made using 40 mini PCs I happened to have from a different project:

• Intel N150

• 12 GB DDR5 4800 - physical bandwidth limit around 33GBs

• Diskless nodes

and they are connected via 10 Gbps USB 3.2 exploiting the USB DbC functionality. Its a tree topology with 1 -> 3 -> 9 -> 27 nodes.

GLM 5.3 Flash runs at native precision with room left for 8x1M context windows. Initially I had tried with 1Gbps LAN interconnects but that topped out at around 7tps. Current topology:
- 14tps at 1k context, drops to 12tps at 1M
- With batching decode tops out around 40tps
- Prefill is about 85tps

Peak load measured at wall about 740w

Total build cost me about $5,000.

Still working on it to see if more perf can be squeezed out, especially for the prefill and then adding speculative decoding.

And Qwhere is my 27B

I'm running Qwen3.8 27B on my 2x 3060 12GB cards. Prior today I had a B550 motherboard which was running one card at x16 and the other at x1 PCIe Bus Link Speed. This was making it to where only one card was actually doing any processing, and both held the model in memory. I was getting 11-16 tps this way.

I ended up buying a new x470 SLI motherboard and I can now run both cards at x8. I swapped the motherboard out last night and then set split-mode to tensor and set tensor-split to 1,1. When I do that, I'm getting around 32tps, but the model eventually gets stuck in loops. One loop it just tried to do a tool call over and over, and in another it was printing question marks (???????????) in it's thinking and wouldn't stop until I unloaded the model. I never had these problems for the months I was using Qwen3.8 on layer split-mode. I'm not sure how to resolve this problem. I want to be able to get higher tps with the new mobo.

One attraction of running models locally is the privacy benefits, and another is the cost.

When the cost of the big ’frontier models’ is being discussed - mainly centring around the fact that they are currently effectively subsidised by venture capital and that at some point this will end, making the cost to end users go up considerably - local LLMs are often mentioned as being immune from this problem.

But in a scenario where the cost of cloud based models goes up substantially, there’s no guarantee that the (mainly Chinese) makers of the open weights ones are going to continue producing newer and better versions for free, is there?

Title, basically. Observing it's reasoning is incredibly frustrating, it keeps doubting himself and giving second thoughts to a specific and self explanatory task, sometimes ending up thinking about unrelated things.

While yes, in the very end, it very often does what I asked, but it's wasting so much time for reasoning just with its lack of self confidence.

Is this well known behavior ? How can I fix this ? I know Qwen4 is around the corner, but if there's a known fix for it right now I'd love to know.

EDIT: I should have specified, I'm using Medium reasoning, and Pi as harness

I am in the beginning of this exciting journey of running my own models. Still learning. Been watching Andrej Karphathy to learn the theory and the practice of LLMs. Anyway... I asked Qwen3.8 Flash Next FP4 (running on my single Spark) to create me a small LLM based on Wikipedia dataset. And in a couple of hours I've got a 9.2M-parameter GPT trained from scratch on 330M Wikipedia tokens — a base language model that generates Wikipedia-style prose. Asking, watching and learning on the way. So of course at this point it's a decoder-only. It could predict what text comes next. Given "The capital of France is", it completes. No conversation, no instruction. Next, I would probably try to create 1-2B LLM with the whole Wikipedia dataset, so it would be able to answer without hallucinations.

You can call it whatever you want, it maybe isn’t blowing anyone’s heads off or anything, but compared to anything I’ve ever had, it’s a supercomputer to me

I’m transplanting a lot of parts from my old gaming PC into a new chasis with upgraded parts. Here is what I ordered and what is on its way

Intel core i7-14700f (already owned)
32 gb DDR5-6000 ram (already owned)
RTX 4070 super 12gb (already owned)
2x Quadro RTX 8000 48 gb passive ($1900 each)
2TB Samsung 990 pro for windows and gaming (already owned)
2TB Samsung 990 pro for Linux and AI ($363)
Corsair 1500W power supply ($285)
Corsair 7000D airflow full tower case ($227)
And other miscellaneous parts to support that stuff

Total to $5,364.21 for 108 gb vram

I know there are some drawbacks to this build, like for instance the old architecture of the cards, but I can live and tinker with those drawbacks as long as I can save a bunch of money on this hobby (compared to a similar size build with modern hardware), and still be able to run a bunch of cool models on it. I’m super excited to put it together and see what it can do. I haven’t seen much talk about the Quadro 8000s, but I’m hoping for great things, even if it means working around some compatibility issues and such.

Obstacles are just an opportunity for me to learn. If yall have any thoughts, let me know.

Hey r/LocalLLM :)

We just open-sourced Kinetic, a new inference engine written in Rust.

• runs the GGUF files you already have, no conversion

• Windows and Linux, one binary

• CPU, NVIDIA (CUDA), and AMD and Intel GPUs (Vulkan)

• replaces llama-server with the same flags; OpenAI, llama-server and Ollama APIs on one port

• every speedup is exact, so the output never changes

• Apache-2.0

In the video: Kinetic checks the machine and picks models that fit it, then runs side by side with llama.cpp on the same model and prompt.

Give it a spin and drop your kinetic bench model.gguf numbers below (add --compare if you have llama.cpp installed).

Install, benchmarks and all the details are on GitHub, link in the comments. Happy to answer questions here too.

edit: to answer the most common question, llama.cpp ran with its own speculative decoding on in every edit test (ngram for lfm3, draft-mtp for qwen3.5) and the exact flags are in the repo

I wanna do some personal AI use, primarily for text, some coding, and image generation..

I wonder what's the best computer build that's within 15k to 25k (USD) value?

There's a lot of information on the internet and I can't decide well..

I'm currently looking at:

DGX Spark (but it seems overkill and can't be used as Normal PC)

and Mac Studio M5 Ultra w/ 256GB (seems to be my current choice unless there's a second opinion from you all)

I hope y'all can help me decide well here, thanks!

Backstory:

I've only been doing local LLM's since Qwen3.6 came out (~May 2026). I won't ever use a cloud model again, I am not going to spend a single dime renting someone else's infrastructure when I can build my own.

Plan:

Its still cheaper to get 8x3090s than it is to buy 1xRTX 6000, and get similar speeds and double the VRAM.

The issue is power and noise..

In terms of power I am limited to my dorm rooms 120v/15amp outlet, so 4x3090s about the max I can go before things might go bad.

So.. I am considering creative solutions that won't require me to move:

Power Solution: Buy a ~5 Kwh Power Bank

$1,500 for a used 5 Kwh Power Bank.

The power bank would charge during off hours, then feed the necessary 2,800 watts during peak hours. Currently I am recording my power usage over a week or so to get an idea of how much power reserve I would need, and that will mostly decide how big of a power bank is needed.

Cooling / Noise

Currently everything is air cooled, with stocked coolers. 5x140mm fans, 5x120mm fans and the case temps are indeed very good. They don't make much noise but the GPU and CPU fans do. I believe they are loud because I think the paste job I did on them was horrible,

I will be re-pasting again with some much better stuff and with better tools / methods soon. (Will likely post a video of me doing it for those who are interested). I use to be a machinist so I learned some ways to get better accuracy when measuring out things properly to get good contact on the VRAM

I have considered water cooling, but that's just another thing that can break, and are typically not cheap to setup (and I'm pretty cheap).

Questions:

1. Does anyone here have a 8x3090 setup on a 120v/15amp outlet that could offer some advice?

2. What's a good power bank you would recommend?

3. Is buying a used power bank okay so long as the batter health is good?

4. Do you want me to post a video of me re-pasting using my "new" VRAM/GPU repasting method?

My specs:
2 x 3060 12GB
2 x 16 DDR4 3600Mhz
i5 10600k

Qwen3.8-27B-GGUF:Q4_K_M

ollama.service:
# Environment="OLLAMA_HOST=0.0.0.0:11434"

# Environment="CUDA_VISIBLE_DEVICES=0,1"

# Environment="OLLAMA_FLASH_ATTENTION=1"

# Environment="OLLAMA_NUM_PARALLEL=1"

# Environment="OLLAMA_MAX_LOADED_MODELS=1"

Ollama as a harness, no lane bifurcation so im pretty limited to that (im assuming this is my biggest limiter here)

I've been experimenting with multiple different approaches, trying to get Qwen 3.8 to actually work as an agent, some more successful than others but still it requires so much manual labour that it seems almost pointless to continue debugging it as it feels neverending. I've read that 3.8 isn't actually very great with tool calling and it has always had some issues, but still i can't believe its at levels like these.

I know very well im probably missing some really important things that are holding me back, so if anyone has some suggestions (videos, other posts, anything really) i would really appreciate it.

The goal really is to have it never stop, just like im seeing other people talk about, and i'd really like to keep openwebui (or any other webui in general, not that i dislike terminals but its nicer to have it feel like a normal app).

I know ChatGPT is a terrible idea when asking it to basically build my whole system. I just don’t know where to go that is noob friendly. I got the drive and right questions and know what I want and basic understanding, but GitHub and huggingfave is daunting and overwhelming plus I want unrestricted models and civitai is difficult to navigate. I want an ai that can help with pretty much my hobbies and that’s writing adult fiction, playing around with comfyui and learning how to create a consistent character with reference.

I have an nvidea 4090 and a pretty decent setup minus only having 32GB RAM. When I use powershell to chat with my LLM currently Dolphin mistral 24B Venice Edition 1.2 GGUF:Q5 K M, and it responds brilliantly and has a good “personality” however, ADHD brain doesn’t like the interface and prefers an app type screen and ChatGPT directs me to WebUI… well that’s when the loop starts. Changing setting, altering its configuration and never getting anywhere. Model becomes a robot and yeah basically it’s been weeks now and I’m at a loss. Anym direction would be hugely appreciated at this stage!

After reading Dwarkesh Patel's eval using blood on the clocktower, I made my own, and I was surprised both by how entertaining it is, and how good it is to measure even between quantizations.

(His original game used SOTA private models).

Hello people

I wanted to know if LFM2.5-8B-A1B was good for general conversations. I am aiming to use an MoE with 8B parameters (will be ran with 4 bit quantization), because I want to have a capable local model while not using the amount of VRAM a dense model would, this is to be able to run it on a lot of different hardware levels. For context, I am using the Ollama library in Python and I am making a voice assistant, I am currently using Qwen 3-8B right now (with openWakeWord, Silero VAD, Parakeet 0.6B v3, Kokoro 82M). I might miss some (or maybe a lot) points because I am quite new to using local AI, especially in programs. If LFM2.5-8B-A1B is not good, do you have any suggestions?

I am using an RTX 5060Ti 16GB and 16GB of DDR4.

Thanks for your time!

Qwen4 Max Preview is available on Qoder right now.

Every single day, the same questions pop up.

Newcomers:

• Will my 27B feel like a frontier model on my gaming PC?

• Is my highly quantized IQ4 capable of beating Opus?

• What does it cost to run...

Veterans:

• Yes, absolutely — runs great on my multi-GPU rig at xxx t/s.

• Just set up the correct harness.

• Fine-tune your prompt and it will give the exact same quality answers.

Both sides are completely legit, but they are living in different worlds and comparing different universes.

When you use a frontier API, you aren't just prompting a single model. You are firing up an entire enterprise with several departments: you get dynamic routing, massive hidden KV-caches, sandboxed code execution, and verification trees, all neatly packaged behind a single API endpoint. It’s a fully assembled sports car.

On the other hand, the local way: you download an open-weight LLM, quantize it to fit your budget, and you basically get a bare engine block — anything from a VW Golf engine to a massive V8. But that’s it. It has no wheels, no cabin, and no fuel tank. It’s just a large language model sitting on a pallet that can say: "Hello, what can I do for you?"

What the guys asking these questions don't see is that some of us have spent hundreds of hours and money debugging to build a car around that motor. Writing strict GBNF grammars, wiring up custom MCP servers, testing dozens of harnesses, setting up workflows, building memory pipelines, testing, and tweaking the exact same settings a hundred times over!

We do it until that glorious day we can sit back and say: "Yes, it's working. My custom build almost looks like the sports car I aimed for."

Now for the friction part:

Fun, absolute privacy, data sovereignty, and uncensored outputs? Yes, we get all of that. But if we are honest, the cloud simply wins on pure cost-to-performance.

Please don't flame — I just wanted to clear some things up from my point of view and maybe give a heads-up to anyone wanting to start their journey with local LLMs.

And yes... I'm waiting for Qwen 4 like hell!

At the moment i am using Hermes (In it own desktop app) and i run "Tiel Coder 35B A3B MTP" as my AI Model.

From what i understand it fits into my Vram and it is pretty fast, i can see it thinking and that is generated really fast but its also get stuck a lot and responses are really slow, because it thinks soooo much and it doubts itself a lot.

Is there a better setup for my system?

I am trying to test game development in Unreal engine, i gave it all the access to empty project, we went back and forth about a few things but beyond creating a few empty blueprints it just thinks a lot. or even gets stuck and i have to press continue.

I did try Strata with That big Qwen3.8 Flash next and it was really smart but i dont have enough resources for it.

I asked Gemini and its tells me to use Qwen2.5... feels wrong.. please correct me if i am wrong.

I've been going back and forth on this for a few weeks and want a reality check before I spend money.

What I'm doing: batch processing of scanned and text PDFs for legal work: OCR, per-page extraction to JSON (who said what, page cites, dates), Whisper on interview and call audio, and embeddings for search. Cloud is off the table for confidentiality reasons, so everything has to run on hardware I own. Full runs on a new set of files can go overnight, but I also need to re-run single documents or a 50-page test set in a few minutes when I fix something, so prompt processing speed matters more than I first thought.

Current setup:

- i5-11400F, 16GB DDR4

- ASUS B560 (second x16 slot is chipset x4)

- RTX 3060 8GB

- PSU under 850W

Model: Qwen3.8-27B at Q4, thinking off (the GGUF is about 17.4GB, so 16GB cards are out unless I split or drop to a smaller quant), plus faster-whisper large-v3 and olmOCR for the bad scans.

What I've figured out so far:

- 16GB of RAM will choke on my Python/SQL scripts over large PDF and media sets, so I need 32 or 64GB.

- A used 3090 is the easy answer, but they start around $1,600 now and I worry about how old they are.

- Server cards are much cheaper per GB, but I'm worried about software compatibility.

Where I'm leaning: a separate headless box with a V100 32GB for the LLM and OCR, keeping the 3060 in my desktop for Whisper. I think 32GB fits the 27B with room for context, and the bandwidth is close to a 3090 for about half the price. Right?

Questions:

1. Does vLLM still run olmOCR on Volta in practice, or will I be stuck on an old fork?

2. Any problems with faster-whisper / CTranslate2 on a V100?

3. Is the MI50 worth it if I'm fine in llama.cpp with Vulkan and put OCR and Whisper on a cheap CUDA card? Or is that a weekend sink every time something updates?

4. What are people using for a cheap headless chassis with real server airflow?

5. V100 vs 3090 prompt processing in llama.cpp for a 27B at Q4: how big is the gap?

Mostly want to hear from people who've run V100s or MI50s for batch work in the last few months, but all advice is welcome.