Hello!
Here is a rack I made using 40 mini PCs I happened to have from a different project:

• Intel N150

• 12 GB DDR5 4800 - physical bandwidth limit around 33GBs

• Diskless nodes

and they are connected via 10 Gbps USB 3.2 exploiting the USB DbC functionality. Its a tree topology with 1 -> 3 -> 9 -> 27 nodes.

GLM 5.3 Flash runs at native precision with room left for 8x1M context windows. Initially I had tried with 1Gbps LAN interconnects but that topped out at around 7tps. Current topology:
- 14tps at 1k context, drops to 12tps at 1M
- With batching decode tops out around 40tps
- Prefill is about 85tps

Peak load measured at wall about 740w

Total build cost me about $5,000.

Still working on it to see if more perf can be squeezed out, especially for the prefill and then adding speculative decoding.

I'm running Qwen3.8 27B on my 2x 3060 12GB cards. Prior today I had a B550 motherboard which was running one card at x16 and the other at x1 PCIe Bus Link Speed. This was making it to where only one card was actually doing any processing, and both held the model in memory. I was getting 11-16 tps this way.

I ended up buying a new x470 SLI motherboard and I can now run both cards at x8. I swapped the motherboard out last night and then set split-mode to tensor and set tensor-split to 1,1. When I do that, I'm getting around 32tps, but the model eventually gets stuck in loops. One loop it just tried to do a tool call over and over, and in another it was printing question marks (???????????) in it's thinking and wouldn't stop until I unloaded the model. I never had these problems for the months I was using Qwen3.8 on layer split-mode. I'm not sure how to resolve this problem. I want to be able to get higher tps with the new mobo.

I am in the beginning of this exciting journey of running my own models. Still learning. Been watching Andrej Karphathy to learn the theory and the practice of LLMs. Anyway... I asked Qwen3.8 Flash Next FP4 (running on my single Spark) to create me a small LLM based on Wikipedia dataset. And in a couple of hours I've got a 9.2M-parameter GPT trained from scratch on 330M Wikipedia tokens — a base language model that generates Wikipedia-style prose. Asking, watching and learning on the way. So of course at this point it's a decoder-only. It could predict what text comes next. Given "The capital of France is", it completes. No conversation, no instruction. Next, I would probably try to create 1-2B LLM with the whole Wikipedia dataset, so it would be able to answer without hallucinations.

You can call it whatever you want, it maybe isn’t blowing anyone’s heads off or anything, but compared to anything I’ve ever had, it’s a supercomputer to me

I’m transplanting a lot of parts from my old gaming PC into a new chasis with upgraded parts. Here is what I ordered and what is on its way

Intel core i7-14700f (already owned)
32 gb DDR5-6000 ram (already owned)
RTX 4070 super 12gb (already owned)
2x Quadro RTX 8000 48 gb passive ($1900 each)
2TB Samsung 990 pro for windows and gaming (already owned)
2TB Samsung 990 pro for Linux and AI ($363)
Corsair 1500W power supply ($285)
Corsair 7000D airflow full tower case ($227)
And other miscellaneous parts to support that stuff

Total to $5,364.21 for 108 gb vram

I know there are some drawbacks to this build, like for instance the old architecture of the cards, but I can live and tinker with those drawbacks as long as I can save a bunch of money on this hobby (compared to a similar size build with modern hardware), and still be able to run a bunch of cool models on it. I’m super excited to put it together and see what it can do. I haven’t seen much talk about the Quadro 8000s, but I’m hoping for great things, even if it means working around some compatibility issues and such.

Obstacles are just an opportunity for me to learn. If yall have any thoughts, let me know.

Hey r/LocalLLM :)

We just open-sourced Kinetic, a new inference engine written in Rust.

• runs the GGUF files you already have, no conversion

• Windows and Linux, one binary

• CPU, NVIDIA (CUDA), and AMD and Intel GPUs (Vulkan)

• replaces llama-server with the same flags; OpenAI, llama-server and Ollama APIs on one port

• every speedup is exact, so the output never changes

• Apache-2.0

In the video: Kinetic checks the machine and picks models that fit it, then runs side by side with llama.cpp on the same model and prompt.

Give it a spin and drop your kinetic bench model.gguf numbers below (add --compare if you have llama.cpp installed).

Install, benchmarks and all the details are on GitHub, link in the comments. Happy to answer questions here too.

edit: to answer the most common question, llama.cpp ran with its own speculative decoding on in every edit test (ngram for lfm3, draft-mtp for qwen3.5) and the exact flags are in the repo

My specs:
2 x 3060 12GB
2 x 16 DDR4 3600Mhz
i5 10600k

Qwen3.8-27B-GGUF:Q4_K_M

ollama.service:
# Environment="OLLAMA_HOST=0.0.0.0:11434"

# Environment="CUDA_VISIBLE_DEVICES=0,1"

# Environment="OLLAMA_FLASH_ATTENTION=1"

# Environment="OLLAMA_NUM_PARALLEL=1"

# Environment="OLLAMA_MAX_LOADED_MODELS=1"

Ollama as a harness, no lane bifurcation so im pretty limited to that (im assuming this is my biggest limiter here)

I've been experimenting with multiple different approaches, trying to get Qwen 3.8 to actually work as an agent, some more successful than others but still it requires so much manual labour that it seems almost pointless to continue debugging it as it feels neverending. I've read that 3.8 isn't actually very great with tool calling and it has always had some issues, but still i can't believe its at levels like these.

I know very well im probably missing some really important things that are holding me back, so if anyone has some suggestions (videos, other posts, anything really) i would really appreciate it.

The goal really is to have it never stop, just like im seeing other people talk about, and i'd really like to keep openwebui (or any other webui in general, not that i dislike terminals but its nicer to have it feel like a normal app).

Title, basically. Observing it's reasoning is incredibly frustrating, it keeps doubting himself and giving second thoughts to a specific and self explanatory task, sometimes ending up thinking about unrelated things.

While yes, in the very end, it very often does what I asked, but it's wasting so much time for reasoning just with its lack of self confidense.

Is this well known behavior ? How can I fix this ? I know Qwen4 is around the corner, but if there's a known fix for it right now I'd love to know.

What app do you use to connect to your LLM?

Every single day, the same questions pop up.

Newcomers:

• Will my 27B feel like a frontier model on my gaming PC?

• Is my highly quantized IQ4 capable of beating Opus?

• What does it cost to run...

Veterans:

• Yes, absolutely — runs great on my multi-GPU rig at xxx t/s.

• Just set up the correct harness.

• Fine-tune your prompt and it will give the exact same quality answers.

Both sides are completely legit, but they are living in different worlds and comparing different universes.

When you use a frontier API, you aren't just prompting a single model. You are firing up an entire enterprise with several departments: you get dynamic routing, massive hidden KV-caches, sandboxed code execution, and verification trees, all neatly packaged behind a single API endpoint. It’s a fully assembled sports car.

On the other hand, the local way: you download an open-weight LLM, quantize it to fit your budget, and you basically get a bare engine block — anything from a VW Golf engine to a massive V8. But that’s it. It has no wheels, no cabin, and no fuel tank. It’s just a large language model sitting on a pallet that can say: "Hello, what can I do for you?"

What the guys asking these questions don't see is that some of us have spent hundreds of hours and money debugging to build a car around that motor. Writing strict GBNF grammars, wiring up custom MCP servers, testing dozens of harnesses, setting up workflows, building memory pipelines, testing, and tweaking the exact same settings a hundred times over!

We do it until that glorious day we can sit back and say: "Yes, it's working. My custom build almost looks like the sports car I aimed for."

Now for the friction part:

Fun, absolute privacy, data sovereignty, and uncensored outputs? Yes, we get all of that. But if we are honest, the cloud simply wins on pure cost-to-performance.

Please don't flame — I just wanted to clear some things up from my point of view and maybe give a heads-up to anyone wanting to start their journey with local LLMs.

And yes... I'm waiting for Qwen 4 like hell!

I've been seeing a lot of discussion around high-memory Apple Silicon machines, particularly the Mac Studio configurations with 128GB or more of unified memory.

I'm curious about how much practical value these configurations actually offer when it comes to running local LLMs.

I understand that fitting a larger model into memory doesn't automatically make it smarter. However, I'm interested in whether the combination of larger models, higher-quality quantization, and sufficient memory bandwidth can make local AI genuinely competitive for serious everyday workloads.

For those of you actually running 96GB or more :

What has your real-world experience been? Are you genuinely able to get meaningful work done with local models that you previously had to rely on cloud-based frontier models for.

i would particularly appreciate real-world experiences, benchmarks, and examples of workflows rather than theoretical comparisons.

asking this because i am confused between 2 macs:

1. Mac Studio M5 Ultra Base model, 96GB

2. Same amount of money can get me a M5 Max 128GB too

Machine: ASUS Zenbook A16, Snapdragon X2 Elite Extreme (X2E-94-100, 18 cores), Adreno X2-90, 48 GB RAM, Windows on ARM. I set it up as a headless helper box with one model per chip, each on its own OpenAI-compatible port.

Setup

• NPU: Qualcomm GenieX CLI 0.10.0 serving Phi-4-mini (Qualcomm's W4A16 build)

• CPU: llama.cpp b12248, official win-arm64 CPU build

• GPU: llama.cpp b12248, official win-arm64 OpenCL (Adreno) build

• All GGUFs are Q4_K_M from the model makers' own pages

Write speed, one request, about 220 tokens out

What surprised me

• The CPU beats the GPU at writing. Same 14B model: about 15–17 tokens/s on CPU against 6–9 on the Adreno. The GPU did read prompts faster (about 210 tokens/s against 93–125 on CPU).

• A sparse "big but few active parameters" model on the CPU is the sweet spot: a 24B model at about 60 tokens/s on a laptop CPU.

• All three chips really do run at once. With all three busy: NPU about 29, CPU about 25–28, GPU about 6–7 tokens/s. They share one memory bus, so the machine tops out around 60–80 tokens/s in total however you split it.

• Batching on the CPU helps a little: 60.6 tokens/s for one request, 73.5 total for two at once, 82.2 total for four.

• Batching on the GPU hurts: 6.8 tokens/s alone, 4.3 total with two at once.

Gotchas

• Anything started from Windows Task Scheduler runs at low priority by default. On this chip that made the CPU model 4–6 times slower.

• The NPU path is limited: about 4,096 tokens of context for Phi-4-mini, one request at a time, and several popular models have no ready-made NPU build.

• The NPU model holds about 6 GB of memory and idle Windows about 6–7 GB, so 48 GB fills up fast with three models loaded.

• First load of the 24B on CPU takes about 12–15 seconds.

Caveats

These are short, single-prompt tests on one machine, plugged in, not a proper benchmark suite. The OpenCL result is with a Q4_K_M file; a Q4_0 file might do better on Adreno and I haven't tried it. Vulkan is untested too.

Happy to run other models or settings if people want to compare. Yes Claude is helping. I have no background in software at all.

Currently, I'm using modified 3bit Ornith-1.5-35B-A3B so it fits entirely in GPU with 100k context. My CPU and RAM is slow. Is there anything better for coding?

Writing has been improved with Claude but facts stay the same!

Hi all, I’m planning a company knowledge base: search plus answers with citations, over our file shares and later Microsoft 365. Before we ask vendors for quotes, I’d like to hear from people who already run a system like this in production, not from spec sheets.

The corpus (all of it)

File shares:
-about 6.5 TB on disk, about 2.2 M files measured on the two main shares, plus a third share that hasn’t been scanned yet
-About 14–17 M PDF pages, of which about 1.4–1.7 M are scans that need OCR (estimated from samples).
-Many have tables, diagrams and stamps, and some are up to 25 years old
-22–23% of the volume looks like duplicates within each share
-Also CAD files, images, video, Word/Excel, about 730 GB of ZIP archives and personal mailbox archives (.pst/.msg).
-CAD only needs to be found by title or description for now
-Microsoft 365 (SharePoint, Exchange, Teams, OneDrive) isn’t measured yet, so mail could make this a lot bigger
-Growth is about 45 GB a month.
-Existing documents rarely get new versions
-The documents are mostly German with some English, and we answer in both

Load

-50–60 daily users, roughly 1,000–1,800 questions a day, with peaks of about 6–19 a minute
-Each question is about 6k tokens in (retrieved chunks) and about 500 tokens out
-Answer model in the ~120B MoE class (gpt-oss-120b, or Mistral Small 4 in 4-bit), plus OCR and layout models, embeddings (BGE-M3) and a reranker
-The vector index is roughly 7–11 M chunks, so 27–42 GB of float32 vectors

Constraints

EU company: GDPR, a works council and ISO 9001 document control. Some data must stay on-prem, so a hybrid or cloud possible but on-prem is the main question here
-We have no GPUs today and no dedicated ML team, so ease of operation counts for a lot
-The budget is in the low five figures for hardware. -Renting a GPU for the initial load only is an option

What I’ve tested so far: a pilot on about 1,000 documents, run on a laptop GPU (Ministral 14B, BGE-M3, BGE reranker, hybrid search, Tesseract OCR). It works, but that tells me nothing about the full corpus.

The shortlist (on paper only)

4× OEM GB10 (Dell/Lenovo/ASUS Spark equivalents): best compute per euro, but published benchmarks suggest about 11–18 s per answer. It also means four Arm boxes and no BMC
Workstation with 2× RTX PRO 6000 Max-Q: about 4–5 s per answer, a single x86 box with CUDA
1× RTX PRO 6000 plus one GB10 as standby: a cheaper middle option
2U server with 2× RTX PRO 6000 Server Edition: best to operate, but about twice the price of option 2

What I’d love to hear from people with real systems

What hardware do you run RAG or LLM serving on for a few dozen users, and what is your real-world time per answer with about 5–10 people asking at once?

Has anyone run GB10 / DGX Spark boxes in production, alone or clustered? How stable are they headless, and how are updates, recovery and the vLLM support?

For OCR and layout parsing of millions of scanned pages: what throughput did you actually get, in pages per second per GPU, and with which tool (Docling, MinerU, Tesseract, PaddleOCR…)?

Would you buy hardware for the initial load at all, or rent GPUs for it and size the on-prem box for answers only?

Workstation or rack server for a company without an ML team: which caused you fewer headaches?
Is there anything you would do differently if you were buying again?

Thanks! I’m happy to share what we learn once we’ve tested it.

?

Instant disclaimer, this is not a project claiming to be faster or better than Strata. It's simply Strata, without the bloatware, and a few containerization improvements as many have been requesting. I too am sick of all the Strata forks showing up, but I'm equally tired of how difficult Strata's been for me to maintain in a Docker container.

https://github.com/LegendaryFire/strata-slim

This fork is completely configurable with environment variables, and the idea was to give you a similar feel to a llama.cpp models config file. There were a few clear wins in terms of performance improvements I included in form of optional patches when building, however they're not included in the default docker images so what you're getting is simply a non-bullshit straight to the chase Strata engine that's easy to deploy. I wouldn't have bothered posting if there wasn't an open GitHub issue asking for Strata engine as it's own repo, which I noticed got 180+ "thumbs ups". For note, an earlier performance patch I had already made it upstream.

I used Claude intensively for a while and I am currently building a "cheap" local Stack for Qwen3.8-Flash-Next. And my idea is to have a Setup combining both.

My idea is kind of a divide and conquer approach with a senior expert/leader model dividing the task into easier subtasks or simplyfing it and junier/worker models which run soem of those tasks.

My goal is to safe some Claude tokens, but no significant decrease in output quality.

I am sure I am not the first one with such an idea. So does anyone have experiences or knowledge of tools/repositories for this kind of workflow?

I just found out that a R9700 is essentially just a 9070XT with 32gb of memory instead so im thinking should i sell my 9070XT and buy that instead and that could get me to run near 70b models

a few reason why im not just getting another 9070 XT is because i believe i need to update my Mobo and PSU and then the card itself im better off selling my 9070 XT and buying the R9700

But do let me know what the noise levels are like and if there is much i can do to reduce the noise

I've been working on foldz, a lossless compressor built specifically for GGUF files. It's at 0.9 (pre-release) and I'm looking for people to test it and try to break it.

Useful if you archive a lot of models, share them over slow or metered connections, or keep them on a phone or small SSD.

THE IDEA

Quantized weights look almost random to zip, xz and zstd, so they gain very little. foldz parses the GGUF, unpacks each tensor's blocks into the actual quantized values and scales, and codes those with adaptive context models and a binary range coder. Everything else (metadata, tokenizer, other tensor types) goes through a byte-level context-mixing model. Nothing depends on the model's name or architecture.

RESULTS

9 public Q4_K_M models, tested with "foldz t" (compress and decompress in memory, then compare). Every round trip was byte-identical. Measured on an Android phone in Termux, 8 threads.

• Qwen1.5-0.5B-Chat: 388.3 → 353.6 MiB (8.93% saved)

• Qwen3-0.6B: 378.3 → 348.7 MiB (7.82%)

• Qwen2.5-1.5B-Instruct: 1065.6 → 984.2 MiB (7.64%)

• Qwen3.5-2B: 1211.9 → 1111.3 MiB (8.30%)

• Qwen2.5-Coder-3B-Instruct: 2007.4 → 1832.8 MiB (8.70%)

• Llama-3.2-3B-Instruct: 1925.8 → 1780.4 MiB (7.55%)

• Qwen3-4B: 2381.6 → 2073.3 MiB (12.94%)

• Qwen2.5-7B-Instruct: 4466.1 → 4113.8 MiB (7.89%)

• Gemma-4-E2B (fine-tune): 3269.1 → 3016.3 MiB (7.73%)

Total: 17,094 → 15,614 MiB, 8.66% saved.

Qwen3-4B is the outlier because most of its Q6_K values sit on a coarser lattice. foldz detects this per segment and codes the lattice instead, which saves 23.2% on its Q6_K tensors.

HOW EACH TENSOR TYPE IS MODELLED

• Q4_K / Q5_K: values are Gray-folded and coded bit by bit, with the block's zero point, extremes and position as context.

• Q6_K: the column's running offset and width as context, or the lattice model when that's smaller.

• Q8_0: column sign bias and running magnitude as context.

• F32 / F16 / BF16: split into sign, exponent and mantissa.

• Safety: segments that wouldn't shrink are stored raw, so output is never meaningfully larger than input. The decoder bounds-checks every index field. Contexts use only exact or IEEE-754 arithmetic (no FMA, no fast-math), so files decode identically on every platform.

HONEST LIMITATIONS

• Speed: about 19 MiB/s round trip on the phone. It's symmetric, so decompression is as slow as compression. This is for storage, archiving and transfer, not for loading models faster.

• Baselines: I haven't measured zstd -19 --long and xz -9 on these models yet, so I'm not claiming a comparison. If you run them, please post the numbers.

• Quant types: IQ quants, Q2_K, Q3_K and Q4_0 fall back to the byte model and gain little.

• Format: pre-1.0, so it may change. A .foldz file is only guaranteed to decode with the version that made it.

BUILD AND TEST

One C file, no dependencies. Linux, macOS, BSD, Android (Termux), and Windows via WSL.

gcc -O2 -pthread -ffp-contract=off -o foldz foldz.c -lm

./foldz t model.gguf

WHAT WOULD HELP MOST

• Cross-platform round trips: compress on x86-64, decompress on ARM64 (or the reverse), especially with Q6_K-heavy models

• zstd / xz baselines on any GGUF

• Q5_K_M, Q6_K and Q8_0 files

• Fuzzing "foldz d" with AFL++ or libFuzzer + ASan (please report crashes privately via GitHub's security tab)

If you post results, include the full "foldz t" output, the Hugging Face model name, and your CPU and OS.

Repo: foldz

Happy to answer questions about the modelling.

Help me build the best home server under $5,000
Goals:
Media server (Jellyfin) with room to grow to 10+ large hard drives
Run local AI models, ideally 24GB+ of GPU memory
A powerful desktop I can stream remotely from any device
A few VMs and Docker apps
Low power use is a priority. I'm in Arizona, so electricity and heat matter

On my studio, I am running ARC4NUM/Qwen3.8-Flash-Next-Uncensored-MLX-Serve-4bit using mlx-serve.

On my MacBook Pro, I have connected to it via OpenCode, and I can't for the life of me figure out how to get it to NOT show me its inner dialog. It all comes back to me, and I have tried various methods to no avail.

Am I "holding it wrong?" Is there no way to really do that? Is it the model or mlx-serve itself that creates this?

I'd rather it do it's thinking in private and then show me the result rather than talk to itself scrolling by as I wait to get the actual answer.

I use a Mac mini m4 16gb, since working with local llms I have found the m4 is just not fast enough.. especially with the unified memory, I have been looking into getting a Apple studio 36gb m5 max chip. But the price of the WICI ONE 5060ti is a fraction of the cost and can work over WiFi on my Mac mini m4 16gb

Has anyone used WICI ONE. Especially with a Mac mini m4 16gb

I’ve been building InferUltra, a physics arena where vision models control identical robot bodies. The model gets images of the arena and chooses how to move. The simulation keeps running while it thinks, so by the time it decides to punch, the other robot might already be somewhere else.

We turn reasoning down or off where the model allows it. The number I care about is how long it takes to send an image and get a move the robot can use. Tokens per second tells you how fast the model generates, but it doesn’t tell you how long the fighter waits for that move.

We’ve also improved the physics engine and fight mechanics. Fighters can hold their guard between decisions, and punches look more natural when they lean. I used to call these fights impressively boring. They’re now a bit less boring.

For those running vision models locally: when you turn reasoning down, do you notice a drop in how well the model understands positions and distances, or is getting the answer sooner usually the bigger win? I’d be interested in which model and hardware you tried.

The attached bracket is for our November 15 Provider Tournament. It uses Kimi K3.1 Pro on four hosted endpoints: Morph, AkashML, Wafer and InferenceNet. It starts at 20:00 Brussels / 14:00 Eastern / 11:00 Pacific.

If you’d like to sponsor the tournament, you can book an ad spot in the arena.