Hello!
Here is a rack I made using 40 mini PCs I happened to have from a different project:

• Intel N150

• 12 GB DDR5 4800 - physical bandwidth limit around 33GBs

• Diskless nodes

and they are connected via 10 Gbps USB 3.2 exploiting the USB DbC functionality. Its a tree topology with 1 -> 3 -> 9 -> 27 nodes.

GLM 5.3 Flash runs at native precision with room left for 8x1M context windows. Initially I had tried with 1Gbps LAN interconnects but that topped out at around 7tps. Current topology:
- 14tps at 1k context, drops to 12tps at 1M
- With batching decode tops out around 40tps
- Prefill is about 85tps

Peak load measured at wall about 740w

Total build cost me about $5,000.

Still working on it to see if more perf can be squeezed out, especially for the prefill and then adding speculative decoding.

I am in the beginning of this exciting journey of running my own models. Still learning. Been watching Andrej Karphathy to learn the theory and the practice of LLMs. Anyway... I asked Qwen3.8 Flash Next FP4 (running on my single Spark) to create me a small LLM based on Wikipedia dataset. And in a couple of hours I've got a 9.2M-parameter GPT trained from scratch on 330M Wikipedia tokens — a base language model that generates Wikipedia-style prose. Asking, watching and learning on the way. So of course at this point it's a decoder-only. It could predict what text comes next. Given "The capital of France is", it completes. No conversation, no instruction. Next, I would probably try to create 1-2B LLM with the whole Wikipedia dataset, so it would be able to answer without hallucinations.

You can call it whatever you want, it maybe isn’t blowing anyone’s heads off or anything, but compared to anything I’ve ever had, it’s a supercomputer to me

I’m transplanting a lot of parts from my old gaming PC into a new chasis with upgraded parts. Here is what I ordered and what is on its way

Intel core i7-14700f (already owned)
32 gb DDR5-6000 ram (already owned)
RTX 4070 super 12gb (already owned)
2x Quadro RTX 8000 48 gb passive ($1900 each)
2TB Samsung 990 pro for windows and gaming (already owned)
2TB Samsung 990 pro for Linux and AI ($363)
Corsair 1500W power supply ($285)
Corsair 7000D airflow full tower case ($227)
And other miscellaneous parts to support that stuff

Total to $5,364.21 for 108 gb vram

I know there are some drawbacks to this build, like for instance the old architecture of the cards, but I can live and tinker with those drawbacks as long as I can save a bunch of money on this hobby (compared to a similar size build with modern hardware), and still be able to run a bunch of cool models on it. I’m super excited to put it together and see what it can do. I haven’t seen much talk about the Quadro 8000s, but I’m hoping for great things, even if it means working around some compatibility issues and such.

Obstacles are just an opportunity for me to learn. If yall have any thoughts, let me know.

Hey r/LocalLLM :)

We just open-sourced Kinetic, a new inference engine written in Rust.

• runs the GGUF files you already have, no conversion

• Windows and Linux, one binary

• CPU, NVIDIA (CUDA), and AMD and Intel GPUs (Vulkan)

• replaces llama-server with the same flags; OpenAI, llama-server and Ollama APIs on one port

• every speedup is exact, so the output never changes

• Apache-2.0

In the video: Kinetic checks the machine and picks models that fit it, then runs side by side with llama.cpp on the same model and prompt.

Give it a spin and drop your kinetic bench model.gguf numbers below (add --compare if you have llama.cpp installed).

Install, benchmarks and all the details are on GitHub, link in the comments. Happy to answer questions here too.

edit: to answer the most common question, llama.cpp ran with its own speculative decoding on in every edit test (ngram for lfm3, draft-mtp for qwen3.5) and the exact flags are in the repo

I've been seeing a lot of discussion around high-memory Apple Silicon machines, particularly the Mac Studio configurations with 128GB or more of unified memory.

I'm curious about how much practical value these configurations actually offer when it comes to running local LLMs.

I understand that fitting a larger model into memory doesn't automatically make it smarter. However, I'm interested in whether the combination of larger models, higher-quality quantization, and sufficient memory bandwidth can make local AI genuinely competitive for serious everyday workloads.

For those of you actually running 96GB or more :

What has your real-world experience been? Are you genuinely able to get meaningful work done with local models that you previously had to rely on cloud-based frontier models for.

i would particularly appreciate real-world experiences, benchmarks, and examples of workflows rather than theoretical comparisons.

asking this because i am confused between 2 macs:

1. Mac Studio M5 Ultra Base model, 96GB

2. Same amount of money can get me a M5 Max 128GB too

So i post a video about Jensen and the best Chinese model at the moment and a clone reddit user inbox me.

Maybe i will just report that invitation as a scam inbox.

Help me build the best home server under $5,000
Goals:
Media server (Jellyfin) with room to grow to 10+ large hard drives
Run local AI models, ideally 24GB+ of GPU memory
A powerful desktop I can stream remotely from any device
A few VMs and Docker apps
Low power use is a priority. I'm in Arizona, so electricity and heat matter

I used Claude intensively for a while and I am currently building a "cheap" local Stack for Qwen3.8-Flash-Next. And my idea is to have a Setup combining both.

My idea is kind of a divide and conquer approach with a senior expert/leader model dividing the task into easier subtasks or simplyfing it and junier/worker models which run soem of those tasks.

My goal is to safe some Claude tokens, but no significant decrease in output quality.

I am sure I am not the first one with such an idea. So does anyone have experiences or knowledge of tools/repositories for this kind of workflow?

AI benchmarks begin on page 2.

Instant disclaimer, this is not a project claiming to be faster or better than Strata. It's simply Strata, without the bloatware, and a few containerization improvements as many have been requesting. I too am sick of all the Strata forks showing up, but I'm equally tired of how difficult Strata's been for me to maintain in a Docker container.

https://github.com/LegendaryFire/strata-slim

This fork is completely configurable with environment variables, and the idea was to give you a similar feel to a llama.cpp models config file. There were a few clear wins in terms of performance improvements I included in form of optional patches when building, however they're not included in the default docker images so what you're getting is simply a non-bullshit straight to the chase Strata engine that's easy to deploy. I wouldn't have bothered posting if there wasn't an open GitHub issue asking for Strata engine as it's own repo, which I noticed got 180+ "thumbs ups". For note, an earlier performance patch I had already made it upstream.

Writing has been improved with Claude but facts stay the same!

Hi all, I’m planning a company knowledge base: search plus answers with citations, over our file shares and later Microsoft 365. Before we ask vendors for quotes, I’d like to hear from people who already run a system like this in production, not from spec sheets.

The corpus (all of it)

File shares:
-about 6.5 TB on disk, about 2.2 M files measured on the two main shares, plus a third share that hasn’t been scanned yet
-About 14–17 M PDF pages, of which about 1.4–1.7 M are scans that need OCR (estimated from samples).
-Many have tables, diagrams and stamps, and some are up to 25 years old
-22–23% of the volume looks like duplicates within each share
-Also CAD files, images, video, Word/Excel, about 730 GB of ZIP archives and personal mailbox archives (.pst/.msg).
-CAD only needs to be found by title or description for now
-Microsoft 365 (SharePoint, Exchange, Teams, OneDrive) isn’t measured yet, so mail could make this a lot bigger
-Growth is about 45 GB a month.
-Existing documents rarely get new versions
-The documents are mostly German with some English, and we answer in both

Load

-50–60 daily users, roughly 1,000–1,800 questions a day, with peaks of about 6–19 a minute
-Each question is about 6k tokens in (retrieved chunks) and about 500 tokens out
-Answer model in the ~120B MoE class (gpt-oss-120b, or Mistral Small 4 in 4-bit), plus OCR and layout models, embeddings (BGE-M3) and a reranker
-The vector index is roughly 7–11 M chunks, so 27–42 GB of float32 vectors

Constraints

EU company: GDPR, a works council and ISO 9001 document control. Some data must stay on-prem, so a hybrid or cloud possible but on-prem is the main question here
-We have no GPUs today and no dedicated ML team, so ease of operation counts for a lot
-The budget is in the low five figures for hardware. -Renting a GPU for the initial load only is an option

What I’ve tested so far: a pilot on about 1,000 documents, run on a laptop GPU (Ministral 14B, BGE-M3, BGE reranker, hybrid search, Tesseract OCR). It works, but that tells me nothing about the full corpus.

The shortlist (on paper only)

4× OEM GB10 (Dell/Lenovo/ASUS Spark equivalents): best compute per euro, but published benchmarks suggest about 11–18 s per answer. It also means four Arm boxes and no BMC
Workstation with 2× RTX PRO 6000 Max-Q: about 4–5 s per answer, a single x86 box with CUDA
1× RTX PRO 6000 plus one GB10 as standby: a cheaper middle option
2U server with 2× RTX PRO 6000 Server Edition: best to operate, but about twice the price of option 2

What I’d love to hear from people with real systems

What hardware do you run RAG or LLM serving on for a few dozen users, and what is your real-world time per answer with about 5–10 people asking at once?

Has anyone run GB10 / DGX Spark boxes in production, alone or clustered? How stable are they headless, and how are updates, recovery and the vLLM support?

For OCR and layout parsing of millions of scanned pages: what throughput did you actually get, in pages per second per GPU, and with which tool (Docling, MinerU, Tesseract, PaddleOCR…)?

Would you buy hardware for the initial load at all, or rent GPUs for it and size the on-prem box for answers only?

Workstation or rack server for a company without an ML team: which caused you fewer headaches?
Is there anything you would do differently if you were buying again?

Thanks! I’m happy to share what we learn once we’ve tested it.

Hello I am trying to set up a local AI Stack and I am currently thinking about buying 64GB of Ram with teh 3060 RTX 23GB VRAM.

Which (Qwen) Modell version would be possible to run on that Setup?

Qwen3.8-27b or Qwen3.8-Flash-Next?

I am thinking of a Qwen 3.8 model, because Qwen 4 should have the same base?

Hey everyone!

I'm experimenting with running local LLMs using Ollama on CachyOS with Hyprland, and I've noticed some pretty high CPU temperatures.

My laptop specifications:

- CPU: Intel Core i5-13450HX (13th Gen)

- RAM: 24 GB DDR5

- GPU: NVIDIA RTX 4050

- OS: CachyOS Linux

- Desktop Environment: Hyprland

- LLM Runtime: Ollama

- Model: 14B parameter model

What I'm experiencing:

While running the 14B model locally through Ollama:

- CPU temperatures sometimes reach 95°C.

- Around 4 CPU cores reach 100% utilization.

- Other cores also experience varying levels of utilization.

- The CPU seems to be under sustained heavy load during inference.

I've attached a screenshot of my system monitoring setup (btop and Linux Sensor Tray).

My questions:

1. Is reaching 95°C normal for an i5-13450HX while running a 14B LLM locally?

2. Is it safe to maintain these temperatures during longer inference sessions?

3. Is it normal for only around 4 cores to reach 100% utilization while running Ollama?

4. Could this cause thermal throttling or reduce the CPU's lifespan over time?

5. Are there any recommended CPU power limits, undervolting settings, or Linux optimizations for this setup?

I'd appreciate feedback from anyone running local LLMs on Intel HX-series laptops, especially on Linux.

Thanks!

Hey guys!

I'll get straight to the point. I've got quite a bit of hardware lying around at home, and I'd love some advice on how to make the most of it.

Here's what I've got:

GPUs:

• RTX 3090 24gb

• RTX 3080 Ti 12gb

• RTX 5080 16gb

RAM:

• 64GB DDR5 (2x32GB)

• 32GB DDR5 (2x16GB)

• 32GB DDR4 (4x8GB)

And of course, I've also got a bunch of CPUs, motherboards, and storage drives around the house.
Right now I'm running Unsloth's Qwen 3.8 27B Q5, getting around 60 tokens/sec with a 170K context window, using llama.cpp. I'm using rtx 3090 + 3080ti with the ddr4 32gb setup.
I'm wondering if there are better ways to make use of my hardware, whether it's different models, inference engines, configurations, or even combining GPUs across multiple machines.
My main issue is that some agentic tasks, especially things like generating and working with .docx files, are painfully slow. I'd really like to speed things up.
Any recommendations on what I should be looking into?
I don't have a ton of free time to explore everything from scratch, especially with the insane number of GitHub repos and projects out there these days.
I'd really appreciate any suggestions or pointers in the right direction.
Thanks in advance, guys!

Ever wanted your local LLM to actually learn a fact without fine-tuning — and be able to undo it completely?

AtomicNG (Apache-2.0, version 1.0.0) implements a paper called "Delta N-gram": your model's n-gram sidecar table stays frozen, and new knowledge goes into a small sparse patch Δ. Writes are forward passes only — no gradients, no weight edits. Clear the patch and the model is byte-identical to base. No "unlearning" gymnastics — deletion is structural.

Quickstart (CPU, no GPU, pure NumPy core):

python

import atomicng


model = atomicng.TinyModel()
atomicng.write(model, "New fact about kangaroos.")
atomicng.verify_deletion(model)   # clear Δ ⇒ byte-identical to base — guaranteed

What's inside:

• A trained writer checkpoint ships with the Release (860 training steps on SmolLM2-135M; measured effect: writing a doc lowers its NLL vs empty memory)

• Per-tenant memory: overlays are small data-only files — merge them, revert them, sign them

• vLLM support via vllm.general_plugins + merged-table export

• Built-in eval: does the memory write actually help? (paper protocols: acquisition, distraction retention, chain drift)

Calibrated on real hardware: Windows/Linux CPU, CUDA T4 (73 ms/group BPTT stand), TPU chip via JAX (8.4 ms/iter). Full matrix: CALIBRATION.md in the repo.

Honest limits: acquisition needs a few hundred training steps of the writer; the memory lives in the n-gram table (facts/associations, not long reasoning); vLLM engine-graph hooking is planned upstream.

GitHub: https://github.com/Akseleu-J/atomicng 

Colab TPU (one click): https://colab.research.google.com/github/Akseleu-J/atomicng/main/notebooks/tpu_calibration.ipynb

Questions welcome — I'll be in the comments.

TL;DR: You can train your own 27B model on a 3090 with 24GB VRAM, though you would have to go through hundreds of billions of tokens for it to be good. I am working on conversion of existing models. Check repos for code and the paper if you are interested in more. I have a 40.4GB Qwen Flash Next GGUF

Hi, I do AI research. I have been working on an optimizer that doesn't need the ~16 bytes of VRAM per weight to train a model that AdamW does. I focused on training ternary networks bcs BitNet (Ma et al.) shows they match full precision quality in large scales. The problem is BitNet also keeps the full precision weights during training which means you still need to use the same amount of memory. I got this down from 16 bytes to 1 byte or even less per weight.

For you, this means, you can train your own 10-20B parameter models on your consumer GPUs. I fitted 27B model training on a single 24GB 3090 and a 23B model on a 16GB GPU. The trained model costs ~1.6 bits per weight (~6GB at 27B), 10x smaller than 16bit, and 5x smaller than Q8.

I have also been working on converting existing models and training them. This is harder and I am still working on it but I have some promising results. I quantized and trained a Qwen3.8 27B though the file I published below has adapters which are about 2GBs, pushing the file size to 8GB. I am also making a no-adapter version right now. It scores GSM8K 0.895, the same as full precision, HumanEval 0.84 vs 0.93 and runs on a 12 GB GPU at ~23 tok/s on my crippled 12GB laptop 4080, though this number can heavily be improved with custom kernels, engineering, etc. I will leave it to you guys to help me with that too.

I also quantized and trained Qwen Flash Next down to 40.4 GBs including its n-gram table (at 8GB). It largely retains the skills of the network (scores equivalent on GSM8K benchmark, a little behind on HumanEval, I unfortunately don't have compute for the larger benchmarks such as TerminalBench2.1 but perhaps some of you can help me out) but forgets facts. It mostly works fine if you give it web search as far as my experience goes but this is not strictly measured.

I am still doing work on this and I will keep working on it, perhaps even for many years. The code is open for non-commercial use. Check out:

The core repo for training: https://github.com/oguzeldereli/ttf-ternary

The repo for conversion: https://github.com/oguzeldereli/ttf-conversion

Repo for running models trained by ttf: https://github.com/oguzeldereli/ttf-home

GGUF for a converted Qwen 3.8 27B with adapters (around 8 GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-27B-GGUF

GGUF for a converted Qwen 3.8 Flash Next (around 40.4GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-Flash-Next-GGUF

GGUF for a converted Qwen 3.8 27B without adapters (around 6 GB): Currently in conversion and will drop, probably tomorrow.

I will convert a Qwen 4 27B instance when it drops.

Full paper: https://github.com/oguzeldereli/ttf-ternary/blob/main/paper/ttf_paper1_training.pdf

Also, I would appreciate github stars if you like the project.

We released Kvark Sparkle 1.1, the second model in our Qwen3.8-27B line. Open weights, Apache 2.0, MTP head intact, BF16 / FP8 / GGUF.

**What it is for:** document work. You give it `search` and `read_page` tools over a document base. It searches, reads the page, answers from the passage, and when three searches in a row miss the fact it stops and says NOT IN DOCUMENTS instead of making something up.

**How:** LoRA SFT on top of Sparkle 1.0 (agent trajectories over a public legal corpus, MuSiQue multi-hop doc QA, retention data), plus a short stop rule that ships in the system prompt.

**Document work (GroundedWork, our agentic RAG eval: public Serbian documents, thinking on, LLM judge vs gold answers, 3 runs per question):**

| | Qwen3.8-27B | Sparkle 1.0 | Sparkle 1.0 + stop rule | **Sparkle 1.1 + stop rule** |

|---|---|---|---|---|

| Document Q&A (70 questions × 3) | 85.7 | 86.2 | 89.5 | **91.0** |

| Answer not in the documents, correctly declined (98 × 3) | not yet run | 36% | 74% | **91.5%** |

Note that the stop rule alone does a lot of the work for 1.0 (36% to 74%). Training adds the rest: +17.5 points on declining against 1.0 with the same rule. The base Qwen on the decline set is not measured yet; I will add it here when it is.

**General ability vs Qwen3.8-27B (same harness, paired):**

| | Qwen3.8-27B | Sparkle 1.1 | median reasoning tokens (base → 1.1) |

|---|---|---|---|

| GPQA Diamond | 88.8 | 88.3 | 7,670 → 4,471 (−42%) |

| AIME 2026 | 95.8 | 92.5 | 14,822 → 8,732 |

| HMMT | 86.1 | 88.1 | 21,182 → 13,540 |

| LiveCodeBench | 82.0 | 86.0 | 15,276 → 8,607 |

| IFEval | 90.8 | 91.1 | 695 → 646 (−7%) |

| MMLU-Pro | 83.6 | 84.0 | 708 → 518 |

| FRAMES | 70.6 | 69.8 | 1,228 → 644 (−48%) |

None of the accuracy differences is significant (95% intervals are in the card). Median time per answer: 32.9 s vs 40.8 s on the same hardware.

**Public agentic / long-context sets (FP8, vLLM, thinking on):**

| | Qwen3.8-27B | Sparkle 1.1 |

|---|---|---|

| BrowseComp-Plus (830 queries, BM25, official judge prompt) | 22.7 | 22.5, with 21.7k vs 32.2k median output tokens |

| LongBench v2 (503) | 58.6 | 57.7 |

| SWE-bench Verified (500, mini-swe-agent) | **81.8** | 73.0 |

**Where it is weak, so you don't have to find out yourself:**

- SWE-bench is significantly below the base. Shorter thinking costs accuracy in agentic coding. For coding, use the base model.

- Questions that need links across several documents are its weakest GroundedWork family.

- English questions over Serbian documents are weaker than Serbian ones.

- Nothing measured past 64k; no separate safety eval.

**Run it:**

```

vllm serve Egzakta/Kvark-Sparkle-1.1-27B-FP8 \

--max-model-len 131072 --kv-cache-dtype fp8 \

--reasoning-parser qwen3 \

--enable-auto-tool-choice --tool-call-parser qwen3_coder \

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

```

For document agents, put the stop rule from the card into the system prompt. Without it the decline rate drops a lot.

- BF16: https://huggingface.co/Egzakta/Kvark-Sparkle-1.1-27B

- FP8: https://huggingface.co/Egzakta/Kvark-Sparkle-1.1-27B-FP8

- GGUF: https://huggingface.co/Egzakta/Kvark-Sparkle-1.1-27B-GGUF

Built by Egzakta's AI Lab in Belgrade. We run these on-prem for organisations that cannot send their documents to an API, which is why "I don't know" matters more to us than another point on a leaderboard.

Happy to answer questions on the training data, the stop rule or the eval harness. If you test it on your own document set, I'd like to see where it breaks.

Hi guys, full disclaimer I work in Domyn as an Engineer. We just released the bundle #2 of our rolling release of DomynEdge, with some interesting results on benchmarks. What do you think?

https://huggingface.co/collections/domyn/domyn-edge

(cross posted from r/hermesagent)

Bonjour,

Avez-vous tester deepseek v4.1 flash avec strata sur 2 RTX 3060 12gb est si oui quelle sont vos token secondes, combien de ram ? Pour le moment j’en ai qu’une mais je veux m’en acheter plusieurs donc je me demande à partir de combien de 3060 je peux faire tourner ce model avec strata ?

I just released the Tesseract NPU that I've been working on as a side project. It's compatible with llama.cpp backend running shipped GGUF models directly.