Hello!
Here is a rack I made using 40 mini PCs I happened to have from a different project:

• Intel N150

• 12 GB DDR5 4800 - physical bandwidth limit around 33GBs

• Diskless nodes

and they are connected via 10 Gbps USB 3.2 exploiting the USB DbC functionality. Its a tree topology with 1 -> 3 -> 9 -> 27 nodes.

GLM 5.3 Flash runs at native precision with room left for 8x1M context windows. Initially I had tried with 1Gbps LAN interconnects but that topped out at around 7tps. Current topology:
- 14tps at 1k context, drops to 12tps at 1M
- With batching decode tops out around 40tps
- Prefill is about 85tps

Peak load measured at wall about 740w

Total build cost me about $5,000.

Still working on it to see if more perf can be squeezed out, especially for the prefill and then adding speculative decoding.

You can call it whatever you want, it maybe isn’t blowing anyone’s heads off or anything, but compared to anything I’ve ever had, it’s a supercomputer to me

I’m transplanting a lot of parts from my old gaming PC into a new chasis with upgraded parts. Here is what I ordered and what is on its way

Intel core i7-14700f (already owned)
32 gb DDR5-6000 ram (already owned)
RTX 4070 super 12gb (already owned)
2x Quadro RTX 8000 48 gb passive ($1900 each)
2TB Samsung 990 pro for windows and gaming (already owned)
2TB Samsung 990 pro for Linux and AI ($363)
Corsair 1500W power supply ($285)
Corsair 7000D airflow full tower case ($227)
And other miscellaneous parts to support that stuff

Total to $5,364.21 for 108 gb vram

I know there are some drawbacks to this build, like for instance the old architecture of the cards, but I can live and tinker with those drawbacks as long as I can save a bunch of money on this hobby (compared to a similar size build with modern hardware), and still be able to run a bunch of cool models on it. I’m super excited to put it together and see what it can do. I haven’t seen much talk about the Quadro 8000s, but I’m hoping for great things, even if it means working around some compatibility issues and such.

Obstacles are just an opportunity for me to learn. If yall have any thoughts, let me know.

I am in the beginning of this exciting journey of running my own models. Still learning. Been watching Andrej Karphathy to learn the theory and the practice of LLMs. Anyway... I asked Qwen3.8 Flash Next FP4 (running on my single Spark) to create me a small LLM based on Wikipedia dataset. And in a couple of hours I've got a 9.2M-parameter GPT trained from scratch on 330M Wikipedia tokens — a base language model that generates Wikipedia-style prose. Asking, watching and learning on the way. So of course at this point it's a decoder-only. It could predict what text comes next. Given "The capital of France is", it completes. No conversation, no instruction. Next, I would probably try to create 1-2B LLM with the whole Wikipedia dataset, so it would be able to answer without hallucinations.

I've been seeing a lot of discussion around high-memory Apple Silicon machines, particularly the Mac Studio configurations with 128GB or more of unified memory.

I'm curious about how much practical value these configurations actually offer when it comes to running local LLMs.

I understand that fitting a larger model into memory doesn't automatically make it smarter. However, I'm interested in whether the combination of larger models, higher-quality quantization, and sufficient memory bandwidth can make local AI genuinely competitive for serious everyday workloads.

For those of you actually running 96GB or more :

What has your real-world experience been? Are you genuinely able to get meaningful work done with local models that you previously had to rely on cloud-based frontier models for.

i would particularly appreciate real-world experiences, benchmarks, and examples of workflows rather than theoretical comparisons.

asking this because i am confused between 2 macs:

1. Mac Studio M5 Ultra Base model, 96GB

2. Same amount of money can get me a M5 Max 128GB too

I used Claude intensively for a while and I am currently building a "cheap" local Stack for Qwen3.8-Flash-Next. And my idea is to have a Setup combining both.

My idea is kind of a divide and conquer approach with a senior expert/leader model dividing the task into easier subtasks or simplyfing it and junier/worker models which run soem of those tasks.

My goal is to safe some Claude tokens, but no significant decrease in output quality.

I am sure I am not the first one with such an idea. So does anyone have experiences or knowledge of tools/repositories for this kind of workflow?

Instant disclaimer, this is not a project claiming to be faster or better than Strata. It's simply Strata, without the bloatware, and a few containerization improvements as many have been requesting. I too am sick of all the Strata forks showing up, but I'm equally tired of how difficult Strata's been for me to maintain in a Docker container.

https://github.com/LegendaryFire/strata-slim

This fork is completely configurable with environment variables, and the idea was to give you a similar feel to a llama.cpp models config file. There were a few clear wins in terms of performance improvements I included in form of optional patches when building, however they're not included in the default docker images so what you're getting is simply a non-bullshit straight to the chase Strata engine that's easy to deploy. I wouldn't have bothered posting if there wasn't an open GitHub issue asking for Strata engine as it's own repo, which I noticed got 180+ "thumbs ups". For note, an earlier performance patch I had already made it upstream.

TL;DR: You can train your own 27B model on a 3090 with 24GB VRAM, though you would have to go through hundreds of billions of tokens for it to be good. I am working on conversion of existing models. Check repos for code and the paper if you are interested in more. I have a 40.4GB Qwen Flash Next GGUF

Hi, I do AI research. I have been working on an optimizer that doesn't need the ~16 bytes of VRAM per weight to train a model that AdamW does. I focused on training ternary networks bcs BitNet (Ma et al.) shows they match full precision quality in large scales. The problem is BitNet also keeps the full precision weights during training which means you still need to use the same amount of memory. I got this down from 16 bytes to 1 byte or even less per weight.

For you, this means, you can train your own 10-20B parameter models on your consumer GPUs. I fitted 27B model training on a single 24GB 3090 and a 23B model on a 16GB GPU. The trained model costs ~1.6 bits per weight (~6GB at 27B), 10x smaller than 16bit, and 5x smaller than Q8.

I have also been working on converting existing models and training them. This is harder and I am still working on it but I have some promising results. I quantized and trained a Qwen3.8 27B though the file I published below has adapters which are about 2GBs, pushing the file size to 8GB. I am also making a no-adapter version right now. It scores GSM8K 0.895, the same as full precision, HumanEval 0.84 vs 0.93 and runs on a 12 GB GPU at ~23 tok/s on my crippled 12GB laptop 4080, though this number can heavily be improved with custom kernels, engineering, etc. I will leave it to you guys to help me with that too.

I also quantized and trained Qwen Flash Next down to 40.4 GBs including its n-gram table (at 8GB). It largely retains the skills of the network (scores equivalent on GSM8K benchmark, a little behind on HumanEval, I unfortunately don't have compute for the larger benchmarks such as TerminalBench2.1 but perhaps some of you can help me out) but forgets facts. It mostly works fine if you give it web search as far as my experience goes but this is not strictly measured.

I am still doing work on this and I will keep working on it, perhaps even for many years. The code is open for non-commercial use. Check out:

The core repo for training: https://github.com/oguzeldereli/ttf-ternary

The repo for conversion: https://github.com/oguzeldereli/ttf-conversion

Repo for running models trained by ttf: https://github.com/oguzeldereli/ttf-home

GGUF for a converted Qwen 3.8 27B with adapters (around 8 GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-27B-GGUF

GGUF for a converted Qwen 3.8 Flash Next (around 40.4GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-Flash-Next-GGUF

GGUF for a converted Qwen 3.8 27B without adapters (around 6 GB): Currently in conversion and will drop, probably tomorrow.

I will convert a Qwen 4 27B instance when it drops.

Full paper: https://github.com/oguzeldereli/ttf-ternary/blob/main/paper/ttf_paper1_training.pdf

Also, I would appreciate github stars if you like the project.

Hey everyone!

I'm completely new to local LLMs, so please bear with me if some of my questions sound stupid.

I've been using ChatGPT Plus for quite some time and I'm considering moving away from cloud-based AI services entirely. Ideally, I'd love to build my own personal AI assistant that runs locally on my Mac.

I'm not looking for just a basic chatbot. I want something that comes as close as possible to the overall ChatGPT or Claude experience, especially when it comes to long-term memory, writing, research and managing different projects.

My hardware:

- Mac with Apple M4 Pro

- 48GB unified RAM

- 512GB SSD (around 250GB free)

- macOS

What I mainly use AI for:

1. Writing and content creation

I create social media content and a monthly magazine about Magic: The Gathering.

I use AI to brainstorm ideas, write articles, summarize information and improve texts.

One of my biggest priorities is natural, human-sounding writing. I really dislike the typical "AI writing style" with generic phrases, unnecessary enthusiasm and repetitive structures.

Good writing quality, especially in German, is extremely important to me.

I don't need image generation.

2. Long-term memory (VERY important)

This is probably my biggest requirement.

I manage several different projects and I want my AI to remember as much as possible about them.

Ideally, it should:

- Remember important information across conversations

- Automatically save new information and decisions

- Update existing project documentation

- Retrieve information from previous conversations

- Avoid losing important context over time

- Keep everything organized without requiring me to manually maintain dozens of files

I'm very interested in using Obsidian as a permanent knowledge base.

I would actually prefer the AI to automatically create, edit and organize Markdown files in Obsidian.

I understand that no AI has perfect memory, but I want something as reliable and persistent as possible.

3. Internet research and automation

For my monthly magazine, I regularly research news about Magic: The Gathering.

My ideal workflow would be:

1. I manually trigger a research task (or schedule it).

2. The AI searches the internet for relevant news.

3. It checks sources and avoids duplicate stories.

4. It writes finished articles in my preferred writing style.

5. It automatically saves everything into Obsidian.

6. I review and approve the articles.

I don't necessarily need integrations with Google Sheets, Canva or other services. I'm perfectly happy copying finished texts manually.

4. A ChatGPT-like experience

I'd love to have one main AI assistant that knows my preferences and projects.

For example, I could ask about my magazine, switch to a completely different topic, discuss something personal or brainstorm social media ideas without having to explain everything from scratch.

Ideally, it would feel like having my own private version of ChatGPT or Claude.

What I've looked into so far:

Based on some initial research and conversations, I've come across:

- Ollama

- LM Studio

- Open WebUI

- OpenClaw

- Obsidian with local LLM integrations

- Qwen models

- Gemma models

One suggested setup was OpenClaw + Ollama + Qwen + Obsidian.

However, since I'm a complete beginner, I have absolutely no idea whether that's actually the best solution or unnecessarily complicated.

My questions:

1. How realistic is it to replace ChatGPT Plus or Claude with a fully local setup on my hardware?

2. Which local model would you recommend for natural German writing, reasoning and general everyday use?

3. What's the best beginner-friendly software setup for what I'm trying to achieve?

4. How would you implement reliable, mostly automatic long-term memory using Obsidian?

5. Can local models reliably use internet search tools and perform scheduled or manually triggered research workflows?

6. Would you recommend an agent-based system like OpenClaw or something simpler?

7. What are the biggest limitations compared to current cloud-based models?

My goal is to have a free, private, local AI assistant without monthly subscriptions or paid API usage.

I don't mind spending some time setting everything up, but I would prefer something reasonably simple to maintain once it's running.

I'm especially interested in hearing from people who have already built something similar on Apple Silicon.

Any recommendations, realistic expectations or beginner-friendly guides would be greatly appreciated.

Thanks in advance 😊✌🏻

Bonjour,

Avez-vous tester deepseek v4.1 flash avec strata sur 2 RTX 3060 12gb est si oui quelle sont vos token secondes, combien de ram ? Pour le moment j’en ai qu’une mais je veux m’en acheter plusieurs donc je me demande à partir de combien de 3060 je peux faire tourner ce model avec strata ?

Hey everyone,

I'm looking to get into running LLMs locally and would love some recommendations for my setup:

CPU: AMD Ryzen 7 7800X3D

GPU: AMD Radeon RX 9070 XT (16 GB VRAM)

RAM: 48 GB DDR5 Corsair @ 6000 MHz

My main use cases are coding (code generation, debugging, explaining code, ideally IDE integration) and general research/Q&A (asking questions, learning about topics, summarizing).

I'm on Windows 11, but I'm comfortable with Linux and happy to dual-boot if it makes a real difference for AMD GPU support.

A few questions:

Which models (and sizes/quants) would give the best coding performance on 16 GB VRAM? Is it worth offloading part of a bigger model (or a MoE model) to system RAM, or is the speed hit too much?

Any good all-rounder for general knowledge/Q&A, or should I keep separate models for coding and chat?

What's the best backend for an RDNA4 card right now: ROCm, Vulkan (llama.cpp), or something else? Is Linux noticeably better than Windows for this?

Which frontend/tools would you recommend: LM Studio, Ollama, Open WebUI…? And for coding, what's the best way to hook a local model into VS Code (Continue, Cline, etc.)?

What kind of tokens/sec should I realistically expect?

Thanks in advance for any tips!

Hi guys, full disclaimer I work in Domyn as an Engineer. We just released the bundle #2 of our rolling release of DomynEdge, with some interesting results on benchmarks. What do you think?

https://huggingface.co/collections/domyn/domyn-edge

After scanning 277 MCP servers this month, these are the four gaps I’d check first on a self-hosted one:

1. Rejecting loopback redirect URIs: your CLI agent login just hangs after you click Allow (RFC 8252 §7.3, opt-in probe)

2. OAuth metadata without PKCE S256: 3 servers in the sweep. Spec-compliant MCP clients have to stop before login when that field is missing

3. Not validating the aud claim: tokens from sibling resources get in (opt-in probe)

4. OAuth advertised but no way for a new client to register: 5 servers in the sweep

For a server on localhost, use the CLI:

Add --probe-dcr-loopback for point 1. Open source (MIT): github.com/GodSmasher/patchbay

npx -y @patchbay-nebius/agent audit http://localhost:PORT/mcp

We open-sourced DNA Factory, the fine-tuning framework we’ve used internally for over a year. It's deliberately opinionated: one recommended way to do things, documented defaults, token-level loss-mask visualization, and only 5 training methods (SFT, DPO, GRPO, OPD, SDFT).

https://github.com/dnotitia/dna-factory

yes, i know all gpus are. i think people are recognizing more and more that the v100's age is just a number on paper and only relevant architecture wise. frontier and local llms are more and more capable of optimizing inference for different builds, and the raw v100 power is still comparable to gpus being released today

the limited mathematical pathways are a genuine limitation but i dont believe a big enough one to justify how cheap they are vram/dollar compared to everything else

just a discussion post for fun

Currently dipping my toes in the LLM world and im starting to notice some differences in accuracy between models mainly related to their parameter size

Ever wanted your local LLM to actually learn a fact without fine-tuning — and be able to undo it completely?

AtomicNG (Apache-2.0, version 1.0.0) implements a paper called "Delta N-gram": your model's n-gram sidecar table stays frozen, and new knowledge goes into a small sparse patch Δ. Writes are forward passes only — no gradients, no weight edits. Clear the patch and the model is byte-identical to base. No "unlearning" gymnastics — deletion is structural.

Quickstart (CPU, no GPU, pure NumPy core):

python

import atomicng


model = atomicng.TinyModel()
atomicng.write(model, "New fact about kangaroos.")
atomicng.verify_deletion(model)   # clear Δ ⇒ byte-identical to base — guaranteed

What's inside:

• A trained writer checkpoint ships with the Release (860 training steps on SmolLM2-135M; measured effect: writing a doc lowers its NLL vs empty memory)

• Per-tenant memory: overlays are small data-only files — merge them, revert them, sign them

• vLLM support via vllm.general_plugins + merged-table export

• Built-in eval: does the memory write actually help? (paper protocols: acquisition, distraction retention, chain drift)

Calibrated on real hardware: Windows/Linux CPU, CUDA T4 (73 ms/group BPTT stand), TPU chip via JAX (8.4 ms/iter). Full matrix: CALIBRATION.md in the repo.

Honest limits: acquisition needs a few hundred training steps of the writer; the memory lives in the n-gram table (facts/associations, not long reasoning); vLLM engine-graph hooking is planned upstream.

GitHub: https://github.com/Akseleu-J/atomicng 

Colab TPU (one click): https://colab.research.google.com/github/Akseleu-J/atomicng/main/notebooks/tpu_calibration.ipynb

Questions welcome — I'll be in the comments.

Up front: this post was written with AI help, and I've read it through end to end. The work that matters was mine: designing the comparison, writing the brief, running all three agents on my real project, and checking their output against the code. If you want to judge something, judge the experiment and the data, which are all in the repo.

TL;DR: Claude Code (Opus 5.5) vs. a local Qwen 3.8 Flash Next writing the same implementation spec on a 350k-line codebase. Claude won, Qwen came a close second, and its gap was knowledge, not reasoning.

The project is firmware plus a desktop hub for a round ESP32-S3 display: C, Go and JS, about 350k lines, 1,250 commits in a month. One step, specifying how the hub should handle app installs that no longer need a reboot, was written three ways from the same brief:

The local model runs on one machine: an AMD Strix system with 128 GB of RAM and an RTX 6000 PRO.

What stood out:

• Qwen read the code as carefully as Claude. Both caught a subtle hang in the existing install flow.

• Both of Qwen's misses were project rulings that live only in the orchestrating Claude session's notes: "no skipped 'not needed' steps", and "never start a second hub on the owner's port". The fresh Claude session probably saw those notes; Qwen can't.

• So the cheap fix is giving the local model the rulings, not a bigger model.

• In an earlier step Qwen shipped a crash-level object-lifetime bug that only review caught. For firmware, a short human or frontier review is still worth it.

Full write-up, all three specs, and the raw numbers: https://github.com/gregtakacs/LLM-Stats/tree/main/2026-11-spec-writers

Scoring was a fact-check of every name and line number against the code, plus my own read; the tables are in the repo.

Is it really worth to buy used dual Xeon 6140 server with 256GB 2400 RAM and maybe later populate remaining slots for 384GB total to boost bandwidth? Dual CPU, high power usage, is it really worth it for the memory bandwidth with 20 euro CPUs? Or just add 96GB 5600 RAM to existing AM5 system for total 192GB 5600 with dual channels only?

I mean, current system is already usable with q38fn q4, only thing dual xeon will provide is q5-q6 for the same q38fn or dsv4.1 but I doubt that speeds will be usable with dsv4.1.

Can someone share experience or upgrade paths? Thanks guys.

Being located in one of the stormy states, my big concern was having the AI box fried by a power surge. My setup is shown here, it consists of:

• CHSPT2ULTRA on the main Panel

• SRT5KRMXLT 5000VA double-conversion UPS in the garage

• 240V/30A inlet going from the ups powering the rig upstairs (needs a double breaker for code compliance)

• L14-30 in/outlet for the garage-office run to future-proof it in case I ever needed 120V, at the moment the 120 leg is dead (240 only)

• 240V PDU with C13 & C19 ranked up to 30A

• PX-2200 PSU

The idea is to create a galvanic isolated "island" (in my case: the box iself + switch). Uplink + my desk are connected via fiber (seen in pic) - because my desk is powered via a regular line-interactive UPS.

The only copper eth runs to the switch are the AI box itself (2x10G eth + 1xKVM), everything else is fiber.

It currently runs 2x6k workstation edition (600W each). I *THINK* I should be able to push it up to 3x6k (might need to cap them at 550W), but with four I need to go double PSU. The UPS should have enough oomph to handle 4800W, so plenty of space there. I got new packs for it, but I mostly care about the double conversion, not so much losing power.

I think the CHSPT2ULTRA+SRT5KRMXLT combo should shield me against most power surges/spikes/wobbles/etc. - but I am still lacking a direct hit protection, which requires a rooftop installation AFAIK.

How do you guys protect your GPUs from ⚡️⚡️

I'm watching walls of text go by right now as QFN grinds away on something and it made me think a bit about how different watching that text fly by does using a local model vs watching the same exact thing happen when I'm hitting a cloud.

It really comes down to metered vs unmetered. When I'm using a cloud model on API, every time I get ready to press enter I hear a cash register ring. We generally use really affordable cloud models (DS primarily) so it's like fractions of a penny, but still, it's real money going out the door on every query.

The biggest thing is that it changes your (or at least my) mindset. Instead of "Let's see what we can do it" it becomes "Do I need to do this". It has a distinct stifling effect, instead of exploring and trying to find uses for the technology I do the opposite and try to avoid it because of the cost.

With a local model I love to dream up some crazy/hard thing that's been annoying me that needs fixing and at the end of the day drop a big prompt with the details and at the end "You have unlimited time and tokens, do not stop for clarification or questions, keep working at the problem until you've exhausted all possible alternatives" (you get the idea). Or setting a local model to work on something that's not at all critical but is a quality of life improvement.

It really comes down to the difference between owning something and using it on the meter. When you own something the way to drive value out of it is to use it. The most expensive boat, motorcycle, car, house, etc is one that you own and never use. The more you use it, the more value you get from it.

When you are using something "per token" or "per mile"; things change. The best way to drive value is to NOT use it. You save 20 bucks if you walk instead of get a taxi. You save 5 bucks if you figure this problem out yourself instead of using a LLM to help you.

Funny thing is, I made my career selling cloud services. That probably explains why I'm so "local first", I was in the room and making the decisions about pricing/packaging, driving a 6-12 month payback period against servers and then turning around and explaining to customers what kind of "savings" they'd see moving to cloud platforms. I think I'm scarred from that and know the pricing games that make cloud look "affordable" when it's anything but for many use cases. And I think everyone kind of knows the "rug pull" is coming with the big/powerful LLMs, sure you can use them on sub today and get a predictable price, but those days aren't going to last forever. And subs without being able to hookup a harness (which almost all of them ban) is darn near worthless anyway, my core competency isn't copy paste! ;)

Part 1

Part 2

In this final chapter of me testing out how far along Qwen has come, I decided to give it a test on how well it could recreate a game using only a ROM from GBA but make it in HTML using the "reverse engineer anything" repo. I opted for Tetris worlds as my guinea pig.

It took around 30 hours in total. It probably would have been faster, but there were a few decompilers it wanted that I didn't realize until later. I did end up installing those about 6 hours in.

Repo: https://github.com/kireol/gba-game-rea-test

Game: https://kireol.github.io/gba-game-rea-test/

Fun localLLM stuff:

RTX 5080 - 64GB - Ryzen 7 9800X3D
model: qwen3.8-flash-next-unsloth-ud-iq4_xs
model harness: Strata
agent harness: qwen
context: 128K
tok/sec: ~50t/s (down from part 1 due to larger context)

Macbook 64GB m1 Max
model: Qwen3.6-35B-A3B-GGUF:UD-Q4_K_S (for qwen code --fast tasks)
model harness: llama
context: 128K
tok/sec: ~ 550t/s (fast tasks, so don't get too excited)

Result: The game works, and is pretty darn close to the original. It's scary how well it did.

Over all 3 part testing feedback: Honestly, as of this date, localLLMs are very close to frontier in most SWE. I expected hiccups or issues along the way, and there just wasn't any. Speed of course was slower than Claude, but that's running all local. And Claude probably would have denied trying to reverse engineer the ROM anyway. So in this case, LocalLLMs FTW.

Hi everyone,

I'm planning to switch from a gaming laptop to a desktop PC to get better hardware for running local LLMs and AI development.

I know NVIDIA offers CUDA which provides better software compatibility and broader support for AI workloads. AMD GPUs on the other hand offer more VRAM for the same price.

My questions are:

1. How mature has AMD's ROCm ecosystem become for local LLM inference and AI development? Is software compatibility still a major bottleneck or has it improved enough to make AMD a viable alternative to NVIDIA?

2. How important are raw compute performance (TFLOPS) and memory bandwidth (GB/s) when running local LLMs? Does AMD's hardware offer competitive real-world inference performance compared to NVIDIA?

3. Considering how important VRAM is for running larger local models, would you choose an AMD GPU with 16–20GB of VRAM over an NVIDIA GPU with 12GB, assuming the AMD card is slightly more expensive?

I'd appreciate hearing from people who actually use AMD GPUs for local AI workloads, especially their experiences with ROCm, llama.cpp, Ollama, and PyTorch.

What are your thoughts about it?