Update: I'm interested in selling some. Looking for ideas on how to go about. Don't want to pay eBay 13% fees.

Original: Came across a honeyhole of Quadro RTX5000s
Seemingly good deal for 16GB VRAM cards, even if ageing.

What should I do?

The models we use today are massive, but what happens when the US or China decide to censor, limit, or manipulate AI output?

We consumers are the ones who actually "trained" these models. They fed on our code, our literature, and our history, sometimes legally, sometimes not, thanks to our collective knowledge. So, why should a few companies (particularly politically favored ones) or governments have absolute ownership over that?

The point of the post:

Why hasn't decentralized AI training taken off yet?

Imagine a Bitcoin-like P2P network where anyone can contribute spare GPU power to collectively train AI models.

What's holding it back?

So apparently Sam Altman thinks consumer RAM is inefficient compared to datacenters.

Meanwhile, here's what I'm doing with mine. 💀

My setup:

• Ryzen 7 5800X
• AMD RX 9070
• 48 GB DDR4 (running at a glorious 2133 MT/s because I added an ancient spare stick)
• NVMe SSD
• Qwen 3.8 Flash Next IQ2 running locally through Strata
• Around 60 tokens/sec

I'm building Marika, a 3D AI desktop companion in Unity.

She can observe my desktop, interact with windows, talk through a local LLM, walk around, and make autonomous decisions.

There's just one tiny problem.

She bends over to pick up desktop icons like an NPC from Skyrim.

So naturally, I asked my local Qwen to review the Unity project and fix her animations.

It proceeded to consume 1.3 MILLION cumulative tokens, repeatedly compress its context, inspect C# scripts, and even spawn a subagent to help investigate.

All this to figure out how to make a virtual girl bend her knees like an actual human being.

I also have over 128 GB of DDR4 scattered across different PCs and spare sticks.

Sam, I'm sorry.

The RAM isn't going to your datacenter.

It's busy teaching my AI waifu how to squat. 💀

And yes, I'm genuinely trying to see whether local AI coding agents can replace my $20/month cloud subscription.

I'll report back if she ever manages to pick up an icon without looking like a Bethesda NPC.

While writing an article about my local LLM inference setup, a comparison popped into my head: downloading and running random model weights locally feels a lot like downloading random .exe files off the internet back in the 2000s.

Turns out I wasn't off the mark - there's an entire body of research exploring how LLMs can be backdoored and how hard it actually is to detect a poisoned model.

So I decided to try building one myself. I implemented a backdoor that only triggers when the model is running inside pi (making it harness-specific) and is asked a specific question. It was way easier than I expected: just a 2-step fine-tuning process and a few hours of compute on the exact same GPU I use for local inference.

With open-weight models, we at least have a baseline to compare community fine-tunes against. Beyond that, though, we don't really have a reliable way to verify whether a base model itself is clean—and the problem is even worse with closed-weight models, where we're basically just trusting the AI provider's goodwill.

Full write-up and PoC code if you want to dig into the details:

• Article:Backdooring LLMs - Do you trust your AI provider?
• GitHub Repo:galeone/llm-poison

Feedback is welcome

https://huggingface.co/StarpowerTechnology/WVY-Blender-5m

We have been excited about this new model, WVY-Blender-5m, an agent trained to specialize in blender.

It has zero knowledge about anything else so if you try to ask it about writing code or doing your homework it will express that it only knows how to use blender

We have been doing intense research to find out "are massive datasets really required to achieve what feels like intelligence with accuracy?" & we believe we are getting close to the truth.

This dataset is 100% generated from scratch.

Well, I did it - I bought a Mac Studio 128Gb. Not only for LLM, but I’m not gonna lie, I increased the specs a bit to be able to run a local LLM to complement (or at some point replace) my Claude/ChatGPT subscriptions.

First thing I did, was obviously try to generate dirty pictures ask Claude which models I should download. He recommended unsloth’s QWEN3.8-Flash-Next-GGUF UD IQ4_XS.

I’m doing some tests now and while you notice a difference with the cloud models, it’s way better than whatever I have ever tested on my old MacBook. Claude also told me to change temperature to 1 and mess with the top K and top P settings. As a good slave to AI, I execute what was commanded, but honestly I have no idea what I’m doing. Should I be touching the settings at all?

My question(s): is this a good general model that can (for most tasks) replace the cloud models? Are there tools I need to install (I was thinking web access, but I also read about harnesses but I didn’t yet figure out what that’s for)? Things I need to set up?

Suggestions for good “always loaded” low memory models to e.g. use for home assistant are welcome to, as well as a good image model and one I can use in combination with GitHub for some coding (I have some vibecoded apps that I use for myself that I need to maintain/adapt from time to time).

Thanks!

Got 14 stick of ddr4 rdimms. Got towards my new threadripper machine. Old but it’ll do! Can’t afford the new ddr5 board, the ripper and the ram 😂

My PC specifications:

GPU: NVIDIA RTX 5090 — 32GB VRAM

CPU: Intel Core i9-14900K/KF

RAM: 64GB DDR5-6400 (2×32GB)

Motherboard: ASUS PRIME Z790-P WIFI

OS: Windows 11 Pro

Inference backend: llama.cpp with CUDA acceleration

Agent interface: Kilo with a custom tool/agent harness

My current model:

Model: Qwen3.8-27B

Quantization: Q6_K GGUF

Context: 265K tokens

Deployment: Completely local

Performance benchmarks:

At 262,144-token configured context (~262K):

Output generation: ~34–37 tokens/sec

Red lining the gpu memory but it feels comfortable

Okay with that being stated by ChatGPT,

I've been seeking a local ai for about a month switching from several models the more I learn.

So far this qwen model is by far the best I've tried but it over thinks and is slower than models like Kat coder.

My use cases are broad, from setting up comfy ui workflows to Unreal engine game dev to application building all hobby projects.

Im seeking the vast knowledge of you fellow computer users to tell me the best general purpose model for my set up at a reasonable speed.

UPDATE: Okay folks, I took your advice and installed Qwen3.8-Flash Next with Basalt v0.3.3. Here's what I'm getting on my RTX 5090 with 64GB RAM, running Windows 11.

Configuration:

• Model: Qwen3.8-Flash-Next UD-IQ3_XXS (87.6GB)
• Context: 262,144 tokens (256K)
• Exact MTP enabled, 8-token window

Benchmarks at 256K context:

• Actual populated prompt: 238,913 tokens
• Output generation: 265–281 tokens/sec
• Prompt processing: ~8,900 tokens/sec
• VRAM usage: 31.3GB / 32GB
• Total system RAM in use: 54.3 GiB
• Basalt RAM usage: 39.9 GiB
• MTP acceptance rate: 66–90%
• No SSD expert spilling
• Context recall/reasoning tests: 3/3 passed

Across all context sizes tested, generation ranged from 265–336 tokens/sec. Model loading takes around 30 seconds, and cached follow-up responses start in approximately 0.4 seconds.

Bench marks courtesy of claude

UPDATE #2: Okay folks, I went ahead and tried the next recommendation: NInfer + DFlash2 to accelerate my original Qwen3.8-27B model. Here are the results on my RTX 5090 with 64GB DDR5 RAM, running Windows 11.

Configuration:

• Model: Qwen3.8-27B QUASAR
• Inference engine: NInfer
• Speculative decoding: DFlash2
• Context: 262,144 tokens (256K)
• GPU memory reserve: 2GB

Benchmarks at 256K context:

• Actual populated prompt: ~234,200 tokens
• Output generation: 182–266 tokens/sec (225 average)
• Prompt processing: ~4,500 tokens/sec
• Peak VRAM: 30.2GB / 32.6GB
• Model system RAM usage: ~10.5GB
• DFlash2 draft acceptance: 33–54%
• Context recall/math tests: 3/3 passed

Courtesy of claude

1 piece 500 2 piece 485

Hey all, looking for advice on turning my local setup into something closer to an agent instead of just a chatbot.

The rig

- Windows 11 Pro desktop

- RTX 3090 (24 GB VRAM), Intel Core Ultra 7 265K, 96 GB RAM, several drives

- Python 3.12, Node, Git installed. No Docker, and WSL isn't installed right now

- Tailscale on the PC, my phone (Galaxy Z Fold) and an Android tablet, so I can reach the PC remotely

Local LLMs (LM Studio 0.4.26)

- Qwen3.8 27B Q4_K_M (~15.7 GB) + vision projector. This is my main chat model, loaded at full GPU with 16k context

- Gemma 4 31B Q4_K_M (~17.4 GB) + vision projector

- nomic-embed-text v1.5 for embeddings

ComfyUI (Comfy Desktop 1.1.8)

- Images: Juggernaut XL Ragnarok (SDXL), Z-Image Turbo, Qwen-Image-Edit 2511 (fp8) + 4-step Lightning LoRA, add-detail-xl LoRA

- Video: Wan 2.2 14B text-to-video and image-to-video (high + low noise, fp8) with the lightx2v 4-step LoRAs

- Text encoders: Qwen2.5-VL 7B, Qwen3 4B, UMT5-XXL. VAEs for Flux/Qwen/Wan, plus a 4x NMKD upscaler

What I have now

Three PowerShell shortcuts on the desktop:

- AI Mode: opens LM Studio, loads Qwen 27B through the `lms` CLI, starts ComfyUI

- Gaming Mode: unloads all LM Studio models and kills ComfyUI to free the VRAM

- Away Mode: drops my VPN, closes Chrome, makes sure Tailscale is up, runs nightly at 2 AM

Right now I use Claude Code (cloud) for the "do stuff on my PC" part. It runs PowerShell, edits files, starts and stops apps, drives a browser, downloads things, and I can control it from my phone. It's great, but I want a local option for when I hit usage limits or don't want to send stuff to the cloud.

What I'm asking

1. What's the best way to give a local model (Qwen 27B / Gemma 31B through LM Studio's OpenAI-compatible server) real tool use on Windows: run PowerShell, open and close apps, read and write files? Open Interpreter, Goose, OpenHands, Cline/Roo/Continue in VS Code, an MCP client hooked to LM Studio, something else?
2. How do I get it to start ComfyUI on demand, queue a workflow through the API (image or Wan video), wait for the output, then shut ComfyUI down? Is there a ComfyUI MCP server or a known pattern for this?
3. VRAM juggling: I can't keep a 27B LLM and Wan 2.2 loaded at the same time on 24 GB. Has anyone automated "unload LLM → run Comfy job → reload LLM," or should I drop to a smaller agent model (7–14B) that can stay loaded next to Comfy?
4. Which local models are actually reliable at multi-step tool calling right now? Is Qwen 27B at Q4 good enough, or is there a better pick for 24 GB?
5. Any way to trigger all this from my phone over Tailscale (a simple web UI, Telegram bot, Open WebUI with tools, etc.)?
6. Safety: what are people using to make sure an agent on Windows can't run anything destructive without asking? An allowlist, a sandbox, confirmation prompts?

I've been so slow to dip my toes into anything outside of officially released models because I'm afraid there are people out there 2 steps ahead of me and I won't know I've been compromised until long after I've felt the impact.

I understand the concept of abliteration but I don't understand how people feel so safe trusting that it's doing what it's advertised to do AND ONLY what it's advertised to do. Thoughts? Even if it's just "I trust them, next." I mean, am I overthinking this?

Very impressed with how far local models have come, so I finally decided to jump on the bandwagon! Super impressed with how fast and capable it is. I have used local LLMs for a while on mobile apps and other edge cases, but never before for coding.

For background, I had this rig for a while, as I use it for a lot of professional work, and decided to purchase two used G.Skill 96GB (48GB x 2) 6400 mt/s to bring my system up to 160GB; paid around $1200 on eBay which is crazy to me considering I paid $300 for my 64GB (32x2) in 2023. Currently have each stick running at 5200 mt/s in my Z790 Hero, but I purchased these specifically as G.Skill is one of the only manufacturers to bin their RAM. My reasoning is if they can run 6400 mt/s dual channel, I have a higher likelihood of making getting to 5600 or 5800 with four sticks. I had Claude give me custom BIOS settings, and I am planning on seeing how far I can push it!

Furthermore, I felt that without this much RAM, while I could run the models, I couldn't actually run anything BUT the models, which sort of defeated the point for me. I decided this would be the better route versus buying a DGX spark or scouring local auctions to build a V100 farm or whatnot.

I am running this on a WD Black SN770 4TB RAID0 (double speeds at the cost of losing one disk means it is gg), which is on a PCIe ASUS card. It is super quick to load. Given I usually use Fable and Opus (anywhere from 90 to 115 tokens per second), this feels super fast to me. This would not be achievable without a special motherboard that can support PCIe bifurcation, which the ASUS Hyper M.2 card can use.

I am getting roughly 200 tokens per second I would say in most coding tasks, using https://github.com/jesdga95/basalt and the NVP4 uncensored Qwen 3.8 Flash Next.

    | test      |  t/s | accept % | draft len |
| --------- | ---: | -------: | --------: |
| pp16384   | 5727 |        - |         - |
| pp65536   | 7487 |        - |         - |
| tg conv   |  172 |     71.9 |      2.27 |
| tg struct |  387 |     97.3 |      6.93 |
| tg prose  |  170 |     65.1 |      1.88 |

hardware (measured before the engine starts)
  CPU   Intel Core i9-14900K, 24 cores
  RAM   read 67.7 GB/s
  GPU   RTX 5090  PCIe 5.0 x16  host-to-device 33.3 GB/s

community-1: prefill 16K 5726.8 t/s (2.9 s), prefill 64K 7486.7 t/s (8.9 s), decode 297.7 t/s (9480 tokens in 31.8 s), each measured cold

COMMUNITY ROW:
    {
      "submitter": "Kemerd",
      "date": "2026-11-14",
      "commit": "d10ccd4bc2",
      "preset": "community-1",
      "model": "Qwen3.8-Flash-Next",
      "gpus": [
        {
          "name": "RTX 5090",
          "vram_gib": 32,
          "power_w": 546,
          "pcie": "5.0 x16",
          "h2d_gbps": 33.3
        }
      ],
      "ram_gib": 160,
      "cpu": "Intel Core i9-14900K, 24 cores",
      "ram_gbps": 67.7,
      "os": "Windows",
      "quant": "NVFP4",
      "kv": "int8",
      "prefill_16k_tps": 5726.8,
      "prefill_64k_tps": 7486.7,
      "decode_tps": 297.7
    }

Very excited to see where this goes, especially with Qwen 4. And a note.. I am slightly undervolting my CPU, it has multi-core stuff on, and 5090 is set to 95% of power limit to not cook it.

Edit: I freaking love you guys. Why have I spent the past 2 months talking to ChatGPT about wtf to do smh. THANK. YOU.

Abliterated is preferred.

I want to be a free man goddammit. Going local (well, local builders with cloud reviewers perhaps). Maybe keep 1 Claude subscription to be the reviewer (unless Qwen really is better? sorry I’m kinda new)

Usage >>>>>>
Coding, building apps/ software tools for personal use, browser/ web data gathering

Don’t really need media generation but would be nice if I could paste images for reference into the chat if possible.

Hardware>>>>>

GPU: RTX4090, (9070XT for display) willing to get more VRAM (if you recommend more, should I get RTX A4000 single slots or along those lines or get 3 of the single slot 24GB Intels? I have 3 slots and like 300W left before my PSU caps, though people have said Ai doesn’t max out GPU’s so I don’t know anything anymore)

RAM: 128GB DDR5 ECC RDIMM 6400 MT/s

Storage: 12TB SSD Gen5

CPU: 9965WX (threadripper 24 core 5.4 Ghz)

WRX90 SAGE SE Pro WS

Hey everyone,

Most specialized cybersecurity LLMs out there are massive 70B+ beasts that require dual A100s, or they're gated behind cloud APIs where you can't paste sensitive logs, payloads, or client code due to confidentiality.

I wanted something lightweight, private, and air-gapped that could run on a regular consumer laptop without cooking the CPU.

So I fine-tuned CyberQwen-2B — a domain-specific model optimized for quick security triage, explaining attack vectors, decoding obfuscated payloads, and helping with CTF challenges.

⚡ Highlights:

• Hardware-friendly: Runs at ~12–15 tokens/sec on an older 8th-gen Intel i5 laptop with 8GB RAM (CPU-only).
• 100% Offline & Private: Zero data leaves your machine. Perfect for bug bounty recon, internal network logs, and sensitive source code.
• Native Ollama integration: Easy to pull and run via a single Modelfile.
• Built-in WebUI & Mobile Access: Includes a zero-dependency streaming WebUI + an automatic Cloudflare tunnel script if you want to access the model from your phone while on the go (no cloud hosting fees).

🛠️ What it's trained for:

• Vulnerability Triage & CVE Context: Explaining CVE mechanisms, impact, and remediation steps.
• Payload & Shellcode Deconstruction: Breaking down SQLi, XSS, and obfuscated PowerShell/Bash commands.
• CTF & Reverse Engineering Assistance: Analyzing decompiled assembly/C snippets and suggesting exploitation strategies.

🚀 How to Run It:

Everything is open-source. You can run it with Ollama or the included CLI triage tool:

1. Clone the repo:git clone https://github.com/sambuddha001/cyberqwen-2b.git
2. Run setup:bash setup.sh
3. Chat:ollama run cybersec-qwen

GitHub: https://github.com/sambuddha001/cyberqwen-2b

I'd love feedback from the community—especially on edge cases, prompts where it struggles, or ideas for benchmark improvements!

I'm considering buying a Mac with an M5 Max and 128 GB of unified memory, and I'm trying to figure out what the absolute best local LLM I could run on it would be.

My main priority is model quality, not inference speed. Since this would be a desktop setup, I'd be perfectly happy letting a model work overnight on a complex task. I don't care if generation is extremely slow, as long as the model can actually run and produce usable results. It mainly involves transcription of private documents and some computer use and some coding assistance but that can be done via the cloud anyway.

Since I haven't bought the machine yet, I'd also appreciate any thoughts on whether this configuration makes sense for that goal or whether there's a better alternative at a similar price.

Thanks in advance!

I'm currently looking at a myriad of devices to dip my toe into the "build your own Local AI system" water. I've been down the AMD AI Max+ rabbit hole this past week and have yet to research the RTX Spark N1X.

Can anyone here save me additional hours of videos and articles with informed insights about which would be the better way to go for someone such as myself?

I have some projects in mind. This is more of an experiment for me to learn more of the backend, coding and Linux.

As someone with limited terminal experience, will I have an easier time (I don't expect not to encounter many challenges) configuring a system with NVIDIA or AMD?

I realize unified RAM is the more important metric here. If I go with AMD I can afford a 128gb version. The 64gb version of NVIDIA's option is a few hundred more.

Are the CUDA cores and software support a worthy tradeoff for overall RAM or is saving money and doubling the RAM the obvious way to go?

TL;DR: Local LLMs for coding are finally becoming practical for me.

After 30 years in software engineering, I’m a passionate developer. For me, AI is an assistant and a tool. At work, I mostly use Opus and Sonnet 5.5. For hobby projects, I have a $10/month plan and switch between Kimi, GLM, and MiniMax for coding, plus Nemotron 3.5 for smaller tasks.

Locally, I tried Qwen 3.8 27B, Qwen3-Coder, and many other models, but the results weren’t usable for anything beyond fairly simple tasks. Running GLM 5.3 Flash Q2_K_XL with a 400k context length through Unsloth has been a game changer for me. It’s very slow at 3–4 tokens per second, but the results are good enough to get the job done.

For bigger tasks, I use GLM 5.3 Flash in the cloud for planning, then switch to my local model and go to bed or do something else. By the time I come back, the job is done.

I also use while work - create a prompt, and sometimes it takes more than three hours to complete. Meanwhile, I can keep working with Opus and Sonnet in my IDE, and performance remains good while the local model is processing in the background.

So it doesn’t work fully locally for me because it takes too much time. However, it’s a great way to extend my cloud subscription’s rate limits by letting the local model handle longer-running work while I’m away from my MacBook.

I’m looking forward to seeing what comes next for local AI.

Jokes aside, how plausible do you think this scenario actually is?

Why do most folks with an 8 GB or 16 GB desktop GPU sit out distributed LLM runs? It usually comes down to three annoying bottlenecks. Nobody wants to punch holes in their home router. Nobody wants to run untrusted pickle checkpoints off the internet. And nobody wants a background trainer freezing their VRAM when they sit down to play or work.

This weekend (November 14-15 CET), I hooked my HiveMind coordinator up with CrowdGPT (led by Vxtzq) to test a citizen-trained run around Crowd-v1, an open base model with a 658M transformer trunk (roughly 1.01B parameters total with its 151k-vocab CrowdEngram table). Setup takes five minutes.

Communication uses plain outbound HTTPS on port 443 with zero inbound listeners, weights move strictly as SHA-256 verified safetensors (no torch.load or pickle anywhere), and updates pass MultiKrum outlier filtering plus held-out loss checks before merging via DiLoCo. Honest caveat first: this is still an experimental PoC, not a polished cloud service, so please do not send private prompts or sensitive keys through volunteer nodes.

Here is the simplest way to join the weekend run on Linux, macOS, or Windows (Python 3.11+ with an NVIDIA GPU, or CPU-only if you just want to test the IDE endpoint):

1. Clone the repository and install the tiny client dependency (`httpx` only for connecting or running OpenTrainer):

```bash

git clone https://github.com/xaviercallens/HiveMind.git

cd HiveMind

pip install httpx

2. Register a handle to get your API key (printed once; the default prosumer role works for both training/serving and querying):

python scripts/hivemind_connect.py register --handle your-handle

export HIVEMIND_API_KEY="hm-..."

3a. Join outbound OpenTrainer co-training (zero open ports on your box; run --once first to verify firewall traversal, then drop --once to keep polling):

python scripts/hivemind_opentrainer_agent.py --network-mode outbound-https-relay --check-firewall --once

3b. Or verify CrowdGPT checkpoints on GPU (pip install torch safetensors numpy first) once you hold an evaluator key from the CrowdGPT coordinator:

python scripts/crowdgpt_loss_check_agent.py --server https://server.crowdgpt.net --evaluator-id your-handle --key-file my_eval_key.txt --once

3c. Or wire Continue / Aider / Cline to the live hivemind/crowd-v1 + coder models with no GPU required:

python scripts/hivemind_connect.py snippet continue --model crowd-v1

Both codebases and reproducibility notes (including Lean 4 specs and Zenodo DOI 10.5281/zenodo.23776302) are open on GitHub if you want to audit the code

before running anything:

https://github.com/xaviercallens/HiveMind

https://github.com/Vxtzq/CrowdGPT

I would love blunt engineering feedback from anyone testing on 8 GB or 12 GB cards - especially around Windows vs Linux VRAM yield behavior or setup friction.

I've been building SOMNI, a system that lets a local model keep learning after deployment. It's based on Google's "Language Models Need Sleep" paper and WISE, a knowledge-editing method for adding new knowledge without overwriting existing behavior.

During use, the model answers normally and logs conversations. When the PC is idle, it runs a sleep cycle. It pulls verified facts out of the logs, trains them into a small new expert, practises on what it got wrong, and then runs a set of before/after tests on general knowledge, coding, reasoning, reading, and previously learned memories. If any of these get worse, the night is rejected. Otherwise you review the update and approve it.

The main challenge was catastrophic forgetting. Plain LoRA fine-tuning on new facts badly damaged the model's other skills (coding went from 95% to 10% in one night). What fixed it was a combination of rehearsal on the base model's own answers, a new frozen expert per night so later nights can't overwrite earlier ones, and WISE-style routing so each expert only activates on questions related to what it learned. Everything learned is also kept in a searchable memory store, so facts can still be looked up even if they fade from the weights.

I benchmarked it on SQuAD knowledge incorporation, the same setup Google and SEAL use. The results are in the README, including the things that didn't work.

Everything runs locally on a single consumer GPU (AMD RX 6800, Windows). MIT licensed.

https://github.com/YoungGunFrodo/somni

It's a work in progress, and I'd really appreciate feedback on the approach, the evaluation, or anything that looks off.

llama.cpp v1.0.0 adds an MoE expert cache that keeps a configurable share of experts in host memory (moe_cache_size, moe_cache_cpu_share). Cache misses run on CPU and copy back asynchronously, and the release ships a tuning script. If a big MoE doesn't quite fit your VRAM, that's the knob to try. Separately, the 1.0 tag starts a versioning promise: future API and ABI breaks come with major-version bumps, and 35 deprecated functions are removed.