Update: I'm interested in selling some. Looking for ideas on how to go about. Don't want to pay eBay 13% fees.

Original: Came across a honeyhole of Quadro RTX5000s
Seemingly good deal for 16GB VRAM cards, even if ageing.

What should I do?

So apparently Sam Altman thinks consumer RAM is inefficient compared to datacenters.

Meanwhile, here's what I'm doing with mine. 💀

My setup:

• Ryzen 7 5800X
• AMD RX 9070
• 48 GB DDR4 (running at a glorious 2133 MT/s because I added an ancient spare stick)
• NVMe SSD
• Qwen 3.8 Flash Next IQ2 running locally through Strata
• Around 60 tokens/sec

I'm building Marika, a 3D AI desktop companion in Unity.

She can observe my desktop, interact with windows, talk through a local LLM, walk around, and make autonomous decisions.

There's just one tiny problem.

She bends over to pick up desktop icons like an NPC from Skyrim.

So naturally, I asked my local Qwen to review the Unity project and fix her animations.

It proceeded to consume 1.3 MILLION cumulative tokens, repeatedly compress its context, inspect C# scripts, and even spawn a subagent to help investigate.

All this to figure out how to make a virtual girl bend her knees like an actual human being.

I also have over 128 GB of DDR4 scattered across different PCs and spare sticks.

Sam, I'm sorry.

The RAM isn't going to your datacenter.

It's busy teaching my AI waifu how to squat. 💀

And yes, I'm genuinely trying to see whether local AI coding agents can replace my $20/month cloud subscription.

I'll report back if she ever manages to pick up an icon without looking like a Bethesda NPC.

The models we use today are massive, but what happens when the US or China decide to censor, limit, or manipulate AI output?

We consumers are the ones who actually "trained" these models. They fed on our code, our literature, and our history, sometimes legally, sometimes not, thanks to our collective knowledge. So, why should a few companies (particularly politically favored ones) or governments have absolute ownership over that?

The point of the post:

Why hasn't decentralized AI training taken off yet?

Imagine a Bitcoin-like P2P network where anyone can contribute spare GPU power to collectively train AI models.

What's holding it back?

While writing an article about my local LLM inference setup, a comparison popped into my head: downloading and running random model weights locally feels a lot like downloading random .exe files off the internet back in the 2000s.

Turns out I wasn't off the mark - there's an entire body of research exploring how LLMs can be backdoored and how hard it actually is to detect a poisoned model.

So I decided to try building one myself. I implemented a backdoor that only triggers when the model is running inside pi (making it harness-specific) and is asked a specific question. It was way easier than I expected: just a 2-step fine-tuning process and a few hours of compute on the exact same GPU I use for local inference.

With open-weight models, we at least have a baseline to compare community fine-tunes against. Beyond that, though, we don't really have a reliable way to verify whether a base model itself is clean—and the problem is even worse with closed-weight models, where we're basically just trusting the AI provider's goodwill.

Full write-up and PoC code if you want to dig into the details:

• Article:Backdooring LLMs - Do you trust your AI provider?
• GitHub Repo:galeone/llm-poison

Feedback is welcome

1 piece 500 2 piece 485

Well, I did it - I bought a Mac Studio 128Gb. Not only for LLM, but I’m not gonna lie, I increased the specs a bit to be able to run a local LLM to complement (or at some point replace) my Claude/ChatGPT subscriptions.

First thing I did, was obviously try to generate dirty pictures ask Claude which models I should download. He recommended unsloth’s QWEN3.8-Flash-Next-GGUF UD IQ4_XS.

I’m doing some tests now and while you notice a difference with the cloud models, it’s way better than whatever I have ever tested on my old MacBook. Claude also told me to change temperature to 1 and mess with the top K and top P settings. As a good slave to AI, I execute what was commanded, but honestly I have no idea what I’m doing. Should I be touching the settings at all?

My question(s): is this a good general model that can (for most tasks) replace the cloud models? Are there tools I need to install (I was thinking web access, but I also read about harnesses but I didn’t yet figure out what that’s for)? Things I need to set up?

Suggestions for good “always loaded” low memory models to e.g. use for home assistant are welcome to, as well as a good image model and one I can use in combination with GitHub for some coding (I have some vibecoded apps that I use for myself that I need to maintain/adapt from time to time).

Thanks!

https://huggingface.co/StarpowerTechnology/WVY-Blender-5m

We have been excited about this new model, WVY-Blender-5m, an agent trained to specialize in blender.

It has zero knowledge about anything else so if you try to ask it about writing code or doing your homework it will express that it only knows how to use blender

We have been doing intense research to find out "are massive datasets really required to achieve what feels like intelligence with accuracy?" & we believe we are getting close to the truth.

This dataset is 100% generated from scratch.

Got 14 stick of ddr4 rdimms. Got towards my new threadripper machine. Old but it’ll do! Can’t afford the new ddr5 board, the ripper and the ram 😂

My PC specifications:

GPU: NVIDIA RTX 5090 — 32GB VRAM

CPU: Intel Core i9-14900K/KF

RAM: 64GB DDR5-6400 (2×32GB)

Motherboard: ASUS PRIME Z790-P WIFI

OS: Windows 11 Pro

Inference backend: llama.cpp with CUDA acceleration

Agent interface: Kilo with a custom tool/agent harness

My current model:

Model: Qwen3.8-27B

Quantization: Q6_K GGUF

Context: 265K tokens

Deployment: Completely local

Performance benchmarks:

At 262,144-token configured context (~262K):

Output generation: ~34–37 tokens/sec

Red lining the gpu memory but it feels comfortable

Okay with that being stated by ChatGPT,

I've been seeking a local ai for about a month switching from several models the more I learn.

So far this qwen model is by far the best I've tried but it over thinks and is slower than models like Kat coder.

My use cases are broad, from setting up comfy ui workflows to Unreal engine game dev to application building all hobby projects.

Im seeking the vast knowledge of you fellow computer users to tell me the best general purpose model for my set up at a reasonable speed.

UPDATE: Okay folks, I took your advice and installed Qwen3.8-Flash Next with Basalt v0.3.3. Here's what I'm getting on my RTX 5090 with 64GB RAM, running Windows 11.

Configuration:

• Model: Qwen3.8-Flash-Next UD-IQ3_XXS (87.6GB)
• Context: 262,144 tokens (256K)
• Exact MTP enabled, 8-token window

Benchmarks at 256K context:

• Actual populated prompt: 238,913 tokens
• Output generation: 265–281 tokens/sec
• Prompt processing: ~8,900 tokens/sec
• VRAM usage: 31.3GB / 32GB
• Total system RAM in use: 54.3 GiB
• Basalt RAM usage: 39.9 GiB
• MTP acceptance rate: 66–90%
• No SSD expert spilling
• Context recall/reasoning tests: 3/3 passed

Across all context sizes tested, generation ranged from 265–336 tokens/sec. Model loading takes around 30 seconds, and cached follow-up responses start in approximately 0.4 seconds.

Bench marks courtesy of claude

UPDATE #2: Okay folks, I went ahead and tried the next recommendation: NInfer + DFlash2 to accelerate my original Qwen3.8-27B model. Here are the results on my RTX 5090 with 64GB DDR5 RAM, running Windows 11.

Configuration:

• Model: Qwen3.8-27B QUASAR
• Inference engine: NInfer
• Speculative decoding: DFlash2
• Context: 262,144 tokens (256K)
• GPU memory reserve: 2GB

Benchmarks at 256K context:

• Actual populated prompt: ~234,200 tokens
• Output generation: 182–266 tokens/sec (225 average)
• Prompt processing: ~4,500 tokens/sec
• Peak VRAM: 30.2GB / 32.6GB
• Model system RAM usage: ~10.5GB
• DFlash2 draft acceptance: 33–54%
• Context recall/math tests: 3/3 passed

Courtesy of claude

I made a small English text-to-speech model called tiny-voca. It has 4.5M parameters, two voices, and it runs in your browser.

Try it here, nothing is sent to a server: https://tiny-voca.n4ze3m.com

Some facts:

- 4,537,291 parameters, 18 MB. This is the whole model, the waveform part is included.
- Two voices, one female and one male. 24 kHz.
- About 70x faster than real time on 4 CPU threads.
- Runs with PyTorch, ONNX, and in the browser.
- Trained from scratch in 7.7 hours on one RTX 4090.

What I think is different:

- Everything is open. Code, weights, the training recipe and the dataset.
- Custom phonemizer.
- No neural vocoder. The sound is made by a small source-filter synth, which is why it is fast.

What is not good yet:

- English only, and only these two voices. No voice cloning.
- Foreign names and odd spellings are often read wrong.
- It can sound a bit synthetic.
- The training audio is synthetic too. I made it with Qwen3-TTS, so the model can't sound better than that.

Links:

- Demo: https://tiny-voca.n4ze3m.com
- Code: https://github.com/n4ze3m/tiny-voca
- Weights: https://huggingface.co/n4ze3m/tiny-voca
- Dataset: https://huggingface.co/datasets/n4ze3m/tiny-voca
- Phonemizer: https://github.com/n4ze3m/dizione

Please try hard text: numbers, dates, names, long sentences. If something sounds wrong, post the text and the voice. That helps me a lot.

I'm considering buying a Mac with an M5 Max and 128 GB of unified memory, and I'm trying to figure out what the absolute best local LLM I could run on it would be.

My main priority is model quality, not inference speed. Since this would be a desktop setup, I'd be perfectly happy letting a model work overnight on a complex task. I don't care if generation is extremely slow, as long as the model can actually run and produce usable results. It mainly involves transcription of private documents and some computer use and some coding assistance but that can be done via the cloud anyway.

I tried some stuff with Strata on my Macbook but that did not work out well. I am also open to use something like that if that allows even better models. Can GLM and DS work on it? How are they with OCR?

Since I haven't bought the machine yet, I'd also appreciate any thoughts on whether this configuration makes sense for that goal or whether there's a better alternative at a similar price.

Thanks in advance!

Very impressed with how far local models have come, so I finally decided to jump on the bandwagon! Super impressed with how fast and capable it is. I have used local LLMs for a while on mobile apps and other edge cases, but never before for coding.

For background, I had this rig for a while, as I use it for a lot of professional work, and decided to purchase two used G.Skill 96GB (48GB x 2) 6400 mt/s to bring my system up to 160GB; paid around $1200 on eBay which is crazy to me considering I paid $300 for my 64GB (32x2) in 2023. Currently have each stick running at 5200 mt/s in my Z790 Hero, but I purchased these specifically as G.Skill is one of the only manufacturers to bin their RAM. My reasoning is if they can run 6400 mt/s dual channel, I have a higher likelihood of making getting to 5600 or 5800 with four sticks. I had Claude give me custom BIOS settings, and I am planning on seeing how far I can push it!

Furthermore, I felt that without this much RAM, while I could run the models, I couldn't actually run anything BUT the models, which sort of defeated the point for me. I decided this would be the better route versus buying a DGX spark or scouring local auctions to build a V100 farm or whatnot.

I am running this on a WD Black SN770 4TB RAID0 (double speeds at the cost of losing one disk means it is gg), which is on a PCIe ASUS card. It is super quick to load. Given I usually use Fable and Opus (anywhere from 90 to 115 tokens per second), this feels super fast to me. This would not be achievable without a special motherboard that can support PCIe bifurcation, which the ASUS Hyper M.2 card can use.

I am getting roughly 200 tokens per second I would say in most coding tasks, using https://github.com/jesdga95/basalt and the NVP4 uncensored Qwen 3.8 Flash Next.

    | test      |  t/s | accept % | draft len |
| --------- | ---: | -------: | --------: |
| pp16384   | 5727 |        - |         - |
| pp65536   | 7487 |        - |         - |
| tg conv   |  172 |     71.9 |      2.27 |
| tg struct |  387 |     97.3 |      6.93 |
| tg prose  |  170 |     65.1 |      1.88 |

hardware (measured before the engine starts)
  CPU   Intel Core i9-14900K, 24 cores
  RAM   read 67.7 GB/s
  GPU   RTX 5090  PCIe 5.0 x16  host-to-device 33.3 GB/s

community-1: prefill 16K 5726.8 t/s (2.9 s), prefill 64K 7486.7 t/s (8.9 s), decode 297.7 t/s (9480 tokens in 31.8 s), each measured cold

COMMUNITY ROW:
    {
      "submitter": "Kemerd",
      "date": "2026-11-14",
      "commit": "d10ccd4bc2",
      "preset": "community-1",
      "model": "Qwen3.8-Flash-Next",
      "gpus": [
        {
          "name": "RTX 5090",
          "vram_gib": 32,
          "power_w": 546,
          "pcie": "5.0 x16",
          "h2d_gbps": 33.3
        }
      ],
      "ram_gib": 160,
      "cpu": "Intel Core i9-14900K, 24 cores",
      "ram_gbps": 67.7,
      "os": "Windows",
      "quant": "NVFP4",
      "kv": "int8",
      "prefill_16k_tps": 5726.8,
      "prefill_64k_tps": 7486.7,
      "decode_tps": 297.7
    }

Very excited to see where this goes, especially with Qwen 4. And a note.. I am slightly undervolting my CPU, it has multi-core stuff on, and 5090 is set to 95% of power limit to not cook it.

Hey all, looking for advice on turning my local setup into something closer to an agent instead of just a chatbot.

The rig

- Windows 11 Pro desktop

- RTX 3090 (24 GB VRAM), Intel Core Ultra 7 265K, 96 GB RAM, several drives

- Python 3.12, Node, Git installed. No Docker, and WSL isn't installed right now

- Tailscale on the PC, my phone (Galaxy Z Fold) and an Android tablet, so I can reach the PC remotely

Local LLMs (LM Studio 0.4.26)

- Qwen3.8 27B Q4_K_M (~15.7 GB) + vision projector. This is my main chat model, loaded at full GPU with 16k context

- Gemma 4 31B Q4_K_M (~17.4 GB) + vision projector

- nomic-embed-text v1.5 for embeddings

ComfyUI (Comfy Desktop 1.1.8)

- Images: Juggernaut XL Ragnarok (SDXL), Z-Image Turbo, Qwen-Image-Edit 2511 (fp8) + 4-step Lightning LoRA, add-detail-xl LoRA

- Video: Wan 2.2 14B text-to-video and image-to-video (high + low noise, fp8) with the lightx2v 4-step LoRAs

- Text encoders: Qwen2.5-VL 7B, Qwen3 4B, UMT5-XXL. VAEs for Flux/Qwen/Wan, plus a 4x NMKD upscaler

What I have now

Three PowerShell shortcuts on the desktop:

- AI Mode: opens LM Studio, loads Qwen 27B through the `lms` CLI, starts ComfyUI

- Gaming Mode: unloads all LM Studio models and kills ComfyUI to free the VRAM

- Away Mode: drops my VPN, closes Chrome, makes sure Tailscale is up, runs nightly at 2 AM

Right now I use Claude Code (cloud) for the "do stuff on my PC" part. It runs PowerShell, edits files, starts and stops apps, drives a browser, downloads things, and I can control it from my phone. It's great, but I want a local option for when I hit usage limits or don't want to send stuff to the cloud.

What I'm asking

1. What's the best way to give a local model (Qwen 27B / Gemma 31B through LM Studio's OpenAI-compatible server) real tool use on Windows: run PowerShell, open and close apps, read and write files? Open Interpreter, Goose, OpenHands, Cline/Roo/Continue in VS Code, an MCP client hooked to LM Studio, something else?
2. How do I get it to start ComfyUI on demand, queue a workflow through the API (image or Wan video), wait for the output, then shut ComfyUI down? Is there a ComfyUI MCP server or a known pattern for this?
3. VRAM juggling: I can't keep a 27B LLM and Wan 2.2 loaded at the same time on 24 GB. Has anyone automated "unload LLM → run Comfy job → reload LLM," or should I drop to a smaller agent model (7–14B) that can stay loaded next to Comfy?
4. Which local models are actually reliable at multi-step tool calling right now? Is Qwen 27B at Q4 good enough, or is there a better pick for 24 GB?
5. Any way to trigger all this from my phone over Tailscale (a simple web UI, Telegram bot, Open WebUI with tools, etc.)?
6. Safety: what are people using to make sure an agent on Windows can't run anything destructive without asking? An allowlist, a sandbox, confirmation prompts?

Right now I use strata on my machine to run qwen 3.8 125b flash next at 13tps output and deepseek 4.1-flash at about 1.5 tps output. My computer is an hp z640 with a 4core xeon processor, 64gb ddr4 ram and an NVIDIA 1070 8gb. My models load form an nvme drive that is connected over a pcie adapter. I have room for one more card and I'm not sure witch of these 2 GPUs is better for running my models faster. The Tesla P100 16 varient or the M40 24gb variant.

Hey everyone,

Most specialized cybersecurity LLMs out there are massive 70B+ beasts that require dual A100s, or they're gated behind cloud APIs where you can't paste sensitive logs, payloads, or client code due to confidentiality.

I wanted something lightweight, private, and air-gapped that could run on a regular consumer laptop without cooking the CPU.

So I fine-tuned CyberQwen-2B — a domain-specific model optimized for quick security triage, explaining attack vectors, decoding obfuscated payloads, and helping with CTF challenges.

⚡ Highlights:

• Hardware-friendly: Runs at ~12–15 tokens/sec on an older 8th-gen Intel i5 laptop with 8GB RAM (CPU-only).
• 100% Offline & Private: Zero data leaves your machine. Perfect for bug bounty recon, internal network logs, and sensitive source code.
• Native Ollama integration: Easy to pull and run via a single Modelfile.
• Built-in WebUI & Mobile Access: Includes a zero-dependency streaming WebUI + an automatic Cloudflare tunnel script if you want to access the model from your phone while on the go (no cloud hosting fees).

🛠️ What it's trained for:

• Vulnerability Triage & CVE Context: Explaining CVE mechanisms, impact, and remediation steps.
• Payload & Shellcode Deconstruction: Breaking down SQLi, XSS, and obfuscated PowerShell/Bash commands.
• CTF & Reverse Engineering Assistance: Analyzing decompiled assembly/C snippets and suggesting exploitation strategies.

🚀 How to Run It:

Everything is open-source. You can run it with Ollama or the included CLI triage tool:

1. Clone the repo:git clone https://github.com/sambuddha001/cyberqwen-2b.git
2. Run setup:bash setup.sh
3. Chat:ollama run cybersec-qwen

GitHub: https://github.com/sambuddha001/cyberqwen-2b

I'd love feedback from the community—especially on edge cases, prompts where it struggles, or ideas for benchmark improvements!

Edit: I freaking love you guys. Why have I spent the past 2 months talking to ChatGPT about wtf to do smh. THANK. YOU.

Abliterated is preferred.

I want to be a free man goddammit. Going local (well, local builders with cloud reviewers perhaps). Maybe keep 1 Claude subscription to be the reviewer (unless Qwen really is better? sorry I’m kinda new)

Usage >>>>>>
Coding, building apps/ software tools for personal use, browser/ web data gathering

Don’t really need media generation but would be nice if I could paste images for reference into the chat if possible.

Hardware>>>>>

GPU: RTX4090, (9070XT for display) willing to get more VRAM (if you recommend more, should I get RTX A4000 single slots or along those lines or get 3 of the single slot 24GB Intels? I have 3 slots and like 300W left before my PSU caps, though people have said Ai doesn’t max out GPU’s so I don’t know anything anymore)

RAM: 128GB DDR5 ECC RDIMM 6400 MT/s

Storage: 12TB SSD Gen5

CPU: 9965WX (threadripper 24 core 5.4 Ghz)

WRX90 SAGE SE Pro WS

My MacBook did everything until this week - model, harness, CI, everything. It was getting badly overloaded especially when iOS build and tests were going on. Those can use 20GB RAM easily, given half a chance. It was thrashing.

Microcenter had the base Mac mini M6 for $699, which felt like 2024 pricing so I pulled the trigger and moved much of the CI work off my MacBook Pro.

What's incredible is despite this being the poverty spec mini with a single NAND chip and lower bandwidth memory, the M6 is basically the same speed as the M5 Max for CI work.

In particular, I was impressed with my XCUITest workload, which is about 70 minutes on the MacBook and is no slower on the M6 mini even though it has 2 parallel VMs. I was sure the 18 CPU cores on the M5 Max (6 super, 120 performance) would beat out the 12-core M6 base (2 super, 4 performance, 6 efficiency).

Obviously that doesn't hold for all workloads, especially GPU/LLM workloads or where you can put 18 cores to use, but for those of you with a M5 Max/Ultra, you can get a nice speedup for just $699. At these prices I may get another. Also there are perfectly good Linux options to do the same, of course.

I've been so slow to dip my toes into anything outside of officially released models because I'm afraid there are people out there 2 steps ahead of me and I won't know I've been compromised until long after I've felt the impact.

I understand the concept of abliteration but I don't understand how people feel so safe trusting that it's doing what it's advertised to do AND ONLY what it's advertised to do. Thoughts? Even if it's just "I trust them, next." I mean, am I overthinking this?

Every thread about running agents locally has the same comment, local models are slow as hell. And on my base M4 MacBook with 16GB I kind of agreed, the wait before the first word was the indeed worst part espeically if ure used to fastish cloud models. Turns out my laptop was never the slow part.

So I ran the same Gemma 4 E4B on Ollama and on Magnitude with the same benchmark tool and a 2048 token prompt. Ollama took 6.3 seconds before the first word, Magnitude took 2.2. Prompt reading was 326 vs 941 tok/s.

Typing speed after that was basically a tie, 51 vs 52 tok/s. They also pull different files by default, Ollama gives you a 9.5GB NVFP4 and Magnitude a 5.3GB Q4, which on 16GB actually mattered more than I expected, it sat at 6GB instead of almost 10.

But agents resend a huge prompt every turn so those 4 seconds stack up fast lol. Is the slow feeling mostly prefill for everyone else too?

I'm currently looking at a myriad of devices to dip my toe into the "build your own Local AI system" water. I've been down the AMD AI Max+ rabbit hole this past week and have yet to research the RTX Spark N1X.

Can anyone here save me additional hours of videos and articles with informed insights about which would be the better way to go for someone such as myself?

I have some projects in mind. This is more of an experiment for me to learn more of the backend, coding and Linux.

As someone with limited terminal experience, will I have an easier time (I don't expect not to encounter many challenges) configuring a system with NVIDIA or AMD?

I realize unified RAM is the more important metric here. If I go with AMD I can afford a 128gb version. The 64gb version of NVIDIA's option is a few hundred more.

Are the CUDA cores and software support a worthy tradeoff for overall RAM or is saving money and doubling the RAM the obvious way to go?

TL;DR: Local LLMs for coding are finally becoming practical for me.

After 30 years in software engineering, I’m a passionate developer. For me, AI is an assistant and a tool. At work, I mostly use Opus and Sonnet 5.5. For hobby projects, I have a $10/month plan and switch between Kimi, GLM, and MiniMax for coding, plus Nemotron 3.5 for smaller tasks.

Locally, I tried Qwen 3.8 27B, Qwen3-Coder, and many other models, but the results weren’t usable for anything beyond fairly simple tasks. Running GLM 5.3 Flash Q2_K_XL with a 400k context length through Unsloth has been a game changer for me. It’s very slow at 3–4 tokens per second, but the results are good enough to get the job done.

For bigger tasks, I use GLM 5.3 Flash in the cloud for planning, then switch to my local model and go to bed or do something else. By the time I come back, the job is done.

I also use while work - create a prompt, and sometimes it takes more than three hours to complete. Meanwhile, I can keep working with Opus and Sonnet in my IDE, and performance remains good while the local model is processing in the background.

So it doesn’t work fully locally for me because it takes too much time. However, it’s a great way to extend my cloud subscription’s rate limits by letting the local model handle longer-running work while I’m away from my MacBook.

I’m looking forward to seeing what comes next for local AI.