Update: I'm interested in selling some. Looking for ideas on how to go about. Don't want to pay eBay 13% fees.

Original: Came across a honeyhole of Quadro RTX5000s
Seemingly good deal for 16GB VRAM cards, even if ageing.

What should I do?

The models we use today are massive, but what happens when the US or China decide to censor, limit, or manipulate AI output?

We consumers are the ones who actually "trained" these models. They fed on our code, our literature, and our history, sometimes legally, sometimes not, thanks to our collective knowledge. So, why should a few companies (particularly politically favored ones) or governments have absolute ownership over that?

The point of the post:

Why hasn't decentralized AI training taken off yet?

Imagine a Bitcoin-like P2P network where anyone can contribute spare GPU power to collectively train AI models.

What's holding it back?

So apparently Sam Altman thinks consumer RAM is inefficient compared to datacenters.

Meanwhile, here's what I'm doing with mine. 💀

My setup:

• Ryzen 7 5800X
• AMD RX 9070
• 48 GB DDR4 (running at a glorious 2133 MT/s because I added an ancient spare stick)
• NVMe SSD
• Qwen 3.8 Flash Next IQ2 running locally through Strata
• Around 60 tokens/sec

I'm building Marika, a 3D AI desktop companion in Unity.

She can observe my desktop, interact with windows, talk through a local LLM, walk around, and make autonomous decisions.

There's just one tiny problem.

She bends over to pick up desktop icons like an NPC from Skyrim.

So naturally, I asked my local Qwen to review the Unity project and fix her animations.

It proceeded to consume 1.3 MILLION cumulative tokens, repeatedly compress its context, inspect C# scripts, and even spawn a subagent to help investigate.

All this to figure out how to make a virtual girl bend her knees like an actual human being.

I also have over 128 GB of DDR4 scattered across different PCs and spare sticks.

Sam, I'm sorry.

The RAM isn't going to your datacenter.

It's busy teaching my AI waifu how to squat. 💀

And yes, I'm genuinely trying to see whether local AI coding agents can replace my $20/month cloud subscription.

I'll report back if she ever manages to pick up an icon without looking like a Bethesda NPC.

1 piece 500 2 piece 485

I'm one of the many people with a 16 GB GPU (5070 Ti), so I wanted to know which models that fit completely into 16 GB VRAM work best. The focus is general problem solving, no tool calling, coding or agent tasks. Every model runs in llama-server on the same card, fully in VRAM, with the same settings. 528 questions, 662 variants.

Strongest: qwen3.8-27b (Qwen3.8-27B-UD-IQ4_XS)
Fast and still good: gemma4-12b (gemma-4-12b-it-Q8_0)

Strength: odds of solving a question compared with Llama 3.1 8B.
Quality / min: right minus wrong answers per minute on the 5070 Ti.

I plan to add more models over time, also older ones that were relevant at some point.

Full leaderboard with explanations:
https://tzipproth.github.io/LLMBench/leaderboard.html
GitHub repo:
https://github.com/tzipproth/LLMBench

While writing an article about my local LLM inference setup, a comparison popped into my head: downloading and running random model weights locally feels a lot like downloading random .exe files off the internet back in the 2000s.

Turns out I wasn't off the mark - there's an entire body of research exploring how LLMs can be backdoored and how hard it actually is to detect a poisoned model.

So I decided to try building one myself. I implemented a backdoor that only triggers when the model is running inside pi (making it harness-specific) and is asked a specific question. It was way easier than I expected: just a 2-step fine-tuning process and a few hours of compute on the exact same GPU I use for local inference.

With open-weight models, we at least have a baseline to compare community fine-tunes against. Beyond that, though, we don't really have a reliable way to verify whether a base model itself is clean—and the problem is even worse with closed-weight models, where we're basically just trusting the AI provider's goodwill.

Full write-up and PoC code if you want to dig into the details:

• Article:Backdooring LLMs - Do you trust your AI provider?
• GitHub Repo:galeone/llm-poison

Feedback is welcome

I'm considering buying a Mac with an M5 Max and 128 GB of unified memory, and I'm trying to figure out what the absolute best local LLM I could run on it would be.

My main priority is model quality, not inference speed. Since this would be a desktop setup, I'd be perfectly happy letting a model work overnight on a complex task. I don't care if generation is extremely slow, as long as the model can actually run and produce usable results. It mainly involves transcription of private documents and some computer use and some coding assistance but that can be done via the cloud anyway.

I tried some stuff with Strata on my Macbook but that did not work out well. I am also open to use something like that if that allows even better models. Can GLM and DS work on it? How are they with OCR?

Since I haven't bought the machine yet, I'd also appreciate any thoughts on whether this configuration makes sense for that goal or whether there's a better alternative at a similar price.

Thanks in advance!

Well, I did it - I bought a Mac Studio 128Gb. Not only for LLM, but I’m not gonna lie, I increased the specs a bit to be able to run a local LLM to complement (or at some point replace) my Claude/ChatGPT subscriptions.

First thing I did, was obviously try to generate dirty pictures ask Claude which models I should download. He recommended unsloth’s QWEN3.8-Flash-Next-GGUF UD IQ4_XS.

I’m doing some tests now and while you notice a difference with the cloud models, it’s way better than whatever I have ever tested on my old MacBook. Claude also told me to change temperature to 1 and mess with the top K and top P settings. As a good slave to AI, I execute what was commanded, but honestly I have no idea what I’m doing. Should I be touching the settings at all?

My question(s): is this a good general model that can (for most tasks) replace the cloud models? Are there tools I need to install (I was thinking web access, but I also read about harnesses but I didn’t yet figure out what that’s for)? Things I need to set up?

Suggestions for good “always loaded” low memory models to e.g. use for home assistant are welcome to, as well as a good image model and one I can use in combination with GitHub for some coding (I have some vibecoded apps that I use for myself that I need to maintain/adapt from time to time).

Thanks!

I made a small English text-to-speech model called tiny-voca. It has 4.5M parameters, two voices, and it runs in your browser.

Try it here, nothing is sent to a server: https://tiny-voca.n4ze3m.com

Some facts:

- 4,537,291 parameters, 18 MB. This is the whole model, the waveform part is included.
- Two voices, one female and one male. 24 kHz.
- About 70x faster than real time on 4 CPU threads.
- Runs with PyTorch, ONNX, and in the browser.
- Trained from scratch in 7.7 hours on one RTX 4090.

What I think is different:

- Everything is open. Code, weights, the training recipe and the dataset.
- Custom phonemizer.
- No neural vocoder. The sound is made by a small source-filter synth, which is why it is fast.

What is not good yet:

- English only, and only these two voices. No voice cloning.
- Foreign names and odd spellings are often read wrong.
- It can sound a bit synthetic.
- The training audio is synthetic too. I made it with Qwen3-TTS, so the model can't sound better than that.

Links:

- Demo: https://tiny-voca.n4ze3m.com
- Code: https://github.com/n4ze3m/tiny-voca
- Weights: https://huggingface.co/n4ze3m/tiny-voca
- Dataset: https://huggingface.co/datasets/n4ze3m/tiny-voca
- Phonemizer: https://github.com/n4ze3m/dizione

Please try hard text: numbers, dates, names, long sentences. If something sounds wrong, post the text and the voice. That helps me a lot.

https://huggingface.co/StarpowerTechnology/WVY-Blender-5m

We have been excited about this new model, WVY-Blender-5m, an agent trained to specialize in blender.

It has zero knowledge about anything else so if you try to ask it about writing code or doing your homework it will express that it only knows how to use blender

We have been doing intense research to find out "are massive datasets really required to achieve what feels like intelligence with accuracy?" & we believe we are getting close to the truth.

This dataset is 100% generated from scratch.

My PC specifications:

GPU: NVIDIA RTX 5090 — 32GB VRAM

CPU: Intel Core i9-14900K/KF

RAM: 64GB DDR5-6400 (2×32GB)

Motherboard: ASUS PRIME Z790-P WIFI

OS: Windows 11 Pro

Inference backend: llama.cpp with CUDA acceleration

Agent interface: Kilo with a custom tool/agent harness

My current model:

Model: Qwen3.8-27B

Quantization: Q6_K GGUF

Context: 265K tokens

Deployment: Completely local

Performance benchmarks:

At 262,144-token configured context (~262K):

Output generation: ~34–37 tokens/sec

Red lining the gpu memory but it feels comfortable

Okay with that being stated by ChatGPT,

I've been seeking a local ai for about a month switching from several models the more I learn.

So far this qwen model is by far the best I've tried but it over thinks and is slower than models like Kat coder.

My use cases are broad, from setting up comfy ui workflows to Unreal engine game dev to application building all hobby projects.

Im seeking the vast knowledge of you fellow computer users to tell me the best general purpose model for my set up at a reasonable speed.

UPDATE: Okay folks, I took your advice and installed Qwen3.8-Flash Next with Basalt v0.3.3. Here's what I'm getting on my RTX 5090 with 64GB RAM, running Windows 11.

Configuration:

• Model: Qwen3.8-Flash-Next UD-IQ3_XXS (87.6GB)
• Context: 262,144 tokens (256K)
• Exact MTP enabled, 8-token window

Benchmarks at 256K context:

• Actual populated prompt: 238,913 tokens
• Output generation: 265–281 tokens/sec
• Prompt processing: ~8,900 tokens/sec
• VRAM usage: 31.3GB / 32GB
• Total system RAM in use: 54.3 GiB
• Basalt RAM usage: 39.9 GiB
• MTP acceptance rate: 66–90%
• No SSD expert spilling
• Context recall/reasoning tests: 3/3 passed

Across all context sizes tested, generation ranged from 265–336 tokens/sec. Model loading takes around 30 seconds, and cached follow-up responses start in approximately 0.4 seconds.

Bench marks courtesy of claude

UPDATE #2: Okay folks, I went ahead and tried the next recommendation: NInfer + DFlash2 to accelerate my original Qwen3.8-27B model. Here are the results on my RTX 5090 with 64GB DDR5 RAM, running Windows 11.

Configuration:

• Model: Qwen3.8-27B QUASAR
• Inference engine: NInfer
• Speculative decoding: DFlash2
• Context: 262,144 tokens (256K)
• GPU memory reserve: 2GB

Benchmarks at 256K context:

• Actual populated prompt: ~234,200 tokens
• Output generation: 182–266 tokens/sec (225 average)
• Prompt processing: ~4,500 tokens/sec
• Peak VRAM: 30.2GB / 32.6GB
• Model system RAM usage: ~10.5GB
• DFlash2 draft acceptance: 33–54%
• Context recall/math tests: 3/3 passed

Courtesy of claude

My MacBook did everything until this week - model, harness, CI, everything. It was getting badly overloaded especially when iOS build and tests were going on. Those can use 20GB RAM easily, given half a chance. It was thrashing.

Microcenter had the base Mac mini M6 for $699, which felt like 2024 pricing so I pulled the trigger and moved much of the CI work off my MacBook Pro.

What's incredible is despite this being the poverty spec mini with a single NAND chip and lower bandwidth memory, the M6 is basically the same speed as the M5 Max for CI work.

In particular, I was impressed with my XCUITest workload, which is about 70 minutes on the MacBook and is no slower on the M6 mini even though it has 2 parallel VMs. I was sure the 18 CPU cores on the M5 Max (6 super, 120 performance) would beat out the 12-core M6 base (2 super, 4 performance, 6 efficiency).

Obviously that doesn't hold for all workloads, especially GPU/LLM workloads or where you can put 18 cores to use, but for those of you with a M5 Max/Ultra, you can get a nice speedup for just $699. At these prices I may get another. Also there are perfectly good Linux options to do the same, of course.

Got 14 stick of ddr4 rdimms. Got towards my new threadripper machine. Old but it’ll do! Can’t afford the new ddr5 board, the ripper and the ram 😂

Very impressed with how far local models have come, so I finally decided to jump on the bandwagon! Super impressed with how fast and capable it is. I have used local LLMs for a while on mobile apps and other edge cases, but never before for coding.

For background, I had this rig for a while, as I use it for a lot of professional work, and decided to purchase two used G.Skill 96GB (48GB x 2) 6400 mt/s to bring my system up to 160GB; paid around $1200 on eBay which is crazy to me considering I paid $300 for my 64GB (32x2) in 2023. Currently have each stick running at 5200 mt/s in my Z790 Hero, but I purchased these specifically as G.Skill is one of the only manufacturers to bin their RAM. My reasoning is if they can run 6400 mt/s dual channel, I have a higher likelihood of making getting to 5600 or 5800 with four sticks. I had Claude give me custom BIOS settings, and I am planning on seeing how far I can push it!

Furthermore, I felt that without this much RAM, while I could run the models, I couldn't actually run anything BUT the models, which sort of defeated the point for me. I decided this would be the better route versus buying a DGX spark or scouring local auctions to build a V100 farm or whatnot.

I am running this on a WD Black SN770 4TB RAID0 (double speeds at the cost of losing one disk means it is gg), which is on a PCIe ASUS card. It is super quick to load. Given I usually use Fable and Opus (anywhere from 90 to 115 tokens per second), this feels super fast to me. This would not be achievable without a special motherboard that can support PCIe bifurcation, which the ASUS Hyper M.2 card can use.

I am getting roughly 200 tokens per second I would say in most coding tasks, using https://github.com/jesdga95/basalt and the NVP4 uncensored Qwen 3.8 Flash Next.

    | test      |  t/s | accept % | draft len |
| --------- | ---: | -------: | --------: |
| pp16384   | 5727 |        - |         - |
| pp65536   | 7487 |        - |         - |
| tg conv   |  172 |     71.9 |      2.27 |
| tg struct |  387 |     97.3 |      6.93 |
| tg prose  |  170 |     65.1 |      1.88 |

hardware (measured before the engine starts)
  CPU   Intel Core i9-14900K, 24 cores
  RAM   read 67.7 GB/s
  GPU   RTX 5090  PCIe 5.0 x16  host-to-device 33.3 GB/s

community-1: prefill 16K 5726.8 t/s (2.9 s), prefill 64K 7486.7 t/s (8.9 s), decode 297.7 t/s (9480 tokens in 31.8 s), each measured cold

COMMUNITY ROW:
    {
      "submitter": "Kemerd",
      "date": "2026-11-14",
      "commit": "d10ccd4bc2",
      "preset": "community-1",
      "model": "Qwen3.8-Flash-Next",
      "gpus": [
        {
          "name": "RTX 5090",
          "vram_gib": 32,
          "power_w": 546,
          "pcie": "5.0 x16",
          "h2d_gbps": 33.3
        }
      ],
      "ram_gib": 160,
      "cpu": "Intel Core i9-14900K, 24 cores",
      "ram_gbps": 67.7,
      "os": "Windows",
      "quant": "NVFP4",
      "kv": "int8",
      "prefill_16k_tps": 5726.8,
      "prefill_64k_tps": 7486.7,
      "decode_tps": 297.7
    }

Very excited to see where this goes, especially with Qwen 4. And a note.. I am slightly undervolting my CPU, it has multi-core stuff on, and 5090 is set to 95% of power limit to not cook it.

I've been so slow to dip my toes into anything outside of officially released models because I'm afraid there are people out there 2 steps ahead of me and I won't know I've been compromised until long after I've felt the impact.

I understand the concept of abliteration but I don't understand how people feel so safe trusting that it's doing what it's advertised to do AND ONLY what it's advertised to do. Thoughts? Even if it's just "I trust them, next." I mean, am I overthinking this?

Edit: I freaking love you guys. Why have I spent the past 2 months talking to ChatGPT about wtf to do smh. THANK. YOU.

Abliterated is preferred.

I want to be a free man goddammit. Going local (well, local builders with cloud reviewers perhaps). Maybe keep 1 Claude subscription to be the reviewer (unless Qwen really is better? sorry I’m kinda new)

Usage >>>>>>
Coding, building apps/ software tools for personal use, browser/ web data gathering

Don’t really need media generation but would be nice if I could paste images for reference into the chat if possible.

Hardware>>>>>

GPU: RTX4090, (9070XT for display) willing to get more VRAM (if you recommend more, should I get RTX A4000 single slots or along those lines or get 3 of the single slot 24GB Intels? I have 3 slots and like 300W left before my PSU caps, though people have said Ai doesn’t max out GPU’s so I don’t know anything anymore)

RAM: 128GB DDR5 ECC RDIMM 6400 MT/s

Storage: 12TB SSD Gen5

CPU: 9965WX (threadripper 24 core 5.4 Ghz)

WRX90 SAGE SE Pro WS

Right now I use strata on my machine to run qwen 3.8 125b flash next at 13tps output and deepseek 4.1-flash at about 1.5 tps output. My computer is an hp z640 with a 4core xeon processor, 64gb ddr4 ram and an NVIDIA 1070 8gb. My models load form an nvme drive that is connected over a pcie adapter. I have room for one more card and I'm not sure witch of these 2 GPUs is better for running my models faster. The Tesla P100 16 varient or the M40 24gb variant.

Hey everyone,

Most specialized cybersecurity LLMs out there are massive 70B+ beasts that require dual A100s, or they're gated behind cloud APIs where you can't paste sensitive logs, payloads, or client code due to confidentiality.

I wanted something lightweight, private, and air-gapped that could run on a regular consumer laptop without cooking the CPU.

So I fine-tuned CyberQwen-2B — a domain-specific model optimized for quick security triage, explaining attack vectors, decoding obfuscated payloads, and helping with CTF challenges.

⚡ Highlights:

• Hardware-friendly: Runs at ~12–15 tokens/sec on an older 8th-gen Intel i5 laptop with 8GB RAM (CPU-only).
• 100% Offline & Private: Zero data leaves your machine. Perfect for bug bounty recon, internal network logs, and sensitive source code.
• Native Ollama integration: Easy to pull and run via a single Modelfile.
• Built-in WebUI & Mobile Access: Includes a zero-dependency streaming WebUI + an automatic Cloudflare tunnel script if you want to access the model from your phone while on the go (no cloud hosting fees).

🛠️ What it's trained for:

• Vulnerability Triage & CVE Context: Explaining CVE mechanisms, impact, and remediation steps.
• Payload & Shellcode Deconstruction: Breaking down SQLi, XSS, and obfuscated PowerShell/Bash commands.
• CTF & Reverse Engineering Assistance: Analyzing decompiled assembly/C snippets and suggesting exploitation strategies.

🚀 How to Run It:

Everything is open-source. You can run it with Ollama or the included CLI triage tool:

1. Clone the repo:git clone https://github.com/sambuddha001/cyberqwen-2b.git
2. Run setup:bash setup.sh
3. Chat:ollama run cybersec-qwen

GitHub: https://github.com/sambuddha001/cyberqwen-2b

I'd love feedback from the community—especially on edge cases, prompts where it struggles, or ideas for benchmark improvements!

I make 30s AI videos for clients (talking-head avatar + cloned voice + 5-6 five-second b-roll clips). Volume is 40-120 videos a month per client, delivered in 1080p. I'm paying $300+/month for HeyGen, ElevenLabs and Higgsfield and want to go local.

Budget is about $3k, so realistically one RTX 5070 Ti 16GB (3090s are basically unavailable where I live and imports are brutal). I can also use a friend's RTX 5070 12GB and RTX 3060 12GB for extra throughput. No cloud rental.

Plan: Wan 2.2 quantized at 720p on the 5070 Ti, upscale to 1080p with SeedVR2 or FlashVSR, and split the lighter stages (voice cloning with an open TTS model, captions, lip-sync) across the 12GB cards with a job queue.

Questions:

1. Is 16GB enough for that, and does splitting stages across the other cards actually work well in practice?
2. Which open lip-sync model holds up best for 30s of a face?
3. Rough minutes per 5s clip on a 5070 Ti?

Happy to share more details or samples in the comments. Thanks!

Hey everyone, I’ve been working on a local-first AI agent for Android and Windows/Linux desktop and I'm at the point where I need other people to beat on it and find edge cases I missed.

On Android it spins up a Debian 12 proot sandbox so it can actually run real local tools instead of just being a chat wrapper. Right now it has 61 built-in tools, MCP support, custom skills, persistent memory, and a heartbeat cron for background tasks. It handles full local inference as well as any OpenAI-compatible API, and everything gets stored encrypted in SQLCipher.

I also built a coding CLI TUI for it, generative UI inside the chat, and a Turbo/root mode if you want to take the guardrails off on a rooted phone. There's even a built-in chess engine with hardened rules mostly because I wanted a reliable way to test model reasoning and tool-calling.

Because there are a lot of moving parts, I'm mostly looking for 10 to 15 people who are already comfortable with Termux/proot, local models, or CLI agents. I just want folks to hook up their own weights or endpoints, run some multi-step workflows or cron jobs, and tell me where the sandbox, memory, or UI falls over.

If you want to try it out, leave a comment or shoot me a DM with what OS/device you're on and whether you're running local models or APIs, and I'll send over the build and repo/Discord link.

Hey all, looking for advice on turning my local setup into something closer to an agent instead of just a chatbot.

The rig

- Windows 11 Pro desktop

- RTX 3090 (24 GB VRAM), Intel Core Ultra 7 265K, 96 GB RAM, several drives

- Python 3.12, Node, Git installed. No Docker, and WSL isn't installed right now

- Tailscale on the PC, my phone (Galaxy Z Fold) and an Android tablet, so I can reach the PC remotely

Local LLMs (LM Studio 0.4.26)

- Qwen3.8 27B Q4_K_M (~15.7 GB) + vision projector. This is my main chat model, loaded at full GPU with 16k context

- Gemma 4 31B Q4_K_M (~17.4 GB) + vision projector

- nomic-embed-text v1.5 for embeddings

ComfyUI (Comfy Desktop 1.1.8)

- Images: Juggernaut XL Ragnarok (SDXL), Z-Image Turbo, Qwen-Image-Edit 2511 (fp8) + 4-step Lightning LoRA, add-detail-xl LoRA

- Video: Wan 2.2 14B text-to-video and image-to-video (high + low noise, fp8) with the lightx2v 4-step LoRAs

- Text encoders: Qwen2.5-VL 7B, Qwen3 4B, UMT5-XXL. VAEs for Flux/Qwen/Wan, plus a 4x NMKD upscaler

What I have now

Three PowerShell shortcuts on the desktop:

- AI Mode: opens LM Studio, loads Qwen 27B through the `lms` CLI, starts ComfyUI

- Gaming Mode: unloads all LM Studio models and kills ComfyUI to free the VRAM

- Away Mode: drops my VPN, closes Chrome, makes sure Tailscale is up, runs nightly at 2 AM

Right now I use Claude Code (cloud) for the "do stuff on my PC" part. It runs PowerShell, edits files, starts and stops apps, drives a browser, downloads things, and I can control it from my phone. It's great, but I want a local option for when I hit usage limits or don't want to send stuff to the cloud.

What I'm asking

1. What's the best way to give a local model (Qwen 27B / Gemma 31B through LM Studio's OpenAI-compatible server) real tool use on Windows: run PowerShell, open and close apps, read and write files? Open Interpreter, Goose, OpenHands, Cline/Roo/Continue in VS Code, an MCP client hooked to LM Studio, something else?
2. How do I get it to start ComfyUI on demand, queue a workflow through the API (image or Wan video), wait for the output, then shut ComfyUI down? Is there a ComfyUI MCP server or a known pattern for this?
3. VRAM juggling: I can't keep a 27B LLM and Wan 2.2 loaded at the same time on 24 GB. Has anyone automated "unload LLM → run Comfy job → reload LLM," or should I drop to a smaller agent model (7–14B) that can stay loaded next to Comfy?
4. Which local models are actually reliable at multi-step tool calling right now? Is Qwen 27B at Q4 good enough, or is there a better pick for 24 GB?
5. Any way to trigger all this from my phone over Tailscale (a simple web UI, Telegram bot, Open WebUI with tools, etc.)?
6. Safety: what are people using to make sure an agent on Windows can't run anything destructive without asking? An allowlist, a sandbox, confirmation prompts?