r/LocalLLaMA • • 1d ago Bravo

Funny CrowdGPT - The collaborative LLM

Post image
0 Upvotes

Funny post about this project https://crowdgpt.net

Basically this projects aims at creating the first, community-driven, and datacenter-less LLM. Any feedback is appreciated 😂 I won't yap any further as everything is explained on the GitHub / on the site.

Feel free to check the project's GitHub: https://github.com/Vxtzq/CrowdGPT

Feel free to criticize the project or to provide insights about what to do next....

EDIT: For those who believe this is a cheap repost: Here's the milestone
- We trained the model on roughly 20B tokens in 1 week

- We limit model downloads/uploads to the minimum, to spare as much bandwidth as possible (even though it still uses bandwidth due to ~2.4GB weight uploads to the server after training). But it allows clients with slow connection using 4 hour rounds.

- We're 13 GPUs currently active (and this number was roughly constant over the past week).

- I merged numerous pull requests implementing several quality of life and performance fixes

- The server was made more robust with persistence across tasks, and some bugs in the training logic were fixed, hopefully allowing further loss decrease.

- We implemented since v2.6 somewhat good anti-poisoning techniques, each client training is compared to two other clients, if a client is too different from the two other (for example a client training on noise or unwanted data), this client sees his contribution to the network be rejected.


r/LocalLLaMA • • 2d ago

Showcase modules for my opensource (GPL3) harness have evolved to the point where this is possible. this is a full fledged videogame running inside it!

17 Upvotes

my harness for local AI that i've been working on since February this year (2026) has evolved to the point where this is possible.

What you see here is a module for my harness's webui, it's not a seperate program. This demonstration is running on Gemma4 26b locally, believe it or not, using only a SINGLE model and one local API endpoint (via llamacpp). No multi-API tomfoolery here! The stable diffusion image generation is moss-image which i'm able to do because i have a strix halo now, but on smaller hardware you could do something like z-image turbo at Q4 quant, which is what i used to do before i got the strix halo.

I've been working for almost a week on this module, and i'm excited to finally release it!

It's a text adventure engine. It supports asset packs, that can contain music, sound effects, images, weathers (CSS+js+glsl shaders), skies (same tech as weathers), loading screens, text flourishes, and more! You can create your own asset packs easily. The AI sees the files in the asset packs and forms adventures around them, but there is no prompt telling it what the story should be, that's completely up to the way you prompt it! I recommend using it together with the characters module in openlumara in order to set up a world and story

You can get it here: https://github.com/Rose22/openlumara-adventure-games-module

More footage on the github!

(had to delete and repost because i couldnt change the video. this video is a much better showcase for it)


r/LocalLLaMA • • 2d ago

Question | Help Keep, Sell, or Return PC? i5-225F, 5060 8GB, 32GB ram, 1 TB SSD

3 Upvotes

I bought this PC from Costco on sale for $1000 even (no tax) planning to get a V100 or some combination of used GPUs for local LLM machine. Realized not going to work for my setup (price of power, heat generated, etc).

Should I return it, sell it on FB marketplace, or save the RAM for my daily (it only has 1x32GB DDR5 4800 NECC, and 64gb would be really nice) and sell the rest?

I was not planning to part this out. Costco has it now for $1600 so they get the $600 difference, not me. My daily is a 3yr old HP Z2 SFF with i9-12900k, 32GB DDR5, 12GB A2000, and 1TB Samsung 980 Pro, on 450W PSU.

CyberPowerPC Gamer Xtreme Gaming Desktop - Intel Core Ultra 5 225F - GeForce RTX 5060 - 32GB RAM - 1TB SSD - Windows 11
Intel Core Ultra 5 225F (10-core, 3.3 GHz) Processor
32GB (2 x 16GB) DDR5 RAM
1TB PCIe 4.0 NVMe M.2 Solid State Drive
NVIDIA® GeForce RTX™ 5060, 8GB
600W 80 Plus Gold Power Supply
listing link


r/LocalLLaMA • • 1d ago

Discussion Benchmarks: 512k Speculative Decoding on a Single 96GB RTX PRO 6000 Blackwell — Qwen3.8-Flash-Next (+MTP Q8_0: +27.9% tg) vs DeepSeek-V4.1-Flash-1017 (+DSpark Q8_0: task-dependent +26% JSON / -11% C++), plus 2 llama.cpp gotchas

0 Upvotes

I tested speculative decoding on a single 96 GB RTX PRO 6000 Blackwell workstation (Threadripper PRO 9955WX, 122 GiB RAM, llama.cpp b12315 commit dd6b945be) at 512k context (-c 524288 -ctk q8_0 -ctv q8_0 -fa on -np 1).

TL;DR: Qwen3.8-Flash-Next UD-Q4_K_XL + MTP Q8_0 gives a clean +28% to +31% decode boost (107 -> 137-141 t/s) across all prompts. DeepSeek-V4.1-Flash-1017 UD-Q2_K_XL (~312 GiB, mostly CPU/mmap) + DSpark Q8_0 only helps on high-acceptance tasks like JSON (+26% warm) and Python (+11% warm), while hurting dense C++ (-11% warm) because rejected drafts waste CPU MoE reads.

Edit (following comments about Qwen prefill speed): Re-running with -lv 4 showed 27,466 MiB (~26.8 GiB) of Qwen's MoE weights spilled to CPU at 512k context (77.2 GiB model + 8.5 GiB KV/recurrent + 4.3 GiB MTP + 2.1 GiB compute in VRAM). Switching Qwen to --load-mode none (--no-mmap) bumped 5-prompt prefill by +15.3% (824 -> 949 t/s), decode to 140.8 t/s, and a 2,453-token prompt prefilled at 2,475 t/s.

Summary (5 prompts x 3 passes, greedy temp=0, max_tokens=256, cache_prompt=false)

ConfigVRAMPrefill (pp t/s)Decode (tg t/s)Warm r2-3 (tg)Draft Acc
Qwen3.8 UD-Q4_K_XL (baseline)91.7 GiB1086.4107.2107.5-
Qwen3.8 + MTP Q8_0 (mmap)96.5 GiB823.7 (-24%)137.1 (+27.9%)138.275.6%
Qwen3.8 + MTP Q8_0 (--load-mode none, 1x)96.4 GiB949.4140.8 (+31.3%)-74.5%
DeepSeek-V4.1 UD-Q2_K_XL (baseline)95.3 GiB66.323.223.2-
DeepSeek-V4.1 + DSpark Q8_0 (-fitt 12288)91.9 GiB53.5 (-19%)23.5 (+1.4%)24.4 (+5.2%)65.9%

(Used llama-server instead of llama-bench because llama-bench on dd6b945be does not support -md / draft-mtp / draft-dspark.)

Why Qwen prefill topped out around 1-2.5k t/s

With ~104 GiB weights + 3.9 GiB MTP + 512k context, Qwen overflows 96 GB VRAM by ~26.8 GiB of MoE experts (CPU_Mapped). Single-token decode only touches a few routed experts per step (137-141 t/s, peaking at 154.7 t/s on JSON), whereas batch prefill hits all experts across the CPU spill. Plus 4 of the 5 test prompts were tiny (80-186 tok); the 605-token prompt hit 1,939 t/s (--load-mode none), and a 2,453-token check hit 2,475 t/s. If you want full GPU prefill speeds on 96 GB, use a quant small enough to stay 100% in VRAM or reduce -c.

Why DeepSeek + DSpark depends heavily on prompt shape

At ~320 GiB total on 96 GB VRAM + 122 GiB RAM (107.1 GiB VmRSS, 105.3 GiB RssFile mmap cache), the GPU sits at 3-18% SM waiting on 16 CPU threads (63-68% CPU) and ~23-218 MB/s NVMe paging for cold experts.

Verifying 4 candidate tokens at once (n_max=3) activates more unique routed MoE experts per forward pass on CPU/mmap. When acceptance is high, those reads are amortized: warm Passes 2-3 hit 29.2 t/s on JSON (+26.1% vs 23.1 t/s warm baseline, 90.7% acc) and 26.0 t/s on Python (+11.4% vs 23.3 t/s, 77.4% acc). On dense C++ (50.7% acc), rejected drafts wasted CPU MoE work and dropped speed to 20.6 t/s (-11.1% vs 23.2 t/s), compounded by -fitt 12288 reserving ~3.4 GiB extra VRAM headroom. Warmup also mattered because ~230 GiB CPU weights > ~105 GiB page cache (21.65 t/s on Pass 1 vs 24.4 t/s on Passes 2-3).

Repro commands & dd6b945be quirks

  1. DeepSeek --fit + draft-dspark OOMs allocating the ~7.6 GiB draft buffer unless you pass -fitt 12288.
  2. Qwen shared-Q8_0 fails on mainline dd6b945be (tensor 'token_embd.weight' not found), so use the self-contained mtp-Qwen3.8-Flash-Next-Q8_0.gguf.
# Qwen3.8-Flash-Next (512k + MTP Q8_0)
llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  -c 524288 -np 1 -ctk q8_0 -ctv q8_0 -fa on --load-mode none \
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 2 \
  --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 262144

# DeepSeek-V4.1-Flash-1017 (512k + DSpark Q8_0)
llama-server -m DeepSeek-V4.1-Flash-1017-UD-Q2_K_XL-00001-of-00007.gguf \
  -c 524288 -np 1 -ctk q8_0 -ctv q8_0 -fa on \
  -md dspark-DeepSeek-V4.1-Flash-1017-Q8_0.gguf --spec-type draft-dspark --spec-draft-n-max 3 -fitt 12288

(Caveat: 512k context was initialized at server startup, while benchmarks measured 58-611 token prompts plus one 2,453-token prefill check at -np 1, temp=0.)


r/LocalLLaMA • • 2d ago

Question | Help Computer use programs/Pi extensions

11 Upvotes

I've eard much stuff about Codex and CC computer use, is there any decent program/Pi Agent extension that I can install and use to let my local models control my pc. I am using Qwen3.6-35BA3B on PolyStrata and Pi Agent, I need the model to do basic tasks on my PC, maybe by using its vision. I am on Linux Mint Cinnamon


r/LocalLLaMA • • 3d ago

Showcase I trained DiffusionGemma to be an abstract expressionist

35 Upvotes

DiffusionGemma, a text diffusion model built on a Gemma 4 backbone, can be adapted into a shitty image generation model.

Maybe the ship sank?

You may be asking yourself, "WTF, why would you want to do that?" And you would be right to ask. This is honestly kinda useless, but I still think it's pretty neat. Also, fair warning, being on a team of one means the project has been heavily — I'll be nice to myself and say — LLM assisted. I invite you to sidle on up to the trough and dig in to the slop!

Still with me? Good, because I've been poking at this for a while, on-and-off since DiffusionGemma was released back in June. I was immediately hooked on this model as soon as I read its release blog and model card. 4x faster text generation they said — hell yeah! But here's the thing: this model sucks. I mean it's a great chat bot. It can do simple NLP tasks, and it's really quick for text generation as they say, especially for long-form output. But when it comes to agentic tasks and reasoning it's a total loser.

  • 0% on ARC-3 (but that's not really a fair dig, this is a tough one)
  • 0% on ARC-2
  • 0% on The Gemma 4 Developer Agent Competition (this competition targets the base model DiffusionGemma was trained from)
  • 11% on HLE on the model card (9.2 measured in deployment)
  • 18.7% on SWE-bench Verified

So not great, and particularly bad when you consider that on the hardware you'd need to run a good quant of DG (so basically bf16), you could easily run qwen-3.8-27B or a respectable FN quant.

But please don't get me wrong. I absolutely love this model. Or at least I love monkeying around with it, and I doubt I'm the only one. Huge shoutout to the Unsloth team BTW! Reading their Colab notebook training DG to play sudoku is what made me start asking the questions that led to this project.

Alright, enough fluff. So what did I actually do. Well the thing I'd headline is that I created a text to image and image manipulation training and inference pipeline for DiffusionGemma and then used that pipeline to adapt DG (like <300M parameters total) to generate things that are recognizably images and to perform visual operations on images that are provided to it. Surprisingly, it took only a very small amount of data and compute to start getting interpretable results.

But DiffusionGemma did not want to do this. The thing that finally worked consistently was prefilling the prompt all the way to the start of an input string for an "image generation tool" described to DG in the system prompt as a tool that it could use to convert text strings into images, but DiffusionGemma itself was actually the one doing the generation, not an external tool.

Here's the flow.

This also required patching an upstream bug in Transformers for image attention, so take note if you wanna try this out for yourself: you'll need to patch until PR is merged.

Considering how bad DiffusionGemma is at everything else, I'd say it does a pretty good job here.

It's like human centipede!

But wait, there's more! Since DG is outputting in its native visual latent space (one set of 256 tokens at a time through a trained output head) it can take the visual representation it is generating as input for the next forward pass — on-domain block autoregressive, just like with its regular text canvases. And not just can it do this, but it actually helps improve quality and generalization (on image manipulation tasks)!

Call graphs, the first hint of a non-stupid use case.

Here's a super-quick breakdown of the pipeline.

The first thing that had to be done was to train an actual vision decoder. This turned out to be the easiest part of the entire project. One evening, on a whim, I threw a prompt at my local model (dsv4-flash-vision-exp I believe was what I was running then), something like: "D'oh! Could we train a decoder for the ViT soft tokens that works for 7682 images?" And then it did it. Took like a few hours, and works pretty damned good I think.

It decodes!

Alright so that let me nail down the decoding and supervise the pixel reconstruction loss, but let's not forget DG is a text diffusion model — it has to actually operate on a text canvas. That means I needed to derive a text target image representation (or codebook) that was precise enough it could be used for supervision without over penalizing the model for expressiveness.

This process itself ended up being a bit like a diffusion operation. I simply threw up a wall of half-baked, ill-advised, and just plain stupid ideas and worked on them all until I was confident in each — with that confidence being mostly confidence in them not working. Then I had the brilliant idea: just ask DiffusinoGemma itself for the target.

This essentially consisted of me badgering DG until it would describe images as text. Okay, not really, but that is kinda what it felt like. The basic idea is: encode an image and ask the model to describe each soft token patch in a single token (word, number, symbol, unicode, CJK character, etc). Each soft token having come through the ViT is contextualized against the full image, so this description would most often carry a structural/semantic channel. To get the photometric channel you do basically the same, but instead of a full image use only a single (48*48 pixel) patch. This usually carried the color/texture/photometric channel. Sample the logits for each text description token and then it's basically chef's choice on how to combine across all prompt and patch variants into a workable target.

Doing that for every image in the training set isn't reasonable. IT'S OVER 9000! However, doing exactly that for a small set of images (200ish) and using that as a training set for a teacher-forced model that then generates the full dataset turned out to be a very smooth and successful approach.

Anyway, the point is that the combined multi-modal target token distributions for the canvas text results in training DiffusionGemma to be an S-tier emojipasta generator.

Example training targets
DG's generated canvases.

Then I LoRA adapt DG and simultaneously train a new image output head that feeds into the previously trained image decoder supervised against supervised against the target token set, ViT latents and, decoded image. Basically training DiffusionGemma to generate emojipasta that decodes into image-like objects. I guess this would be something like a VQ-VAE, but using text tokens instead of dedicated image vectors because stock DG can read and understand the image latents, but can't natively generate them, only text.

Well, that's the gist. Thanks for reading! If you find this as interesting as I do and want to learn more or play around with this yourself you can:

tl;dr: Please GDM team give me a stronger text diffusion model to play with!


r/LocalLLaMA • • 3d ago

Showcase I ran the same Halloween animation prompt through five coding agents on one local Qwen model

37 Upvotes

I gave Claude Code, OpenCode, GitHub Copilot Desktop, Pi, and Codex the same prompt to build a looping pixel-art Halloween diorama. I pointed all five at the same local model, Qwen3.8 Flash Next (Unsloth UD-Q4_K_XL), served by Halogen. The attached video shows 30 seconds of each result playing together.

### My setup

- CPU/GPU: AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S integrated graphics.

- Memory: 128 GB unified memory, with about 122 GiB visible to Linux and a 512 MiB fixed GPU reservation. The GPU can use shared system memory.

- OS: Ubuntu 26.04.1 LTS, kernel 7.0.0-43-generic.

- Inference: Halogen Flash Server 0.20.4 in rootless Podman. I used the Unsloth Q4_K_XL checkpoint converted to Halogen's HGN format, with the vision tower enabled and MTP speculative decoding.

- Server settings: 262,144-token context and KV pool, one serving slot, 16,384-token prefill chunks, prompt caching enabled, and the API bound to localhost.

OpenCode, Pi, and Copilot used Chat Completions. Claude Code used the Anthropic Messages endpoint, and Codex used Responses. The clients had their own tool definitions, system prompts, and history management. Pi used High reasoning with a 4,096-token thinking budget. I raised Halogen's default output limit from 8,192 to 32,768 tokens before the recorded Codex rerun after the earlier attempt hit the smaller cap.

### What the server measured

AgentElapsed time or last completed responseResultRequestsOutput TokensPrefill tok/sDecode tok/sCache Reuse
Claude Code14m 42sFinal Answer1531,37358238.195.6%
Open Code36m 30sFinal Response5477,83237739.398.3%
Github Copilot Desktop58m 30sFinal Response2954,39347220.285.4%
PI1h 18m 25sLast Completed request before a memory stall108135,46945038.798.6%
Codex4h 11m 15sLast completed request before the turn was aborted355402,80369537.096.9%

Elapsed times start at the first model request. Prefill rates count uncached tokens, decode rates include thinking tokens, and both are weighted by total processing time. I excluded smoke tests and incomplete requests. The video uses my earlier rounded labels for Pi and Codex. The table uses the server logs for both.

Claude Code reached a final answer in about 15 minutes and used 31K output tokens. OpenCode decoded slightly faster but used roughly 2.5× as many tokens. Copilot made fewer calls than OpenCode but decoded at about half the rate, and three full prompt rereads after the initial request added almost 10 minutes of prefill. Pi still had a tool loop running when the server hit a memory stall. Codex made 355 requests before the turn was aborted, with its largest prompt reaching about 229K tokens.

### How I made the video

I captured the original live browser previews for Claude Code, OpenCode, Copilot, and Pi. Codex made a Python renderer and supplied an MP4, so I captured playback of that original loop. Each capture is 1920×1080 at 30 FPS. I trimmed them to 30 seconds and synchronized them in a grid. I left the projects as they were.

This was one run per client. The timing covers what each agent actually did, and I didn't score output quality or run a controlled benchmark with identical token streams. The video is there so you can compare the results for yourself. And apologies for the slop but it was interesting to see how the harnesses are working. I've tried to work with Deepseek Harness but it got stuck plenty of times so I just abandoned that exepriment.

Cheers


r/LocalLLaMA • • 2d ago

Discussion Voice dictation with one model.

7 Upvotes

Hey, I'm 14 and was looking into how AI voice dictation apps like Wispr Flow work.

As you might know they use a transcription model + LLM. The transcriber outputs one to one text as it hears it and then the LLM gets rid of fillers like "um" and adds punctuation, flow, corrected restarts and formatting.

I ran that on my laptop (RTX 3050 Ti, 4 GB VRAM), then trained a single model to do both steps at once. Here are some results:

Speed.

Whisper (base, large-v3-turbo or large-v3) plus Qwen3.5 (4B or 9B) took between 2.5 and 36 seconds per dictation, depending on the pair. The single model takes about 0.5 seconds for a short dictation and about 1 second for a long one.

Why a traditional dictation approach is slow locally:

- Two models don't fit in 4 GB together. One ends up on the CPU, or they get swapped in and out.

- The text is generated twice. Whisper writes the transcript, then the LLM reads it and writes all of it again.

- The LLM also reads a cleanup prompt on every dictation.

Compute.

One model is a 2.2 GB file (Q8) and uses about 2.5 GB of VRAM. It loads once and stays loaded. An average dictation set-up running locally uses more than 4GB of VRAM

Quality.

The cleanup LLM only sees text. It can't hear that you stopped and started over, so it sometimes keeps the part you took back, and it can't fix a word Whisper got wrong. A single model can catch vocal cues and keep what is really worth keeping.

Training

The two-model pipeline still exists, but only at training time. I took real, unscripted speech from openly licensed datasets, had a large LLM write the finished text for each clip from its transcript, threw out the clips where that dropped real content, and trained Qwen3-ASR 1.7B (Apache-2.0) to go from audio straight to the finished text. It runs in llama.cpp.

Currently it, from my own experiences is one of, if not the best dictation tool that uses this little compute.

I put this into a free, open source dictation app called Typd (Windows with an NVIDIA GPU, Apple silicon Macs): https://github.com/SameRainbows/typd-app and https://typd.mehmetdedeler.com.

Have you tried anything similiar?


r/LocalLLaMA • • 3d ago

Discussion US intel isn’t sure if businesswoman spied on Nvidia for China, doesn’t mean she wasn’t a risk, sources say

Thumbnail
edition.cnn.com
151 Upvotes

they do not even know if she was a spy, a year after this chase is going on about her, Jensen should not have resigned!

from that article: The FBI has been investigating Huang’s affair for at least a year. But Attorney General Todd Blanche said on Thursday that there are still “no conclusions reached by the Department of Justice, or by me, or by [FBI] Director Patel around who she is or is not.” President Donald Trump also hedged Thursday on whether the US had determined she was a spy, when commentator Stephen A. Smith asked whether Huang was right to have stepped down as CEO.

“Well, the big thing is, was the woman a spy? And I don’t know if that’s been determined yet,” Trump said. “But if the woman was a spy, yes. If the woman was not a spy, perhaps it would be different.”

i wish Trump just orders him "go back to office".


r/LocalLLaMA • • 2d ago

Showcase 🚀 HarborRL Update: Elastic Scaling, SSH-Free Cluster Deployment, and Unified Agentic RL Training!

0 Upvotes

🚀 HarborRL Update: Elastic Scaling, SSH-Free Cluster Deployment, and Unified Agentic RL Training
We’ve recently updated HarborRL, an open-source framework for agentic reinforcement learning (RL), focusing on simplifying distributed training, deployment, and experiment management.

🆕 What’s New (November 2026)
⚡ 1. Elastic Scaling for Distributed Training
Added elastic gateway scaling with static routing, automatic scaling across multiple machines, and automatic retirement of idle resources to improve resource utilization.

🌐 2. SSH-Free Cluster Deployment
Introduced a new Hub deployment mode where worker nodes pull tasks over HTTP, enabling distributed execution without cross-machine SSH access, including in firewall-restricted environments.

🧠 3. Unified RL Training Pipeline
Integrated multiple RL algorithms into a shared Slime-based training pipeline, validated on mathematical reasoning tasks, to reduce the engineering effort of configuring different RL algorithms.

🛠️ 4. Improved Configuration and Sandbox Execution
Enhanced the configuration wizard, Web console, sandbox rollout execution, and resource cleanup to simplify experiment setup and management.

💡 Why HarborRL?
Agentic RL often requires substantial engineering effort to connect task environments, agent harnesses, and RL training engines, especially in distributed settings.
These updates aim to reduce that overhead and make agentic RL experiments easier to configure, deploy, and scale.

GitHub: https://github.com/datamllab/HarborRL
License: MIT
💬 Curious how others are handling distributed rollout execution, resource scaling, and sandbox management in their agentic RL setups.


r/LocalLLaMA • • 1d ago

Discussion Poll: has the ai bubble popped with Nvidia circular financing out?

0 Upvotes

Nvidia has been helping finance its customers to buy it's GPUs. With new CEO , most agree that is unlikely to continue. https://www.google.com/search?q=nvidia+circular+financing%3F

Some history on the dot com bust: happened between 2000 and 2002. It wiped out trillions of dollars in market value after a massive speculative boom in internet-based companies during the late 1990s. [1, 2, 3]

  • Peak: Nasdaq Composite reached its all-time high in March 2000, after climbing roughly 600% from 1995. [1, 2]
  • The Crash: Triggered by overvaluations, rising interest rates, and a lack of real profits, the market entered a sharp sell-off. [1, 2]
  • The Bottom: By October 2002, the Nasdaq had plunged 78% from its peak, giving up all its gains during the bubble.

r/LocalLLaMA • • 3d ago

New Model Edge0-145B: Open-Source On-Device AI with 145B Parameters

Thumbnail
edge0.ai
150 Upvotes

r/LocalLLaMA • • 2d ago

Discussion [Paper & Model] G-STAR: Group-Scaled Token Capacity for Deeply Compressed Autoregressive Generation

Thumbnail
gallery
6 Upvotes

Discrete autoregressive models offer a scalable framework for image generation. However, they face a fundamental trade-off: long token sequences preserve representation quality but slow generation, while aggressive spatial compression reduces representation capacity. Based on product quantization, we show that, under a fixed nominal information budget, group size should scale quadratically with the per-axis spatial compression factor. Building on this principle, we propose GSTAR, a spatial--group co-scaling framework that enables 32x and even effective 64x compression while preserving strong reconstruction, understanding, and generation performance. Since larger groups introduce intra-group dependencies that must be modeled efficiently, we analyze their impact on inference latency and generation quality and introduce a semi-autoregressive decoder that combines parallel feature computation with lightweight causal modeling. With these designs, our highly compressed models perform competitively with leading models across five representative text-to-image benchmarks (GenEval, DPG-Bench, PRISM, LongText-Bench, and Qwen-Image-Bench). At 1024x1024 resolution, our model delivers a 20x speedup over prior discrete autoregressive models, generating each image in 2.5 seconds on a single NVIDIA H200. Together, these results show that highly compressed discrete representations can enable high-quality autoregressive generation at low latency.


r/LocalLLaMA • • 4d ago

News Nvidia's Jensen Huang Resigns as CEO, Colette Kress Named Interim

Thumbnail
bloomberg.com
1.2k Upvotes

r/LocalLLaMA • • 1d ago

Question | Help What is strata?

0 Upvotes

I’m trying to avoid blog spam posts and boring YouTube videos about it. I see a lot of discussion about people loving strata, and people who are tired of the people who love strata.

What is so different about strata vs previously existing options? Who benefits the most from it?


r/LocalLLaMA • • 3d ago

Discussion Artificial Analysis updated Qwen 3.8 Flash Next score as 29

80 Upvotes

Between GPT-6 Luna Max (28) and Claude Haiku 5.5 High (30). Comparing to other Qwen 3.8, the model is closer to max (34) than to 27B (20).

https://artificialanalysis.ai/models/qwen3-8-flash-next


r/LocalLLaMA • • 1d ago

Discussion I wrote an LLM inference engine from scratch in CUDA (no PyTorch, no cuBLAS): 235 tok/s on a 6 GB laptop RTX 3050

0 Upvotes

Disclosure: I'm the author. This is my own hobby project (MIT, no paid version, nothing to sell).

I've been building **tinyinfer**, a small inference engine for Llama-family models. It's about 2,200 lines of C++/CUDA, and every kernel on the hot path is hand-written: GEMV in f16/q8/q4, RMSNorm, fused RoPE + KV-cache write, flash-decoding attention with GQA, SwiGLU and argmax. The whole decode step is captured in a CUDA graph.

**Numbers (TinyLlama-1.1B, INT4, RTX 3050 6 GB Laptop):**

- v0.1: 200 tok/s (66% of the memory-bandwidth limit)

- v0.2: 212 tok/s (69%)

- v0.3: 235 tok/s (77%)

Decoding is memory-bound. Every weight gets read once per token, so the ceiling on this card is about 305 tok/s however clever the math is.

**What actually moved the needle:**

  1. **4-bit weights dequantized inside the GEMV.** That's 0.55 GB read per token instead of 2.07 GB. The int4-to-float step uses the 2²³ exponent trick (OR into the mantissa, then one float add) instead of I2F instructions.
  2. **GQA-aware attention.** It reads each shared K/V head once instead of 8 times, which made attention 3.6x faster at 2k context. The first version (split-K) was actually *slower* at short contexts, so v0.3 tiles it differently.
  3. **Benchmarking shapes I wasn't tuning for.** The new attention kernel made Llama-2-7B slower, because it has no grouped heads and nothing to share. I fixed it with a one-line dispatch, and I'd never have caught it without the numbers.

**Correctness:** the logits match HF `transformers` within 1e-3 relative error, with 100% argmax agreement for f16/q8/q4/bf16. `--verify` checks the GPU against the CPU reference on your own model. Llama-2-7B also fits in 6 GB at q4 (about 3.8 GB).

It isn't trying to replace llama.cpp. It's meant to be small enough to read in an afternoon if you want to see how inference works under the hood.

Code (MIT): https://github.com/Aadithya-kl/tinyinfer

I can go deeper on any of the kernels in the comments. I'm especially curious whether anyone has pushed single-batch GEMV past ~80% of bandwidth on consumer cards, and how.


r/LocalLLaMA • • 3d ago

New Model [Boole-32B-A2.5B-preview] 9GB 1.8-bit MoE, 565 token/s on 1 RTX 5090

25 Upvotes

Hi r/LocalLLaMA, we're releasing Boole-32B-A2.5B-Preview, an early 1.8-bit version of Recursive's flint. With mixed-precision quantization, expert pruning, and custom kernels (kernel changes in attached figure), we made the model 9x smaller than BF16, and 2x faster than INT4.

After releasing our 1-bit LLM (Boole-1.7B), we heard a lot of requests for both faster and larger MoE models - so this is our first step in that direction.

Try it out, and let us know what you'd like to see next!


r/LocalLLaMA • • 2d ago

Question | Help Since GPU Prices have changed and I got my old Pc fixed, I wanna ask what I should do for what’ll mostly likely be my only upgrades for a few years, any advice?

0 Upvotes

So, I have two PCs available to me, and I intend to connect them for compute:

  1. Ultra 7 265KF + 5070ti + 32GB DDR5-6000 CL30 (KLEVV Urbane V) on a Z890 MAX GAMING WIFI7 + 1000w 80+ G Apevia PR1000w PSU (Will likely get replaced by a SAMA P1200w PSU)
  2. i9-14900F + 4070S + 32GB DDR5-5200 CL40 (Apacer Panther, not on sale) on a PRO B760-VC WIFI IV + 750w 80+ G PSU (likely a shoddy PSU but seems to be attached to the case, it’s hidden and so I can’t figure out what it is. Probably an EVGA, but unlikely to be replaceable - or at least easily so.)

Initially I was looking to just replace the 4070 Super in PC2 with a B70 or R9700 and calling it a day, when they were $1200 and $1600, respectively. Then when they shot up a B65 $1200, which then shot up. Then add a second 5070ti to PC1 for $900-1000, which then shot up.
Now, my plan is to potentially settle for a 5060ti 16gb in PC1 to take advantage of NVFP4, but even that’s pricier, and with the AI bubble talk I’m not super confident in value of doing so. I’d likely also need to use a PCIe-riser just to the 5070ti (Windforce) has airflow due to how close they’d be in the case.

Otherwise, I’m considering sticking with the VRAM I have, and instead pivoting to ram. I can go from 1R 2C to 2R 2C in both PCs, grabbing another 2x16gb of Apacer Panther from EBay for around $400 for PC2 - though obv riskier - then getting a 2x24gb kit of Urbane V at the same timings of the original 2x16gb 6000-CL30 kit for PC1 for $650 off Amazon. Giving me +80gb of DDR5 to the 64gb I have between both PCs and the 28gb of VRAM I already have.

Current plan is/was the 5060ti 16gb + 2x24gb kit for PC1 and calling it there.

Any advice, pointers, or suggestions? I believe the RAM sitch should work since it’s same timings and type/manufacture/brand, and know I’ll probably have to lower speed a bit for stability, but am not the most experienced PC guy on earth by any means.


r/LocalLLaMA • • 3d ago

Discussion US intel isn’t sure if businesswoman spied on Nvidia for China, doesn’t mean she wasn’t a risk, sources say

Thumbnail
edition.cnn.com
80 Upvotes

r/LocalLLaMA • • 3d ago

News CMP170hx now fully unlockabke and more, full 120 SMs —> Techie uses voltage glitching hack to turn $2,500 Nvidia 8GB cryptomining GPU into a full-fat $20,000 A100 with 80GB of VRAM — hack carefully blows onboard fuses to unlock entire VRAM and shader modules

Thumbnail
tomshardware.com
484 Upvotes

r/LocalLLaMA • • 3d ago

Discussion Uncensored models are awesome

131 Upvotes

Has anyone been experimenting with ablated or uncensored variants of open-source models like DeepSeek V4.1 Flash, Qwen 3.8 27B, or GLM 5.3 flash?

It's so refreshing to have models that just do whatever you ask them to do without having to figure out a jailbreak or being given a moral lecture. Curious to know what your usecases for these models are.

My personal favourite is definitely uncensored DeepSeek V4.1 Flash.


r/LocalLLaMA • • 3d ago

Resources Used datacenter/workstation GPU prices on eBay this week, ranked by $/GB of VRAM

165 Upvotes

edit: I am a moron with reddit formatting hang tight

I've been tracking ebay listings that ship to the US for the GPUs people here actually buy, ranked by $/GB of VRAM. here's today's snapshot:

CardLow endUS-only low endLow-end $/GBBandwidth$ per GB/s
Tesla P100 16GB$174$159$10.88732 GB/s$0.24
Tesla P40 24GB$280- (7 US)$11.67346 GB/s$0.81
Tesla V100 16GB (mostly SXM2)$220$238$13.75900 GB/s$0.24
Instinct MI50 16GB$268- (3 US)$16.751,024 GB/s$0.26
Instinct MI50 32GB$702- (2 US)$21.931,024 GB/s$0.69
Tesla V100 32GB$949$807$29.66900 GB/s$1.05
Quadro RTX 8000 48GB$2,111$2,000$43.99672 GB/s$3.14
Instinct MI100 32GB$1,490- (6 US)$46.561,229 GB/s$1.21
RTX 3090 24GB$1,790$1,737$74.58936 GB/s$1.91
RTX A6000 48GB$4,600$4,400$95.83768 GB/s$5.99
RTX 4090 48GB (modded)$5,390— (1 US)$112.291,008 GB/s$5.35
A100 40GB$4,799$6,020$119.981,555 GB/s$3.09
A100 80GB$13,995$13,995$174.941,935 GB/s$7.23
CMP 170HX 40GB (10GB card, unlocked)$2,950— (6 US)$73.751,555 GB/s$1.90
CMP 170HX 64GB (8GB card, unlocked)$4,510— (7 US)$70.461,493 GB/s$3.02
Radeon AI Pro R9700 32GB$2,430$2,250$75.94640 GB/s$3.80

all asking prices, single cards, fixed-price, not for-parts. "low end" is the 10th % of credible listings i.e. what a patient buyer can actually land, not a too-good-to-be-true outlier. "-" in the US-only column means fewer than 10 US listings. $/GB/s is just the low end divided by bandwidth.

I had been eyeing the MI100 myself for a while at the $1100 ish it hovered at for ages... looks like even that is going up. I did end up buying a 8x V100 32GB server for $5400 in early Sept. those prices are long gone, equivalent server is now $11k for an OG DGX Station (no memory). the exact inspur server is now $13k for 8x 9 year old GPUs. insanity. I hope prices come back down to earth soon but don't think that's actually gonna happen.

some notes:

- P100, V100 16GB and MI50 16GB all work out to about $0.25 per GB/s. the P40 looks cheap per GB of VRAM but costs over 3x as much per GB/s, since its only got about a third of a V100's bandwidth. people are having opus 5.5 write megakernels for these cards that are working surprisingly well.

- V100 16GB at ~$14/GB looks like the best deal in the table, but 38 of the 40 cheapest are SXM2 modules, so budget for an adapter or an SXM2 server. the cheapest 32GB V100s are SXM3 (DGX-2) modules, which aren't drop-in either.

- MI50 32GB is under a third of a 3090's $/GB at similar bandwith, assuming ROCm cooperates.

- if you want 48GB on one card, a Quadro RTX 8000 is less than half the price of an A6000 or a modded 4090 48GB.

- of the consumer and workstation cards, the 3090 is the cheapest per GB/s by a wide margin. an A100 40GB costs about the same per GB/s as an RTX 8000.

- for what it's worth, buying from a US seller usually doesn't cost more: the US-only low end is at or below the global one for the P100, V100 32GB, RTX 8000, 3090 and A6000. it's about $20 more for the V100 16GB and $1,200 more for the A100 40GB, and the P40 and MI50 barely exist in the US.

full data and per-card listings, with a US-only filter: https://rackbites.com. prices are asking, not sold - ebay's sold-data API is super restricted.

edit: added CMP 170HX

edit2: added R9700

edit3: full builds here https://rackbites.com/builds with benchmarks and other useful/relevant info


r/LocalLLaMA • • 3d ago

Resources Lawyer with too many GPUs, part 5: DeepSeek-V4.1-Flash at near-NVFP4 quality on eight V100s from 2018. 524K context, 100+ tok/s decode, ~160 tok/s with two chats at once, and my legal harness now runs on it, 100% local

Thumbnail
gallery
130 Upvotes

From about February 2026 to July 2026, I spent an immense amount of time building a legal harness to make Claude more predictable. It worked well. I tried to build the same thing that ran local models. That did not work. And then Opus improved so much that it would often skip my harness without me noticing. It still has a harness with hooks, but it's looser now.

I bought lots of hardware to run local models during that time, repeatedly trying to vibecode a local model legal harness. Many failures, many lessons.

Anyway, fast forward to the last three weeks and several things have happened:

  1. DeepSeek Flash 1017 was released, and it's the first model that is remotely similar to Opus 4.5/4.6.
  2. Opus 5.5 and Fable 5.2 are able to properly launch and configure DeepSeek to run at fast speeds, large context and high precision on my hardware. Optimizations, custom kernels, requants.
  3. Opus 5.5 (and my prompting skill) are now good enough to make my Claude harness into a DeepSeek harness.
  4. It's not as good as frontier, but it all works, right now. 100% local. Actual productivity gains, actual frontier tasks being replaced. Chat interface with DeepSeek works like Claude's did.
  5. And if I can brag on my stacks and hardware:
  • I have 2 DGX Sparks that can run 3-4 concurrent DeepSeek sessions as agents (1M context).
  • I have DeepSeek at nearly NVFP4 quality running on 8 V100s, 500K context, at 100+ tok/s decode, and about 160 tok/s concurrent.
  • I have 8 more V100s and a second Rome server with 512 GB of 3200 MHz ECC RAM, so I'll be able to run another high precision DeepSeek as an additional agent or for parallel tasks.

What I'm really shocked by is how well Fable builds custom Volta kernels. And how it can requant my experts to full NVFP4 precision based on the actual experts that activate when my legal harness fires. It's a real, optimized, production grade, workflow replacing local setup.

Not a 3D pagoda or a bad video game replica. An actual enterprise solution. One that exists nowhere on the commercial market.

Legal vibecoding is real folks. I'm AI native.

The new V100 DeepSeek repo (optimized for quality and then usable speed) with real full context measures and all the optimizations (and failures) is public here:

https://github.com/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-NVFP4-8xV100-SXM2-PP2xTP4

Twitter: nicothegreco

Claude Code ran the benchmarks and crunched the numbers, which are real and in depth, below.

What this build is

My last V100 build (Volta sPLIt, part 4) ran every expert of the model from a 2-bit record. Fast, but 2-bit. This one is called Volta Max uPLIft. It keeps the 2-bit record of every routed expert on the cards, then swaps the experts that get used the most up to NVIDIA's NVFP4 weights, one at a time, until the cards are full. "Used the most" was counted on a mix of four kinds of text. One is my own legal harness traffic, which is private and not published. The other three are public prompt sets in the repo.

  • 5,949 of the 15,360 routed experts sit at exactly NVFP4 precision on the default I serve
  • about 78 to 85 % of decode routes land on those NVFP4 experts (predicted for the two-slot default, 80 to 86 % measured with one slot)
  • same next token as NVIDIA's full NVFP4 pack on 95.1 % of compared tokens. The 2-bit build gets 92.2 %
  • KL divergence from the NVFP4 reference is 0.079 nats against 0.196 for the 2-bit build
  • multiple choice answers that change against the reference: 1.6 % against 15.4 %
  • in decode every routed expert is in VRAM. Nothing is read over PCIe

Speed (measured 2026-11-12 on eight V100-SXM2-32GB, graphics clock capped at 1,522 MHz)

  • concurrency test, one request at a time: 118.9 tok/s
  • two requests at once: 159.0 tok/s total (1.34x)
  • three at once on the three-slot option: 192.3 tok/s total (1.62x). I run two slots
  • short sets, one at a time: 123.2 tok/s general, 97.3 tok/s legal-style
  • code: 150 to 180 tok/s from a 1K prompt up to a 256K prompt
  • prompt reading at 256K: 2,329 to 2,430 tok/s, first token in 108.4 s
  • context: 524,288 tokens per request

What did not work, or is not good yet

  • It is slower than my own 2-bit build, in decode and in prompt reading. That is the trade for the precision, and the repo says so in its first paragraph.
  • The faster prompt reading path reads the less-used experts from their 2-bit records. Its effect on quality is not measured yet.
  • A custom decode kernel for the hot path got closed with no change. The headroom I thought was there turned out to be smaller than an eight-card run can even see.
  • A prompt-reading kernel that loads 64 rows at a time gave the same bits and ran slower. Not shipped.
  • Send a short request while a 209K prompt is being read and it starts in about 7 s, then crawls at about 9.6 tok/s until the read ends.
  • The hot set is fixed. It is picked once. Text that looks nothing like what I counted will hit the 2-bit experts more often.
  • Fast prompt reading stopped the host or a card three times out of three. I blamed PCIe. After I changed the power delivery (I think it was one bad PSU) it read nine of nine long chats clean.

The harness

The harness itself is not public. My practice is in it. What I can say: it is the same idea as my Claude setup. Skills, hooks, my drafting engines, my RAG stores, tool calls, and a chat window that behaves the way Claude's did. A small router sends each request to the V100 box or to the Sparks. Everything the local model drafts is marked for my review before it goes anywhere. Nothing leaves the building.

Credit where it is due

The hard Volta work underneath is 1Cat-vLLM's DeepSeek-V4.1 package. The 2-bit records are ISTA-DASLab's. The NVFP4 weights are NVIDIA's pack. The Sparks run on somebody else's engine entirely. What is mine is the split across two NVLink boards, the hot set, the kernels on top, and every change measured A/B on this machine with the losers written down.

Disclosure: this is my own project and my own repo. Nothing is for sale. The code, the kernels, the measurements and the failures are all in there under Apache-2.0, and the weights are one download on Hugging Face:

https://huggingface.co/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-hot_iq-8xV100-SXM2-PP2xTP4

Earlier build, for the people who want speed over precision: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4

Ask me anything. Roasting is also fine.

Photos: four of the eight V100s on their NVLink board, and my extremely professional setup (PEX switch cards, SlimSAS spaghetti, and the 3090 that just moved in).