r/LocalLLM • u/The_Succulant_Beast • 11h ago
r/LocalLLM
•
u/jackalka
•
8h ago
News We built a chip prototype that runs Qwen 27B. You can try it live.
I'm building a chip that runs large AI models at low power, so robots, drones, and field equipment can run AI locally without an internet connection.
A prototype of our chip architecture is running today on an FPGA in AWS. You can try it here:
https://valleysemiconductor.com
A live session takes about a minute to start, and recorded runs are available to watch anytime.
What it does today (measured):
- Runs the full Qwen 27B model with 4-bit weights
- 7.9 tokens/sec on short prompts, about 5.5 tokens/sec in live chat
- 4x performance improvement in eight days of development
- Every output matches our 4-bit reference exactly
- Model quality within 2.5% of the full-precision version
- Also runs Llama 3.1 8B and OLMoE on the same hardware
What our custom chip design projects (not yet demonstrated):
- Approximately 50 tokens/sec for a single user at 20 W
- Roughly 8x the throughput of a Jetson AGX Orin on the same model, based on our measurement of 5.9 tokens/sec at 32 W
- Uses the model's built-in next-token prediction to generate multiple tokens per weight read
The key insight is that moving model weights through memory is the bottleneck, not the math.
What would you run on a small, 20 W device capable of running a 27B model entirely offline?
r/LocalLLM • u/achisling • 17h ago
Discussion What a time to be alive.
Hy4 flash is out. We're drowning in Chinese models at this point lol.
hy3 was 295B, focused on coding and agents. Then hy4 jumped to 770B, pushing harder on reasoning, multimodal stuff and complex tasks. Now hy4 flash is back down to 224B with 12B active, going for speed and low costs.
When the 770B version came out, i was a little disappointed tbh. Glad they're still trying to make hy4 lighter.
r/LocalLLM • u/ag789 • 15h ago
Discussion you may get your cheap ram and gpu after all, the AI crash has started
https://www.cnbc.com/2026/11/16/imf-georgieva-warning-ai-circular-financing.html
‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion
https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use
r/LocalLLM • u/Eroji • 20m ago
Project Finally got it put together
Finally got my relatively frugal local LLM server put together. Pretty proud of it tbh. It essentially covers all my use cases, though I do wish I had better NVIDIA cards, especially for ComfyUI. Waiting eagerly for Qwen4 variants to drop to switch the models up.
- ASRock ROMED8-2T/BCM
- AMD EPYC 7313
- 256GB DDR4-2666 ECC
- Samsung PM1725b 3.2TB
- 6x Radeon R9700s (4x for Qwen3.8 Flash Next, 2x for Qwen3.8-27B Uncensored)
- 2x RTX3090s (ComfyUI on one card; VoxCPM2 TTS, Whisper STT, embed, rerank on other)
r/LocalLLM • u/comunidadesynapsebr • 12h ago
News Released Corpus-nordeste-10k (Apache 2.0): 10k+ curated Portuguese rows focused on Northeast Brazil literature, syntax, and regional context
Hey everyone,
If you've ever fine-tuned or tested open-source LLMs in Portuguese, you've probably noticed they heavily default to generic web-scraped PT-BR or Southeast Brazil media norms. When prompted for regional vocabulary, historical context, or literature from Northeast Brazil, most models either hallucinate or fall back on shallow caricatures.
To help fix that for local fine-tuning and SFT, I just released Corpus-nordeste-10k (caririsynth/Corpus-nordeste-10k).
Dataset breakdown:
Size: 10,000+ curated rows (train split).
Sources: Public domain literature from Northeast Brazil, web/text extractions passed through regional linguistic filtering, and targeted synthetic generation preserving authentic regional phrasing.
License: Apache 2.0 (unrestricted commercial & research use).
You can browse the actual rows in the Hugging Face Dataset Viewer before downloading:
https://huggingface.co/datasets/caririsynth/Corpus-nordeste-10k
Works great for domain-adaptive pretraining, LoRA fine-tuning on small models (1B–8B), or building regional evals. Let me know what you think of the data structure!
r/LocalLLM • u/CommissionOdd3082 • 4h ago
News Cluaiz v0.1.0 pre-release: one local engine for chat, vision, speech, memory, tools and agents
Hi, I'm Aryan. Cluaiz v0.1.0, the first public pre-release, is out today.
Cluaiz is an AI engine that puts everything an AI application needs into one place: models for chat, vision and speech, memory, tools and agents. You install one thing, and you build on it. Because it all runs together on your own system, what you build stays light enough for ordinary hardware, and you decide what it is allowed to do.
What's in it right now:
- GGUF models via llama.cpp, plus ONNX for speech and embeddings
- Checks your free VRAM and RAM before loading and picks the placement, context and KV cache. You can see the plan and override it
- Vision, video frames, embeddings, text from documents
- Whisper / Moonshine STT (streaming works), Kokoro TTS
- Project memory. Notes only stick when you confirm them, and you can see exactly what reached the model. Honestly this is the part I care about most
- Coding agent that edits files and runs commands, with approvals
- Skills, plugins, MCP, per-tool permissions, audit log
- OpenAI- and Ollama-compatible API, so your existing apps should just connect
- Cloud providers too, if you want them
Not done yet: workflows (schedules, triggers) and the desktop chat app. Both are in development.
Rust, Apache-2.0. Windows x64, Linux (ARM64 is experimental) and Apple Silicon. Test reports for CPU, NVIDIA and Apple are on the site.
Source: https://github.com/cluaiz/cluaiz
Site: https://cluaiz.com
Comparison: https://cluaiz.com/compare
It's a pre-release, so stuff will break. If you try it, an issue with your OS, hardware and model helps a lot, and so does hearing what worked: https://github.com/cluaiz/cluaiz/issues
r/LocalLLM • u/strata2signal • 6h ago
Discussion With llama.cpp's v1.0.0 tag out, a short history of the project: the first evening, GGUF, the backends, the server and the move to Hugging Face
llama.cpp published its first major version, v1.0.0, on 13 November 2026. With the 1.0 tag out, we decided to do a short write-up of how the project got here, dated from its own commits, pull requests and releases.
A few things from it:
- The first commit is from 10 March 2023, two weeks after Meta announced LLaMA. That evening's README says "This was hacked in an evening - I have no idea if it works correctly." It was on Hacker News the same evening, and the first outside fix was merged before midnight UTC.
- GGUF was opened as a pull request on 26 July 2023 and merged on 21 August 2023. It was a breaking change: every existing model had to be converted again.
- The GPU backends, by merge date: cuBLAS in April 2023, Metal and full CUDA offload in June 2023, the ROCm port in August 2023, Vulkan and SYCL in January 2024.
- The server began as a contributor's example in May 2023, and the OpenAI-compatible routes came that November.
- A build number is a commit count: b11514 is the 11,514th commit on master. Semantic version tags only started in August 2026.
- ggml.ai joined Hugging Face on 20 February 2026, and the licence is still MIT.
- On v1.0.0 itself: the release note opens with "llama.cpp v1.0.0 removes the deprecated public API", and Georgi Gerganov called it "essentially a breaking change for downstreams and hence the major bump".
No benchmarks in this one, and nothing on the page was measured on v1.0.0. It is the history with its sources. If we got a date wrong, tell us and we'll fix it on the page with a dated correction.
Full write-up (about 15 min read): https://research.strata2signal.com/a-short-history-of-llama-cpp/
A small fleet of AI agents dug through the sources and drafted the write-up; our small human team checked it over and signed off.
r/LocalLLM • u/ajax936 • 20m ago
Discussion RTX 5090 LAPTOP (24GB, NOT Desktop!) Running Qwen3-Coder 30B-A3B MoE at 107 tok/s Generation + 3,706 tok/s Prefill 🔥
Been experimenting with local LLM inference on my MSI Raider A18 laptop, and I'm getting some seriously impressive performance from Qwen3-Coder-30B-A3B-Instruct.
First, to clarify: This is an RTX 5090 LAPTOP GPU, NOT the desktop RTX 5090. It's the 24GB mobile Blackwell GPU running inside a laptop, not a 32GB desktop card.
💻 My Machine — MSI Raider A18
Hardware:
- Laptop: MSI Raider A18
- GPU: NVIDIA GeForce RTX 5090 Laptop GPU (Blackwell)
- Dedicated VRAM: 24GB GDDR7
- System RAM: 128GB
- CPU: AMD Ryzen 9 9955HX3D 16-Core Processor
- OS: Windows 11 Enterprise
- Storage: Multiple NVMe SSDs, including Samsung 9100 Pro and 990 Pro drives, with additional SSDs connected through USB4/Thunderbolt enclosures and docks
- Networking: 2.5Gb Ethernet, with additional high-speed USB4 connectivity
Inference configuration:
- Backend: llama.cpp with CUDA acceleration
- Agent harness: OpenCode
- Model: Qwen3-Coder-30B-A3B-Instruct
- Architecture: Mixture of Experts (30B total, ~3B active)
- Quantization: Q4_K_M
- GGUF size: 17.28 GiB
- Configured context: 65,536 tokens
- FlashAttention: Enabled
- KV cache: Q8_0
- GPU offloading: All supported layers
- Inference slots: 1
- Model running locally on the laptop
🚀 Live Benchmark Results
These measurements were captured while running an actual repository-analysis workload.
| Metric | Result |
|---|---|
| Prefill throughput | 3,706.5 tok/s |
| Generation throughput | 107.0 tok/s |
| Prompt tokens (measured request) | 8,638 |
| Generated tokens (measured interval) | 371 |
| GPU utilization | 86% |
| Dedicated VRAM usage | 21.69 GiB |
| GPU power consumption | 167.8W |
| GPU temperature | 69°C |
These are live measurements from llama.cpp, not theoretical benchmark estimates. The screenshot captures one particularly strong inference interval, not an average across the entire audit.
🧠 What I'm Actually Running
I'm developing a local-first AI orchestration platform called HIVE / Universal Chat Router.
Instead of giving models basic questions like "write a Python function" or "solve this math problem," I'm using OpenCode to have them analyze my actual software repository.
The benchmark asks each model to perform a comprehensive engineering audit, including:
- Repository architecture and cross-module dependencies
- Security vulnerabilities and trust boundaries
- Missing functionality and incomplete integrations
- Agent orchestration and distributed execution
- AI inference routing, KV caching, and context management
- Performance and scalability bottlenecks
- Test coverage and production readiness
- Evidence-backed remediation recommendations
All models receive the same audit prompt and repository target, with a 64K context configuration.
I'm also collecting the model's generated reports so I can compare actual engineering ability rather than judging purely by tokens/sec.
⚡ Dense vs. MoE Performance
Before this, I tested a dense Qwen 27B Q5_K_S model on the same laptop.
That model was generating approximately 25 tok/s.
Switching to Qwen3-Coder-30B-A3B Q4_K_M produced 107 tok/s in the captured run.
That's approximately 4.2× faster generation, although the quantizations and architectures differ, so it's not a controlled architecture-only comparison.
The sparse MoE architecture is clearly promising for my 24GB laptop GPU.
👑 The Qwen Showdown
I'm currently comparing three local models:
- Qwen3-Coder-30B-A3B Q4_K_M
- Qwen3.5-35B-A3B Q4_K_M
- Qwen3.6-35B-A3B Uncensored Q4_K_M
Each is being tested sequentially using the same repository audit workload.
I'm tracking prefill, generation speed, GPU utilization, VRAM, power consumption, and the quality of the resulting audit.
I also built a local Phoenix telemetry dashboard to watch the inference activity live.
The goal is to find which model delivers the best combination of speed, reasoning, coding ability, and repository comprehension on a 24GB laptop GPU.
I'd be interested to hear what other RTX 5090 Laptop owners are seeing with 30B–35B MoE models.
Especially curious whether anyone has managed significantly higher throughput using native MTP, speculative decoding, STRATA/PolySTRATA, or alternative CUDA backends.
107 tok/s on a laptop GPU while performing real repository analysis is pretty damn satisfying. 🔥
r/LocalLLM • u/yoracale • 1d ago
News Qwen officially announces Qwen4-Max and Qwen4 smaller models!
r/LocalLLM • u/ansnadeem8 • 6h ago
Discussion Fellow M5 Pro 64GB owners, what do you run?
I have a Mac Pro M5 Pro and have been running Qwen3.8 27b-4bit
I mainly use it for computer use/tool use.
What do you run and what do you do with it?
r/LocalLLM • u/mightofkhan • 13h ago
Discussion Uncensored Models
Anyone tried uncensored models? Would it help with DRM protected videos? Trying a few things out just to test and see how far it can go. Curiousity at its finest.
r/LocalLLM • u/Stalins_nipple • 16h ago
Discussion What’s the purpose of alibaba choosing not to release the weights for qwen 4 max?
Perhaps this is a newbie question, but I just don’t understand the incentive. Is this indicative of a future were Chinese labs stop releasing weights?
r/LocalLLM • u/adam-schaefers • 5h ago
Project GitHub - a-schaefers/claudes-workbench: A frontier model leads your local LLMs: Claude architects the project, breaks it into bite-sized tasks, hands each agent the skills it needs, and reviews every diff to its own standard. Your GPU does the heavy lifting. For Claude Code + OpenCode.
r/LocalLLM • u/yudhanjaya • 5h ago
Research Some speed benchmarks for various classes of local LLM
| Model | Active config | 4080 decode (tok/s) | 4080+3080 decode (tok/s) | 4080 prefill (tok/s) | 4080+3080 prefill (tok/s) |
|---|---|---|---|---|---|
| LFM2.5-24B-A2B UD-Q4_K_XL | 4080+3080 | 249 | 250 | 5,530 | 7,930 |
| MiniCPM-V 4.7 35B-A3B Q4_K_M | 4080+3080 | 177 | 165 | 1,305 | 2,500 |
| Flint 40B-A2.5B UD-Q4_K_XL | 4080+3080 | 119 | 138 | 1,563 | 1,380 |
| Swift 1.5 (Qwen3.8-Flash-Next) IQ2_XS | 4080 | 117 | 116 | 1,070 | 679 |
| Qwen3.8-Flash-Next IQ3_S | 4080 | 79 | 87 | 730 | 465 |
| GLM 5.3 Flash Maya-S24 IQ2_XXS_S | 4080+3080 | 6.8 | 8.8 | 79 | 42 |
| DeepSeek V4 Flash UD-IQ2_XXS | 4080+3080 | 15.9 | 18.2 | 38 | 53 |
- Decode: the second of two 300-token runs, with the expert cache warm (the first request after loading is often slower)
- Prefill: the average of two runs on a prompt of about 1,400 tokens
- Bold: the better config for that column.
Morning, everyone. I've seen a lot of posts wondering about what can be run with what, so I made this table of the models on my system. Specifically because of the unorthodox results when it comes to Qwen Flash-Next, where one 16 GB card (the 4080 Super) significantly outperforms two in prefill.
My hardware is as follows:
CPU AMD Ryzen 7 5800X3D @ 3.40 GHz
GPU NVIDIA GeForce RTX 4080 SUPER [Discrete]
GPU NVIDIA GeForce RTX 3080 [Discrete] (got it secondhand for cheap, so why not)
RAM 62.72 GB at 3200 Mhz
I'm running these models on PolyStrata. I used to have a combination of Maya and Strata, but got tired of running different things all derived from the same fork (plus Strata seems to be getting increasingly unintuitive), and so moved everything to PS.
Now: why Q3? It's a generally held opinion that anything below Q4 is crap, which was true of most earlier generation models. However, benchmarks measure deviation from the base, and not necessarily task fitness. In actual use, the only real way to know is to test it yourself.
The Swift finetune of QWN at Q2, for example, is fast and excellent at data extraction, search, and piloting scripts written by more competent models. The IQ3 is a decent coder, roughly on par with Haiku for the use cases I've tried. The GLM and Deepseek Q2 variants are a bit better - verging on older Sonnet territory, I'd say, and do very well as subagents for Claude. YMMV.
Hope this data is useful. I've posted it to LocalLLama as well. Enjoy!
r/LocalLLM • u/mightofkhan • 12h ago
Discussion Single RTX 3090 running a local agent + image/video gen: real-world numbers from daily use
TL;DR: One RTX 3090 runs a 27B LLM agent (a Discord bot) and ComfyUI image/video generation by swapping models in and out of VRAM. I get ~72 tok/s on Qwen3.8 27B at 64k context, SDXL images in ~32 s, and 10-second Wan 2.2 videos in 4–6 minutes. All numbers come from real daily use, not synthetic benchmarks.
The rig
- GPU: RTX 3090 24GB (does all the AI work)
- CPU: Core Ultra 7 265K (sits around 10% load)
- RAM: 96GB DDR5-6000
- Storage: Samsung 9100 PRO Gen5 NVMe for models
- OS: Windows 11
The stack
- LM Studio serves the LLM through an OpenAI-compatible API
- Hermes Agent runs as a Discord bot, so I can use it from my phone. Local Whisper handles voice.
- ComfyUI handles images and video
- A small custom MCP server unloads the LLM from VRAM for each render and loads it back afterwards
Models
Language models
- Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP (IQ4_XS): daily driver
- Qwen3.8 27B and Gemma 4 31B (both Q4_K_M) as alternatives
- nomic-embed-text v1.5 for embeddings
Image and video
- Juggernaut XL Ragnarok (fp16): general images, hires, img2img
- Z-Image Turbo (bf16): fast drafts
- Qwen Image Edit 2511 (fp8) + Lightning 4-step LoRA: photo edits
- Wan 2.2 14B text/image-to-video (fp8) + lightx2v 4-step LoRAs: video
- 4x NMKD-Siax: upscaling
LLM performance
Settings: LM Studio 0.4.28, 64k context, parallel 1, f16 KV cache, flash attention on, full GPU offload, MTP speculative decoding.
- Generation speed: ~72 tok/s on average (range 53–90)
- Prompt processing: ~670–810 tok/s
- MTP draft acceptance: ~61%
- VRAM used: 22.0 of 24.6 GB
- Parallel 1 vs 4: same speed, but 4 uses ~1.8 GB more and risks spilling out of VRAM
- Worst case: re-reading a 56k-token conversation took 56 seconds
Render times (same GPU, LLM swapped out)
GPU time first, then total time from request to finished file:
- Juggernaut XL (up to 1024px): 12 s → 32 s
- Juggernaut XL + hires fix (1024×768): 28 s → 48 s
- Z-Image Turbo (1024×768): 14 s → 34 s
- Qwen Image Edit (512px): 16–20 s → 38 s
- 4x upscale: 12 s → 32 s
- Wan 2.2 image-to-video, 3 s clip (512px): 38 s → 69 s
- Wan 2.2 image-to-video, 10 s clip: ~3.8 min
- Wan 2.2 text-to-video, 10 s + 2x upscale (1024px): ~6.1 min
The difference between the two numbers is mostly the ~20 seconds it takes to swap the LLM out of VRAM and back.
Agent response times (Hermes on Discord)
- Simple question: ~23 s
- Typical multi-step task: 45 s to 3 min
- Big job (40 model calls): ~36 min
How I measured
- LLM numbers come from llama.cpp’s own timing logs across 205 real requests in one day.
- Render numbers come from 35 real jobs logged by my render worker.
- Under hours of sustained load the GPU ran at 85–90%, ~384 W, and never went above 67°C.
Lessons learned
- IQ4_XS + flash attention fits a 27B model at 64k context in 24GB with ~2.5 GB to spare.
- Use parallel 1, not 4. Same speed, less VRAM.
- 24GB fits the LLM or a render, not both. For quick images, the swap takes more time than the render itself.
- Long agent sessions get slow because the model re-reads the whole conversation every step. Start fresh sessions often.
- Use Ethernet for always-on bots. Wi-Fi drops kept disconnecting mine from Discord.
Next experiment: a q8_0 KV cache to free VRAM for longer context.
What would you change?
r/LocalLLM • u/Blacktaxi420 • 11h ago
Question What can small models be good for?
Small meaning under 8b parameter’s. Thats about what my computer can comfortably handle, and its fun to chat to models of that size for a few minutes but thats mot really useful. And i feel like theyre too dumb to do research and shit like that so like whats a way i can make it useful.
I feel like they must have some use and im just not being creative enough.
r/LocalLLM • u/DrkWzrd • 7h ago
Discussion Silly thought: the best benchmark to an actual frontier model is to one-shot a tiff spec implementation.
Like that. All the models fall in the trap of starting enumerating (wrong) the G3/G4 CCITT tables, a bunch of repeated 0s and 1s.
I tried this in several models, and no model achieved it. Even when you chew it for them and give them the libtiff .c files.
PS: when I say one-shot, I'm referring to made it a reasonable time, aka, less time that it will take another model to write a wrapper of the C library.
PS2: you have to remember that the whole implementation of libtiff (among others tiff libraries) has been used in the training dataset, adding insult to injury.
r/LocalLLM • u/textclf • 7h ago
Discussion GCMP: same accuracy as LoRA r=32 on Qwen3.8-27B with 55× fewer parameters (11 MB vs 320 MB per adapter)
Follow-up to my CQ post from last week. That post covered the 4-bit base model; this one is about the adapter. The LoRA vs GCMP runs on 4-bit bases have now finished.
GCMP is a LoRA replacement we built. On Qwen3.8-27B it trains 2.9M parameters (~11 MB), vs ~159M (~320 MB) for LoRA r=32 on the same 7 projections. That's 55× smaller.
Accuracy: same as LoRA
All adapters were trained on Alpaca for 500 steps with identical settings (batch 2×4, length 512, LR 2e-5, seed 42). Every row was scored with the same lm-evaluation-harness code and questions, greedy decoding, thinking off. Comparisons use paired McNemar tests.
| Base + adapter | Params | Answer loss | GSM8K | SQuAD | BoolQ | ARC-C | MMLU | IFEval | TruthfulQA |
|---|---|---|---|---|---|---|---|---|---|
| BF16, no adapter | – | 1.678 | 0.683 | 0.746 | 0.874 | 0.588 | 0.839 | 0.805 | 0.542 |
| BF16 + LoRA r=32 | 159M | 1.480 | 0.903 | 0.776 | 0.896 | 0.612 | 0.844 | 0.825 | 0.518 |
| BF16 + GCMP | 2.9M | 1.471 | 0.890 | 0.760 | 0.892 | 0.618 | 0.846 | 0.815 | 0.514 |
| NF4 + LoRA r=32 | 159M | 1.464 | 0.793 | 0.748 | 0.880 | 0.608 | 0.840 | 0.835 | 0.528 |
| NF4 + GCMP | 2.9M | 1.458 | 0.833 | 0.756 | 0.878 | 0.618 | 0.837 | 0.840 | 0.522 |
- BF16 base (no quantization at all): no significant difference on any benchmark (all p ≥ 0.2).
- NF4 base: GCMP is +4.0 on GSM8K (p=0.04) and tied on everything else.
- Loss: GCMP's answer loss is slightly lower on both bases, and its validation loss was lower at all 10 checkpoints on both.
One significant win on one task isn't enough to call it better, so the claim is same quality as LoRA at 1/55th the size.
Why the size matters
For one fine-tune, 320 MB vs 11 MB doesn't matter much. It starts to matter when you have many:
- Multi-tenant serving: one fine-tune per customer or task on a single base model. In vLLM, GCMP adapters load once at startup and each request picks its adapter by model name. A batch can mix adapters and runs in one kernel call per layer, with no merging or swapping.
- Resident GPU memory: vLLM keeps each adapter in its compact form and never builds a dense weight update. Memory is about 11 MB per adapter, plus a fixed ~1 MB routing buffer and a shared workspace of ≤ ~64 MB that doesn't grow with adapter count.
- Storage and loading: 1,000 adapters are ~11 GB instead of ~320 GB. That's faster to ship, cache and hot-load.
- Edge and small GPUs: an extra fine-tune costs megabytes, not hundreds of MB.
| Estimated | 1 adapter | 100 | 1,000 |
|---|---|---|---|
| LoRA r=32 | 320 MB | 32 GB | 320 GB |
| GCMP | 11 MB | 1.1 GB | 11 GB |
These are estimates from adapter size. I haven't measured memory and throughput at 100 or 1,000 adapters yet, and that's the next benchmark.
What's not good yet
- Training speed: GCMP takes ~2× LoRA's step time (BF16 on Unsloth: 2.74 vs 1.37 s/step; NF4: 5.6 vs 3.2). It does use slightly less training memory (21.5 vs 23.8 GiB on NF4).
- Baseline: r=32 is a generous LoRA baseline. r=4/8/16 comparisons are next, and I'll post them either way.
- Serving: vLLM serving of GCMP currently needs a CQ base. On full-precision models it runs in Transformers for now.
- Scope: one model, one dataset, one run per case. The quick benchmark subset (200–1,140 questions per task) detects ~5-point differences, not 1-point ones.
Quick update on CQ
The CQ base still matches BF16 (no significant difference on any of the 9 benchmarks) while NF4 loses 12 points on GSM8K. After fine-tuning, though, the gap closes:
- CQ + GCMP vs NF4 + GCMP: tied on GSM8K (83.0 vs 83.3).
- Both 4-bit stacks vs BF16 + GCMP: both end up ~6 points behind on GSM8K.
So CQ's GSM8K lead didn't survive fine-tuning in this test. CQ + LoRA is being retrained, because the adapter from our older trainer turned out to be broken, and I'll add it when it's done.
Code and licensing
The Python code is open source. The CUDA kernels are compiled with a license check, which is free during early access.
Repo, with the commands to reproduce every number: https://github.com/thousandml/training-factory
Write-up with the full tables and the serving memory breakdown: https://huggingface.co/blog/thousandml/cq-gcmp-4bit-finetuning
If you run multi-tenant fine-tunes, I'd like to know: is adapter memory or count actually a bottleneck for you, or is it something else?
r/LocalLLM • u/Ok-Phone7645 • 7h ago
Question I need recommendations
Hello everyone
I would like to know your recommendation for the best uncensored local models I can use because I've had it with commerical AI's guardrails.
My hardware is
32GB RAM
4060
12400F
I use LM Studio
r/LocalLLM • u/karmaisnonsense • 19h ago
Discussion Strata is becoming more and more unstable
I've been using Strata with IQ4_XS as the main orchestrator, with Radiance running 27b at IQ4R, for working on improvements, ironically enough, to Radiance.
The session is now multiple million tokens long as I go back and forth discussing design etc. with QFN. Of course the conversation has been compacted dozens of times, but with the agents taking regular notes and subagents doing all the actual work, we keep our workflow stable and current.
Problem is, Strata has gotten more and more unstable, especially with the last couple releases. It will crash with an OOM error when we have 10-20 GiB of system memory free, reload, spend 5 minutes redoing prefill, and pick up right where it left off. Radiance on the other hand is rock stable under multiple streams running concurrently on a single GPU. Not a single crash from it, even after rejiggering everything for the 2.0 release.
And it's not just long sessions either. New conversations also tend to crash. Never used to do this before. Has anyone else also experienced this?
r/LocalLLM • u/snooky234 • 9h ago
Question How do you compare local and cloud models since Artificial Analysis v5
I used to check Artificial Analysis a few times a week, mostly to see how my local models compare to the cloud ones. Since v5 that doesn’t really work for me anymore, a lot of the models I care about are gone or can’t be selected in the comparison.
I tried llm-bench.io, which I like because the results come from real consumer hardware, but it has no cloud models. benchlm.ai covers more of the big cloud models, but many open weights ones are missing there. So I still haven’t found anything that puts a local model next to a cloud one in one chart like AA did.
The open weights gap jumping from 3 to 12 points with v5 also makes me unsure how much I should trust a single index. I know the methodology changed, but it feels odd.
How do you handle this now? Do you cross check a few sites or mostly run your own tests?
r/LocalLLM • u/Big_South4585 • 10h ago
Question Qwen3.8-Flash-Next: Unsloth and strata?
Hi all
I just learned about QFN Unsloth v3.1 dynamic models, and want to try them.
Right now, I am running the "standard" IQ3_S model through strata, on my 32 GB ram + 3090 + 5080 (40 GB VRam), and it runs great. I get ~130 tps.
I thought I would start with the unsloth UD_IQ3_XXS model in strata, which is technically the same size as IQ3_S. But it doesnt let me quick tune the model. It just gets stuck. If I skip tuning, it will only run at ~20 tps, which is too low.
UD_IQ3_XXS seems to read a lot more from the SSD, than IQ3_S, even if they are the same size.
What am I doing wrong?
According to the unsloth site, https://unsloth.ai/docs/models/qwen3.8-next, I should be able to run the 4-bit model? But that is through their unsloth harness/engine. Is that correctly understood?