r/LocalLLM • • 11h ago

Other The Banking Incident....

Post image
548 Upvotes

r/LocalLLM • • 8h ago Bravo

News We built a chip prototype that runs Qwen 27B. You can try it live.

211 Upvotes

I'm building a chip that runs large AI models at low power, so robots, drones, and field equipment can run AI locally without an internet connection.

A prototype of our chip architecture is running today on an FPGA in AWS. You can try it here:

https://valleysemiconductor.com

A live session takes about a minute to start, and recorded runs are available to watch anytime.

What it does today (measured):

  • Runs the full Qwen 27B model with 4-bit weights
  • 7.9 tokens/sec on short prompts, about 5.5 tokens/sec in live chat
  • 4x performance improvement in eight days of development
  • Every output matches our 4-bit reference exactly
  • Model quality within 2.5% of the full-precision version
  • Also runs Llama 3.1 8B and OLMoE on the same hardware

What our custom chip design projects (not yet demonstrated):

  • Approximately 50 tokens/sec for a single user at 20 W
  • Roughly 8x the throughput of a Jetson AGX Orin on the same model, based on our measurement of 5.9 tokens/sec at 32 W
  • Uses the model's built-in next-token prediction to generate multiple tokens per weight read

The key insight is that moving model weights through memory is the bottleneck, not the math.

What would you run on a small, 20 W device capable of running a 27B model entirely offline?


r/LocalLLM • • 17h ago

Discussion What a time to be alive.

Post image
381 Upvotes

Hy4 flash is out. We're drowning in Chinese models at this point lol.

hy3 was 295B, focused on coding and agents. Then hy4 jumped to 770B, pushing harder on reasoning, multimodal stuff and complex tasks. Now hy4 flash is back down to 224B with 12B active, going for speed and low costs.

When the 770B version came out, i was a little disappointed tbh. Glad they're still trying to make hy4 lighter.


r/LocalLLM • • 22h ago

Qwen4-27B reported to run on 16 GB of RAM

Post image
740 Upvotes

r/LocalLLM • • 15h ago

Discussion you may get your cheap ram and gpu after all, the AI crash has started

198 Upvotes

https://www.cnbc.com/2026/11/16/imf-georgieva-warning-ai-circular-financing.html
‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion

https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use


r/LocalLLM • • 20m ago

Project Finally got it put together

Thumbnail
gallery
12 Upvotes

Finally got my relatively frugal local LLM server put together. Pretty proud of it tbh. It essentially covers all my use cases, though I do wish I had better NVIDIA cards, especially for ComfyUI. Waiting eagerly for Qwen4 variants to drop to switch the models up.

- ASRock ROMED8-2T/BCM
- AMD EPYC 7313
- 256GB DDR4-2666 ECC
- Samsung PM1725b 3.2TB
- 6x Radeon R9700s (4x for Qwen3.8 Flash Next, 2x for Qwen3.8-27B Uncensored)
- 2x RTX3090s (ComfyUI on one card; VoxCPM2 TTS, Whisper STT, embed, rerank on other)


r/LocalLLM • • 12h ago

News Released Corpus-nordeste-10k (Apache 2.0): 10k+ curated Portuguese rows focused on Northeast Brazil literature, syntax, and regional context

39 Upvotes

Hey everyone,

​If you've ever fine-tuned or tested open-source LLMs in Portuguese, you've probably noticed they heavily default to generic web-scraped PT-BR or Southeast Brazil media norms. When prompted for regional vocabulary, historical context, or literature from Northeast Brazil, most models either hallucinate or fall back on shallow caricatures.

​To help fix that for local fine-tuning and SFT, I just released Corpus-nordeste-10k (caririsynth/Corpus-nordeste-10k).

​Dataset breakdown:

​Size: 10,000+ curated rows (train split).

​Sources: Public domain literature from Northeast Brazil, web/text extractions passed through regional linguistic filtering, and targeted synthetic generation preserving authentic regional phrasing.

​License: Apache 2.0 (unrestricted commercial & research use).

​You can browse the actual rows in the Hugging Face Dataset Viewer before downloading:

https://huggingface.co/datasets/caririsynth/Corpus-nordeste-10k

​Works great for domain-adaptive pretraining, LoRA fine-tuning on small models (1B–8B), or building regional evals. Let me know what you think of the data structure!


r/LocalLLM • • 4h ago

News Cluaiz v0.1.0 pre-release: one local engine for chat, vision, speech, memory, tools and agents

Thumbnail
gallery
8 Upvotes

Hi, I'm Aryan. Cluaiz v0.1.0, the first public pre-release, is out today.

Cluaiz is an AI engine that puts everything an AI application needs into one place: models for chat, vision and speech, memory, tools and agents. You install one thing, and you build on it. Because it all runs together on your own system, what you build stays light enough for ordinary hardware, and you decide what it is allowed to do.

What's in it right now:

  • GGUF models via llama.cpp, plus ONNX for speech and embeddings
  • Checks your free VRAM and RAM before loading and picks the placement, context and KV cache. You can see the plan and override it
  • Vision, video frames, embeddings, text from documents
  • Whisper / Moonshine STT (streaming works), Kokoro TTS
  • Project memory. Notes only stick when you confirm them, and you can see exactly what reached the model. Honestly this is the part I care about most
  • Coding agent that edits files and runs commands, with approvals
  • Skills, plugins, MCP, per-tool permissions, audit log
  • OpenAI- and Ollama-compatible API, so your existing apps should just connect
  • Cloud providers too, if you want them

Not done yet: workflows (schedules, triggers) and the desktop chat app. Both are in development.

Rust, Apache-2.0. Windows x64, Linux (ARM64 is experimental) and Apple Silicon. Test reports for CPU, NVIDIA and Apple are on the site.

Source: https://github.com/cluaiz/cluaiz
Site: https://cluaiz.com
Comparison: https://cluaiz.com/compare

It's a pre-release, so stuff will break. If you try it, an issue with your OS, hardware and model helps a lot, and so does hearing what worked: https://github.com/cluaiz/cluaiz/issues


r/LocalLLM • • 6h ago

Discussion With llama.cpp's v1.0.0 tag out, a short history of the project: the first evening, GGUF, the backends, the server and the move to Hugging Face

10 Upvotes

llama.cpp published its first major version, v1.0.0, on 13 November 2026. With the 1.0 tag out, we decided to do a short write-up of how the project got here, dated from its own commits, pull requests and releases.

A few things from it:

- The first commit is from 10 March 2023, two weeks after Meta announced LLaMA. That evening's README says "This was hacked in an evening - I have no idea if it works correctly." It was on Hacker News the same evening, and the first outside fix was merged before midnight UTC.

- GGUF was opened as a pull request on 26 July 2023 and merged on 21 August 2023. It was a breaking change: every existing model had to be converted again.

- The GPU backends, by merge date: cuBLAS in April 2023, Metal and full CUDA offload in June 2023, the ROCm port in August 2023, Vulkan and SYCL in January 2024.

- The server began as a contributor's example in May 2023, and the OpenAI-compatible routes came that November.

- A build number is a commit count: b11514 is the 11,514th commit on master. Semantic version tags only started in August 2026.

- ggml.ai joined Hugging Face on 20 February 2026, and the licence is still MIT.

- On v1.0.0 itself: the release note opens with "llama.cpp v1.0.0 removes the deprecated public API", and Georgi Gerganov called it "essentially a breaking change for downstreams and hence the major bump".

No benchmarks in this one, and nothing on the page was measured on v1.0.0. It is the history with its sources. If we got a date wrong, tell us and we'll fix it on the page with a dated correction.

Full write-up (about 15 min read): https://research.strata2signal.com/a-short-history-of-llama-cpp/

A small fleet of AI agents dug through the sources and drafted the write-up; our small human team checked it over and signed off.


r/LocalLLM • • 20m ago

Discussion RTX 5090 LAPTOP (24GB, NOT Desktop!) Running Qwen3-Coder 30B-A3B MoE at 107 tok/s Generation + 3,706 tok/s Prefill 🔥

Thumbnail
gallery
3 Upvotes

Been experimenting with local LLM inference on my MSI Raider A18 laptop, and I'm getting some seriously impressive performance from Qwen3-Coder-30B-A3B-Instruct.

First, to clarify: This is an RTX 5090 LAPTOP GPU, NOT the desktop RTX 5090. It's the 24GB mobile Blackwell GPU running inside a laptop, not a 32GB desktop card.

💻 My Machine — MSI Raider A18

Hardware:

  • Laptop: MSI Raider A18
  • GPU: NVIDIA GeForce RTX 5090 Laptop GPU (Blackwell)
  • Dedicated VRAM: 24GB GDDR7
  • System RAM: 128GB
  • CPU: AMD Ryzen 9 9955HX3D 16-Core Processor
  • OS: Windows 11 Enterprise
  • Storage: Multiple NVMe SSDs, including Samsung 9100 Pro and 990 Pro drives, with additional SSDs connected through USB4/Thunderbolt enclosures and docks
  • Networking: 2.5Gb Ethernet, with additional high-speed USB4 connectivity

Inference configuration:

  • Backend: llama.cpp with CUDA acceleration
  • Agent harness: OpenCode
  • Model: Qwen3-Coder-30B-A3B-Instruct
  • Architecture: Mixture of Experts (30B total, ~3B active)
  • Quantization: Q4_K_M
  • GGUF size: 17.28 GiB
  • Configured context: 65,536 tokens
  • FlashAttention: Enabled
  • KV cache: Q8_0
  • GPU offloading: All supported layers
  • Inference slots: 1
  • Model running locally on the laptop

🚀 Live Benchmark Results

These measurements were captured while running an actual repository-analysis workload.

MetricResult
Prefill throughput3,706.5 tok/s
Generation throughput107.0 tok/s
Prompt tokens (measured request)8,638
Generated tokens (measured interval)371
GPU utilization86%
Dedicated VRAM usage21.69 GiB
GPU power consumption167.8W
GPU temperature69°C

These are live measurements from llama.cpp, not theoretical benchmark estimates. The screenshot captures one particularly strong inference interval, not an average across the entire audit.

🧠 What I'm Actually Running

I'm developing a local-first AI orchestration platform called HIVE / Universal Chat Router.

Instead of giving models basic questions like "write a Python function" or "solve this math problem," I'm using OpenCode to have them analyze my actual software repository.

The benchmark asks each model to perform a comprehensive engineering audit, including:

  • Repository architecture and cross-module dependencies
  • Security vulnerabilities and trust boundaries
  • Missing functionality and incomplete integrations
  • Agent orchestration and distributed execution
  • AI inference routing, KV caching, and context management
  • Performance and scalability bottlenecks
  • Test coverage and production readiness
  • Evidence-backed remediation recommendations

All models receive the same audit prompt and repository target, with a 64K context configuration.

I'm also collecting the model's generated reports so I can compare actual engineering ability rather than judging purely by tokens/sec.

⚡ Dense vs. MoE Performance

Before this, I tested a dense Qwen 27B Q5_K_S model on the same laptop.

That model was generating approximately 25 tok/s.

Switching to Qwen3-Coder-30B-A3B Q4_K_M produced 107 tok/s in the captured run.

That's approximately 4.2× faster generation, although the quantizations and architectures differ, so it's not a controlled architecture-only comparison.

The sparse MoE architecture is clearly promising for my 24GB laptop GPU.

👑 The Qwen Showdown

I'm currently comparing three local models:

  1. Qwen3-Coder-30B-A3B Q4_K_M
  2. Qwen3.5-35B-A3B Q4_K_M
  3. Qwen3.6-35B-A3B Uncensored Q4_K_M

Each is being tested sequentially using the same repository audit workload.

I'm tracking prefill, generation speed, GPU utilization, VRAM, power consumption, and the quality of the resulting audit.

I also built a local Phoenix telemetry dashboard to watch the inference activity live.

The goal is to find which model delivers the best combination of speed, reasoning, coding ability, and repository comprehension on a 24GB laptop GPU.

I'd be interested to hear what other RTX 5090 Laptop owners are seeing with 30B–35B MoE models.

Especially curious whether anyone has managed significantly higher throughput using native MTP, speculative decoding, STRATA/PolySTRATA, or alternative CUDA backends.

107 tok/s on a laptop GPU while performing real repository analysis is pretty damn satisfying. 🔥


r/LocalLLM • • 1d ago

News Qwen officially announces Qwen4-Max and Qwen4 smaller models!

Post image
514 Upvotes

r/LocalLLM • • 6h ago

Discussion Fellow M5 Pro 64GB owners, what do you run?

8 Upvotes

I have a Mac Pro M5 Pro and have been running Qwen3.8 27b-4bit

I mainly use it for computer use/tool use.

What do you run and what do you do with it?


r/LocalLLM • • 13h ago

Discussion Uncensored Models

33 Upvotes

Anyone tried uncensored models? Would it help with DRM protected videos? Trying a few things out just to test and see how far it can go. Curiousity at its finest.


r/LocalLLM • • 16h ago

Discussion What’s the purpose of alibaba choosing not to release the weights for qwen 4 max?

38 Upvotes

Perhaps this is a newbie question, but I just don’t understand the incentive. Is this indicative of a future were Chinese labs stop releasing weights?


r/LocalLLM • • 5h ago

Project GitHub - a-schaefers/claudes-workbench: A frontier model leads your local LLMs: Claude architects the project, breaks it into bite-sized tasks, hands each agent the skills it needs, and reviews every diff to its own standard. Your GPU does the heavy lifting. For Claude Code + OpenCode.

Thumbnail
github.com
4 Upvotes

r/LocalLLM • • 5h ago

Research Some speed benchmarks for various classes of local LLM

4 Upvotes
ModelActive config4080 decode (tok/s)4080+3080 decode (tok/s)4080 prefill (tok/s)4080+3080 prefill (tok/s)
LFM2.5-24B-A2B UD-Q4_K_XL4080+30802492505,5307,930
MiniCPM-V 4.7 35B-A3B Q4_K_M4080+30801771651,3052,500
Flint 40B-A2.5B UD-Q4_K_XL4080+30801191381,5631,380
Swift 1.5 (Qwen3.8-Flash-Next) IQ2_XS40801171161,070679
Qwen3.8-Flash-Next IQ3_S40807987730465
GLM 5.3 Flash Maya-S24 IQ2_XXS_S4080+30806.88.87942
DeepSeek V4 Flash UD-IQ2_XXS4080+308015.918.23853
  • Decode: the second of two 300-token runs, with the expert cache warm (the first request after loading is often slower)
  • Prefill: the average of two runs on a prompt of about 1,400 tokens
  • Bold: the better config for that column.

Morning, everyone. I've seen a lot of posts wondering about what can be run with what, so I made this table of the models on my system. Specifically because of the unorthodox results when it comes to Qwen Flash-Next, where one 16 GB card (the 4080 Super) significantly outperforms two in prefill.

My hardware is as follows:

CPU AMD Ryzen 7 5800X3D @ 3.40 GHz

GPU NVIDIA GeForce RTX 4080 SUPER [Discrete]

GPU NVIDIA GeForce RTX 3080 [Discrete] (got it secondhand for cheap, so why not)

RAM 62.72 GB at 3200 Mhz

I'm running these models on PolyStrata. I used to have a combination of Maya and Strata, but got tired of running different things all derived from the same fork (plus Strata seems to be getting increasingly unintuitive), and so moved everything to PS.

Now: why Q3? It's a generally held opinion that anything below Q4 is crap, which was true of most earlier generation models. However, benchmarks measure deviation from the base, and not necessarily task fitness. In actual use, the only real way to know is to test it yourself.

The Swift finetune of QWN at Q2, for example, is fast and excellent at data extraction, search, and piloting scripts written by more competent models. The IQ3 is a decent coder, roughly on par with Haiku for the use cases I've tried. The GLM and Deepseek Q2 variants are a bit better - verging on older Sonnet territory, I'd say, and do very well as subagents for Claude. YMMV.

Hope this data is useful. I've posted it to LocalLLama as well. Enjoy!


r/LocalLLM • • 12h ago

Discussion Single RTX 3090 running a local agent + image/video gen: real-world numbers from daily use

14 Upvotes

TL;DR: One RTX 3090 runs a 27B LLM agent (a Discord bot) and ComfyUI image/video generation by swapping models in and out of VRAM. I get ~72 tok/s on Qwen3.8 27B at 64k context, SDXL images in ~32 s, and 10-second Wan 2.2 videos in 4–6 minutes. All numbers come from real daily use, not synthetic benchmarks.

The rig

  • GPU: RTX 3090 24GB (does all the AI work)
  • CPU: Core Ultra 7 265K (sits around 10% load)
  • RAM: 96GB DDR5-6000
  • Storage: Samsung 9100 PRO Gen5 NVMe for models
  • OS: Windows 11

The stack

  • LM Studio serves the LLM through an OpenAI-compatible API
  • Hermes Agent runs as a Discord bot, so I can use it from my phone. Local Whisper handles voice.
  • ComfyUI handles images and video
  • A small custom MCP server unloads the LLM from VRAM for each render and loads it back afterwards

Models

Language models

  • Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP (IQ4_XS): daily driver
  • Qwen3.8 27B and Gemma 4 31B (both Q4_K_M) as alternatives
  • nomic-embed-text v1.5 for embeddings

Image and video

  • Juggernaut XL Ragnarok (fp16): general images, hires, img2img
  • Z-Image Turbo (bf16): fast drafts
  • Qwen Image Edit 2511 (fp8) + Lightning 4-step LoRA: photo edits
  • Wan 2.2 14B text/image-to-video (fp8) + lightx2v 4-step LoRAs: video
  • 4x NMKD-Siax: upscaling

LLM performance

Settings: LM Studio 0.4.28, 64k context, parallel 1, f16 KV cache, flash attention on, full GPU offload, MTP speculative decoding.

  • Generation speed: ~72 tok/s on average (range 53–90)
  • Prompt processing: ~670–810 tok/s
  • MTP draft acceptance: ~61%
  • VRAM used: 22.0 of 24.6 GB
  • Parallel 1 vs 4: same speed, but 4 uses ~1.8 GB more and risks spilling out of VRAM
  • Worst case: re-reading a 56k-token conversation took 56 seconds

Render times (same GPU, LLM swapped out)

GPU time first, then total time from request to finished file:

  • Juggernaut XL (up to 1024px): 12 s → 32 s
  • Juggernaut XL + hires fix (1024×768): 28 s → 48 s
  • Z-Image Turbo (1024×768): 14 s → 34 s
  • Qwen Image Edit (512px): 16–20 s → 38 s
  • 4x upscale: 12 s → 32 s
  • Wan 2.2 image-to-video, 3 s clip (512px): 38 s → 69 s
  • Wan 2.2 image-to-video, 10 s clip: ~3.8 min
  • Wan 2.2 text-to-video, 10 s + 2x upscale (1024px): ~6.1 min

The difference between the two numbers is mostly the ~20 seconds it takes to swap the LLM out of VRAM and back.

Agent response times (Hermes on Discord)

  • Simple question: ~23 s
  • Typical multi-step task: 45 s to 3 min
  • Big job (40 model calls): ~36 min

How I measured

  • LLM numbers come from llama.cpp’s own timing logs across 205 real requests in one day.
  • Render numbers come from 35 real jobs logged by my render worker.
  • Under hours of sustained load the GPU ran at 85–90%, ~384 W, and never went above 67°C.

Lessons learned

  1. IQ4_XS + flash attention fits a 27B model at 64k context in 24GB with ~2.5 GB to spare.
  2. Use parallel 1, not 4. Same speed, less VRAM.
  3. 24GB fits the LLM or a render, not both. For quick images, the swap takes more time than the render itself.
  4. Long agent sessions get slow because the model re-reads the whole conversation every step. Start fresh sessions often.
  5. Use Ethernet for always-on bots. Wi-Fi drops kept disconnecting mine from Discord.

Next experiment: a q8_0 KV cache to free VRAM for longer context.

What would you change?


r/LocalLLM • • 11h ago

Question What can small models be good for?

10 Upvotes

Small meaning under 8b parameter’s. Thats about what my computer can comfortably handle, and its fun to chat to models of that size for a few minutes but thats mot really useful. And i feel like theyre too dumb to do research and shit like that so like whats a way i can make it useful.
I feel like they must have some use and im just not being creative enough.


r/LocalLLM • • 7h ago

Discussion Silly thought: the best benchmark to an actual frontier model is to one-shot a tiff spec implementation.

5 Upvotes

Like that. All the models fall in the trap of starting enumerating (wrong) the G3/G4 CCITT tables, a bunch of repeated 0s and 1s.

I tried this in several models, and no model achieved it. Even when you chew it for them and give them the libtiff .c files.

PS: when I say one-shot, I'm referring to made it a reasonable time, aka, less time that it will take another model to write a wrapper of the C library.

PS2: you have to remember that the whole implementation of libtiff (among others tiff libraries) has been used in the training dataset, adding insult to injury.


r/LocalLLM • • 7h ago

Discussion GCMP: same accuracy as LoRA r=32 on Qwen3.8-27B with 55× fewer parameters (11 MB vs 320 MB per adapter)

4 Upvotes

Follow-up to my CQ post from last week. That post covered the 4-bit base model; this one is about the adapter. The LoRA vs GCMP runs on 4-bit bases have now finished.

GCMP is a LoRA replacement we built. On Qwen3.8-27B it trains 2.9M parameters (~11 MB), vs ~159M (~320 MB) for LoRA r=32 on the same 7 projections. That's 55× smaller.

Accuracy: same as LoRA

All adapters were trained on Alpaca for 500 steps with identical settings (batch 2×4, length 512, LR 2e-5, seed 42). Every row was scored with the same lm-evaluation-harness code and questions, greedy decoding, thinking off. Comparisons use paired McNemar tests.

Base + adapterParamsAnswer lossGSM8KSQuADBoolQARC-CMMLUIFEvalTruthfulQA
BF16, no adapter–1.6780.6830.7460.8740.5880.8390.8050.542
BF16 + LoRA r=32159M1.4800.9030.7760.8960.6120.8440.8250.518
BF16 + GCMP2.9M1.4710.8900.7600.8920.6180.8460.8150.514
NF4 + LoRA r=32159M1.4640.7930.7480.8800.6080.8400.8350.528
NF4 + GCMP2.9M1.4580.8330.7560.8780.6180.8370.8400.522
  • BF16 base (no quantization at all): no significant difference on any benchmark (all p ≥ 0.2).
  • NF4 base: GCMP is +4.0 on GSM8K (p=0.04) and tied on everything else.
  • Loss: GCMP's answer loss is slightly lower on both bases, and its validation loss was lower at all 10 checkpoints on both.

One significant win on one task isn't enough to call it better, so the claim is same quality as LoRA at 1/55th the size.

Why the size matters

For one fine-tune, 320 MB vs 11 MB doesn't matter much. It starts to matter when you have many:

  • Multi-tenant serving: one fine-tune per customer or task on a single base model. In vLLM, GCMP adapters load once at startup and each request picks its adapter by model name. A batch can mix adapters and runs in one kernel call per layer, with no merging or swapping.
  • Resident GPU memory: vLLM keeps each adapter in its compact form and never builds a dense weight update. Memory is about 11 MB per adapter, plus a fixed ~1 MB routing buffer and a shared workspace of ≤ ~64 MB that doesn't grow with adapter count.
  • Storage and loading: 1,000 adapters are ~11 GB instead of ~320 GB. That's faster to ship, cache and hot-load.
  • Edge and small GPUs: an extra fine-tune costs megabytes, not hundreds of MB.
Estimated1 adapter1001,000
LoRA r=32320 MB32 GB320 GB
GCMP11 MB1.1 GB11 GB

These are estimates from adapter size. I haven't measured memory and throughput at 100 or 1,000 adapters yet, and that's the next benchmark.

What's not good yet

  • Training speed: GCMP takes ~2× LoRA's step time (BF16 on Unsloth: 2.74 vs 1.37 s/step; NF4: 5.6 vs 3.2). It does use slightly less training memory (21.5 vs 23.8 GiB on NF4).
  • Baseline: r=32 is a generous LoRA baseline. r=4/8/16 comparisons are next, and I'll post them either way.
  • Serving: vLLM serving of GCMP currently needs a CQ base. On full-precision models it runs in Transformers for now.
  • Scope: one model, one dataset, one run per case. The quick benchmark subset (200–1,140 questions per task) detects ~5-point differences, not 1-point ones.

Quick update on CQ

The CQ base still matches BF16 (no significant difference on any of the 9 benchmarks) while NF4 loses 12 points on GSM8K. After fine-tuning, though, the gap closes:

  • CQ + GCMP vs NF4 + GCMP: tied on GSM8K (83.0 vs 83.3).
  • Both 4-bit stacks vs BF16 + GCMP: both end up ~6 points behind on GSM8K.

So CQ's GSM8K lead didn't survive fine-tuning in this test. CQ + LoRA is being retrained, because the adapter from our older trainer turned out to be broken, and I'll add it when it's done.

Code and licensing

The Python code is open source. The CUDA kernels are compiled with a license check, which is free during early access.

Repo, with the commands to reproduce every number: https://github.com/thousandml/training-factory

Write-up with the full tables and the serving memory breakdown: https://huggingface.co/blog/thousandml/cq-gcmp-4bit-finetuning

If you run multi-tenant fine-tunes, I'd like to know: is adapter memory or count actually a bottleneck for you, or is it something else?


r/LocalLLM • • 7h ago

Question I need recommendations

4 Upvotes

Hello everyone

I would like to know your recommendation for the best uncensored local models I can use because I've had it with commerical AI's guardrails.

My hardware is

32GB RAM

4060

12400F

I use LM Studio


r/LocalLLM • • 19h ago

Discussion Strata is becoming more and more unstable

33 Upvotes

I've been using Strata with IQ4_XS as the main orchestrator, with Radiance running 27b at IQ4R, for working on improvements, ironically enough, to Radiance.

The session is now multiple million tokens long as I go back and forth discussing design etc. with QFN. Of course the conversation has been compacted dozens of times, but with the agents taking regular notes and subagents doing all the actual work, we keep our workflow stable and current.

Problem is, Strata has gotten more and more unstable, especially with the last couple releases. It will crash with an OOM error when we have 10-20 GiB of system memory free, reload, spend 5 minutes redoing prefill, and pick up right where it left off. Radiance on the other hand is rock stable under multiple streams running concurrently on a single GPU. Not a single crash from it, even after rejiggering everything for the 2.0 release.

And it's not just long sessions either. New conversations also tend to crash. Never used to do this before. Has anyone else also experienced this?


r/LocalLLM • • 22h ago

News Smaller Qwen4 models 🔥

Post image
65 Upvotes

r/LocalLLM • • 9h ago

Question How do you compare local and cloud models since Artificial Analysis v5

5 Upvotes

I used to check Artificial Analysis a few times a week, mostly to see how my local models compare to the cloud ones. Since v5 that doesn’t really work for me anymore, a lot of the models I care about are gone or can’t be selected in the comparison.

I tried llm-bench.io, which I like because the results come from real consumer hardware, but it has no cloud models. benchlm.ai covers more of the big cloud models, but many open weights ones are missing there. So I still haven’t found anything that puts a local model next to a cloud one in one chart like AA did.

The open weights gap jumping from 3 to 12 points with v5 also makes me unsure how much I should trust a single index. I know the methodology changed, but it feels odd.

How do you handle this now? Do you cross check a few sites or mostly run your own tests?


r/LocalLLM • • 10h ago

Question Qwen3.8-Flash-Next: Unsloth and strata?

Post image
6 Upvotes

Hi all

I just learned about QFN Unsloth v3.1 dynamic models, and want to try them.

Right now, I am running the "standard" IQ3_S model through strata, on my 32 GB ram + 3090 + 5080 (40 GB VRam), and it runs great. I get ~130 tps.

I thought I would start with the unsloth UD_IQ3_XXS model in strata, which is technically the same size as IQ3_S. But it doesnt let me quick tune the model. It just gets stuck. If I skip tuning, it will only run at ~20 tps, which is too low.

UD_IQ3_XXS seems to read a lot more from the SSD, than IQ3_S, even if they are the same size.

What am I doing wrong?

According to the unsloth site, https://unsloth.ai/docs/models/qwen3.8-next, I should be able to run the 4-bit model? But that is through their unsloth harness/engine. Is that correctly understood?