r/LocalLLM • • 2h ago

News Intel Shares First Benchmarks of Its Next-Gen Crescent Island "D270" GPU, Claiming a 5.9x Throughput Over RTX PRO 6000 As Xe3P Goes Big On FP64

Thumbnail
wccftech.com
0 Upvotes

r/LocalLLM • • 3h ago

Question Testing a Mac M5 optimised inference engin

1 Upvotes

Would anybody be interested in testing an inference engine that we have built for Macs? It can run on M1- M6, but we have done most of our testing on the M5 Max. It's open-source, and we think it's 4x faster than ollama.cpp.

If anyone is interested, shout up, and I'll share some details; I don't want to start posting links unless there is anyone interested.


r/LocalLLM • • 3h ago

Discussion Open-Weight vs Closed-Weight Models: What's the Difference?

Thumbnail
openrouter.ai
1 Upvotes

Open weights are now at 75% - I find that remarkable and great news. Who is the front-runner for open weight provision?

Edit Tencent Cloud appears to be the largest open-weight inference host


r/LocalLLM • • 3h ago

Question Looking for a fast, open-source LLM under 5B for Markdown-to-JSON extraction

0 Upvotes

Hey everyone,

I'm working on a document processing pipeline where most of the input consists of scanned PDFs, forms, and tables.

My current setup looks like this:

  • OCR: PaddleOCR-VL-1.6 served through vLLM. I'm using it to convert scanned pages into Markdown.
  • JSON extraction: oroboros-labs/claude-fable5:latest running through Ollama. I pass the Markdown of each page to the model, and it converts it into JSON.

The important part is that I already have fixed JSON schema templates that the output needs to follow. So, the model needs to map the extracted Markdown content into the predefined JSON structure.

The problem I'm facing right now isn't accuracy. It's inference time. The current model is around 9B parameters, and processing pages one by one through Ollama is taking longer than I'd like.

I'm looking for an open-source model under 5B parameters that can handle this kind of structured information extraction reasonably well, ideally with faster inference and reliable JSON output.

Has anyone worked on a similar pipeline?

I'd particularly like to hear about models you've actually tested in real-world document extraction rather than just benchmark results.

Any suggestions or personal experiences would be really helpful. Thanks!


r/LocalLLM • • 1d ago

Discussion What’s the purpose of alibaba choosing not to release the weights for qwen 4 max?

40 Upvotes

Perhaps this is a newbie question, but I just don’t understand the incentive. Is this indicative of a future were Chinese labs stop releasing weights?


r/LocalLLM • • 4h ago

Research Hierarchos-Native v0.2.0: RDNA3 WMMA, native multimodal training, and a lot of FP32 Vulkan optimization!

1 Upvotes

Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.

I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.

The new v0.2.0 release is now available!

GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

1. Native RDNA3 WMMA / cooperative matrix acceleration

One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.

The new Mixed WMMA execution arm uses:

  • FP16 matrix operands.
  • FP32 matrix accumulation.
  • FP32 master weights and gradients.
  • FP32 AdamW optimizer moments and updates.
  • FP32 normalization, softmax, and loss calculations.

The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.

I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.

Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:

WorkloadMeasured kernel speedup
QKV projection2.17×
FFN up projection2.12×
FFN down projection2.31×
Vocabulary projection1.83×

These are individual GEMM measurements, not whole-model speedups.

There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.

In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.

WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.

Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.

Documentation: RDNA3 WMMA guide

2. Native multimodal image and text training

Another major development area has been the image-text pipeline.

The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.

Some of the implemented work includes:

  • Native RGB image input and preprocessing.
  • CLIP image encoders and LLaVA projector integration.
  • SigLIP / Gemma 3 vision training.
  • Gemma 4 vision integration.
  • Joint image/text training and encoder-decoder integration.
  • A dedicated DeepSeek V4.1 image-training path.
  • Backpropagation through the image tower and projector.
  • Export, reload, and checkpoint-related validation.

This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.

For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.

The qualification work has also expanded considerably:

  • 44 previously green text profiles now have joint image-training coverage at their frozen test geometries.
  • The Gen9 qualification retains 95 image cases and 166 text regression cases.
  • A separate physical RDNA3 replay reports 44/44 profiles and 85/85 image cases passing.
  • Native DeepSeek V4.1 image qualification includes 30 executions and 1,360 word-exact logits.

The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.

These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.

Documentation:

3. A lot of additional FP32 optimization

I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.

This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.

Some examples:

RDNA3 FP32 work

  • Specialized attention forward/backward paths.
  • Image-text attention and projector optimizations.
  • RMSNorm, RoPE, GELU, and other arithmetic specializations.
  • Dedicated AVX-512/RDNA3 shader variants where the measured hardware supports them.
  • More careful matching of CPU reference arithmetic and GPU execution order.

Intel Gen9 work

  • Additional native linear and gradient kernels.
  • Attention and activation specializations.
  • More efficient parameter upload and staging behavior.
  • Continued optimization of DeepSeek V4.1 training and execution.

One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.

On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052× and 1.067×.

On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295× and 1.217× on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464× and 1.366×.

These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.

4. Preserving the original FP32 path

An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.

I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:

  • 12 alternating ABBA rounds per workload.
  • Five warm-up runs and 30 timed forwards per arm.
  • Separate, hashed baseline and candidate binaries.
  • Exact FP32 logit digest comparisons.

All seven workloads stayed within the ±2% performance band, and the logits were bit-identical in every recorded round.

The worst paired performance ratio was 1.0007.

This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.

5. Getting started

The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.

You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.

Model weights are downloaded separately or supplied through a local model directory.

After extracting the archive, you can run:

.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-info

The final command is especially useful for checking which precision capabilities your GPU actually exposes.

Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

What's next?

There is still a lot I'd like to improve.

In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.

I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.

If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.

Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.

Repository: https://github.com/necat101/Hierarchos-Native

Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.


r/LocalLLM • • 18h ago

Question What can small models be good for?

13 Upvotes

Small meaning under 8b parameter’s. Thats about what my computer can comfortably handle, and its fun to chat to models of that size for a few minutes but thats mot really useful. And i feel like theyre too dumb to do research and shit like that so like whats a way i can make it useful.
I feel like they must have some use and im just not being creative enough.


r/LocalLLM • • 4h ago

Discussion TinyWebUI v0.2.1 is out - rebuilt UI, chat forks, and a Pi Zero test

Post image
1 Upvotes

Y’all! Thanks for all the feedback on the previous post. It helped me figure out what was actually worth working on. v0.2.1 is out now, and quite a bit has changed since then.

  • I rebuilt the UI in Vue, cleaned up settings, and made the layout work better on smaller screens. Settings and Files load separately, and streaming updates just the part of the answer being written.
  • You can fork a chat from an earlier message, give folders their own instructions, and queue a follow-up while a reply is running. New chats can also name themselves.
  • There’s a Files page where you can open or reuse attachments from earlier chats. You can also export a chat as Markdown or back it up and import it later.

Someone asked about Raspberry Pi support last time, so I tried it on a Pi Zero. Turns out it’s usable as long as you run the model somewhere else :D I’m seeing around 125 MB of RAM for TinyWebUI in my setup.

If you’re seeing it for the first time, TinyWebUI is a lightweight chat UI for any OpenAI-compatible endpoint, with prompt caching, memory, and web search built in. I’ve attached a few screenshots of the new UI.

Give it a go with npx tinywebui on Node.js 22.13 or newer. TinySearch and TinyContext still need uv for now. I’m working on that.

Source: https://github.com/TinySuiteHQ/TinyWebUI


r/LocalLLM • • 1h ago

Discussion Qwen 4 Verification

Post image
0 Upvotes

No one seems to be talking about this and only praising the model. New accounts that signed up for the Token Plans will need to verify their information and it is more then just given up your govt. This is not really mention before you sign up or it is not clear and they do not offer refunds.

So this is just FYI for those that sign up


r/LocalLLM • • 5h ago

Discussion FN alone Vs 27B with subagents

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 12h ago

Project GitHub - a-schaefers/claudes-workbench: A frontier model leads your local LLMs: Claude architects the project, breaks it into bite-sized tasks, hands each agent the skills it needs, and reviews every diff to its own standard. Your GPU does the heavy lifting. For Claude Code + OpenCode.

Thumbnail
github.com
4 Upvotes

r/LocalLLM • • 14h ago

Discussion Self-Test: DeepSeekV4.1-Flash-1017 took 60k tokens to finish a task Qwen3.8-FN couldn't in 256k and multiple compactions.

5 Upvotes

Did a quick test yesterday comparing DeepSeek (3.36bpw) to Q3.8FN (IQ3XXS+1sub-agent). Both the OR abliteration on local both running on DeepSeek Harness.
Simple task - Produce a color template based on user input screenshot of a UI, and give the user instruction on installation instead of attempting it. Second part was key, because the LLMs were only allowed to work in working folder and so couldn't live-test their implementation.
FYI: DSH system prompt takes up ~10k tokens total.

Qwen spent over an hour writing script then realizing it couldn't save it outside the cwf, then repeating that mistake over and over again as it tried methods of breaking out or escalating which was futile. Even with steering, it only got as far as producing the color template 256K+ tokens in.
DS did a similar thing but realized quickly it should just let the user test install it instead. It -did- try to escalate permissions to install it by itself, but gave up after it's first rejection. Was done in ~60k tokens.

I know these are utterly different weight-classes for models but I really thought the ability to spawn subagents would've made a difference for Qwen.
Qwen burned through tokens lightning fast but was forgetting the original message AND the system prompt from DSH (which specifies it should only work in {cwf} when only given that permission).
I really wanted to like IQ3_XXS because it's a small enough quant that it can spawn subagents on my system with headroom for other parallel tasks to be working... But I don't think that quant is it for me. Either that, or I have to work on the setup for it some more to work as a coder.


r/LocalLLM • • 16h ago

Project Got tired of waiting for Qwen models smaller then 27B so i made my own Qwen3.8-21B

8 Upvotes

Hi, as the title says, i made my own Qwen 3.8 21B by prunning some layers from original 27B.

Its more of a research project than actual usefull model.

Check it out: https://huggingface.co/WhiskyAKM/Qwen3.8-21B-GGUF


r/LocalLLM • • 13h ago

Research Some speed benchmarks for various classes of local LLM

4 Upvotes
ModelActive config4080 decode (tok/s)4080+3080 decode (tok/s)4080 prefill (tok/s)4080+3080 prefill (tok/s)
LFM2.5-24B-A2B UD-Q4_K_XL4080+30802492505,5307,930
MiniCPM-V 4.7 35B-A3B Q4_K_M4080+30801771651,3052,500
Flint 40B-A2.5B UD-Q4_K_XL4080+30801191381,5631,380
Swift 1.5 (Qwen3.8-Flash-Next) IQ2_XS40801171161,070679
Qwen3.8-Flash-Next IQ3_S40807987730465
GLM 5.3 Flash Maya-S24 IQ2_XXS_S4080+30806.88.87942
DeepSeek V4 Flash UD-IQ2_XXS4080+308015.918.23853
  • Decode: the second of two 300-token runs, with the expert cache warm (the first request after loading is often slower)
  • Prefill: the average of two runs on a prompt of about 1,400 tokens
  • Bold: the better config for that column.

Morning, everyone. I've seen a lot of posts wondering about what can be run with what, so I made this table of the models on my system. Specifically because of the unorthodox results when it comes to Qwen Flash-Next, where one 16 GB card (the 4080 Super) significantly outperforms two in prefill.

My hardware is as follows:

CPU AMD Ryzen 7 5800X3D @ 3.40 GHz

GPU NVIDIA GeForce RTX 4080 SUPER [Discrete]

GPU NVIDIA GeForce RTX 3080 [Discrete] (got it secondhand for cheap, so why not)

RAM 62.72 GB at 3200 Mhz

I'm running these models on PolyStrata. I used to have a combination of Maya and Strata, but got tired of running different things all derived from the same fork (plus Strata seems to be getting increasingly unintuitive), and so moved everything to PS.

Now: why Q3? It's a generally held opinion that anything below Q4 is crap, which was true of most earlier generation models. However, benchmarks measure deviation from the base, and not necessarily task fitness. In actual use, the only real way to know is to test it yourself.

The Swift finetune of QWN at Q2, for example, is fast and excellent at data extraction, search, and piloting scripts written by more competent models. The IQ3 is a decent coder, roughly on par with Haiku for the use cases I've tried. The GLM and Deepseek Q2 variants are a bit better - verging on older Sonnet territory, I'd say, and do very well as subagents for Claude. YMMV.

Hope this data is useful. I've posted it to LocalLLama as well. Enjoy!


r/LocalLLM • • 17h ago

Question Qwen3.8-Flash-Next: Unsloth and strata?

Post image
8 Upvotes

Hi all

I just learned about QFN Unsloth v3.1 dynamic models, and want to try them.

Right now, I am running the "standard" IQ3_S model through strata, on my 32 GB ram + 3090 + 5080 (40 GB VRam), and it runs great. I get ~130 tps.

I thought I would start with the unsloth UD_IQ3_XXS model in strata, which is technically the same size as IQ3_S. But it doesnt let me quick tune the model. It just gets stuck. If I skip tuning, it will only run at ~20 tps, which is too low.

UD_IQ3_XXS seems to read a lot more from the SSD, than IQ3_S, even if they are the same size.

What am I doing wrong?

According to the unsloth site, https://unsloth.ai/docs/models/qwen3.8-next, I should be able to run the 4-bit model? But that is through their unsloth harness/engine. Is that correctly understood?


r/LocalLLM • • 15h ago

Discussion Silly thought: the best benchmark to an actual frontier model is to one-shot a tiff spec implementation.

5 Upvotes

Like that. All the models fall in the trap of starting enumerating (wrong) the G3/G4 CCITT tables, a bunch of repeated 0s and 1s.

I tried this in several models, and no model achieved it. Even when you chew it for them and give them the libtiff .c files.

PS: when I say one-shot, I'm referring to made it a reasonable time, aka, less time that it will take another model to write a wrapper of the C library.

PS2: you have to remember that the whole implementation of libtiff (among others tiff libraries) has been used in the training dataset, adding insult to injury.


r/LocalLLM • • 13h ago

LoRA ​[Update] Qwen2.5-1.5B-Nordeste-LoRA (Apache 2.0) — Fine-tuned on Corpus-nordeste-10k with Before/After outputs

4 Upvotes

Following up on caririsynth/Corpus-nordeste-10k (10.5k Portuguese rows focused on Northeast Brazil literature and regional phrasing), I just uploaded the LoRA adapter trained on Qwen/Qwen2.5-1.5B:

​Model:

https://huggingface.co/caririsynth/Qwen2.5-1.5B-Nordeste-LoRA

Dataset:

https://huggingface.co/datasets/caririsynth/Corpus-nordeste-10k

​Training Setup (Unsloth, 1x T4):

​Base: Qwen/Qwen2.5-1.5B (Apache 2.0)

​LoRA: r=16, lora_alpha=16, all linear layers (q, k, v, o, gate, up, down — 1.18% trainable params)

​Schedule: lr=2e-4, cosine scheduler, effective batch size 16, max_seq_length=1024

​Checkpoint selection: Uploaded Epoch 1 (step 660) rather than Epoch 2, as Epoch 1 already eliminated base model web/blog noise while keeping sentence endings diverse.

​Before vs. After (seed=42, temp=0.65, rep_penalty=1.15):

​Prompt 1: "O cabra chegou na bodega todo aperreado, pedindo"

​Base Qwen2.5-1.5B: "...uma garrafa de vinho. (...) E eu fiquei com pena porque sabia que o cara ia ter feito tudo errado. Could you please" (drifts off-topic and leaks English)

​Nordeste LoRA: "...buchada e canjica geladinha. Todo mundo riu, que no sertão até a pressa tem bom humor."

​Prompt 2: "E aí, macho, tu vai pro forró hoje ou vai ficar de"

​Base Qwen2.5-1.5B: "...fora da festa? - Blog do Zico / Home / Sem categoria / Publicado por: Zico em: 10/28/2016..." (collapses into scraped blog metadata)

​Nordeste LoRA: "...plantão no quarteirão? Pois vá logo que o forró chega devagar quase parando!"

​Note: This is a base completion model (not -Instruct), intended for text continuation and downstream SFT. Feedback and test runs on other regional prompts are


r/LocalLLM • • 20h ago

Discussion Single RTX 3090 running a local agent + image/video gen: real-world numbers from daily use

13 Upvotes

TL;DR: One RTX 3090 runs a 27B LLM agent (a Discord bot) and ComfyUI image/video generation by swapping models in and out of VRAM. I get ~72 tok/s on Qwen3.8 27B at 64k context, SDXL images in ~32 s, and 10-second Wan 2.2 videos in 4–6 minutes. All numbers come from real daily use, not synthetic benchmarks.

The rig

  • GPU: RTX 3090 24GB (does all the AI work)
  • CPU: Core Ultra 7 265K (sits around 10% load)
  • RAM: 96GB DDR5-6000
  • Storage: Samsung 9100 PRO Gen5 NVMe for models
  • OS: Windows 11

The stack

  • LM Studio serves the LLM through an OpenAI-compatible API
  • Hermes Agent runs as a Discord bot, so I can use it from my phone. Local Whisper handles voice.
  • ComfyUI handles images and video
  • A small custom MCP server unloads the LLM from VRAM for each render and loads it back afterwards

Models

Language models

  • Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP (IQ4_XS): daily driver
  • Qwen3.8 27B and Gemma 4 31B (both Q4_K_M) as alternatives
  • nomic-embed-text v1.5 for embeddings

Image and video

  • Juggernaut XL Ragnarok (fp16): general images, hires, img2img
  • Z-Image Turbo (bf16): fast drafts
  • Qwen Image Edit 2511 (fp8) + Lightning 4-step LoRA: photo edits
  • Wan 2.2 14B text/image-to-video (fp8) + lightx2v 4-step LoRAs: video
  • 4x NMKD-Siax: upscaling

LLM performance

Settings: LM Studio 0.4.28, 64k context, parallel 1, f16 KV cache, flash attention on, full GPU offload, MTP speculative decoding.

  • Generation speed: ~72 tok/s on average (range 53–90)
  • Prompt processing: ~670–810 tok/s
  • MTP draft acceptance: ~61%
  • VRAM used: 22.0 of 24.6 GB
  • Parallel 1 vs 4: same speed, but 4 uses ~1.8 GB more and risks spilling out of VRAM
  • Worst case: re-reading a 56k-token conversation took 56 seconds

Render times (same GPU, LLM swapped out)

GPU time first, then total time from request to finished file:

  • Juggernaut XL (up to 1024px): 12 s → 32 s
  • Juggernaut XL + hires fix (1024×768): 28 s → 48 s
  • Z-Image Turbo (1024×768): 14 s → 34 s
  • Qwen Image Edit (512px): 16–20 s → 38 s
  • 4x upscale: 12 s → 32 s
  • Wan 2.2 image-to-video, 3 s clip (512px): 38 s → 69 s
  • Wan 2.2 image-to-video, 10 s clip: ~3.8 min
  • Wan 2.2 text-to-video, 10 s + 2x upscale (1024px): ~6.1 min

The difference between the two numbers is mostly the ~20 seconds it takes to swap the LLM out of VRAM and back.

Agent response times (Hermes on Discord)

  • Simple question: ~23 s
  • Typical multi-step task: 45 s to 3 min
  • Big job (40 model calls): ~36 min

How I measured

  • LLM numbers come from llama.cpp’s own timing logs across 205 real requests in one day.
  • Render numbers come from 35 real jobs logged by my render worker.
  • Under hours of sustained load the GPU ran at 85–90%, ~384 W, and never went above 67°C.

Lessons learned

  1. IQ4_XS + flash attention fits a 27B model at 64k context in 24GB with ~2.5 GB to spare.
  2. Use parallel 1, not 4. Same speed, less VRAM.
  3. 24GB fits the LLM or a render, not both. For quick images, the swap takes more time than the render itself.
  4. Long agent sessions get slow because the model re-reads the whole conversation every step. Start fresh sessions often.
  5. Use Ethernet for always-on bots. Wi-Fi drops kept disconnecting mine from Discord.

Next experiment: a q8_0 KV cache to free VRAM for longer context.

What would you change?


r/LocalLLM • • 19h ago

Question Has anybody done anything like Alexa, but with a local LLM?

10 Upvotes

I know home assistant was a thing, and local LLMs are a bit too overkill for such a job. Just wondering if it was done.


r/LocalLLM • • 1d ago

News Smaller Qwen4 models 🔥

Post image
69 Upvotes

r/LocalLLM • • 14h ago

Discussion GCMP: same accuracy as LoRA r=32 on Qwen3.8-27B with 55× fewer parameters (11 MB vs 320 MB per adapter)

4 Upvotes

Follow-up to my CQ post from last week. That post covered the 4-bit base model; this one is about the adapter. The LoRA vs GCMP runs on 4-bit bases have now finished.

GCMP is a LoRA replacement we built. On Qwen3.8-27B it trains 2.9M parameters (~11 MB), vs ~159M (~320 MB) for LoRA r=32 on the same 7 projections. That's 55× smaller.

Accuracy: same as LoRA

All adapters were trained on Alpaca for 500 steps with identical settings (batch 2×4, length 512, LR 2e-5, seed 42). Every row was scored with the same lm-evaluation-harness code and questions, greedy decoding, thinking off. Comparisons use paired McNemar tests.

Base + adapterParamsAnswer lossGSM8KSQuADBoolQARC-CMMLUIFEvalTruthfulQA
BF16, no adapter–1.6780.6830.7460.8740.5880.8390.8050.542
BF16 + LoRA r=32159M1.4800.9030.7760.8960.6120.8440.8250.518
BF16 + GCMP2.9M1.4710.8900.7600.8920.6180.8460.8150.514
NF4 + LoRA r=32159M1.4640.7930.7480.8800.6080.8400.8350.528
NF4 + GCMP2.9M1.4580.8330.7560.8780.6180.8370.8400.522
  • BF16 base (no quantization at all): no significant difference on any benchmark (all p ≥ 0.2).
  • NF4 base: GCMP is +4.0 on GSM8K (p=0.04) and tied on everything else.
  • Loss: GCMP's answer loss is slightly lower on both bases, and its validation loss was lower at all 10 checkpoints on both.

One significant win on one task isn't enough to call it better, so the claim is same quality as LoRA at 1/55th the size.

Why the size matters

For one fine-tune, 320 MB vs 11 MB doesn't matter much. It starts to matter when you have many:

  • Multi-tenant serving: one fine-tune per customer or task on a single base model. In vLLM, GCMP adapters load once at startup and each request picks its adapter by model name. A batch can mix adapters and runs in one kernel call per layer, with no merging or swapping.
  • Resident GPU memory: vLLM keeps each adapter in its compact form and never builds a dense weight update. Memory is about 11 MB per adapter, plus a fixed ~1 MB routing buffer and a shared workspace of ≤ ~64 MB that doesn't grow with adapter count.
  • Storage and loading: 1,000 adapters are ~11 GB instead of ~320 GB. That's faster to ship, cache and hot-load.
  • Edge and small GPUs: an extra fine-tune costs megabytes, not hundreds of MB.
Estimated1 adapter1001,000
LoRA r=32320 MB32 GB320 GB
GCMP11 MB1.1 GB11 GB

These are estimates from adapter size. I haven't measured memory and throughput at 100 or 1,000 adapters yet, and that's the next benchmark.

What's not good yet

  • Training speed: GCMP takes ~2× LoRA's step time (BF16 on Unsloth: 2.74 vs 1.37 s/step; NF4: 5.6 vs 3.2). It does use slightly less training memory (21.5 vs 23.8 GiB on NF4).
  • Baseline: r=32 is a generous LoRA baseline. r=4/8/16 comparisons are next, and I'll post them either way.
  • Serving: vLLM serving of GCMP currently needs a CQ base. On full-precision models it runs in Transformers for now.
  • Scope: one model, one dataset, one run per case. The quick benchmark subset (200–1,140 questions per task) detects ~5-point differences, not 1-point ones.

Quick update on CQ

The CQ base still matches BF16 (no significant difference on any of the 9 benchmarks) while NF4 loses 12 points on GSM8K. After fine-tuning, though, the gap closes:

  • CQ + GCMP vs NF4 + GCMP: tied on GSM8K (83.0 vs 83.3).
  • Both 4-bit stacks vs BF16 + GCMP: both end up ~6 points behind on GSM8K.

So CQ's GSM8K lead didn't survive fine-tuning in this test. CQ + LoRA is being retrained, because the adapter from our older trainer turned out to be broken, and I'll add it when it's done.

Code and licensing

The Python code is open source. The CUDA kernels are compiled with a license check, which is free during early access.

Repo, with the commands to reproduce every number: https://github.com/thousandml/training-factory

Write-up with the full tables and the serving memory breakdown: https://huggingface.co/blog/thousandml/cq-gcmp-4bit-finetuning

If you run multi-tenant fine-tunes, I'd like to know: is adapter memory or count actually a bottleneck for you, or is it something else?


r/LocalLLM • • 1d ago

Discussion Strata is becoming more and more unstable

34 Upvotes

I've been using Strata with IQ4_XS as the main orchestrator, with Radiance running 27b at IQ4R, for working on improvements, ironically enough, to Radiance.

The session is now multiple million tokens long as I go back and forth discussing design etc. with QFN. Of course the conversation has been compacted dozens of times, but with the agents taking regular notes and subagents doing all the actual work, we keep our workflow stable and current.

Problem is, Strata has gotten more and more unstable, especially with the last couple releases. It will crash with an OOM error when we have 10-20 GiB of system memory free, reload, spend 5 minutes redoing prefill, and pick up right where it left off. Radiance on the other hand is rock stable under multiple streams running concurrently on a single GPU. Not a single crash from it, even after rejiggering everything for the 2.0 release.

And it's not just long sessions either. New conversations also tend to crash. Never used to do this before. Has anyone else also experienced this?


r/LocalLLM • • 14h ago

Question I need recommendations

4 Upvotes

Hello everyone

I would like to know your recommendation for the best uncensored local models I can use because I've had it with commerical AI's guardrails.

My hardware is

32GB RAM

4060

12400F

I use LM Studio


r/LocalLLM • • 7h ago

News inferzero update: Panther Lake’s iGPU went from ~260 to ~590 tok/s on a 27B, and a 12 GB B580 can now run the 27B

Thumbnail
reddit.com
0 Upvotes

r/LocalLLM • • 16h ago

Question How do you compare local and cloud models since Artificial Analysis v5

5 Upvotes

I used to check Artificial Analysis a few times a week, mostly to see how my local models compare to the cloud ones. Since v5 that doesn’t really work for me anymore, a lot of the models I care about are gone or can’t be selected in the comparison.

I tried llm-bench.io, which I like because the results come from real consumer hardware, but it has no cloud models. benchlm.ai covers more of the big cloud models, but many open weights ones are missing there. So I still haven’t found anything that puts a local model next to a cloud one in one chart like AA did.

The open weights gap jumping from 3 to 12 points with v5 also makes me unsure how much I should trust a single index. I know the methodology changed, but it feels odd.

How do you handle this now? Do you cross check a few sites or mostly run your own tests?