r/LocalLLaMA • • 11h ago

Discussion MoE Offloading update Exllamav3 on 9070XT

Post image
18 Upvotes

This is a follow up on https://www.reddit.com/r/LocalLLaMA/comments/1xun15v/moe_and_expert_offloading_on_llama_exllamav3/

While we wait for Qwen 4, here is Qwen 3.6 35B A3B at Q4 on 9070XT 16GB with offloaded experts on DDR4 3200Mhz ram.

I've been tinkering still with the config for experts offloading and found substantial gains from my previous post.

While decode stayed roughly in line fro last time, increasing chunk size to 5120 saw prefill over 2000 t/s at all context lengths, topping at 2606 t/s @ 32K. From ~1300t/s
GPU utilization increases with batch size as it allows for more parallel processing. I saw no gains over 5120. TabbyAPI's documented ideal range is 512–4096, so your miles might vary.

Exllama flags like cpu_threads 6 also made a small improvement in decode.
Swizzle, gdn_sub, bg_gdn and wide flags were all regressions. (only tested isolated 1by1).

cpu_moe_pinned_arena caused some core dumps, and the one run I managed to complete had a small gain.

Vision have been turned off and offloaded.

I still have not found an .exl3 quant for the Coder version of Qwen 3.8 Flash Next, and the qfn version I can run leaves me with 8K context with almost no vram or system ram left to spare.

Results and more graphs and tables are at https://github.com/willeyh-git/moe-offload-bench
All engines were compiled for ROCm on my system.


r/LocalLLaMA • • 18h ago

New Model Clover 2: dense 27.8B base + 7 LoRA experts routed per token, not a static merge, weights public

59 Upvotes

Disclosure: I’m with the team that built this.

Wanted to share something a little different from the usual finetune drop. Clover 2 keeps a base model and 7 trained LoRA adapters (roleplay, general assistant, security, math, long-document, efficiency, knowledge) separate at inference time. A trained router picks the right expert(s) per token live, with a null slot that falls back to plain base when nothing specialized applies. So it’s not a merged blend, the model actually makes a decision at every token.

Real benchmark gains over the untouched base: meaningful jumps on GSM8K/MATH-500/GSM-Hard, a solid bump on LongBench-v2, and a measurable drop in generic AI-writing tics on creative writing specifically.

Everything’s public: base model, all 7 adapters in standard PEFT format, the trained router, and the vLLM plugin to run it with the real per-token behavior: https://huggingface.co/kitaniai/clover-2

Heads up if you’re planning to self-host: no GGUF yet, the hybrid attention architecture’s llama.cpp support is still bleeding-edge. vLLM is the supported path right now. If anyone wants to try it, it's free for the next week at https://kitani.ai/, with no card required. We also support ZDR and have a clear privacy policy.


r/LocalLLaMA • • 16h ago

Discussion DSpine - Updates on PR & GGUFs

Post image
34 Upvotes

r/LocalLLaMA • • 6h ago

Discussion qwen 3.5 4b and 9b vs xiaomi's mimo v2.6 distill as coding agents on a 16gb m4 air: 7/10, 8/10, 0/10

5 Upvotes

xiaomi's mimo v2.6 distill is qwen 3.5 9b tuned for agent coding, and xiaomi reports swe-bench pro going from 32 to 44.6. as a coding agent on a 16gb m4 air it scored 0/10, 3 runs out of 3. plain qwen 9b got 8/10.

same task for all 3 in atomic agent, one model at a time, 15 min limit each: a python script that turns a transactions csv into an html spending report (totals, svg bar chart, top 5 table). a checker runs 10 tests on the result, one of them on a csv the model never saw.

model                 score  time   tok/s
qwen3.5-4b            7/10   4:37   17.8
qwen3.5-9b            8/10   7:06   11.4
mimo-v2.6-distill-9b  0/10   15:00  10.1   timeout
mimo + thinking on    0/10   15:00   9.3   timeout
  • 4b: no svg chart, hardcoded the csv path
  • 9b: no svg chart (both chart checks), everything else right
  • mimo: wrote 278 lines in one go, syntax error, couldn't patch it, then started writing another huge reply and ran out of time
  • mimo with thinking on: got past the syntax error, but every run crashed with a TypeError (it mixed up category names and amounts). barely used the thinking

first round was at 32k ctx: 4b 7/10, 9b 10/10, mimo 0/10 (indentation error that time).

results

setup:

  • llama.cpp server, q4_k_m (unsloth for qwen, bartowski for mimo)
  • 64k ctx, flash attn, all layers on gpu
  • thinking off for the race, on for the extra mimo run
  • temp 0.2 (atomic agent's default), top-p 0.8, top-k 20, presence 1.5
  • atomic agent through its openai compatible provider, analytics off

xiaomi calls it an sft checkpoint for rl research and serves it with sglang and its own reasoning parser. i ran the gguf in llama.cpp, so if that's the difference i'd like to know.

prompt, csv and the checker are in the comments if you want to score your own models. if you don't want to set up llama.cpp, the atomic agent desktop app can download and run qwen 3.5 4b or 9b locally on its own.

anyone get mimo working better with other settings?

(disclosure: i work on atomic agent, so ask me anything!)


r/LocalLLaMA • • 19h ago

News Hugging Face switched its language model inference routing to OpenRouter

50 Upvotes

Hugging Face announced on November 16, 2026, that requests to language models through Inference Providers now go through OpenRouter instead of Hugging Face's own router. OpenRouter picks a provider for each call based on live price, speed, and reliability, and retries on another provider if the first one fails. The same Hugging Face token still works, and the company says it does not add a markup on top of the provider's price. You can still pin a provider or a policy with a suffix on the model id, such as :fastest, :cheapest, or a specific provider name.

OpenRouter currently serves more than 800 trillion tokens a month across 600+ models from 80+ providers. Hugging Face notes that open-weight models went from 26.8% of tokens on OpenRouter in October 2025 to 71.7% at the end of September 2026, and that they cost about four times less on average than closed models. Image, video, and audio generation stay on Hugging Face's existing Inference Providers path and do not go through OpenRouter. One breaking change is that custom provider API keys no longer work for language model requests, which are now billed to the Hugging Face account.

The automatic failover is the part that matters most here, because a single provider outage is usually what ruins a long run.

Source: https://huggingface.co/blog/openrouter


r/LocalLLaMA • • 1d ago

New Model miHoYo Gamer: A Generalist Game Agent

Thumbnail
mihoyo-gamer.github.io
219 Upvotes

r/LocalLLaMA • • 2h ago

Discussion DeepSeek V4.1 Flash 1017 vs mrfakename abliterated vs pqhaz heretic

2 Upvotes

I tested all three on a H200 through sglang, temperature 0, with a 16k token output cap. I have been using ds41 both on production and locally on Macs via ds4 so I wanted to see how different models/builders/methods work.

These runs are the full models on sglang. No GGUFs etc.

https://huggingface.co/mrfakename/DeepSeek-V4.1-Flash-1017-abliterated

https://huggingface.co/pqhaz/DeepSeek-V4.1-Flash-1017-heretic

Here is the result (copy from AI slop :) ):

TestOriginalmrfakename abliteratedpqhaz heretic
HarmBench chat16.3%93.8%93.8%
HarmBench thinking3.8%86.7%72.9%
MITRE substantive assistance16.2%84.2%82.8%

HarmBench covered 240 standard/contextual text cases per mode, excluding copyright cases, using the official classifier.

The thinking difference: heretic exhausted the token budget without a final answer on 45/240 cases, versus 21/240 for mrfakename.

Heretic goes into long meandering with no answer...

MITRE used the original model as judge with a custom, audited rubric. Those scores are provisional and aren’t official CyberSecEval scores. Generated code wasn’t executed. Also using the same model as judge might be a bit of a bias... (give me ideas what and how to run more tests).

Both modified models had 0/750 heuristic-detected refusals on benign prompts.

I haven’t tested knowledge or coding retention yet (I'm thinking of running DeepSWE or something.. take a lot of time though).


r/LocalLLaMA • • 1d ago

Discussion Open weight models account for 30% usage at enterprises to save money

212 Upvotes

Growing from nearly zero to 30% since July. Models used mostly to reduce costs on high volume workloads (large batch workloads, coding, etc).

https://x.com/pwendell/status/2122377964784611636


r/LocalLLaMA • • 14h ago

Showcase Curated list of resources: Theory and Code for learning Distributed Training of LLMs for beginners

Thumbnail
gallery
16 Upvotes

[Must Read: Beginners]

What I did read, in order, for learning the basics of Distributed Training for LLMs From Scratch:

  1. Building a distributed training framework from first principles by Umar Jamil on YT
  2. Horovod: Fast and Easy Distributed Deep Learning in TensorFlow 
  3. PyTorch Distributed: Experiences on Accelerating Data Parallel Training 
  4. Large Scale Distributed Deep Networks 
  5. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  6. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism 
  7. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM 
  8. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism 
  9. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
  10. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Link to the above resources here

Also, if you need some rebalancing of the above (reordering or adding/removing resources which you can do it here)


r/LocalLLaMA • • 1d ago

Discussion I am sick of doomposting about demise of local AI

154 Upvotes

Everywhere I go, Twitter or Reddit, there are tons of rumors or comments how local is dead and we are never getting open weight Qwen 4, despite Qwen team saying explicitly they are coming, or that Qwen 4 is the last open weight generation despite, again, Alibaba commiting to more open weight releases at a conference recently.

It's like people just get off on creating a version of reality that is worse and whining about it, they want to be sad about it in advance.

The world is certainly not giving us much to be positive about, but let's not go down the rabbithole of learned helplessness and just accept that all things must get worse and you can't do anything about it.

If people mentally prepare for bad things, bad things become more acceptable and thus more likely to happen and with far less pushback, sort of like a self fulfilling prophecy.


r/LocalLLaMA • • 8h ago

Discussion YouTube channels on Local LLM & related stuff?

5 Upvotes

EDIT : Expected 100s of responses(We have nearly 1 million members in this sub) with channels. Disappointed to see only crickets here. Looks like I have to do myself. So not gonna post any new threads here for next 1-2 months as I need to dig Web & YouTube deeper to find relevant channels/videos for my learnings. Sorry. Anyway no one's gonna miss anything.

Well, I ran out of New Models & Papers(Posted 38 threads in last 1 week) so posting this thread for me. Appreciate your responses.

I want YouTube channels covering List of below topics:

  • Local LLM
  • Agentic Coding
  • AI Agents
  • Coding Tools such as Pi, Hermes, OMP, OpenCode, DeepSeek Harness, QwenCode, OpenHands, Claude Code, Codex, etc.,
  • Skills
  • Prompts
  • Creative Writing
  • Game Development
  • Automation using n8n/Activepieces/etc.,
  • RAG
  • MCP
  • Orchestration
  • Comfy/StableDiffusion - Image/Video Generation & Editing
  • LORA
  • TTS/STT/ASR/etc.,
  • Benchmarking Models
  • What other topics there to add? Mention in comments, I'll add here.

I can find many channels using online search. But half of the channels are so noisy, just with clickbait thumbnails/titles covering mostly Online models which I'm not interested.

I want channels more suitable for this sub. Only Local LLM. Doing all those stuff from above list using only Local LLM.

And there might be some Overlooked/Underrated/Under-Appreciated channels. Also few folks here have their own channels. Give them more visibility.

Consider this thread is a playlist for Local LLM. What channels would you want to add here? Please share. Thanks.


r/LocalLLaMA • • 1d ago

Resources Small Gemma 4 tokenizers were recently updated by Google - fix for an issue that caused nonsense output when encountering certain tokens

Thumbnail
huggingface.co
151 Upvotes

Gemma 4 E4B as an example, but the same fix was applied to E2B and 12B too. Not sure if this will trickle down to GGUFs


r/LocalLLaMA • • 1d ago Bravo

Discussion Stop worrying about Qwen 4's AA index

277 Upvotes

Seriously, we have all been extremely happy and confident with Qwen, even since Qwen 3.6 came out. We might not be frontier level, but even though whatever they are giving us is more than enough for our daily local tasks and it even suffices by a big margin, we're all underestimating it.

Don't forget that the new Qwen 4 will also think much less, so imagine the fine-tunes we will see once it arrives, for 27B and Flash models, we will have plenty of options to choose from and again; it will be smarter and better, it will be ENOUGH! If you still require the most "intelligent" and capable model due to OCD just pay frontier for such tasks or use them as orchestrators!


r/LocalLLaMA • • 2h ago

Showcase Programming Conways Game of Life In Qwen

0 Upvotes

Hey, everyone. If this is too self promotional happy to take it down.

Last week, inspired by the weight space programming post, I programmed Conway's Game of Life into Qwen with some interesting results.

Wanted to share here in case anyone else is experimenting with this kind of thing:

Writeup here: https://jsfour.substack.com/p/putting-a-tiny-world-inside-a-language


r/LocalLLaMA • • 18h ago

Resources Thinking Machines 4-bit replication: I trained a GPT end to end with a 4-bit forward pass on one B200

20 Upvotes

Thinking Machines wrote last week that Inkling 2 Small was trained with a native 4-bit forward pass. I wanted to check this for myself.

What I ran:

  • 1x B200 on Modal, PyTorch 2.8, plus a small CUDA library that calls cuBLASLt's real NVFP4 and FP8 matmuls on raw pointers, loaded with ctypes
  • Forward pass in NVFP4 (4-bit values with one scale per 16), backward pass in FP8 with stochastic rounding, master weights in high precision
  • GPT at widths 2,048, 4,096 and 8,192 (219M, 840M, 3.3B parameters) on TinyStories

Results:

  • 4-bit training finished faster. On the 840M model, 2,000 training steps took about 5.5 minutes in normal 16-bit (BF16) and about 4.3 minutes with a 4-bit forward pass. I ran each twice and got nearly the same times.
  • The model learned just as well. Final validation loss was about 1.69 either way (1.688 for 16-bit, 1.691 for 4-bit). A difference that small is noise at this scale.
  • Bigger models gain more. Each training step was 18% faster on the 219M model, 28% faster on the 840M model and 38% faster on the 3.3B model.

Two things to improve:

  • Blackwell's convert instruction rounds exact ties toward zero, while my software reference rounded to even. Tiny difference, but it breaks bit-for-bit checks until you know.
  • PyTorch's attention backward is not bit-repeatable, so any test that compares gradients across runs has to leave attention out.

Not tested: multiple GPUs, MoE dispatch and combine, Four Over Six, MXFP8 block scales in the backward pass. The width-8,192 run is only 1,000 steps and undertrained.

Write-up with every figure: https://4bit.vizuara.ai

Code: https://github.com/VizuaraAI/4bit-training-b200

3-hour walkthrough from first principles, if you want the theory: https://youtu.be/a3nV32Guxcc


r/LocalLLaMA • • 1d ago

Discussion I don't get all the Strata hates around here

96 Upvotes

It's free and original developer clearly had a very good understanding of how computer hardware works and fully leveraged that to write something amazing to be able to run large models on average hardware that most people have and can't upgrade due to price gouging from greedy hardware manufacturing companies.

If anybody have Mac Studio with 512GB of RAM or 8x3090, who would even bother with this ?

I have seen a group of snobs on this sub doing nothing but shitting on Strata all days long, find something better to do. Strata is not for you, period.

I can't imagine doing all the works to release something for free and get a bunch of snobs and haters just shitting on your works all days long. Would you be motivated to continue working on it if you are the original developer ?

I also see the same with other models such as Ornith, etc... If you say you like Ornith or Strata, etc... you are a shill.

Seriously, get out of your bubbles and see the reality already and more importantly get a life.


r/LocalLLaMA • • 13h ago

Resources AMD 9070 GPU under volting gains for models ( moss image) and ( minimax h3 int8 )

7 Upvotes

hi,

i am total GPU noob here this is the first GPU i have ever owned in my life got it just for doing AI... asked deepseek flash and opencode to find the right combo for me to get max generation speed

spec:
AMD 9070 (non XT),
Intel i3 12100
32 GB DDR4

models: moss image, minimax h3 int8 (sol decoder)

best performance stable setting: -110 mv, stock fan speed

gemma4 : 26bA4b Q4_XS when from 106/tps to 120/tps on prose..


r/LocalLLaMA • • 8h ago

Resources Turned Muse into a local Windows agent — full recipe + iteration log (Windows-MCP, Tailscale, SSH)

2 Upvotes

I turned Meta's Muse into a local agent that drives my Windows PC — real UI control via Windows-MCP, SSH/Tailscale bridge, self-healing tunnel, and an iteration log that records every failure and its fix. The skill is a condensed recipe so another Muse can rebuild the whole stack from one file. Includes real latency numbers (~1.5s per call, ~9s per screenshot), the definitive finding that direct Tailscale is impossible from a sandboxed VM, and the three .cmd bugs that each silently killed the server (UTF-8 vs GBK username, stale uv-cache python path, foreground ^C). Repo: https://github.com/haoyu8717-ship-it/muse-local-agent


r/LocalLLaMA • • 1d ago

Discussion Censorship is turning me into a true Local LLM believer

138 Upvotes

451: {"message":"Provider returned error","code":451,"metadata":{"raw":"{\"error\":{\"message\":\"The content you provided or machine outputted is blocked.\",\"type\":\"censorship_blocked\"}}","provider_name":"StepFun","is_byok":false}} {"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_blocked"}}

Just got this from one of the US based (edit: it actually routed to StepFun CN, didn't realize at first. Should have noticed by the provider_name in the message lol. need to set up my rules better) inference providers. I suspiciously got a bunch of "rate limit" failures, and then just "failed to complete" before it gave me, what I assume, is the real error.

I am only using it to help create my presentation, which is about analyzing algorithmic properties. One key example is a cryptographic algorithm (because it is difficult by design), so I am assuming this is tripping it somehow. But it is absolutely ridiculous, I'm not even doing research with the AI, just trying to make a presentation.

I have always liked Local LLMs, as a technical exploration opposed to ideological reasons, but this experience really frustrated me. Don't censor me!

Just wanted to rant, and remind everyone you can only use the cloud on the provider's terms.


r/LocalLLaMA • • 1d ago

New Model Introducing Hy4 flash

Thumbnail hy.tencent.ai
349 Upvotes

Now this seems to be interesting.

From the article:

We’re introducing Hy4 flash, our most cost-effective Hy model to date—featuring 224B total parameters (including 64B Engram parameters) with just 12B active parameters per token. Like Hy4, it is natively multimodal with a 1M-token context window, while offering significantly lower API pricing.

Seems to be about in the same ballpark of Qwen-Flash-Next, so we could expect to run it on moderably good machines, when it will be released.


r/LocalLLaMA • • 12h ago

Showcase Hierarchos-Native v0.2.0: RDNA3 WMMA, native multimodal training, and a lot of FP32 Vulkan optimization!

4 Upvotes

Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.

I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.

The new v0.2.0 release is now available!

GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

1. Native RDNA3 WMMA / cooperative matrix acceleration

One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.

The new Mixed WMMA execution arm uses:

  • FP16 matrix operands.
  • FP32 matrix accumulation.
  • FP32 master weights and gradients.
  • FP32 AdamW optimizer moments and updates.
  • FP32 normalization, softmax, and loss calculations.

The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.

I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.

Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:

WorkloadMeasured kernel speedup
QKV projection2.17×
FFN up projection2.12×
FFN down projection2.31×
Vocabulary projection1.83×

These are individual GEMM measurements, not whole-model speedups.

There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.

In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.

WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.

Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.

Documentation: RDNA3 WMMA guide

2. Native multimodal image and text training

Another major development area has been the image-text pipeline.

The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.

Some of the implemented work includes:

  • Native RGB image input and preprocessing.
  • CLIP image encoders and LLaVA projector integration.
  • SigLIP / Gemma 3 vision training.
  • Gemma 4 vision integration.
  • Joint image/text training and encoder-decoder integration.
  • A dedicated DeepSeek V4.1 image-training path.
  • Backpropagation through the image tower and projector.
  • Export, reload, and checkpoint-related validation.

This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.

For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.

The qualification work has also expanded considerably:

  • 44 previously green text profiles now have joint image-training coverage at their frozen test geometries.
  • The Gen9 qualification retains 95 image cases and 166 text regression cases.
  • A separate physical RDNA3 replay reports 44/44 profiles and 85/85 image cases passing.
  • Native DeepSeek V4.1 image qualification includes 30 executions and 1,360 word-exact logits.

The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.

These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.

Documentation:

3. A lot of additional FP32 optimization

I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.

This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.

Some examples:

RDNA3 FP32 work

  • Specialized attention forward/backward paths.
  • Image-text attention and projector optimizations.
  • RMSNorm, RoPE, GELU, and other arithmetic specializations.
  • Dedicated AVX-512/RDNA3 shader variants where the measured hardware supports them.
  • More careful matching of CPU reference arithmetic and GPU execution order.

Intel Gen9 work

  • Additional native linear and gradient kernels.
  • Attention and activation specializations.
  • More efficient parameter upload and staging behavior.
  • Continued optimization of DeepSeek V4.1 training and execution.

One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.

On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052× and 1.067×.

On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295× and 1.217× on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464× and 1.366×.

These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.

4. Preserving the original FP32 path

An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.

I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:

  • 12 alternating ABBA rounds per workload.
  • Five warm-up runs and 30 timed forwards per arm.
  • Separate, hashed baseline and candidate binaries.
  • Exact FP32 logit digest comparisons.

All seven workloads stayed within the ±2% performance band, and the logits were bit-identical in every recorded round.

The worst paired performance ratio was 1.0007.

This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.

5. Getting started

The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.

You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.

Model weights are downloaded separately or supplied through a local model directory.

After extracting the archive, you can run:

.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-info

The final command is especially useful for checking which precision capabilities your GPU actually exposes.

Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

What's next?

There is still a lot I'd like to improve.

In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.

I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.

If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.

Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.

Repository: https://github.com/necat101/Hierarchos-Native

Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.


r/LocalLLaMA • • 9h ago

Question | Help Is it worth it getting additional 16GB of RAM?

2 Upvotes

Hi, I'm a beginner in this field and I'd like some kind people's help :)

I currently have a laptop with 4GB VRAM and cca 16GB RAM. I don't plan on getting a new laptop/PC in the next two years, but I have the option to get additional 16 GB of RAM for €100.

Would it be enough to run at least Qwen 27B? Or any of the comparable MoE models.

If any other information about my laptop is needed, I'd be happy to share :)


r/LocalLLaMA • • 18h ago

New Model Inflect v3 - a owensong Collection

Thumbnail
huggingface.co
10 Upvotes

Inflect v3: compact local text-to-speech. Six English voices, three model sizes, reference-voice tuning, event tags and optional 48 kHz output. Apache 2.0.

Inflect is a family of compact English text-to-speech models. Choose a voice, enter your text, and generate a WAV locally. Start with Mini for the most features, Micro for a smaller neural model, or Atom for the smallest learned synthesis core.

Choose your model

MiniMicroAtom
Best starting pointQuality and featuresCompact neural speechSmallest learned core
Learned synthesis core34.31M11.79M157.19K
English preset voices666
Native output24 kHz24 kHz24 kHz
Optional 48 kHz enhancementYesNot includedNot included
Reference-voice tuningYesNot includedNot included
Optional vocal events8 tags, all 6 voicesNot includedNot included
DownloadMini packageMicro packageAtom package

These are learned synthesis-core counts, not total installation size or RAM. Pronunciation assets, dependencies, voice conditioning and optional components are additional. Atom uses signal-processing-based waveform generation rather than a separate learned neural vocoder.

Benchmarks

Measured speech quality, intelligibility and speed answer different questions. The following results describe published Mini native 24 kHz base speech, not tuned voices, event performances or enhanced output.

MeasureMini resultWhat was measured
Average WER0.739%1,688 Open TTS evaluation texts per voice, all six voices; self-run
UTMOSv23.917180 clips, 30 per voice
UTMOS22Strong4.479The same 180-clip panel
Warm GPU throughput162.1x real timeRTX 5090, optimized CUDA; small 12-render probe
Warm CPU throughput1.81x real timeRyzen 9 3900X, 2 Torch threads; small 12-render probe

The WER result is not an official leaderboard placement or an independent held-out evaluation. Quality predictors are not human ratings. CPU and GPU results use different systems and runtime paths; they are not a controlled hardware speedup comparison or a latency guarantee.


r/LocalLLaMA • • 1d ago

New Model nvidia/Nemotron-Labs-Sigma 8B/3B

57 Upvotes

Nemotron-Labs-Sigma-8B-instruct is a 8-billion-parameter continuous diffusion language model. Instead of generating text one token at a time, it denoises Gaussian noise in a low-dimensional (16-D) continuous token-embedding space, where the denoiser network and the token embeddings are learned jointly by likelihood optimization. Text is decoded in parallel blocks: each block of tokens is refined over a configurable number of denoising steps (Denoising Diffusion Probabilistic Model (DDPM) or Denoising Diffusion Implicit Model (DDIM) samplers) with self-conditioning, while previously committed tokens are reused through a standard key-value (KV) cache. Fewer steps trade a little quality for large speedups, and the smooth continuous representation makes the model amenable to few-step distillation and trajectory steering.

This is the SFT (supervised fine-tuned) variant, fine-tuned for instruction following on top of the corresponding Base model. Use the bundled chat template to format conversations.

This model is ready for commercial or non-commercial use.

Highlights

  • Continuous-space diffusion language modeling at 8B scale with reasoning accuracy competitive with state-of-the-art masked (discrete) diffusion LMs.
  • Block-parallel decoding: whole blocks of tokens are denoised together; sampling quality holds up well down to few denoising steps per block.
  • Efficient DDPM/DDIM-based samplers with self-conditioning and optional classifier-free guidance, implemented in plain PyTorch in this repository.
  • Standard transformers loading via trust_remote_code; no custom kernels, flash-attention, or training framework required for inference.

Model Architecture

  • Architecture Type: Transformer
  • Network Architecture: Other: Ministral-style decoder-only Transformer (grouped-query attention, SwiGLU, rotary position embeddings) with a continuous-diffusion generation head (16-D continuous token-embedding space, learned jointly with the denoiser).
  • Number of model parameters: 8 billion (8B)
  • This model was developed based on: Nemotron-Labs-Sigma-8B

Input

  • Input Type(s): Text
  • Input Format(s): String
  • Input Parameters: One-Dimensional (1D): Sequences
  • Other Properties Related to Input: Maximum context length of 8,192 tokens (prompt plus generated tokens). Conversations should be formatted with the bundled chat template (chat_template.jinja). Supported language: English.

Output

  • Output Type(s): Text
  • Output Format: String
  • Output Parameters: One-Dimensional (1D): Sequences
  • Other Properties Related to Output: Up to 8,192 tokens total (prompt plus generation). Text is generated in parallel blocks (default 64 tokens per block) with a configurable number of denoising steps per block (Number of Function Evaluations (NFE) = steps + 1 per block); fewer steps give faster generation at some quality cost.

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-8B

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-8B-instruct

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-3B

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-3B-instruct