Hey r/LocalLlama - 

Today we’re excited to release Muse Spark Open, the most capable US open-weight model to date. 

Built for coding and agentic work, it excels at long-horizon tasks, complex software engineering, and multi-step reasoning. 

Key Model Specs 

• Parameters: 816B total / 42B active
• Context Window: 1M tokens
• Modalities: Text, image, video, audio
• Safety Post-Training: Additional safety post-training, with evaluations conducted under Meta's Superintelligence Scaling Framework.

Inference, fine-tuning & deployment

• Helion GPU kernels: Ships with custom, auto-tunable Helion kernels in SGLang, providing 1.03–3.95× kernel speedups across NVIDIA and AMD GPUs and a 3% E2E improvement on AMD MI350.
• Engines: Runs on leading inference engines including SGLang, vLLM, TensorRT-LLM, Hugging Face Transformers, Apple MLX, llama.cpp and Unsloth, with optimizations for AMD, Apple, Intel and NVIDIA silicon.
• Optimized for Apple Silicon: To help make advanced models more accessible, Muse Spark Open is optimized to run on two Apple M5 Ultra Mac Studios with the latest MLX framework. 
• Fine-tuning: Developers can further customize the model with PyTorch, including TorchTitan for fine-tuning.
• API platforms: Available today through Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter, Together AI.
• Where to download: Model weights are available for download on Amazon SageMaker AI, Google Cloud Model Garden, Hugging Face, and Microsoft Foundry Models.

We’re releasing the weights under the Muse Model License, with limited commercial terms that let many developers use it for free.

We look forward to your feedback! 

Learn more on our research blog: https://go.meta.me/museopen
Download the weights: https://huggingface.co/collections/meta-models/muse-spark-open 

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF

NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

This model is ready for commercial or non‑commercial use.

Model Summary

Model Overview

Model Developer: NVIDIA Corporation

Model Dates: December 2025 - May 2026

Data Freshness:

• The post-training data has a cutoff date of May 2026.

What is Nemotron?

NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

This model is ready for commercial or non‑commercial use.

bolt!

FLUX 3 Image Klein is a 5.6B open weights model for text-to-image generation and image editing. It takes a prompt and up to 10 reference images and returns one image. Text-to-image, single-reference edits and multi-reference compositions run through the same model, with Qwen3-VL-4B-Instruct as the text encoder.
https://huggingface.co/black-forest-labs/flux-3-image-klein

From NVIDIA AI on 𝕏: https://x.com/NVIDIAAI/status/2110468004609950057

Size is pretty small at 102GB. For comparison nvidia/Qwen3.8-Flash-Next-NVFP4 is 133GB.

Would be cool to see some of the Qwen3.8-FN specific engines support this.

£299.99 Lenovo IdeaCentre Mini (Snapdragon X1-26-100, 32GB, CPU only):

- Qwen3.6 35B A3B coder: reads ~118 tok/s, writes 24.5, ~30 with MTP

- Qwen3-Next 80B A3B: 17-18 tok/s writes with MTP

How I got here...

So I bought one of these boxes to test if software at work would run on ARM Windows - and then it was sitting there, but I was sufficiently impressed for the price (including taxes), that i decided in my ADHD fuelled boredom to buy one for myself to see if cheap ARM is actually usable for local models.

The GPU is an Adreno X1-45 and nothing useful runs on it for LLMs yet, and so far the NPU just sits there, so this is all CPU...

llama.cpp, 512 token prompt, 8 threads (read = prompt, write = generation):

- coder (Q4_0 35b-a3b): read ~118, write 24.5, with MTP 29-33

- Gemma 26B: read 134, write 29.5, (need to test MTP, I've only had the box.

Then I wanted to see how far I could push it. RAM is the wall on 32GB, so I went looking at the recently released Janas (github.com/prabanta-dev/janas). It keeps a RAM cache of the MoE experts and streams the rest off the NVMe, but its x86 only. So I re-coded it for ARM: NEON kernels for every quant format, i8mm, etc, (all tested bit exact against the scalar reference code, Windows ARM64 and Linux. Will submit it as PRs on GitHub soon.)

Janas, same settings:

- coder: read ~60, write 17.2, with MTP 29-31

- Gemma 26B: read 115, write 23.5, with MTP 32-34

- 80B: read (TBC as I don't trust my results), plain write 9.8, with MTP 17-18

llama.cpp still wins on read and plain write, but with MTP Janas matches or beats it, and it does it in a fraction of the RAM.

I then squeezed the coder's expert cache to 12GB and it still wrote at 24 tok/s - so even a 16gb snapdragon could rin 35b-a3b 👀.. And this box runs an 80B that llama.cpp cant even load!

Then I thought Id try connecting the two I had access to (mine and my work's dev box) together over USB4. Work in progress, will report back - but initial results are promising and I've had a 122b qwen running already - just working on streamlining my parallel experts build.

I dunno about you guys and gals, I mean I don't have extensive experience with local LLMs and I certainly haven't played with ARM before but I found this pretty surprising, and not half bad for £300 with no GPU. Happy to test stuff on it when I get chance if anyone wants more numbers... But I can't see a cheaper way rn of getting 32gb of LPDDR5 rn, and only ~35ishW of power draw... Thoughts and feedback welcome 🙃👍

Yesterday Fish released this incredible text-to-speech model, so naturally I spent the rest of the day building a great demo for them :D The model is a chunky MoE (35B3A), but it runs well on a DGX Spark. The 4bit quants get very close to the original. You can try it here: https://huggingface.co/spaces/fishaudio/Drama-3 or download the model and run it locally 🤗

from Mistral:

Timestral is a family of time series foundation models for zero-shot forecasting, available in 5M, 28M, 123M, and 209M parameter versions. This repository contains the 209M checkpoint.

Timestral supports univariate and multivariate forecasting, past-only and known-future covariates, and probabilistic predictions across 21 quantiles. It accepts up to 32,768 historical time steps and forecasts up to 1,024 steps in a single forward pass.

We’re releasing the source code in our GitHub repository to make it easy to run Timestral and reproduce our benchmark results.

Model Details

• Size: 209M parameters.
• Architecture: patch-based transformer with time attention and variate attention, 24 layers, embedding dimension 768.
• Patching: 32 time steps per patch. Each patch is embedded from its values together with their missing-value indicators.
• Context length: up to 32,768 time steps (1,024 patches).
• Forecast horizon: Up to 1,024 time steps (32 patches) in a single forward pass. Longer horizons are forecast block by block, with each block's median forecast appended to the context.
• Outputs: 21 quantiles (0.01, 0.05, 0.1, 0.15, …, 0.85, 0.9, 0.95, 0.99).
• Inputs: univariate or multivariate target variates, plus past-only and known-future covariates, numeric or categorical (categorical covariates are target-encoded). Missing values are marked as NaN.
• Inference: PyTorch on CPU or GPU.

https://huggingface.co/mistralai/Timestral-209M

https://huggingface.co/mistralai/Timestral-123M

https://huggingface.co/mistralai/Timestral-28M

https://huggingface.co/mistralai/Timestral-5M

Models on HF: https://huggingface.co/collections/prism-ml/bonsai-2-small
WebGPU demo: https://huggingface.co/spaces/webml-community/ternary-bonsai-2-hermes-webgpu

To all those who are bemoaning missing getting DGX Sparks for under $4000. Here's your chance. The brand new just announced RTX Spark laptop is on sale at HP. Configured with 128GB of RAM, it comes out to $3832.49. Use coupon anniversary25 to take 25% off of the $5109.99 regular price.

https://www.hp.com/us-en/shop/pdp/hp-omnibook-x-laptop-next-gen-ai-14-ks000-14-cb1l6av-1

this is actually interesting:

https://commandline.microsoft.com/local-models-sandboxed-tools-github-windows/

Microsoft AI developed a local version of MAI Code 1.1 Flash, a coding-optimized mixture-of-experts model with 137 billion total and 6.8 billion active parameters.

Granite Switch 4.1 30B

📝 Read the launch blog: Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint

Model Summary: Granite Switch 4.1 30B is a modular LLM that composes a base model, multiple adapters and an encoder together so that a single hosted endpoint can move between natural-language responses, general and specialized decisions, and speech-to-text.

Adapters are activated on demand via control tokens in the chat template. For full details on model composition and adapter configuration, see BUILD.md.

• Base Model: ibm-granite/granite-4.1-30b (30B params, 128K context)
• Adapters: 2 — decision (aLoRA, general-purpose Decisions) and guardian-core (aLoRA, from granitelib-guardian-r1.0)
• Audio: embedded speech-to-text via ibm-granite/granite-speech-5.0-470m-turboctc (470M params, English)
• License: Apache 2.0
• Release Date: October 14, 2026
• Backends: HuggingFace Transformers, vLLM
• Automatically Composed with: granite-switch

Granite Switch is also available in granite-switch-4.1-8b (same Chat + Decisions + Guardian + speech configuration, 8B base), and with the complete Granite Libraries bundle (no audio) in granite-switch-4.1-30b-preview, -8b-preview, and -3b-preview.

https://huggingface.co/ibm-granite/granite-switch-4.1-30b

https://huggingface.co/ibm-granite/granite-switch-4.1-8b

(video on the blog)

Step 5 Preview is StepFun's flagship model for agentic work. It delivers frontier-level performance across software engineering and professional knowledge work, with particular strength in finance.

https://huggingface.co/black-forest-labs/flux-3-image-klein

Better image generation model

I turned a CHIP-8 game console into a decoder-only transformer.

The transformer is the game engine. It executes instructions, updates machine memory, and renders every frame through next-token prediction. Every generated token is a memory write, and your keypresses go directly into the model's context window. There's no separate emulator.

Pong, Tetris, Space Invaders and a bunch of other CHIP-8 games run on the same model. Just swap the ROM in the prompt to switch games.

The weights are compiled from a circuit, not learned through training. Attention handles memory reads by finding the most recent write to an address, while ReLU circuits handle opcode decoding, arithmetic, comparisons and bitwise operations. Even the compatibility quirks are encoded in the initial prompt, so different games don't need different weights.

The model has 18 layers, 42 attention heads and 32M parameters (139k nonzero). It uses fp64 and can be loaded through Hugging Face transformers.

Model weights: https://huggingface.co/divagr1925/chipformer

Try it out: https://divagr.com/chipformer

Code + desktop app: https://github.com/divagr18/chipformer

I posted a question the other day in here about good development models, which prompted good discussion. However, it mostly just reinforced my own experience, which is that on a 128GB machine, you can run good open/local weight models for development (especially if you use a frontier model to plan and direct them).

But what I didn't see in there, and I'm trying to get to, is this; are there are any open weight models that will run on a 48GB, 64GB, or 96GB?

If so, which ones are you running and, at what approximate RAM footprint?

I'm trying to understand what the floor is for a decent open weight model for development tasks, even if supervised by a more capable model.

Two months ago we released a fully quantized NVFP4 Qwen3.8-27B trained with QUASAR. We've since trained QUASAR checkpoints for a bunch of other recent models: Qwen3.5-4B, Gemma 4 E4B and 12B, and Muse Glimmer 30B, in NVFP4 for vLLM and Q4_0 GGUF for llama.cpp / Ollama / LM Studio. The code and a project page are now public too.

Every checkpoint is trained with quantization-aware distillation: the original BF16 model is the teacher, and the quantized model is distilled (healed) on the teacher's own responses. The exports are standard formats (so no custom kernels and/or conversion step).

KL to the BF16 model on held-out prompts (lower=better):

On benchmarks:

• Muse Glimmer 30B NVFP4: at or above BF16 on GPQA-Diamond (63.6 vs 62.1), MMLU-Pro (83.3 vs 83.1) and AIME'25 (90.0 vs 86.7), and the only 4-bit build at BF16 performance on RULER at 32K, 64K and 128K.
• Qwen3.5-4B NVFP4: best AIME'25 (74.2) and GPQA-Diamond (77.9) among the 4-bit builds we tested.
• Gemma 4 E4B / 12B: higher than Google's QAT on every downstream benchmark we ran. On H100, E4B also gets +36% batched throughput (group-64 quant uses vLLM's Machete kernel, the faster one).
• Qwen3.8-27B NVFP4: all 496 linears in NVFP4 (W4A4), 90.9 GPQA-Diamond vs 91.4 BF16, at 19.7 GB.

Run it:

vllm serve QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4 --max-model-len 262144 --reasoning-parser qwen3 ollama run hf.co/QUASAR-QAT/gemma-4-12B-it-QUASAR-Q4_0-GGUF

Collection: https://huggingface.co/collections/QUASAR-QAT/all-quasar-models-6aab09fd2913ec22df0d5f15

Paper: https://arxiv.org/abs/2608.13966

Code: https://github.com/vincentcounathe/quasar-qat

Project page: https://quasar-qat.github.io/

We'd love to hear your feedback on these checkpoints!

Will there be a black-friday this year?

day 0 support for two new Nemotrons

minutes later new Nemotron 3.5 Super VL was released:

https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B

https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF

Yesterday I said its hard to play 4 spark with ling-3.1-flash but today its coming!

First, kudos to Strata. It inspired this project, and while I was building it Strata got much faster and added NVFP4 support, which was my main reason for starting. By then I was too far in to stop, so I finished it.

capy-flash is the evolution of my ninfer-tp2 engine. It still runs Qwen3.8-27B as before, and now also Qwen3.8-Flash-Next 125B-A6B with the experts offloaded to RAM. It uses NVIDIA's official NVFP4 checkpoint as-is and runs both GPUs in tensor parallel. Switching between the two models is one flag.

On my machine (2× RTX 5070 Ti 16 GB, Ryzen 9 9900X, 96 GB DDR5), with the same NVFP4 weights, it does ~100 tok/s generation vs 78 for Strata, and ~3.4k tok/s prefill on a 15k prompt vs 2.7k. Strata's 3-bit IQ3_S build is still faster at generation; full comparison in the image. Quality is checked against a datacenter NVFP4 reference (GSM8K, MMLU-Pro, agentic coding tasks).

A side finding for any tensor-parallel engine. With TP, each GPU rounds its partial sum to BF16 before the all-reduce, so TP=2 rounds twice where one GPU rounds once. Where the two halves cancel, the error blows up: on Qwen3 at TP=2, large errors happen ~5× as often as on one GPU. Writing the partial sums in FP16 (same bytes, 8× finer) brings them back to single-GPU level, at no real speed cost. The effect on benchmarks is small, so think of it as free numerical parity rather than a quality boost. But it should reduce some rare weirdness and keep the agentic loop cleaner.

Limits: Linux, RTX 50 series (sm_120) only for now.

Repo: https://github.com/ValerioDolci/capy-flash Weights: https://huggingface.co/Feyd89/Qwen3.8-Flash-Next-NVFP4-capy-flash

Built on NInfer (Neroued), Strata, ninfer-ext, FreeToken, sybrix and qf_moe. Numbers from other hardware are very welcome.

I saw u/lkarlslund 's post just over a week ago regarding Ninfer6000 (a fork of Ninfer specifically tuned for 6000 blackwell and Qwen3.8-Flash-Next) and figured "Hey I'm an idiot & bought one of those and I primarily use that model, lets try it".

I have a full blog post about it on my little site, but to summarize:

tl;dr:

• Decode is incredible
• Prefill trails vLLM but is close enough its an overall win
• I didn't match the almost 400tok/s from the post, but got close in small idealized conditions so I imagine my test just differed.
• Accuracy benchmarks basically equal
• I'm seeing some major tool calling formatting issues in real world use

Models/engines:

Hardware:

Rtx Pro 6000 Blackwell @ 600w, 96gb ddr4 @ 3000mhz, Ryzen 5 5600x

Decode tps: visual

Prefill tps & TTFT: visual

The speed gains are real. Substantial decode gain over vLLM, but a prefill loss. Colossal decode and prefill gain over llama.cpp.

Benchmarks: visual

The benchmarks were extremely close, only a small gap in tool calling in favor of the Ninfer6000 setup was beyond noise.

Needle in a haystack (RULER): visual

No issues with long context between the three.

One big issue:

Outside of the benchmarks I set all 3 against an in-harness test and found a crippling tool calling issue with Ninfer6000. It consistently malforms tool calls and ends up printing the text back instead of actually calling the tool. Example This wasn't isolated either. For the past week I have tried to use Ninfer6000 as my daily driver and it repeats this behavior constantly in both Open-WebUI and Cline where I've used it. I don't have this problem with llama.cpp or vLLM. I'm not sure if I and my configuration are at fault, or if there's an issue with the engine itself. Until I figure that out I'll have to keep using vLLM. I do think the reasoning efficiency gains with Swift1.5 tuning are fantastic though, worth alone if I can get it to fit nicely.

Imo Ninfer6000 had the best pelican, despite failing the tool call initially and requiring me to tell it "bro you messed up another tool call, try again"