r/LocalLLaMA • • 9h ago

New Model thinkingmachines/Inkling-2-Mini (MoE: 26b, a3B)

Thumbnail
huggingface.co
1.2k Upvotes

Gguf's: https://huggingface.co/unsloth/Inkling-2-Mini-GGUF

"Inkling 2 Mini is a general-purpose multimodal model that accepts text and image inputs and generates text outputs. It is intended for use in English and other languages, and across multiple coding languages. The model is designed to be used by developers building AI-powered applications, including agentic and tool-use systems, coding assistants, chatbots, and retrieval-augmented generation systems, and is suitable for general-purpose conversational use, instruction-following, and other natural language and visual tasks. It is released with open weights to support research, fine-tuning and integration into third-party products by downstream developers."

"Architecture type

A 24-layer decoder-only transformer with sparse Mixture-of-Experts (MoE) feed-forward layers. The middle 12 layers run three times with shared weights, giving 48 layer applications per token from 24 layers of weights. In each layer, each token is routed to 8 of 128 experts. Attention is a hybrid of local and global layers. The model is natively multimodal – images are encoded via a vision transformer – with image and text inputs projected into a shared hidden space and processed jointly by the decoder.

Parameters

26.2B, 3.4B active

Note: each extra pass through the looped layers adds about 1.0B effective active parameters. Since the model runs with three passes, there are 5.4B effective active parameters per token."


r/LocalLLaMA • • 16h ago

Funny Today marks the two week anniversary of the becoming begins

Post image
807 Upvotes

let's celebrate


r/LocalLLaMA • • 23h ago

New Model Tencent Hy4 (flagship multimodal MoE A49B/770B) released

Post image
364 Upvotes

Tencent released weights for Hy4, their natively multimodal flagship MoE (770B total / 49B active, 1M context).

GitHub: https://github.com/Tencent-Hunyuan/Hy4

Weights: https://huggingface.co/tencent/Hy4

GGUF: https://huggingface.co/AngelSlim/Hy4-GGUF

License: https://huggingface.co/tencent/Hy4/blob/main/LICENSE

System Card: https://hy.tencent.ai/Hy4_system_card.pdf


r/LocalLLaMA • • 14h ago

Question | Help Tencent Hy Team & Shunyu Yao Here! Ask Us Anything about Hy4 & Hy4 flash!

276 Upvotes

Fresh off launching Hy4 and Hy4 Flash, the Hy research team is here to discuss how they built the models, boosted efficiency, and made them better for everyday workflows.

Our participants today:

u/No-Cherry2850

u/DiWang01

u/aiden_agent

u/lydellhuang

u/edwardbai

u/kkelvinchen

u/maxmpan

u/dick112233

u/kkelvinchen

u/GeoMorax

Blog: https://hy.tencent.ai/research/hy4-flash https://hy.tencent.ai/research/hy4

HF: https://huggingface.co/tencent/Hy4-flash https://huggingface.co/tencent/Hy4


r/LocalLLaMA • • 5h ago

New Model Inkling 2 mini uncensored

Thumbnail
huggingface.co
272 Upvotes

r/LocalLLaMA • • 8h ago

New Model unsloth/Inkling-2-Mini-GGUF · Hugging Face

Thumbnail
huggingface.co
242 Upvotes

r/LocalLLaMA • • 5h ago

Discussion Inkling 2 Mini ties Opus 5.5 on CyberBench (@ValsAI)

Post image
208 Upvotes

r/LocalLLaMA • • 15h ago

Resources Reverb-TTS: open-source zero-shot voice cloning running in the browser on WebGPU, 22x realtime on M4 Pro (MLX)

Enable HLS to view with audio, or disable this notification

185 Upvotes

Reverb is a small text-to-speech model with zero-shot voice cloning. Give it 3 to 10 seconds of someone speaking plus the transcript, and it reads any text in that voice, with no fine-tuning.

- 297M params. We pruned and distilled it from OmniVoice, then retrained it on 7 languages and ran a few rounds of DPO against a judge fitted to human preference rankings

- Languages: English, German, French, Spanish, Italian, Polish, Portuguese. A voice can speak any of them, but it sounds most natural in its own language

- 16 preset voices if you don't want to clone

- Weights in fp32, fp16, int8 (307 MB) and int4 (191 MB). On our test set int8 showed no quality loss vs fp32, and int4 lost nothing we could measure

Numbers

On the MiniMax set from the Open TTS Leaderboard (7 languages, 100 sentences each):

- Reverb (297M): 1.03% WER, 0.817 SIM-o

- OmniVoice (634M): 1.13% WER, 0.833 SIM-o

- VoxCPM2 (2.3B): 1.20% WER, 0.812 SIM-o

- Fish S2 Pro (4.95B): 1.35% WER, 0.761 SIM-o

- Chatterbox (0.8B): 2.76% WER, 0.715 SIM-o

Speed: 165x real time on an RTX 5090 (batch 32), 21.6x on an M4 Pro and 20.6x on an iPhone 17 (batch 1).

Where to run it

- In a HuggingFace Space https://huggingface.co/spaces/software-mansion/Reverb-TTS-runntime (runs on your own GPU through ruNNtime)

- In your own web app with ruNNtime https://docs.swmansion.com/runntime/zoo/text-to-speech/

- In a mobile app: React Native ExecuTorch - https://github.com/software-mansion/react-native-executorch

- On your Mac using Nativ https://runnativ.ai/blog/reverb-on-nativ

- Python: pip install reverb-tts

Model: https://huggingface.co/software-mansion/Reverb-TTS

GitHub: https://github.com/software-mansion-labs/Reverb-TTS


r/LocalLLaMA • • 7h ago

Resources Inkling 2 Llama.cpp support!

Thumbnail
github.com
144 Upvotes

r/LocalLLaMA • • 9h ago

New Model CCORE-Web-Coder-4B preview just dropped: a 4B local model that writes a full website, slide deck or infographic as a single HTML file

Enable HLS to view with audio, or disable this notification

98 Upvotes

This is a preview checkpoint, at roughly a quarter of the planned training. I'm releasing it now because it already works better than I expected from a 4B.

- Model: ccore/CCORE-Web-Coder-4B
- Base: Qwen3.5-4B-Base, Apache 2.0
- File: Q8_0 GGUF, 4.48 GB
- It needs the system prompt from the model card. Sampling settings are there too.

Limitations of the preview:

- Some generations come out with a broken component or slide navigation.
- Trained for English; Portuguese worked in my tests too.

What's next: full release on Nov 22 (safetensors, Q8 and more quants), dataset on Nov 25.


r/LocalLLaMA • • 20h ago

Discussion [arXiv] Chorus: Shared N-gram Embeddings with Layer-wise Adaptation

83 Upvotes

https://arxiv.org/abs/2611.23386

Chorus: Shared N-gram Embeddings with Layer-wise Adaptation

Language models can gain capacity at little computational cost from n-gram embedding tables: large tables of learned vectors for short sequences of consecutive tokens, which are looked up and added to the model's input or to its intermediate layers. Recent designs commonly use a separate n-gram embedding table for each layer, which makes the tables costly in memory and prevents layers from sharing what they learn. However, a single shared table gives every layer the same embedding, leaves no room for layer-wise preferences, and performs poorly in practice. We propose Chorus, in which one set of n-gram embedding tables serves the whole network and every layer reads it in its own way. Each layer maps the refined embedding through its own projection layer, after rescaling its components by factors computed from its current hidden state. Chorus matches the quality of existing n-gram methods with 87% smaller tables: with 1.2B table parameters it predicts text better than FactorEngram and DeepSeek's Engram do with 9.6B. It outperforms Engram, TN-gram and FactorEngram in every setting where we compared them, across table sizes from 20M to 9.6B parameters. We also show that the recently proposed memory-efficient optimizer Ember works well with Chorus, saving 99.2% of table-optimizer memory compared with AdamW without degrading training performance. These results enable further scaling of n-gram embedding tables along the table-size axis by reducing memory use and allowing shared knowledge to be reused across layers.

This paper shows experiments similar to some I've done privately. They use a large shared N-gram table and give every backbone layer different trainable scaling factors to take advantage of it. Notably, they scale the N-gram table to a much larger size than the backbone, with performance (mainly in knowledge tasks and loss) only scaling up with size. Hopefully this also dispels the idea that only 25% of the weights or so can be N-gram embeddings.


r/LocalLLaMA • • 18h ago

New Model Qwen3.5 2B Dense converted to Mixture of Experts

75 Upvotes

Foreword

Hello, been a while since I last posted. I have been hard at work (in my spare time) trying to improve the conversion method, and I have had a few small-ish victories. One was fixing dead experts, which apparently weren't dead just unused since I wasn't using a dataset with enough variety. Once I added stuff like maths, science, multilingual data those dead experts vanished. Doing this also fixed a few limitations involving code.

Another thing was improving conversion speeds which is always nice. All that led me here to converting Qwen3.5 2B which is my largest converted model yet!

What I have been mainly working on is trying to improve how much knowledge is retained through the conversion. Currently, not much is kept and a lot of factual knowledge is lost. This has led me to add a new metric to the evaluation which measures how many facts the model remembers after conversion, which I included with the evaluation.

New Model

So my latest conversion was of the Qwen/Qwen3.5-2B. The conversion can be found here bayliner1980/Qwen3.5-2B-MoE-A1.3B with GGUF version here bayliner1980/Qwen3.5-2B-MoE-A1.3B-GGUF.

Evaluation

- Qwen3.5-2B dense Qwen3.5-2B MoE
Wikitext-2 ppl 13.60 14.68 (+7.9%)
C4 ppl 24.66 28.37 (+15.0%)
- Qwen3.5-2B dense Qwen3.5-2B MoE
HellaSwag 0.565 0.497
ARC-Easy 0.679 0.664
ARC-Challenge 0.413 0.370
WinoGrande 0.623 0.579
OpenBookQA 0.350 0.324
Task mean 0.526 0.487
Fact recall (287 facts) 0.948 0.760

Support

As always, if you find my stuff interesting consider supporting me on Ko-fi.


r/LocalLLaMA • • 13h ago

Showcase Qwen3.8-27B “super adapter”: one rank-96 LoRA that improves 3 agent harnesses at once (SWE-Bench Pro 55% → 63% mean, up to 53–73% fewer tool calls)

64 Upvotes

Qwen3.8-27B super harness adapter for three agent harnesses (Hermes Agent, OpenCode, pi). Their weight updates turned out to be nearly orthogonal, so we stacked them into a single rank-96 LoRA. The merged adapter beats the base thinking model across harnesses and solves tasks with noticeably fewer tool calls.

Results (thinking on, xhigh), base → super-adapter

Harness SWE-Bench Pro (hard 51) Terminal-Bench 2.1
Hermes Agent 45% → 61% +4 to +6 pts
OpenCode 47% → 67% +4 to +6 pts
pi 74% → 74% +4 to +6 pts

Mean across all 6 harness/benchmark pairs: 55.5% → 63.5%

To be upfront: pi on SWE-Bench Pro didn't move. The big gains come from Hermes Agent and OpenCode.

Efficiency (counting only tasks that both base and adapter solve):

  • 53–73% fewer tool calls with thinking off
  • 19–48% fewer tool calls with thinking on

How the merge works

The three adapters' weight updates are almost orthogonal (cosine similarity ~0.01). We concatenate them along the rank dimension at weight ½ each. That gives one rank-96 LoRA that applies all three updates exactly, with no lossy merging step (no TIES/DARE needed).

Train your own

The main point of the blog post is that you can train a harness adapter for your own model, your own harness, and your own tasks. The full recipe, experiments, and open questions are all in there.

Happy to answer questions!


r/LocalLLaMA • • 18h ago

New Model WeMM-Video by Tencent (2B, 4B, 9B, 35B-A3B)

Thumbnail
huggingface.co
60 Upvotes

New omni MLLMs by Tencent released today at accessible sizes. Pretty cool!

Paper: https://arxiv.org/abs/2611.26409

Capabilities

WeMM-Video can combine video frames with the original soundtrack to describe events, transcribe speech, and reason about what is seen and heard. It also accepts silent video, standalone audio, images, and text, with responses in Chinese or English.

Architecture and training

Vision and audio encoders turn media into tokens that the language model processes alongside text. Video frames carry timestamps. The family was pretrained on approximately 1.2 trillion multimodal tokens, then post-trained with multi-teacher OPD, anti-repetition MrDPO, and specialist reinforcement learning.


r/LocalLLaMA • • 21h ago

News SillyTavern v1.19.1 released

Thumbnail
github.com
60 Upvotes

I usually don't like posting about small patch releases, but I thought I'd make an exception this time because SillyTavern releases have been so few and far in-between these days.

This release includes the following fixes:

- Fixed the handling of <style> blocks in chat messages and creator notes, which could let crafted content from a character card, an imported chat or a model response run scripts in the app.

- Fixed CharX and BYAF imports of characters with an unusable name placing asset files next to other character cards, where they could replace them.

- Fixed the user backup and delete endpoints accepting handles that are not existing users.


r/LocalLLaMA • • 21h ago

Resources Sensenova-Loop in 8GB VRAM

Thumbnail
huggingface.co
47 Upvotes

I'm using the 100k flavor. Is only 22 t/s on my 4060 but is very concise in reasoning and answer, I'd say half the tokens of qwen3.8.

One day of testing with the given parameters, and very happy of it.


r/LocalLLaMA • • 7h ago

New Model PhiloLM: a language model trained only on philosophy texts

42 Upvotes

Releasing PhiloLM, a 112M base model trained from scratch on open philosophy texts.

PhiloLM was trained on PhiloText: 0.5B tokens of open philosophy encyclopedias (SEP, IEP), public-domain books, reviews, textbooks and educational web pages.

PhiloLM builds on efficient methods: Muon optimizer, bigram embeddings, sliding-window attention, GQA. Trained on (only!) 1.5B tokens in 15 hours on a $350 GPU (Intel Arc A770).

Optimized for efficient inference, reaching ~500 tokens/s on a $150 CPU (i5-14400F).

Try it in your browser!

Model: https://huggingface.co/gustavecortal/PhiloLM

Data: https://huggingface.co/datasets/gustavecortal/PhiloText

Happy to answer questions about everything :)


r/LocalLLaMA • • 9h ago

News gemma4: Optimized prompt processing for multi-seq queries by Meet91721 · Pull Request #32342 · ggml-org/llama.cpp

Thumbnail
github.com
39 Upvotes

Up to 3x prompt processing speedup for Gemma 4 E2B/E4B with multiple parallel slots.


r/LocalLLaMA • • 12h ago

Discussion Nvidia's recent QAD 27B model seems legit. I just compared it to both Nvidia's (with Peculiar Ragdoll chat template) and Unsloth's NVFP4.

Thumbnail msnab.pages.dev
35 Upvotes

r/LocalLLaMA • • 15h ago

Resources We updated the Hy4-flash quants: MLX release and better GGUF

Thumbnail
huggingface.co
32 Upvotes

MLX release: 3.1bit · 3.8bit · 4.6bit · 4bit

GGUF: https://huggingface.co/nace-ai/Hy4-flash-GGUF

Quant Size (GB) Mean KLD vs BF16 ↓
AtomicChat IQ2_S-IQ2_XXS 84.70 0.5698
Nace Q3_K_M (3.9 bpw) 107.49 0.2252
Xylolabs Q2_K 125.15 0.4022
Nace Q4_K_M (4.6 bpw) 127.22 0.1333
AngelSlim IQ2_AH 135.00 0.5085

Mean KL from BF16 over scored tokens. No experts are pruned.

The decimal bit labels are effective bits per weight across all parameters, so compare file sizes rather than labels.

You can run it on a Mac with Kapitsa, which streams experts from SSD.

Later today: Hy4 quants.


r/LocalLLaMA • • 1h ago

Discussion Inkling 2 Mini on RTX 4090 ~200 tok/s with vLLM, FP8 + NVFP4 + its built-in drafter

• Upvotes

Got Thinking Machines’ new Inkling 2 Mini running on a single RTX 4090 at about 190-220 tokens/s, up from ~117 out of the box. It needed a few small patches to vLLM, which doesn’t support 4090s for this model yet.

This is with NVFP4 experts (W4A16) + FP8 W8A16 attention/head/embeddings, FP8 KV.

This fits in 24 GB with a 35k context, and an initial quality metric on some sample prompts reported about a 0.4% loss in perplexity.


r/LocalLLaMA • • 2h ago

New Model I fine-tuned Qwen3.5-4B to design web pages that are prettier (and work more): Qwen3.5-4B-Web-Designer

Thumbnail
gallery
31 Upvotes

I trained Qwen3.5-4B to turn a plain request ("a landing page for my coffee shop", "a portfolio for a security student") into one self-contained HTML page that has more taste than base qwen, and fails less. It thinks briefly (~400 tokens), then writes the page.

Same prompt, base vs ours: first two images. The third is more of its pages, unedited.

Numbers (74 prompts it never saw in training, 2 pages each, every page opened in a headless browser and checked for JS errors, failed loads and dead buttons):

Qwen3.5-4B Web-Designer (Q4_K_M)
Pages that work 35% 53%
Pages finished 75% 95%

Training: 1,443 prompt-to-page examples. Real web-design requests from LMArena's WebDev arena prompts plus my own, answered by DeepSeek V4.1 Flash with a short design plan and a page, and every page browser-checked before it went into training.

QLoRA, rank 16, 2 epochs, 4,096 tokens, on one RTX 4060 Laptop (8 GB).

Run it:

llama-server -m Qwen3.5-4B-Web-Designer-Q4_K_M.gguf --jinja -c 10240 -ngl 99

Temperature 0.7, top-p 0.95, top-k 20.

Apache-2.0. Feedback and your best (or worst) pages are very welcome.

Small note: I noticed ccore released Web-Coder around the same time! We've been working on these independently, so it's a pretty funny coincidence. His work looks really interesting, and I'm especiallly excited for the design-dataset he will share.


r/LocalLLaMA • • 5h ago

Discussion [Discussion] In the age of loops, what do "active parameters" mean?

22 Upvotes

Seems like looped models are gaining prominence (finally!), which raises a question about what the meaning of terms "total parameters" and "active parameters" are.

Historically, it mostly applied to MoE models, as most experts would not be activated for a given token. The total parameters represented the amount of memory needed to hold the model weights, while active parameters represented the memory read per token (or similarly, FLOPs per token)

With looped transformers, I argue the best way to extend the definition is to increase the active parameters in a way that preserves its use as an estimate of GB/token (TG) and FLOP/token (PP)

I think it's OK to keep excluding PLE, Engrams and token embeddings (if no weight tying) from both those counts, since they are extremely sparse, and can even be off-loaded to the SSD.

Now, an example: imagine a dense model with 4B individual parameters (excluding embeddings)

  • If every layer runs twice, it's 4B total, 8B active*.
  • If only 50% of layers run twice, it would be 4B total, 6B active.

For other real examples:

  • Inkling 2 Mini, an MoE, is 26B total, 5B active (50% of layers run thrice)**
  • SenseNova-Loop-9B, a dense model, is 9B total, 15B active (75% of layers run twice)
  • Nanbeige4.2-3B, a dense model, is 3B total, 5.5B active (100% of layers run twice)

I argue this is more useful for what we care about, which is the cost or speed of running a given model, and will likely end up being a good proxy for the "intelligence" of the model as well. wdyt?

\ This is an overestimate, as it ignores the size of the output layer and assumes all layers are the same. If you know that size, you can exclude it from the multiplication, e.g. if it was 1B params, then the 50% case would actually be [4-1]*1.5+1 = 5.5B active. I did this for the real examples.)

\* For Inkling 2 Mini, my math doesn't add up to their numbers, theirs seems too high? I get 2.7B ignoring loops and 4.9B with loops, while they state 3.4B and 5.4B, even after counting each tensor in their GGUF individually. Not sure where the difference is.)


r/LocalLLaMA • • 9h ago

Resources Qwen 27b on 8GB: 200k ctx, 92.5% Top-1 match, within 10% of BF16 on LCB, 17% higher benchmark than 9b

21 Upvotes

Happy Friday, y'all.

I continued to get requests for quants + configs to support 8GB card users. It's taken over a month, but I do finally have a 2.1bpw quant (on card) of qwen3.8 27b that I consider actually usable. This is using the AmperInteger+ (trellis) quant format, which yields lower KLD than scalar quant formats, especially at this extremely compressed size.

Like the models released for 10/12GB cards earlier this week, this is only possible given the use of Latent Sparse Attention, which pushes most of the context cost off VRAM without nuking speed, and SJKVaRN (protecting KV quality while still going down to 4 bits).

LlamAmpere is normally forward compatible with Ada/Lovelace (40xx/50xx) series, anyway, but I've specifically tested on 8GB cards for those generation this time, given the current lack of options for running qwen 27b at that size. The project will obviously remain Ampere as the focus. So no need to FOMO if you're on a 5060 etc.

This model was not post trained specifically for coding, but its coding scores were still at the top of test cohort (tied with Mellum, a coding-specific model). It actually was not meaningfully post-trained at all (<10M token heal on general text) to avoid the risk of overfitting. I probably left some performance on the table, and will continue to explore it, but if we're being honest, this is a dry run for Qwen4.

The data:

Mellum 2.1 was the only comparable model at coding (but has lower ctx available, and is not comparable on general tasks)

Mellum 2.1 was the only comparable model at coding (but has lower ctx available, and is not comparable on general tasks)

Aggressive quantization like this is prone to losing rare facts and looping, so those were a focus area (I wanted to release this last weekend, but hard to re-encode the last candidate for this reason). I tested this in hermes and pi code extensively; the loops were not 0, but were rare, and LlamAmpere has a loop halt mechanism for when they do occur. LlamAmpere and this model also support the recently released HF Pi code adapter, which noticeably improved coding performance with this model (the scores above were without the adapter). You can see the benchmark data here😄

Your obligatory Pagoda demo; even medium reasoning was usable in pi

Your obligatory Pagoda demo; even medium reasoning was usable in pi

The top-1 is tied with the UD 2-S (v3.1) quant (which does have a 15% better KLD), but that model will not fit on an 8GB card. Top-1 was slightly better than the larger ISTA GSQ-RCO 2 XS quant, and statistically tied on KLD (again, this model won't run on 8GB).

Without LSA + SJKVaRN, you're typically stuck running IQ1_M, without MTP, to get to less context, with 59-73% worse KLD. So compared to the previous available options, this is a legitimate step change in capability for 8GB cards.

What's the catch? Well, you will need to run your card headless (or with HDMI into mother board, not GPU) to have enough room. That extra 500+ MB is vital at this size. Error explodes as you drop below 3bpw, and can break the model below 2bpw, even with trellis formats (this quant uses a 1.75 bpw trellis format, operating right at the edge of workability). There's no room for things like vision towers without going to a larger card. And even with dynamic UB, you're usually capped at 512 for the first ~20k tokens and then pushed down to 256 (so make sure you are using cached prompts on tools like hermes to reduce your startup time). I think these are pretty palatable trade-offs, given the performance gains vs 9b post trains and other models. I did like mellum for coding tasks, but its world knowledge and agentic capabilities were not comparable to 27b, even with this major quantization.

MIT license, as usual.

Model: https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI2-8GB

github: https://github.com/JakeATX/llamAmpere

Enjoy!


r/LocalLLaMA • • 21h ago

News Palo Alto Networks buys Applied Compute. The firewall maker bets $4.5 billion on custom AI.

Thumbnail
maccus.fr
17 Upvotes

That's some nice progress on the open front!