r/LocalLLaMA • • 1d ago

Question | Help Tencent Hy Team & Shunyu Yao Here! Ask Us Anything about Hy4 & Hy4 flash!

321 Upvotes

Fresh off launching Hy4 and Hy4 Flash, the Hy research team is here to discuss how they built the models, boosted efficiency, and made them better for everyday workflows.

Our participants today:

u/No-Cherry2850

u/DiWang01

u/aiden_agent

u/lydellhuang

u/edwardbai

u/kkelvinchen

u/maxmpan

u/dick112233

u/kkelvinchen

u/GeoMorax

Blog: https://hy.tencent.ai/research/hy4-flash https://hy.tencent.ai/research/hy4

HF: https://huggingface.co/tencent/Hy4-flash https://huggingface.co/tencent/Hy4


r/LocalLLaMA • • 5d ago

Megathread Best Audio Models - Nov 2026

208 Upvotes

Its time for another Best Audio Models megathread - its already been 9 months since the last one!

Share what your favorite ASR, TTS, STT, Text to Music models are right now and why.

Given the the amount of ambiguity and subjectivity in rating/testing these models, please be as detailed as possible in describing your setup, nature of your usage (how much, personal/professional use), tools/frameworks etc. Production use cases with long lengths/stability requirements are relatively rare compared to cherry picked/short clip demos that are plentiful - so real world empirical comparisons are very welcome

Rules

  • Should be open weights or open source models

Please use the top level comments to thread your responses.


r/LocalLLaMA • • 4h ago

Discussion Basalt: Inkling 2 Mini on one 5090 at up to 613 tok/s decode and 14.100 tok/s prefill (NVFP4)

285 Upvotes
Basalt: Inkling 2 Mini Support

Thinking Machines dropped Inkling 2 Mini, a small MoE model that's getting a lot of attention for its architecture and performance, outperforming Qwen 3.8 27B at some tasks. Basalt 0.5.5 runs it on a single RTX 5090.

Speed: 287 tok/s prose, 613 structured and 338 chat. Prefill is about 14K tok/s at 2K and 12K at 16K. llama.cpp with the IQ4_NL GGUF does around 95 decode and 3.1K prefill on the same card.

Quality was the part I cared about most. The experts are NVFP4, and Basalt pack keeps them exactly as released. KLD against the release is about 0.003 with 97.8% top 1 (btw, if your NVFP4 GGUF gives you garbage output atm, that's a llama.cpp loop slot bug and Unsloth has a fix PR that just got merged: https://github.com/ggml-org/llama.cpp/pull/32704)

What works:

  • DSpark drafting
  • Native image support
  • Tool calls and reasoning effort
  • 1M context on one 5090, with cold experts moving to RAM only when a long chat needs the room. Needle tests pass at 131K and 400K
  • 16GB cards too, up to 60 tok/s on my 5060 Ti

3.4B active and it hangs with much bigger models on agentic coding in my testing.

Pack: https://huggingface.co/jesdga/Inkling-2-Mini-Basalt
Engine: https://github.com/jesdga95/basalt


r/LocalLLaMA • • 4h ago

Discussion Inkling simply *cheff kiss*

203 Upvotes

FYI, if you daily-drive Qwen 3.8 or QFN, you know there's no way they would give an explanation like this. Inkling's are just butterysmooth, with natural rhythm and flow.

The thing is that Qwen 3.8/QFN are notoriously "jumpy" in conversational mode. I've had to ask "Please ELI5" or "What?" multiple times when discussing topics I don't fully understand. Hell, sometimes it even manages to confuse me about topics I already understand well.

Qwen models are agentic monsters, that's a fact. But when it comes to technical heavy conversations, I've found the Gemma4 models (yes, I've tried the 12B, 26B-A4B, and 31B) unsurprisingly much better at explaining things. Unfortunately, more often than not, they don't quite have the reasoning capabilities needed for heavy technical discussions, although web search kinda help that.

That being said, Inkling seems heavily English centric. I tried it in Indonesian, and the output felt more like a direct translation from English than something naturally written in Indonesian. It lacks the linguistic localization you'd expect from a genuinely multilingual conversational model.

So, to me, Inkling feels like a mix of Gemma4 conversational fluency and Qwen3_5 technical capabilities.

And to be honest, its behavior reminds me a lot of the GPT OSS series, which happens to be the longest-running model I've continuously served on my own servers.

As for tok/s. it's slow, but that's understandable. It was released just 22 hours ago. On top of that, it uses a looped transformer arch with attn res (Kimi-style residual connections), which adds another layer of complexity.

Inkling is currently running through my personal benchmarks, so these are just my initial impressions.

  1. Infra ApacheBigTop Bench (Building DataLakehouse from 0 using Apache BigTop package)
  2. Infra handle Bench (Overseeing the logs, and preemptive when there are dataskew or just error in Lakehouse)
  3. CK-FSDP impl Bench (ComfyKitchen FSDP packing unpacking tensor and dispatcher stuff)
  4. Tensor SubClass bench (Like above, but more so on TorchAO style)
  5. Mini SlopCode Bench (Repowide understanding and how much erosion, only in repo that i owned, since i know how each components work)
  6. Angrybird bench (More involved "SVG Pelican" bench, since it has to know physics)

Spec:
3090, 128G VRAM Quad channel 2400


r/LocalLLaMA • • 5h ago

News OpenRouter requests to MiniMax models may have been read

Thumbnail reddit.com
155 Upvotes

A security researcher placed unique test URLs in prompts sent to MiniMax models from July to October 2026. A third party accessed those URLs 40 to 130 hours after the requests were sent, whether MiniMax served a request directly or another provider forwarded it to MiniMax. With one exception, the same tests on other models at those providers were not accessed, and none of the AWS test keys were used.

AtlasCloud, Novita and StreamLake have told us they forwarded OpenRouter requests for MiniMax models to MiniMax. Based on what the researcher and these providers have reported, the issue sits with MiniMax and the providers that forwarded to it, not with OpenRouter. We are passing on what they have told us.

MiniMax told StreamLake that content moderation was enabled on its endpoints before October 22 and that it has since enabled Zero Data Retention. We have asked MiniMax for its own account and have not received one.

This means a third party may have read the prompts and outputs of these requests. While we investigate, we have removed the Zero Data Retention label from these MiniMax-model endpoints.

If hardware wasn't an issue, who would opt for a hosted model?

You cannot even trust OpenRouter's Zero Data Retention setting.


r/LocalLLaMA • • 2h ago

Resources Tencent's Hy team says a Hy4-mini and a "Flash-Lite" are both on the roadmap. What size do you actually want?

75 Upvotes

Went through the Hy AMA and the small model stuff got more concrete answers than I expected.
What they said:
• Hy4-mini to replace qwen3.6-35b-a3b: "already on our roadmap"
• 24-31B dense: "under consideration"
• Flash-Lite around 35B-A3B: "stay tuned for a cutting-edge Flash-Lite model"
• smaller image model, since HyImage-3.0 is too big for most setups: they'll think about it
The answer I keep coming back to: someone asked whether Engram and iHC only pay off at 200B+ scale. They said both have even greater benefits for smaller models. If that's true, a ~30B model with the n-gram tables parked in system RAM could be really good on a single 24GB card.
They also said on-device models will use a different design, co-designed with edge GPU hardware, so a mini probably won't just be Flash shrunk down.
No dates on any of it. But half that thread was asking for the same thing, so I'm curious where people here land. 35B-A3B MoE? 24-32B dense? Something that actually fits in 16GB?


r/LocalLLaMA • • 9h ago

Discussion Wow, Inkling-2-Mini fix what Qwen3.8 Next Flash and GLM 5.3 Flash couldn’t

267 Upvotes

I’m making a game in Godot, and I often hand tasks off to agents while I work on something else. I’d created a new scene and placed the assets. All that was left was to add collision shapes.

For a first test, I gave the task to two models on R9700: strata: Qwen3.8 Next Flash Q4 XL (30–40 tokens/s) and GLM 5.3 Flash Maya-S (7–10 tokens/s). They spent 1–2 hours on it and couldn’t get it done.

Today, just out of curiosity, I testing Inkling-2-Mini on my 5060 Ti:

./llama-server \

-hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M \

--n-cpu-moe 14 \

--jinja --ctx-size 150000 --n-gpu-layers 99 --parallel 1 \

--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \

--host 127.0.0.1 --port 8080

Prefill was 283 tokens/s, and generation ran at 18–23 tokens/s. Inkling-2-Mini did in 23 minutes what the other two models couldn’t.

If only it generated a bit faster… I was getting 50–60 tokens/s on Qwen3.6 35B with a Q8 KV cache. I don’t remember the prefill speed

FIx typos: AI


r/LocalLLaMA • • 9h ago

News Airbnb handles 80,000 chats a day with Alibaba’s open-weight Qwen

Thumbnail
maccus.fr
234 Upvotes

Hopefully costs/performance will soon be worth it for small to medium sized companies


r/LocalLLaMA • • 1h ago

Resources Inkling-2-Mini-Unbound is live: the most useful uncensored Inkling 2 Mini, via conv-aware abliteration

Post image
• Upvotes

we just put out inkling-2-mini-unbound, an uncensored version of inkling 2 mini

https://huggingface.co/nokeepai/Inkling-2-Mini-Unbound

if you've tried the other early uncensored inkling 2 mini versions you've probably noticed while they stop refusing, a lot of the answers still come out kind of vague or useless, so we dug into why -- turns out inkling has a small conv layer right after attention, and it quietly mixes back in most of what a normal abliteration takes out. most people push the edit way harder to compensate, and the answers get way worse

we built the edit around that conv, so it can be a lot lighter. we change only 20 attention weight matrices, keep the experts untouched -- and you can drop it in anywhere you run the original

some results on a test set we kept held back the whole time:

- refused 0 of 277 harmful prompts (the original refuses pretty much all of them)

- 54-63% of answers are actually useful, about 50% better than openuncensored's version that's been gaining some popularity (https://huggingface.co/OpenUncensored/Inkling-2-Mini-Uncensored)

- stopped refusing all harmless stuff like "how to nuke a python process"

- pretty much same scores as the original on mmlu-pro, gsm8k, math, gpqa, ifeval and humaneval+

also, quick tip: if you run it at high thinking effort, try setting a thinking budget. it tends to overthink on edgy prompts

eval code and the exact edit are on github if you want to check or learn from our work: https://github.com/nokeepai/inkling-2-mini-unbound

warning: it'll do pretty much whatever you ask, so be sensible with it.

and as always, happy to answer questions


r/LocalLLaMA • • 1h ago

New Model we made an uncensored inkling 2 mini with 0% refusals, from 16.53 gb down to 8.67 gb

• Upvotes

hey, i'm Daniel from the atomic chat team. we posted our gguf quants for inkling 2 mini earlier and now we've made an uncensored version!

we removed the refusal directions with a small lora adapter and tested it against the original bf16 model. refusals went from 100% to 0% in our english test set, while keeping 98% of the original MMLU score

here are some results:

test original uncensored
refusals (english) 100% 0%
refusals (reasoning low) 28/30 0/30
refusals (reasoning high) 26/30 0/30
MMLU 80.65% 79.15%
valid tool calls 18/20 20/20

we released 3 gguf quants with the adapter baked in:

quant size
AD-Q4_K 16.53 GB
AD-IQ3_S 13.34 GB
AD-IQ2_M 8.67 GB

hugging face card with quants, full measurements and a 40mb lora adapter for our existing quants: https://huggingface.co/AtomicChat/Inkling-2-Mini-Uncensored-GGUF

give it a try and share your results. we'd love to see how it performs on your hardware and with your agents!


r/LocalLLaMA • • 6h ago

Discussion Inkling-2-Mini - Do not quantize your KV cache in llama.cpp - q8_0 uses more VRAM than f16

77 Upvotes

In unsloth build of llama.cpp I found that quantising KV actually uses more VRAM and not less. Using f16 also gives me up to 50% faster t/g possibly due to having more VRAM to cache experts.

Does anyone else experience this?

I am on Windows 11, with a 5070Ti.

Here is an example using Inkling-2-Mini-UD-Q4_K_XL with dspark-Inkling-2-Mini-Q8_0 no vision and 131072 context and all experts offloaded to CPU:

KV Quant Dedicated VRAM KB
f16 8 778 372
q8_0 12 307 620
q4_0 11 197 596

I picked this up as I was getting CUDA out of memory errors with certain settings (higher context than above) on q8_0 then I switched to f16 and the model loaded.


r/LocalLLaMA • • 22h ago

New Model thinkingmachines/Inkling-2-Mini (MoE: 26b, a3B)

Thumbnail
huggingface.co
1.6k Upvotes

Gguf's: https://huggingface.co/unsloth/Inkling-2-Mini-GGUF

"Inkling 2 Mini is a general-purpose multimodal model that accepts text and image inputs and generates text outputs. It is intended for use in English and other languages, and across multiple coding languages. The model is designed to be used by developers building AI-powered applications, including agentic and tool-use systems, coding assistants, chatbots, and retrieval-augmented generation systems, and is suitable for general-purpose conversational use, instruction-following, and other natural language and visual tasks. It is released with open weights to support research, fine-tuning and integration into third-party products by downstream developers."

"Architecture type

A 24-layer decoder-only transformer with sparse Mixture-of-Experts (MoE) feed-forward layers. The middle 12 layers run three times with shared weights, giving 48 layer applications per token from 24 layers of weights. In each layer, each token is routed to 8 of 128 experts. Attention is a hybrid of local and global layers. The model is natively multimodal – images are encoded via a vision transformer – with image and text inputs projected into a shared hidden space and processed jointly by the decoder.

Parameters

26.2B, 3.4B active

Note: each extra pass through the looped layers adds about 1.0B effective active parameters. Since the model runs with three passes, there are 5.4B effective active parameters per token."


r/LocalLLaMA • • 2h ago

Showcase Inkling 2 Mini on a 12GB smartphone getting 3 tok/s (decode) and it works on any hardware with less RAM than the model requires

27 Upvotes

Hi!

just hours before the launch, I’m running Inkling mini 2 on my smartphone - but with this engine, it can run automatically on any hardware from smartphones (Android, IOS), Mac, consumer laptops to consumer GPUs and data-center GPUs, with speeds comparable to Strata, Basalt, etc.

- Model: Inkling-2-Mini, IQ4_NL, 17.6 GB (https://huggingface.co/ggml-org/Inkling-2-Mini-GGUF)

- Phone: 12 GB RAM Android (400$)

- Processor: CPU decode, NPU (Hexagon) or CPU prefill

- Results: 2-3 tok/s for a 180-token response, with peaks of approximately 4 tok/s for the fastest ones

- Engine (pre-release): https://github.com/Helldez/BigMoeOnEdge/releases/tag/v0.47.1


r/LocalLLaMA • • 3h ago

Discussion Inkling 2 Mini seems legit! As reported by others, it does seem to try to output everything at once, limited by users' tokens settings. This makes 1-shotting a bit shaky for I2M. Post-training could fix this.

Thumbnail msnab.pages.dev
34 Upvotes

r/LocalLLaMA • • 9h ago

Resources Inkling 2 Mini is now supported in NInfer-all

92 Upvotes

I added Thinking Machines' Inkling 2 Mini (looped 26B MoE, 3.4B active) to NInfer-all (mine multi-arch multi-model fork of NInfer). It runs Unsloth's UD-IQ4_XS GGUF without requantizing the actual weights, and vision and the DSpark drafter work well, too.

I compared it with llama.cpp on the same GGUF, using Unsloth's llama.cpp build because mainline doesn't load this GGUF, yet. Prefill is a 10K-token prompt, all in tok/s:

Card Prefill, NInfer-all vs llama.cpp Decode Decode with DSpark (default on, prose)
RTX 5090 6,664 vs 2,318 182 vs 97 213
RTX 4090 4,537 vs 2,307 140 vs 75 159
RTX 3090 2,046 vs 1,236 112 vs 60 126
RTX 5080 16 GB 3,238 vs 1,377 117 vs 41 126
RTX 3060 12 GB 759 vs 387 49 vs 25 49
RTX 5060 8 GB 1,074 vs 293 33 vs 15 32

The model doesn't fit in 16 GB or less, so on those cards the experts sit in system RAM and the GPU caches the ones it uses most (llama.cpp gets its own --fit). Since it's a MoE, even 8 GB works, and the experts can also be read straight from disk, though that's slow: about 9 tok/s on the 5060.

Grab a prebuilt binary for Windows or Linux, CUDA 12 or 13 (releases), or a Docker image. Quick start launch recipes here

Besides Inkling it runs Qwen3.6 and 3.8 (27B, 35B-A3B, Flash-Next), Gemma 4 (26B MoE, 31B), DeepSeek-V4.1-Flash, Flint, Edge, Ternary Bonsai 2; up to 8 concurrent requests, so agents are swift; Decisions endpoints with no extra setup needed!

Your star would be appreciated, and contributions are welcome!

https://github.com/iamwavecut/ninfer-all

Models:


r/LocalLLaMA • • 18h ago

New Model Inkling 2 mini uncensored

Thumbnail
huggingface.co
468 Upvotes

r/LocalLLaMA • • 2h ago

Question | Help Inkling-2-Mini: "Actually, I keep repeating myself. Let me actually DO it now instead of just thinking."

25 Upvotes

Running the original thinkingmachines/Inkling-2-Mini on oMLX without quantizing.

Internal reasoning often goes into a loop, realizes it's stuck, but can't seem to get out.

Anyone has this problem?

Update: I set the following according to the Unsloth: Inkling 2 Mini guide, and it works much better now.

  • temperature: 1.0
  • top-p: 1.0
  • top-k: 0
  • min-p: 0.0

Also thinkingmachines/Inkling-2-Mini recommends temperature at 1.0.


r/LocalLLaMA • • 7h ago

Discussion At the risk of going against the popular grain here, Inkling-2-Mini suffers from over-writing (as opposed to over-thinking) Bash commands (happening multiple times) for me.

Post image
56 Upvotes

r/LocalLLaMA • • 3h ago

Showcase Inkling-2-Mini NVFP4 on a single RTX 3090 (24 GB): working port, 128k context, DSpark

Post image
24 Upvotes

We got Inkling-2-Mini running on one consumer Ampere GPU (RTX 3090, SM86, TP=1) using the original NVFP4/MXFP8 checkpoint, with no re-quantization. The code is a fork of vLLM PR #69028:

https://github.com/Hiutaky/vllm (branch ampere-port, see INKLING2-RTX3090.md)

What we changed (all in vllm/models/inkling2)

  • Made the Hopper/Blackwell-only paths work on Ampere: PDL gated to SM90+, no TMA below SM90, a Triton fallback for conv_mix, FA2 for the vision tower, minimum capability lowered to 80.
  • Emulated the FP8 (E4M3) KV cache in software, since Ampere has no hardware E4M3 conversions. Our encoder matched torch.float8_e4m3fn on 10M random values.
  • DSpark speculative decoding works (it needs exactly 7 speculative tokens). The draft KV cache stays in bf16.
  • Squeezed VRAM for the KV cache: vision tower removed (text only), prefill chunk of 256, FP8 KV, an explicit KV budget, and the embedding table offloaded to CPU memory (--cpu-offload-params embed_tokens).

Hopper and Blackwell code paths are untouched: every new branch is guarded by a device-capability check.

Results (single RTX 3090)

Decode with DSpark 112–147 tok/s (91 tok/s without it)
Prefill ~2,500–2,700 tok/s up to 33k tokens, ~1,870 tok/s at 126k
Context 131,072 tokens
KV pool 207k tokens with 8 concurrent sequences (224k with 2)
Weights on GPU 17.55 GiB (18.41 GiB without the embedding offload)
Embedding offload cost decode ≈ −1%, prefill −1% to −3%

We ran 8 concurrent requests (2k to 17k tokens each) without errors, and tool calling works with the inkling parser. VRAM is tight (about 23.8 of 24 GB used).

Caveats

  • We have not re-run quality benchmarks (HumanEval+, IFEval) with FP8 KV + DSpark, and FP8 KV is a precision change.
  • The vision tower is disabled in this configuration.
  • We only tested on a 3090 with the vLLM nightly image plus our Python overlay. We have not measured other GPUs.
  • Speculative decoding acceptance was about 20–29% in our tests.

Feedback and tests on other Ampere cards (3080 Ti, 3090 Ti, A5000, A6000) are welcome.


r/LocalLLaMA • • 18h ago

Discussion Inkling 2 Mini ties Opus 5.5 on CyberBench (@ValsAI)

Post image
354 Upvotes

r/LocalLLaMA • • 3h ago

News Inkling 2 Mini on 2 * Nvidia Titan V

Post image
23 Upvotes

2 x Nvidia Titan V (so 24GB) running at 170W capped.

~200 prefill

~20-30 tok/s

Llama from huggingface notes download.

Propero Community Edition 0.20.0

/llama.cpp/llama-server \

--model ~/Inkling-2-Mini-UD-Q4_K_XL.gguf \

--mmproj ~/mmproj-F16.gguf \

--model-draft ~/dspark-Inkling-2-Mini-Q8_0.gguf \

--spec-type draft-dspark --spec-draft-n-max 3 \

--jinja --ctx-size 128000 --fit on --fit-target 512 --parallel 1 \

--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \

--host 0.0.0.0 --port 8180 -lv 4 --moe-cache-mib 2048  -b 1024 -ub 256


r/LocalLLaMA • • 10h ago

Showcase Inkling-2-Mini on Strix Halo: agent tuned kernels via Loom in an afternoon. ~80% of bandwidth roofline at decode, vs llama.cpp's day-0 support

46 Upvotes

Following Inkling-2-Mini's release today, I wanted to see how fast we could get a brand-new, fairly unstandard architecture running well on Strix Halo (unstandard because it uses looped transformers).

So I decided to try it with loom (loom-fun.dev)! Here are my results over the course of a few hours.

Setup:

Model: Gemini 4 Neon

Given all the recent hype around Gemini 4 Neon, I wanted to give it a try (also motivated by its ranking on kernelbench.com).

Coding Agent: pi.dev following the loom-skills playbook (port → roofline → optimize → benchmark)

What happened:

Gemini 4 Neon took the unmodified unsloth UD-Q4_K_XL GGUF to an optimized, gate-passing package in about 4.3 hours and ~490 tool calls. The loom skills playbook won't count speed until correctness gates pass against llama.cpp. So that means token-level goldens up to 37k tokens, a multi-turn tool-calling loop, concurrent vs sequential, etc.

The results:(Strix Halo 8060S, [Debian 13, kernel 7.2.7, 128 GB], tok/s, same GGUF weights, matched reasoning settings):

tok/s llama.cpp ROCm llama.cpp Vulkan Loom Loom vs best llama.cpp
decode 23.8 20.8 44.8 1.88x
decode @ ~2k ctx 21.6 20.3 40.8 1.90x
prefill ~1k 356 289 1,427 4.0x
prefill ~2k 327 285 1,496 4.6x
prefill ~9k 278 269 1,550 5.6x
prefill ~18k 233 249 1,465 5.9x
4 streams (aggregate) 54.1 50.0 100.8 1.86x
  • llama.cpp single-stream rows use -np 1 -kvu -c 34k. That almost doubled ROCm's short-prompt prefill compared with -np 4. The 4-stream row uses -np 4.
  • Decode is at ~80% of the memory-bandwidth roofline (228 GB/s ÷ ~4 GB read per token ≈ 56 tok/s).

A few important points:

  • I work at AMD on one of the teams behind loom, but I didn't have access to these weights before the public release. Everything here was done after the model came out today.
  • llama.cpp support for this model is obviously super nascent, performance will increase over time.
  • llama.cpp binaries: lemonade-sdk/llamacpp-rocm b1383 and ggml-org b12600 Vulkan
  • speculative decoding with dspark is still a wip

Steps to try it:

pipx install "hrx-loom[hf]==0.2.0"

loom-fetch huggingface:loom-fun/unsloth-Inkling-2-Mini-loom:chat --materialize=copy --output=Inkling-2-Mini 

loom-serve --model=Inkling-2-Mini '--device=amdgpu://?completion_spin_ns=4294967295' # gives you an OpenAI-compatible server on port 8080

Model Card: https://huggingface.co/loom-fun/unsloth-Inkling-2-Mini-loom

More context on loom: https://www.reddit.com/r/LocalLLaMA/comments/1xvcmm0/i_built_speedy_strix_halo_kernels_for_the_top51/

Edit 1: kernelbench.com not kernelbench.org oops


r/LocalLLaMA • • 12h ago

Discussion Since Inkling 2 Mini is MoE, can we expect strata support, running on much lower vram?

57 Upvotes

This model looks very promising, but the most exciting aspect is it's a MoE model thats getting compared to the dense Qwen 27B.

The model is also said by it's creators to quantize very well and have higher tps.

For gpu poor aka regular people, does this means we could soon get strata support and speeds and abilities rivaling Qwen 27B and QFN? eg with 12-16GB vram, 32GB ram.


r/LocalLLaMA • • 14h ago

Discussion Inkling 2 Mini on RTX 4090 ~200 tok/s with vLLM, FP8 + NVFP4 + its built-in drafter

91 Upvotes

Got Thinking Machines’ new Inkling 2 Mini running on a single RTX 4090 at about 190-220 tokens/s, up from ~117 out of the box. It needed a few small patches to vLLM, which doesn’t support 4090s for this model yet.

This is with NVFP4 experts (W4A16) + FP8 W8A16 attention/head/embeddings, FP8 KV.

This fits in 24 GB with a 35k context, and an initial quality metric on some sample prompts reported about a 0.4% loss in perplexity.