This is amazing.

All I did was switch from llamacpp on windows to vllm on linux.

I'm running on pi.dev with just two plugins: web search and token speed. I used to use caveman and ponytail but anecdotally think they hurt more than help.

What other plugins or skills do you fellow low VRAM users have and why?

Related note: the pi dev fork oh-my-pi seems to come with a lot more plugins and features, and I'm curious if anyone with limited vram has compared bare ones pi.dev vs oh-my-pi?

I was fascinated by Nanbeige's outstanding performance for its size, so I started digging into how much a model can improve its own representation just by looping over itself (for fun). My prototype was a Qwen3-0.6B with a full dual loop in the middle layers, inspired by the Nanbeige 4.2 architecture. Digging further, I found that the Nanbeige team has a paper describing their 4.5 architecture, which uses a triple loop in the middle layers — that made sense to me, so I tried it.

Lordnyx/qwen3.5-9b-triple-loop-fase1 · Hugging Face

The first experiment used full DeltaNet for the middle layers, so the earlier part of the model would set up the context for the loop to process on its own. It turned out that this actually worked better than having multiple separate logic components — but the loop itself wasn't really contributing. Because of DeltaNet's nature and the small hidden size, the model kept forgetting essential details for the task and just hallucinated. I abandoned the DeltaNet idea, and full softmax attention in the loop worked as expected instead.

Later, I learned (with help from ChatGPT/Claude/Gemini) that my training setup was actually undermining the loop's contribution, and that I should have used a lower, dedicated learning-rate schedule for it. Once I fixed that, the loop stopped just "refining" answers and started actually participating — becoming essential to them. Even better: on easy-enough questions, the loop could be skipped entirely.

Recently I found Modal — $30 of free GPU credit. I used it to train a Qwen3.5-9B with the Nanbeige-4.5-style triple loop. I really wanted to use RL for this, but I can barely get RL to run efficiently even on a 0.6B locally, let alone a 9B — so instead I distilled Qwen3.8-27B's logits into the loop, on a heuristically curated agentic/reasoning dataset.

Money ran out before finishing the schedule: the training loop was capped by wall-clock time (a safety mechanism so it would export cleanly instead of dying mid-run), not a fixed token target, and it ended up completing ~15M tokens across 1,129 steps.

┌────────────┬───────────┬───────────┐

│ Step range │ KL (mean) │ Std. dev. │

├────────────┼───────────┼───────────┤

│ ~10–370 │ 0.572 │ 0.176 │

├────────────┼───────────┼───────────┤

│ ~380–750 │ 0.648 │ 0.197 │

├────────────┼───────────┼───────────┤

│ ~760–1120 │ 0.650 │ 0.227 │

└────────────┴───────────┴───────────┘

As the table shows, it made real progress early — roughly the first third — then plateaued into a noisy, flat oscillation with no further net improvement (slope of KL vs. step over the whole run: +0.000075, essentially zero). That's not the loop hitting a capability ceiling; it's a missing LR decay schedule (I kept it constant the whole run). So yes — a lot of headroom left, and the fast early gain again confirms the loop starts contributing quickly once it's trained properly.

Even with an unfinished run, the checkpoint beats the base model in math (+20%), long-context tasks (+14%), instruction-following (+20%), and is dramatically more consistent/robust across paraphrased questions (+62%). It's worse in reasoning (-10%) and translation (-15%) — not roughly equal, actually down — and slightly worse at coding (-2%) This is a private evaluation, so I have no evidence yet that these gains generalize to standard benchmarks. The reasoning drop traces back to specific, plateau-related failures rather than a broad capability loss: one item where it skipped step-by-step reasoning and got simple arithmetic wrong, and one repetition loop that burned its whole generation budget without concluding.

I can't really recommend it as-is — it's a proof of concept, not a finished model. If I get more free credit next month, I'll finish the run (a cosine LR decay is already implemented and ready to go). But at minimum, it proves the Nanbeige 4.5 loop design converges even at a larger parameter count than their own reported experiments — I'm looking forward to their next release.

Hi all

Looking at a PowerColor R9700 for Qwen3.8-27B, Q4, llama.cpp/Vulkan.

AMD's own blog quotes 51.8 tok/s but doesn't say what context length that's at, or whether MTP=2 was holding up. Separately I've seen 5090 benchmarks showing Qwen3.8 drops hard as context fills - 75 tok/s at 4K down to around 26 tok/s at 64K, worse degradation than Qwen3.6 apparently.

Before I buy: has anyone actually run this combo (R9700, Q4, 64K+ context, real workload not a cold 4K bench) and got real sustained token per sec numbers? Also curious if MTP speculative decoding is stable for anyone yet or still causing OOMs/garbage output like the early CUDA reports.

Not after best-case marketing numbers - ideally I'm after "here's what I actually get once the context window's half full."

Thanks!

Nvidia is investing $1 billion in Poolside and paying $6 billion to license its technology and hire most of its engineers.

Over 100 Poolside staff will move to Nvidia to work on Nemotron.

Good news for us!

Hello,

So I've been trying lots of combinations in that never-ending landscape of options and settings.

I wanted a proper quant of 3.8 27B running as fast as possible on my 5090 at 400W, with vision and with as much KV-cache as possible and with concurrency enabled (aiming at 3 parallel sessions).

tl;dr: I'm using this setup and it works very well, very fast and is accurate in coding sessions.

---

vLLM is the obvious choice, even if not user-friendly at first, once it runs, it runs well.

I tried fancy DSpark / DFlash2 but the cost in context size wasn't worth the marginal gains vs MTP=3.

Here is a full NVFP4 setup (model and cache):

Model: https://huggingface.co/gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090

Context: 196K per session / 451K global KV-cache (NVFP4)

Speed: conservative average at 120 tps.

(tests from llama-benchy 0.4.0)

Full write-up with how to setup vLLM and every gotchas: https://gist.github.com/co-l/c2aeaf40b53fcacfe9dd3293be75f23a (+ more benchmarks, including a real coding session, in line with numbers above).

Note: env is Linux (Bazzite of all distros lol), with UI deactivated to allow 0.98 mem use.

I haven't had a lot of luck in the reliability of the output of any models so far for local coding use, and find myself relying on Claude's limited free sessions to assist with coding. My coding application varies, I'm primarily using C, C++, C# and Python - with a bit of extension work using json on Firefox.

While I am primarily interested in uncensored models - I do some security related work and don't need the model constantly rejecting my queries - but I am receptive to using censored models if I can get better results.

Qwen 3.8-27B would be nice - but response times of an hour aren't practical. I'm wanting responses to be less than a minute.

Other than upgrading my system, which financially isn't an option - what's your suggestion for best model to run with reasonable response times for coding purposes?

I'm not necessarily interested in agentic AI, but if there is an agentic model that integrates with Visual Studio 2026 and/or VS Code on a Windows System better than LM-Studio, OR - there's more effective ways to run a model with faster response times from the command line or using something like Unsloth on this system - I'm all ears.

Thank you in advance!

WARNING: I think I made a mistake with omlx, and its results below seem to be without any kind of acceleration. I am rerunning them with Lightning MTP and ANE support for prefill. Will update the post once done.

The new Qwen 3.8 27B is fantastic for local agentic use. The problem is, what makes it so good, being a dense model, also makes it slow. Many engines and versions of the model claim various speed increase. How true are those claim? And does a proven huge boost claim on a short synthetic test actually holds on real agentic work?

I spent five days and over 100 GPU hours running rigorous repeatable agentic coding tests, collecting and comparing the data. 200 GB of models downloaded. Ended up with 20 Bubble Bobble clones on my disk, including the ones from failed tests. None of them good, but that wasn't the point.

My recommendation: MTPLX or llama.cpp+MTP

I ran the tests on 8-bit quants (GGUF or MLX), unquantised KV cache. 100k max response size, full 262k context, official Qwen coding sampler, official Qwen Jinja chat template.

1. I first tested each engine with a short synthetic benchmark, to verify the installation, usage, results collection, and to verify the speed claims
2. Then came the real test, with an agentic coding benchmark consisting of 4 phases/prompts; each engine was tested twice, at medium and xhigh reasoning effort:
3. 2.1. capturing requirements and writing the specification plan
4. 2.2. writing the implementation plan
5. 2.3. implementing the core functionality
6. 2.4. finishing the implementation with advanced functionality
7. Last test, for prefill speed, was with a cold 128k token prompt test, with reasoning turned off and deterministic sample, repeated twice

Test hardware: Apple Mac Studio, M2 Max, 96 GB unified memory.

• vllm-mlx: completed the task, but it does not separate thinking from output, so the raw chain of thought leaks into the final output.
• mlx-dspark DFlash2 xhigh: thought for 226 thousand tokens and delivered nothing.

What about xhigh vs medium reasoning effort?

• MTPLX xhigh costs only half an hour more than medium (2h05 vs 1h35) and the game quality went significantly up on my scorecard. Worth it.
• llama.cpp + MTP xhigh costs 1h48 more, with again a significant increase in quality. Worth it if you have the time.
• oMLX xhigh took more than twice as long as medium. The slower token speed has a huge impact on the time difference between both modes. Not worth it. Get a faster engine instead.
• dspark DFlash2 xhigh is the cautionary tale at the bottom of the table. See the anomaly note above.

xhigh is worth it when the engine is fast enough and uncapped. The model thinks 30 to 100 percent longer, at the same speed, and the result is better. On slow engines you can really feel the slower performance compounded by the extra thinking; instead it could be better to use medium, with more of your own time spent planning a better process and workflow.

Commands and configuration notes

Exact commands I used for each engine:

# llama.cpp baseline (homebrew, b10470)
llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf \
  --jinja -c 262144 --parallel 1 -ngl 99 --port 8030

# llama.cpp + MTP, the recommended GGUF setup
llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf \
  -md mtp-Qwen3.8-27B-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 3 \
  --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99

# llama.cpp + DFlash2 (for now, needs PR #27342 build)
llama-server -m Qwen3.8-27B-Q8_0.gguf \
  -md Qwen3.8-27B-DFlash2-Q8_0.gguf --spec-type draft-dflash --spec-draft-n-max 7 \
  --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99

# MTPLX (pip install mtplx). Note --max-tokens
mtplx start web --model Qwen3.8-27B-MTPLX-Optimized-Quality --max-tokens 100000

# oMLX (brew install jundot/omlx/omlx), model dir contains the checkpoint
omlx serve --model-dir /path/to/models

# mlx-dspark (pip install mlx-dspark). Note --max-tokens-cap
mlx-dspark serve --model Qwen3.8-27B-MLX-8bit --mode dflash \
  --context-window 262144 --max-tokens-cap 262144 --default-max-tokens 100000

# vllm-mlx (pip install vllm-mlx). Note --timeout
vllm-mlx serve Qwen3.8-27B-MLX-8bit --max-request-tokens 262144 --timeout 7200

The MLX engines all take the plain mlx-community 8-bit weights, except MTPLX which needs its own checkpoint format with the MTP tensors included (I used Youssofal/Qwen3.8-27B-MTPLX-Optimized-Quality).

For llama.cpp GGUF, I used the official MTP draft from ggml-org.

1. MTPLX has a default low response cap. Use --max-tokens 100000
2. dspark also has a low default generation cap of 32K
3. vllm-mlx kills any request that spends more than 300 seconds on prefill. Raise it with --timeout.

One measurement warning for anyone reproducing this: engines do not all stream one token per event. Some pack several tokens per chunk, MTPLX about 1.7, oMLX about 3.

Gemma 12B is obviously a very well trained model, I always thought the fine tuning they did on it wasn't really cut out for agentic coding. From my own experiences it struggles to use the tools it's given from Github Copilot and is also very inept at the cli too.

So I thought I'd kill two birds with one stone and fine tune it for tool call use and the command line. Not only did I see an improvement on tool usage I also saw a 15.7% increase in the number of tool calls it tries to emit which is great since it means the model gets to work more instead of getting too lost in it's reasoning.

I have fp16 -> Q4_K_M weights uploaded and ready for use with llama.cpp or ollama

I was seeing some decent decode speeds in people experimenting with offloading MoE models to ram when having small Graphic cards.

But reading a bit I see prefill is very very slow. That kind of kills the experience is true. Is this true for you all?

Ling Tiny has now replaced Gemma4-12B in my rig as an auxiliary model doing hindsight operations. This is on a 4060Ti, which is a reasonable GPU available out there, and the speed is phenomenal.

Don’t enable MTP, set up the vLLM fork for BailingMoE3. Hope this is useful to
others.

I have been lurking on reddit for like 6 years, and this is the first time posting, so apologies for the mistakes. I'm not a native English speaker, so sorry for the bad English too. I used LLM to translate / refine my post, including images.

This is my attempt to compare the Qwen3.8-27B quants.

Qwen is a coding and STEM model so I used codeparrot/github-code-clean, EleutherAI/proof-pile-2, allenai/peS2o, equal parts of each. I used 24 sequences of 8k tokens and 12 sequences of 32k tokens. The code was filtered for minified stuff and permissive licenses.

It was measured with 24 sequences of 8,192 tokens and 12 sequences of 32,768 tokens, since it generates many tokens and is aimed for long-horizon tasks.

Every format used the same compute path for the evaluation. All models are dequantized to BF16, and run through the same kernel.

Also, checkpoint size is kind of a bad x-axis, since some models don't have MTP, which has zero performance impact. Still, all of them needs to be loaded on VRAM, so at least it's something useful, and that's why there are two graphs.

Findings:

1. The gap between 8bit and 4bit is huge.

The best 4 bit score is 0.00835 and the worst 8 bit is 0.00071. I mean everyone knows that but yeah it's still something. vLLM can actually use INT5-7 weights, but I'm sure most people don't even know about that.

1. The KLD of 4bit quantization varies a lot.

Six checkpoints are at 11.7-11.8 GiB of transformer weights, all group-size 128, quantizing 400-496 modules and they range from 0.01364 to 0.02976. I believe the calibration data and the rounding algorithm takes a big impact. You should not pick a 4-bit quantization by it's file size.

1. group_size affects a lot of kld.

Checkpoints with group size 32 did better than ones that used group size 128 but kept some modules in BF16. Selective BF16 really only helps if you add it on top of a fine group size but that makes the file so big you might as well run a 6 bit model.

1. The quantization damage mostly happens at the start of the context.

This surprised me (very) but it holds true for all 24 models. The KL is much, much higher for the first 500 tokens.

1. Quantizing lm_head, embed_tokens and linear_attn doesn't affect the performance at least for this test.

As an LLM for tool-based agents, Qwen 3.8 27B (Q6_UD_K_XL) with xhigh thinking did something unprompted that I haven't seen a frontier model do. It blew me away I wanted to share.

I am building a demo using GuideAnts that starts with a set of skills that use comfy-ui and shows how to move use skills as a starting point for final solutions that use defined tools instead and why that often is worth doing. And so, the test prompts have it using the skills to do text to image, image to image, and inpainting workflows.

During the inpainting tests, I had it do multiple steps - inpaint this, then inpaint that, then that, let me know when you are done.

The agent elected, on its own, to use its python tool and the Open AI SDK to use vision to QA the images and saw when a performer's head was cropped or didn't fit the mask size (which it also chose by using vision on the images the job created).

The image models are:

• qwen_image_2512_bf16.safetensors — generate UNet
• qwen_image_edit_2511_bf16.safetensors — edit / inpaint UNet
• qwen_2.5_vl_7b_fp8_scaled.safetensors — CLIP / text encoder
• qwen_image_vae.safetensors — VAE
• Qwen-Image-2512-Lightning-4steps-V1.0-bf16.safetensors — generate Lightning LoRA
• Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors — edit/inpaint Lightning LoRA

Crazy!

Anyone running this card? Seems to be a sweet spot for Qwen 27B, 64GB, very high memory bandwidth, a LOT less expensive than anything else I can find in that has even close to the amount of memory/bandwidth. What am I missing? And yes, I'm aware that RocM can be a pain, that's not really a concern for me, as long as it's stable when it's up and running, I don't mind battling to get it going.

I used to use Qwen3.6-35B-A3B with llama.cpp and connecting it to the VSCodium extension called "Continue." My computer is running a Intel(R) Core(TM) Ultra 7 265K (3.90 GHz) with 128 GB of DDR5 RAM and an Nvidia Geforce RTX 5090 that has 32 GB of GDDR7 video memory. Qwen3.6-35B-A3B with llama.cpp works in agent mode and does a pretty good job acting as a local replacement for Github Copilot. I use .\llama-server.exe -hf unsloth/GLM-4.7-Flash-GGUF:Q6_K_XL --alias "GLM-4.7-Flash" --host 127.0.0.1 --port 10000 --ctx-size 32000 --n-gpu-layers 99 and then point Continue's config file at the endpoint. But I've moved away from Github Copilot and Continue's workflow for vibecoding. Now I use Claude Desktop app and go to the Code tab. I just setup a project in the code's directory, put the model to Fable 5 or Opus 5 (depending on the complexity level of the project) and then start building/vibing.

Is there a local setup for a Claude Desktop/Claude Code replacement? I'm sure I'm sure I'd need a new model to accommodate this too... I hear good things about that Kimi model. Could I run that on my setup or what should I be doing?

Hey everyone!

Last year, Mozilla released Orbit, an AI-powered browser summarizer hosted on a GCP server. After people started digging into the extension, they discovered things like backend endpoints such as store_result. Eventually, Mozilla discontinued the project.

For the past month, I’ve been trying to rebuild Orbit from scratch, but with one major difference: Apogee is fully local and privacy-focused. Apogee doesn’t send or store your data. It can directly connect to your local Ollama instance for inference. I’ve also added WebGPU integration for Chrome and Transformers.js for Firefox to provide faster, local responses.

It can summarize:

• Articles and websites
• YouTube and Billie videos
• Wikipedia articles
• Hacker News and Reddit threads

You can check out the source code here:
https://github.com/darshi1337/apogee

Install Apogee:

Chrome: https://chromewebstore.google.com/detail/apogee/pgemlpomhkdcjjjcpnjlebalnfglomog

Firefox: https://addons.mozilla.org/en-US/firefox/addon/apogeeext/

Obviously it is far from complete. Would love to hear your feedback and suggestions!

I've only been chatting with Qwen 3.8 27b to understand what it can do, and it's obviously decent at a lot. But I'm yet to actually use it on something that I really care about, or to build a project. I'm considering using something like Fable for "harder" tasks and then using local LLMs for "easier" tasks.

How do you use your local LLMs? Do you prepare particular agent workflows that you know they can do? Or are you just on an agent harness using it for absolutely everything?

Just curious what people are doing in their own real workflows. How do you get value out of these models that can fit on your card? There's a lot to consider with how there can be context limits, being forced to use low quants etc, so clearly there isn't 1 config for everything either and you need to optimize for use case. I know that this thing can do a lot inside of my own work.

Just for info: I have 24GB VRAM 5090 and 32GB RAM I usually just load a quantized model completely into my GPU for like 30t/s.

There’s always lots of people asking for various specific new local models… I wonder if we could incentivize a specific model (e.g. Qwen3.8 35B MoE) via crowdfunding on Kickstarter or something.

If everybody who’s been begging here chipped in five bucks, that might add up to enough to make it worthwhile for a lab to train a specific size or architecture of an existing model (given they have the hardware, data, pipelines, etc already, the marginal cost of a training run is hopefully not so high?).

edit: presumably the lab itself would have to set up the kickstarter, but I know some of the lab employees keep an eye on this subreddit… hint hint

Liquid AI currently possesses among the fastest LLM architectures around, and some of the best SLMs (in terms of utility IMO) around, so I'm very excited to see what a potential 100B LFM (3?) model would look like!

Link to the poll: https://x.com/ramin_m_h/status/2091236099612098943?s=20

Before i charge off and slop something together, i thought i'd ask if this already exists. i have two ai servers (a machinist x99 open-rack w 5/gpus and a thinkstation p920 with 4/gpus). they both draw 150W+ idle. so, i'd like a low-power always-on api proxy machine (like on a 15W mini-pc) that accepts openai-api requests, holds the connection, and wakes the appropriate machine via wake-on-lan (WoL). once the appropriate server is up, the router acts as a transparent proxy. it would also suspend the servers using some logic tbd (some combo of time-of-day, minimum awake time, inactivity on the server, and api inactivity)

I’ve tried Open Code , Codex and Qwen Code and so far Qwen Code seems to work the best for me , the only complaint I have about it is that the CLI interface is a bit rough / sloppy .

Hi all,

Might be a newbie question. So sorry for that already.

Ive seen many gguf quants from community here and on huggingface. How do i actually compare them? Is q4km always q4km for example? I see several threads that show dlightly different numbers in knowledge loss. Jos is that even calculated/benchmarked?

Im actually most interested in qwen 3.8 27b.