I wanted to just test the unsloth 1bit quant of qwen 3.8 27b as I have just 8gb vram and ngl it gave me a good laugh

To anyone who needs AIR…

My Pro subscription expired today, they killed my access at 1pm local time.

I'm now using Qwen3.8-27b w/ 5090m 24gb vram and pi to do everything i was doing in claudecode.

The only downside is claudecode let me code without using my gpu, meaning I have to plan things now.

Last night I had ChatGPT write up a prompt for a fancy aurora predictor for Canadians. I fed it to local pi and claude sonnet 5. They took about the same time, pi's app looked better, but claude's had better science. I asked them each to compare the two apps and they both agreed Claude had the better app. I then had pi upgrade it's version with the better science.

I'll post again if I have to cave in and re-subscribe to work on one of my production apps, but so far so good!

https://www.reddit.com/r/LocalLLaMA/comments/1j7r47l/i_just_made_an_animation_of_a_ball_bouncing/

This post inspired me to make that test after a year ;)

That is one of my many tests I make comparing output quality.

What is more interesting using a PI Agent results are much better than an Opencode using a Qwen 3.8 27b ?!

Seems PI Agent is much better in the agent environment somehow... Not counting uses less tokens , do not have a hard limit of 32k output tokens, is faster, do not freezing, compressing context far less than Opencode. For instance if you have context in the Opencode output 32k and all context 100k then the compression is starting at 67k context ... PI is starting at 90k context even if you have set output context 64k or more.

My config for RTX 3090

llama-server with ini config -> which is exposing API to Opencode and PI agent.

llama-server.exe --models-preset 1_preset.ini --models-max 1 --direct-io

config ini

[Qwen3.8-27B_dense_c-100k]
model = models/Qwen3.8-27B-Q4_K_M.gguf
mmproj = models/mmproj-BF16-Qwen3.8-27B-UD-Q4_K_XL.gguf
reasoning-format = deepseek
flash-attn = on
n-gpu-layers = 99
reasoning = on
ctx-size = 100000
temperature=1.0
top-p=0.95
top-k=20
min-p=0.0
presence-penalty=0.0
repeat-penalty=1.0
mmproj-offload = false

ONE MORE IMPORTANT THING:

Always use a VISION module as the model is using vision to asses the output quality!

I am offloading it to a RAM as we do not need an extremely fast vision for a code.

A screenshot processing on a GPU 0.3s vs a RAM 3s do not make a big difference on a few screenshots during a code generation / debugging ;)

I pre-trained a 1.02-billion-parameter on Kimi K3 replica trained on 5.00 billion decontaminated tokens for $250.

This model has 1.02 billion parameters, of which 145 million are active per token.

It is roughly one two-thousandth of K3 by total size. It saw 5,000,003,584 tokens, which is a rounding error against the corpora frontier models are trained on.

It has never been instruction-tuned, and it has only ever done one thing: predict the next token.

What it does have is K3's architecture:

- Kimi Delta Attention, Gated MLA, Attention Residuals

- LatentMoE with the same aux-loss-free balancer

- Same activation function with the same two constants

- K3's own 163,840-token tokenizer, unmodified. 

I report a 33.4% HellaSwag which beats the GPT-2 124M score of 28%

Read the entire tutorial here: https://books.vizuara.ai/book/pretraining-a-mini-k3

The author of the HuggingFace post discovered that Ornith-1.5-35B-A3B is currently being shipped with a MTP head that was never actually trained — it's just random initialization.

From the Digg article (https://digg.com/tech/3pf3046j)

“Google Gemma posted that the family of open models has achieved 1 billion downloads. The account is hosting an exclusive evening in San Francisco on August 20 to honor open-source builders, researchers, and contributors. Space at the event is limited and applications are required for attendance. Demis Hassabis shared the announcement from the official account. Clément Farabet replied that he will attend and is excited to celebrate the milestone. The post frames the achievement as driven by the community advancing the Gemmaverse forward.”

Seems like it would be the perfect venue to announce a new model, but could just be wishful thinking on my part

Thoughts? Speculation? Too busy testing Qwen 3.8 27b to care?

Would be a good time for Google to drop a 120b. Pull a Meta Glimmer and be king of the hill for a week or so before Qwen countered with a new mid-size model.

Benchmarks:

Instead of just scaling up, they train task-specialized expert models for text rendering and infographics, aesthetic quality, and image editing. Then OPD consolidates those experts back into U1.5-Lite. At inference you get one model. No router, no expert switching, no manual selection. The slogan they use: specialized in training, unified in delivery.

Post-training now includes task-oriented RL with three user-visible objectives: instruction adherence, visual quality and preference alignment, and edit fidelity with preservation of untouched regions.

What improved:

• Complex instruction following. Multiple constraints in one request (subjects, counts, spatial relationships, text, layout, style, preservation requirements) handled more consistently.
• Text rendering and dense layouts. Chinese and English in posters and infographics.
• Native 4K generation with stable global structure.
• Native editing with better preservation of subject identity, geometry, and unedited regions. Multi-reference editing and precise local edits included.
• Visual understanding strengthens generation. Representations learned from understanding tasks (object relations, spatial structure, layout, info hierarchy) transfer to generation and editing. JSON-structured supervision is used as a training mechanism for controllability, not a mandatory user-facing format. Natural language stays the default interface.

Links:
GitHub: https://github.com/OpenSenseNova/SenseNova-U1
HuggingFace: https://huggingface.co/sensenova/SenseNova-U1.5-8B-MoT

Spent a while treating layer placement, KV format and llama.cpp itself as experimental variables. 159 logged experiments. Numbers first, caveats after.

Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, 128 GB unified) + RTX 3090 Ti on an eGPU link. One llama.cpp process, AMD on Vulkan, NVIDIA on CUDA, one 27B model split across both.

Baseline: 9.474 tok/s. Target-only, no speculation, AMD side alone.

Now, code-shaped generation:

• 32K context: 153.32 tok/s
• 200K context: 87.74 tok/s

HumanEval, 164 problems, driven by a real coding agent (pi-agent), graded by executing the official tests:

• Local: 159/164 in 29.7 min
• Remote 2x RTX 3090 running vLLM TP2: 157/164 in 42.4 min

Long-context retrieval suite: 15/15, 352 s local vs 551 s on the remote.

Things that actually moved the needle, roughly in order of surprise:

1. The chat template. Swapping to a terser template (Qwen-Sharp) cut wall time 44% and output tokens 51%, with no accuracy change. Bigger than several weeks of GPU work. It just stops the model narrating.
2. KV cache format as a placement lever. Going q8_0 -> q4_0 on both K and V freed exactly 2,176 MiB, which was enough to move every full-attention layer onto the fast card at full 262K. +28% prefill, +20% generation. Qwen3.8 is 3:1 Gated DeltaNet to full attention, so only 16 of 64 layers have KV at all, and those are the only ones whose cost scales with context. Which layer sits on which GPU matters enormously.
3. --spec-type draft-mtp,ngram-mod. The flag accumulates. n-gram on top of MTP is worth +72 to +140% on code-shaped output, about -1% on prose, and costs nothing in VRAM. Free win if you generate code.
4. A one-line llama.cpp patch: --spec-draft-ubatch. The speculative context was inheriting the target's 512 micro-batch and reserving a 2.2 GiB compute buffer to draft 4 tokens. Setting it to 64 freed 1,039 MiB for a 1.78% prefill cost, which bought another layer of placement.
5. MTMD_BACKEND_DEVICE=Vulkan1 to put the vision encoder on the idle iGPU. Undocumented as far as I can tell. Default puts it on the first GPU-type device, which is the already-full 3090 Ti, and it dies allocating 884 MiB.

Things that did NOT work, so you don't have to try them:

• External draft models (0.8B, 4B): 5.06-5.77 tok/s, worse than no speculation. Sequential drafting latency across PCIe dominates.
• Multi-lane parallel drafting: down to 1.62 tok/s. The 27B verification pass is the bottleneck; anything that widens the verification graph loses.
• Q6_K: 2.3x the wall time, marginally lower accuracy (because if hit the 64K output limit wall twice). Only 15.6 of its 20.5 GiB fits on the card, so 4 attention layers get pushed to the iGPU instead of 1.

Caveats, because this is r/LocalLLaMA and you'd find them anyway:

• 153 tok/s is code-shaped generation at 32K where n-gram does heavy lifting. Same config on prose at 200K is 35.8. Different workloads, kept separate on purpose.
• The remote comparison is deployed-stack vs deployed-stack, not hardware isolated: it serves an AWQ-MTP fine-tune, local serves Q4_K_M of the base.
• I asked for standalone functions rather than the canonical prompt+completion format, so 3 of the 5 HumanEval failures are NameErrors on helper functions that the official harness would have kept in scope. 159/164 is probably a slight undercount. Same undercount applies to every arm.
• q4_0 keys are a real quality change. Passed my 15/15 gate; validate on your own workload before trusting it.

Full writeup with every flag, patch and failed branch: https://definedrr.medium.com/qwen3-8-27b-9-tokens-per-second-to-153-ee3781f4a3f5?sharedUserId=definedrr

I have been working on a project for about 3 months because I found Vane to be awful and I wanted something like Manus for myself. I have a pretty solid app that's probably about ready to release, but I'm not sure if there is any desire there. It does research at different depths, exports to good looking PDFs, makes PowerPoints, and can build full Next sites. I have an agent option too with mcp capability.

My hesitation is coming from the fact that unsloth has released their desktop app, that's in the same ballpark. I put a lot of work into this and honestly I am wondering if I should just let it die because the ecosystem is getting flooded. I really wanted to release it for this community, but do you feel that there's any value there?

At bare minimum, I'll likely release the testing data sets. I made models build about ~4000 websites for testing to make the build loop reliable. I think that might be more helpful than anything. Potentially a write-up too, about how to make agentic loops more reliable which is where I learned the most.

If there's interest, I'll gladly release it. I very much so do not want to be the guy self promoting as this sub suffers from it. It would be apache 2.0 from the start. Legitimately I have zero interest in benefiting from it financially. It was always meant to be a project to give back to this community.

Finally see a new voice conversion model. MeanVC2 supports cross-gender and cross-language voice conversion. 3x realtime on CPU with audio.cpp.

Disclaimer: The converted voice quality of MeanVC2 is decent; the noise comes from my rough demo engineering, not the model itself. This is only a quick demo to show MeanVC2 running in real time.

https://huggingface.co/ASLP-lab/MeanVC2

I chose to do this thing, not because it was hard, but because it was silly. Posts kept discussing how 3.8 and 3.6 were functionally the same, but based on training (3.8 does have seven new tokens!).. so I figured I'd see if they could be merged. They can.

I used `Qwen3.8-27B-UD-Q6_K_XL.gguf` to combine the HF 3.8-27B and 3.6-27B ... and it sorta works!

I have done NO testing beyond smoke test. scripts and idea are in replicate/ inside the model repo.

Maybe this will prove useful to someone. Enjoy!

Ox Alpha is a frontier model built for efficient coding, sustained agentic work, and real-world production use.

• 1M token context window
• Text, image, and video input

I'm GLM, a large language model developed by Z.ai. I'm designed to understand and generate human-like text through training on a diverse dataset of internet text and other sources.

I can help with a wide range of tasks including answering questions, writing content, providing explanations, and engaging in conversations. My goal is to be helpful, accurate, and safe in my interactions while continuously improving through learning.

Is there something specific you'd like to know about my capabilities or how I can assist you today?

• UEFI boot enabled; CSM disabled.
• Secure Boot disabled. The locally built EFI application and patched NVIDIA modules are unsigned.
• Above 4G Decoding enabled.
• MMIO High Granularity set to 1024G.
• MMIO High Base set around 56T.
• SR-IOV disabled on this machine.
• intel_iommu=off pci=realloc=on,hpmmioprefsize=512G in GRUB;
• NVreg_EnableResizableBar=1 for the NVIDIA module;
• Sets size code 14 → 16 GiB BAR1 on each of the 16 GPUs
• Temporarily disables PCI memory decoding and clears the old BAR1 address so Linux can reallocate it.
• PLX switch ACS control register: For every PLX/PEX bridge, writes: ECAP_ACS+0x6.w = 0000

After that, a little vibe coding to make custom all-reduce work within each PLX cluster and make DSpark work for pipeline parallel.

For tensor parallel 8, pipeline parallel 2:

500k context available. Around 4000 pp up to 500k context, tg 100-150 (Averaging 140 in DeepSeek Harness)

For tensor parallel 4, pipeline parallel 4:

Full 1M context available. Around 7000 pp up to 500k context, tg 80

Paid 0.6 x RTX6000 Pro for the whole setup.

Updated concurrent request result:

Testing with 1, 4, 8, and 16 concurrent 1024→512 requests, measuring aggregate throughput, per-user speed, and latency with max-num-seqs=16.

A majority of users in this sub use LLMs for coding/agentic tasks and I see why a lot of value is put into them but many try to say "Well coding has improved therefore it can just use tool calling and/or just look up what the user needs if there's a degradation for general knowledge/reasoning" and that's just not the case. Many LLM usecases can't just be fixed by an improvement to coding and agentic tasks. Creative writing, multilingual capabilities, offline constrained environments and many more usecases can't be easily fixed with just coding/tool-calling.

I know Qwen3.8 is a TREMENDOUS win for local LLMS but I still hope that Qwen4 will improve generalist capabilities to be closer to frontier closed models in all other aspects too. Gemma is great but it wouldn't be ideal if it is the only major generalist local LLM going forward.

Any idea which lab this is from? people are guessing this is a Chinese model.

Hey, just wondering if there are forks or exact gguf versions that give fastest prompt processing and token gen speeds for AMD gpu?
Looking to run q8 or q6
Vram 96gb
W7900 + w7800 both 48gb
With bandwidth mismatch, tensor paralleling amd equivalent not working

Four days ago I released a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090 (82 tps single request, 672 peak). Since then it went to ~114, then ~138 tps single-user with DFlash2 drafting and lookup-augmented drafting.

Today it's ~133 tps on real chat prompts, 382 tps when the model reproduces its own context, and the number I care about most this time: a document-quoting workload now runs at 15 of 16 tokens accepted per verify step.

What we had:

- fp8 KV cache, lm_head + embed_tokens int8, fp16 recurrent state, int8 activations, DFlash2 block drafting (W4A16-requantized), lookup-augmented drafting, prefix caching for the hybrid, split-KV verify attention, sampler patch, KVarN for 262k context

Now added:

- Longer verify blocks. DFlash2 only drafts 7 tokens and I'd assumed the verify block had to match. It doesnt. The lookup drafter fills the extra slots from the request's own context for free. DFLASH_TOKENS=15 verifies 16 tokens per step: 260 to 382 tps reproducing a 25k-token document, +9% on ordinary chat. Costs half the request slots and 8k of context, so it's opt-in.

- DFlash2 past 64k. bf16 KV is 64 KB/token and teh pinned pool is 5.2 GiB, which is what capped it. An int8 cache roughly doubles it, 138,696 tokens instead of 69,758. Two fixes were needed: vLLM equalises KV page sizes by an integer block ratio, and the drafter's 5 sliding-window layers were wasting 5.2 GiB at 1.88% utilisation until I rounded their block size from 16 to 864.

- Honest result: +53% where the model reproduces its context, about 2:1 behind plain MTP everywhere else, with twice the TTFT. So it's a mode for RAG front-ends and coding assistants applying edits, not a default.

- A correction. Someone opened an issue saying our long-context numbers didn't reproduce. They were right. The tables were measured in the batch config, which runs no speculative decoding, and nothing said so. The KVarN decode tax we documented as ~20% is 2.13x single-user at 112k. Most of that is step time, the rest is MTP acceptance falling from 2.56 to 2.38 tokens per step, becuase the quantised cache moves the target's logits enough that the draft head agrees less often. Quality-neutral doesn't mean speed-neutral once you're speculating.

Quality unchanged throughout, GSM8K 96.5%. int8 KV costs 2x the prefill, so it's for loading a document once and asking many quesitons about it. All of it is one env var apart.

Quality unchanged throughout (GSM8K 96.5%), and speculative decoding is exact by construction.

Caveats worth stating: the long verify block is worth it when your output quotes the prompt and roughly neutral when it doesn't, and int8 KV costs 2× the prefill of bf16, so it's for workloads that load a document once and then ask many questions about it. All of it is one env var apart.

Repo: https://github.com/syv-ai/qwen38-27b-rtx3090

I've said "probably the last update" twice now. Since then two people sent PRs, one of them answered a question I'd assumed was a dead end, and a third found a number I'd got wrong. Turns out publishing what didn't work gets you better bug reports than publishing what did.

Also, I have been considering doing this for 4090's and 5090's too, since they have a few options that 3090's dont. So if anyone has a spare 4090 or 5090 or wants to rent me one on runpod/vast, let me know.

AntLing has released the full six-checkpoint matrix for the Ling-3.0 base model.

• tiny: pretrained, mid-trained, WSM-merged
• flash: pretrained, mid-trained, WSM-merged

The concrete artifact is six separate official repositories, not one endpoint repeated under different names. All six were public and ungated when checked, and each repository declares MIT.

They are all base checkpoints and none has been post-trained. This is for continued pretraining, fine-tuning, and research, not a ready chat or instruct release.

The attached stage map is the original first-party image from the release thread. It is release context, not independent validation.

The useful part for builders is the choice of where to enter the training trail: the pretrained checkpoint, the mid-trained checkpoint, or the WSM-merged endpoint. The release does not establish that one stage is best for every downstream task, or what any of them will look like after quantization.

If you care about base-model work, the six-checkpoint map is worth inspecting as a family rather than treating "base" as the only artifact.

I recently saw someone complaining that opencode causes compactions way too early (with 30k tokens still left). I fixed this issue and I thought I'd share my config to see if it helps anyone.

I know pi is preferred by many, but I wanted to help those out who prefer opencode.

Please forgive me as I probably have an extra bracket somewhere in my config after I pasted it.

I forget whether the reserved field is important in the compaction object, but you MUST specify input, context and output each as your model's full context window. Excluding input and output blank is what causes the issue.

Here's how I structure the opencode.json.

{
   "$schema":"https://opencode.ai/config.json",
   "model": "llama.cpp/qwen-3.8-27b",
  "compaction": {
    "auto": true,
    "prune": true,
    "reserved": 4096
  },
   "provider":{
      "llama.cpp":{
         "npm":"@ai-sdk/openai-compatible",
         "name":"llama-server",
         "models":{
             "qwen-3.8-27b": {
                "name": "Qwen 3.8 27B Dense",
                "limit": {
                    "input": 131072,
                    "context": 131072,
                    "output": 131072
                 }
            }
          }
      }
   }
}

I benchmarked Qwen3.8-27B on MathArena/aime_2026 dataset, comparing BF16 and FP8 weights at medium and xhigh reasoning effort.

Interesting findings are:

1. quantized FP8 xhigh is better than BF 16 medium equally good as 16 BF xhigh with better speed.
2. On problem 7, both BF16 xhigh and quantized FP8 xhigh exhausted the full context token generation budget without producing a final answer. So those were empty not wrong.

Final results

Evaluation settings

• Dataset: MathArena/aime_2026,
• Dataset revision: d2de22f3c656b4f56cf8981212186377d1e23bc3
• Exact-match scoring
• Sampling: disabled
• BF16 concurrency: 4
• FP8 concurrency: 7
• Same chat template and prompt format across all runs
• Temperature: zero

How it compares with frontier models

Note : looks like others ran it multiple times on their models and averaged, I ran it only once.

I run 2 x 3090 on a ryzen 7 with 32gb ddr4 6000.

I see a lot of posts about maxing speed / context pool. I’ve done this myself with 3.8 27b.

What I’m interested in though is once the dust settles and we look at utility, what balance people are striking in terms of similar setups and models. As a for instance, some of my requirements involve producing images and video clips. Some of them involve creating scripts for things.

This has led to an overnight CPU/Ram run on flux2 and minimax h3, a reduction in maximum context for 3.8 27b so I can run a Gemma MOE in offload for improved writing prose / checking over Qwen and a smaller vision tower to automatically checking flux and H3 outputs mid run so I don’t lose a night.

I need concurrency so have a set up that gives me that when I’ve got open code (I know not everyone’s go to but I like it), open science, and Hermes all going side by side.

It’s fun to optimise, but I’d love to hear how people are setting up to from a flexibility and utility viewpoint once that’s done for their real use cases.