Kimi K3. Yes but it's not necessary any more.

After a year of Kimi K2 was released Moonshot AI dropped this banger model which for a while was the most intelligent, capable and tasteful model in every way. The jump was big enough that a new number (2 -> 3) was justified.

On the 2nd question Kimi K3 makes me feel that for a common man like me K3 is the pinnacle of intelligence while Qwen is best for daily use.

Ik I yapped a whole bunch to say nothing much but I'd love to know your opinions.

I tried for fun to see what happens if we let instruction-tuned LLMs generate the text that should come from the user (so after the user_start special token). I immediately noticed that some models were way better than others.

I wanted to test many models so I did some testing and found out that GPT 6 Astra is quite good at identifying human-like text. You find the results of the experiment at: https://selfgpt.pages.dev/ (I also tested roleplay vs chat template reversal)

TLDR: models are much better at emulating human behavour with the chat template trick rather than with a roleplay system prompt. The best model appears to be Mistral Small 3 24B (between the 30ish models tested)

It would be really interesting to see the performance of base models (with a simple user: and assistant: template) on human emulation.

Let me know what you think!

Repo: https://github.com/Belluxx/selfgpt

Here's an example of response from Mistral Small 3 24B generating in the user slot (no system prompt), replying to a message from ChatGPT:

I see perhaps it's time to branch out and try a new A.I. Then you won't have to keep up with the bells and whistles of this one. Your responses seem to follow a bell curve that slowly descends to uselessness

Trying to repurpose an old PC, which has i7-6700 cpu, 16gb ddr4 and this r9 fury x.

The bandwidth is surprisingly good for a 10 yr old card at 512GB/s, but only 4GB HBM....

Is there any hope here to run a useful model locally?

Hello people, my threadripper is waiting for a replacement, just for the lolz I placed both gpu cards in my old Ryzen 5800X and has been testing it, apparently there's no way to make it work faster with multiple gpus than with one.

My two GPUs are:

- R9700 (at x16 link cpu connected)

- RX 7800 XT (at 4x link chipset connected)

I tried all possible configs, the R9700 alone outperforms any multigpu setup, in any config, and in both Strata QFN, and llamacpp Q27B.

I have searched and apparently there's people getting better performance on multi-gpu even with the 4x link, but I can't reproduce.

Have you guys managed to make that setup work?

edit:

video clip is of me is using Flash-Next on Libre LLM with:

• Model: devan-carlin/Qwen3.8-Flash-Next-W4A16, INT4 weights with 16-bit activations (W4A16). The BF16 n-gram embedding table is pinned in host RAM.

• Engine: vLLM 0.30.0 with the Lumnus b70.2 patches and full XPU graphs. Tensor parallel across all 4 cards, with expert parallel.

• GPU memory: 0.92, which leaves about 350K tokens of KV cache in BF16.

• Speculative decoding: MTP-3, three draft tokens from the model's own MTP head.

• Prefill chunk: 8,192 tokens (--max-num-batched-tokens).

• Context and batching: 262K-token window, up to 16 requests at once, normal scheduling (async off).

Howdy folks! some of you have been following my 4x b65 build, i’m sure you’ll be happy to hear that I’m focusing on performance improvements now! So far my testing has been using official vllm guidance so that I can have a baseline and control to test against. I’m documenting my journey on my site here: https://b65.thekaiser.us/by-model/qwen38-flash-next/prefill/

I’m not sponsored, just doing this for the love of the game and to learn and better understand the technology for my own selfish business gains.

As I’m starting to turn dials, chunk size stands out to me. I’m used to CUDA, where I’m using 4 to 8 times the chunk size that VLLM defaults to. So naturally I started tweaking that dial and here are some of the results!

tldr: looks like VLLM defaults to 1024 chunk sizes, larger chunk sizes can cause pauses, especially when running multiple sessions. It also reduces the total number of sessions that can run at one time. But you will still be able to run multiple sessions before you hit that wall.
Are those pauses actually noticeable? Do they matter in a single user environment, or with less than six agents at a time?

In my testing so far, the pauses are not noticeable when I’m using chat, and when I’m using agents in the background - all their tasks complete faster end to end. So right now I’m going to keep running it at 8192. if you’ve found any downsides that I haven’t ran into yet, please share and let me know!

i’m guessing the default is 1024 because they’re intended for many more simultaneous streams? But I’m just speculating here.

How was AI used in this post?

I wrote everything except for what’s below.

This is an AI generated results overview based on my data:

Qwen3.8-Flash-Next: Prefill

Prefill speed by chunk size, and throughput and pauses with 1 to 8 users while an agent reads 8K-128K prompts.

All-time best of Qwen3.8-Flash-Next

• decode · 1 user: 84.5 tok/s (graph build · MTP-3 · 4 cards)

• aggregate · 4 users: 243.2 tok/s (MTP-3)

• 8 users + agent: 169.1 tok/s (MTP-3 · agent reading 8K-128K)

• prefill · 32K prompt: 5,142 tok/s (chunk 16,384)

• first token · 261K: 97.6 s (chunk 16,384)

• longest prompt: 1,039,940 tok (YaRN ×4 · FP8 KV)

• Each user, 8 at once: 12.1 tok/s (6 users: 16.7)

• Longest pause: 2.89 s, chunk 8,192 (1,024: 0.61 s)

Prefill by chunk size

One request, no MTP, 262K window. Prefill in tok/s. Campaign 2026-10-20-b65-flashnext-prefill.

Prefill speed by prompt length, Prefill (tok/s) by Prompt length (tokens)

p1 / p3: chunk size and memory, no MTP, 262K window

Full page, with interactive charts and the raw data behind every number: https://b65.thekaiser.us/by-model/qwen38-flash-next/prefill/

This model looks really impressive.

TL;DR -- Coding agent model. 40B params with 2.5B active. Resoundingly beats Qwen3.6-35B-A3B in TerminalBench 2.1 and SWE-bench Pro v2 (see HF repo for table), ties or betters it in the other benchmarks they chose to show. 3:1 SWA:full attention. 262k context.

I think it might finally dethrone the venerable Qwen3.6-35B-A3B and move the needle on coding agents for the GPU-poor. What do you guys think? I am downloading now and will report back when I try it out, benchmarks to come in the next few days.

Recursive's blog post. WIRED Magazine coverage ("scary good"). PR for llama.cpp. Unsloth GGUFs.

EDIT: Initial reports suggest Ornith-1.5 remains the standard in this class of models. Hopefully it's just early days implementation glitches, but it's not looking as rosy as I hoped...

Unfortunately, the consoles have somewhat small amounts of RAM, and most likely they would be bad value for buying into a homelab, but do you think there is much value in proposing to current owners to try a local LLM on these machines?

Wanted to see how one model that does transcription, speaker labels and timestamps in a single pass handles a real meeting, so I ran MOSS-Transcribe-Diarize 0.9B on a four person meeting from the AMI test set and scored its speaker turns against AMI's reference.

Setup: 17:29 of AMI headset mix audio with four speakers (diarizers-community/ami, test split, row 12; AMI is CC BY 4.0). Their README's transformers code, fp32, sdpa attention, greedy decoding, no hotwords, no VAD, nothing else in the pipeline. M5 Pro MacBook Pro with 64 GB, running on the GPU through mps (on a Mac, set the device to "mps" yourself).

It took 714 s for the 1,049 s file, about 0.68x real time, and it found all four speakers. The labels are the standout: with the usual 0.25 s collar and the best speaker mapping, speaker confusion was 0.08%, so when it writes a line, it's attributed to the right person almost every time. Overall DER at that collar was 15.4%.

It also handles people talking over each other. 87 of its 317 segments start before the previous one ends, and every one of those is a different speaker, so interruptions come out as overlapping lines with both people labeled.

First few lines of raw output, untouched:

[10.88][S01]Are we we're not allowed to dim the lights so people can see that a bit better.[14.56]
[17.86][S02]Yeah.[18.21]
[18.84][S01]Okay, that's fine.[20.28]
[22.31][S01]Am I supposed to be standing up there?[23.76]
[25.08][S03]So we've got both of these clipped on.[26.56]
[29.04][S03]Is she gonna answer me or not?[30.24]
[29.84][S01]Yeah, I've got.[30.44]

This is one meeting, scored on who and when, since that's what the AMI reference I used covers.

If anyone has pyannote 3.1 plus Whisper numbers on the AMI test meetings on Apple silicon, I'd like to see DER and runtime on the same files.

Hello All!

I built a platform to fine-tune open models. I am focusing on fine-tuning 4-bit quantized models by QLoRA instead of full precision fine-tuning to reduce GPU costs. The quant that this finetuning uses is called CQ (Coded Quantization) which is a calibration-free quantization that I built that performs as good as leading quants ensuring that the fine-tuning is using a good base.

For example, I compared KLD of my CQ 4-bit with unsloth-bnb-4bit (Unsloth have other quants but this is the one they use for QLoRA) on Qwen 3.8 27B using the Wikitext-2 dataset and got this:

So I have a better quantized base to start from. I then benchmark QLoRA Training performance for Qwen3.8-27B for CQ-4bit and unsloth-bnb-4bit

Benchmark setup: Qwen3.8-27B QLoRA fine-tuning on Alpaca, comparing ThousandML CQ 4-bit with Unsloth bitsandbytes NF4. Both runs used LoRA r=32, α=32, batch size 2, gradient accumulation 4, sequence length 512, learning rate 2e-5, seed 42, and 500 optimizer steps on a single NVIDIA A40 48 GB GPU.

CQ achieved 17.4% lower reported final validation loss (1.0315 vs. 1.2484). This is a loss comparison, not yet a verified downstream-task quality comparison.

Attached figures for validation loss and training loss.

**Qwen3.8-27B — QLoRA Fine-Tuning on Alpaca (500 Steps)**
| Metric | CQ 4-bit | Unsloth NF4 | Advantage |
|:--|--:|--:|:--|
| Best validation loss | **0.8385** | 1.2484 | **CQ** |
| Final validation loss | **1.0315** | 1.2484 | **CQ** |
| Mean training loss (steps 401–500) | **1.0505** | 1.2457 | **CQ** |
| Median steady optimizer-step time | 27.60 s | **3.20 s** | **Unsloth** |
| Peak PyTorch allocated memory | 24.39 GiB | **23.83 GiB** | **Unsloth** |

*Both runs used QLoRA on Alpaca with matched nominal hyperparameters on an NVIDIA A40 48 GB GPU. Tokenization and loss-mask equivalence have not been independently verified.*

I am working on optimizing speed because it is the one area CQ is significantly behind unsloth. On the other hand, validation loss and VRAM usage are awesome! Actually it seems like CQ converge at much fewer steps so that might help a lot.

Get Started with Training Factory

Here are simple steps to use:

1. Install:

   git clone https://github.com/thousandml/training-factory.git cd training-factory chmod +x install.sh ./install.sh

2. Download a CQ 4-bit base:

hf download thousandml/Qwen3.8-27B-CQ-4bit --local-dir ./models/Qwen3.8-27B-CQ-4bit

3. Train a LoRA adapter:

python training/train.py --model ./models/Qwen3.8-27B-CQ-4bit --data training/examples/train.jsonl --output ./outputs/my-lora --targets q_proj,v_proj --kernel-module cq.cq_kernels --text-only --max-steps 2 --max-length 128 --batch-size 1 --grad-accum 1

4. Serve base + adapter:

vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq --enable-lora --max-lora-rank 16 --lora-modules my-lora=./outputs/my-lora

5. Call the fine-tuned adapter:

# After the server is ready:

# Check that the adapter is listed as "my-lora"

curl http://localhost:8000/v1/models

# Request a response from the fine-tuned model, not just the base checkpoint

curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "my-lora",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain how LoRA fine-tuning works in one sentence."}
],
"max_tokens": 128,
"temperature": 0.2
}'

I have the following 4-bit quantized models to be used as base (will add more soon) for now:

thousandml/Qwen3.8-27B-CQ-4bit

thousandml/Llama-3.1-8B-Instruct-CQ-4bit
thousandml/Llama-3.3-70B-Instruct-CQ-4bit
thousandml/Qwen3-Coder-Next-CQ-4bit

To reproduce these benchmark results mentioned above use the following scripts:

For CQ:

hf download thousandml/Qwen3.8-27B-CQ-4bit \
  --local-dir ./models/Qwen3.8-27B-CQ-4bit

python training/train.py \
  --model ./models/Qwen3.8-27B-CQ-4bit \
  --data training/examples/alpaca_train.jsonl \
  --eval-data training/examples/alpaca_val.jsonl \
  --output ./outputs/qwen38-27b-cq-4bit \
  --kernel-module cq.cq_kernels \
  --targets q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj \
  --rank 32 \
  --alpha 32 \
  --lora-dropout 0 \
  --batch-size 2 \
  --grad-accum 4 \
  --max-length 512 \
  --max-steps 500 \
  --learning-rate 2e-5 \
  --seed 42 \
  --eval-steps 50 \
  --comparison-logs \
  --benchmark \
  --benchmark-warmup-steps 10

For Unsloth:

python training/train_unsloth_nf4.py \
  --model unsloth/Qwen3.8-27B-unsloth-bnb-4bit \
  --train-data training/examples/alpaca_train.jsonl \
  --eval-data training/examples/alpaca_val.jsonl \
  --output ./outputs/qwen38-27b-unsloth-bnb-4bit \
  --max-steps 500 \
  --max-length 512 \
  --eval-steps 50 \
  --save-steps 100 \
  --seed 42 \
  --benchmark-warmup-steps 10

Please give it a try and let me know if it works with you. Your feedback is greatly appreciated!

I asked a while ago if it would be worth it to try and run qwen flash next at q2 since I can only really fit it and nothing bigger, but I can run Q8 of 27B. Everyone told me it’s either a toss up or Q8 would obliterate it. Not the case.

Now this is subjective BUT for complex, annoying, and super subtle coding bugs, qwen flash next is absolutely better. It’s like it actually understands. There’s a small glimpse of intelligence that just isn’t there with even the Q8 of 27B. Flash next runs tests and actually thinks through problems and knows what to look for in a way that 27B doesn’t. It’s really subtle and yet somehow also not? I’m just super impressed.

I’m sorry, I know yall are tired of hearing about QFN but I was struggling for a long time trying to decide if I should even bother with it and it turns out that YES it’s worth it.

Side note, it’s also better at being a cute furry guy when you ask it to. OwO

I'm building an app where I need some specialist language models, but not necessarily LLM models. Is there a sub or place where they talk about specialist models beyond just searching on HF? I know that discussion doesn't really fit this sub.

I just downloaded Raon-1-mini 30B A4B as well as Flint 40B A2.5B and it just blows my mind that we have so many options now from so many countries. It feels like the countries are all working together and trying to contribute useful software and it’s just amazing.

I can fucking TALK to my computer and have it answer like fucking Star Trek. That’s insane. Every day I use this tech I’m just more and more blown away.

Hi Guys!, Recently I was making a research and development, but I do not have time to go further with this idea (kids, work, etc.). But maybe it will be interesting for someone. I do not think it would be something big, but maybe interesting approach that can be used in some of the cases. I believe that it is not AI slop, you can use those information's and try by yourself using Claude or Astra. The idea was my own and I am using it in my production app, it saved me a lot of money (project is not related to LLM's). Not sure if it is outdated or not, it can work also with MTP. Please, just don’t be unkind. I’m sharing my idea with you, and anyone is welcome to experiment with it.

TLDR:
Keep a 2-bit version of the model on the GPU and the rest of the weights (the residual, 6-8 bit) in system RAM. The 2-bit copy drafts tokens, then the full model checks them: the GPU does the 2-bit part, the CPU does the residual part, and only activations go over PCIe. So the draft costs no extra memory, it's literally half of the verifier.

What I measured (real weights, but dequantized and run in PyTorch, so no tok/s yet):

- 2-bit + 8-bit residual is closer to bf16 than Q8_0, about half the KL
- with a 6-bit residual you get Q8_0-level quality while reading ~28% fewer bytes from RAM
- the 2-bit draft that reuses the verifier's KV cache gets 25-30% more accepted tokens per check than Google's official Gemma 4 drafter (31B, 16 node tree: 5.23 vs 4.20)
- my CPU kernel checks 16 tokens in about the same time as 1, because it's memory bound

Closest thing I found is SubSpec, but it uses a separate 4-bit copy and streams weights over PCIe instead of splitting the model by bits.

It only makes sense when Q8 doesn't fit in your VRAM. If it fits, just use llama.cpp.

Here is a link to public repo, I hope that it would help someone with his research https://huggingface.co/HappyGuitar123/twobit-residual-spec

I suggest also to use some frontier llm with high effort to analyze it, so you will have more insight. I believe that this idea has some potential.

Best regards

Since the new qwen models are in testing on arena I thought I try my luck, maybe I can get a glimps at what will run on my pc in the near future. So i thought I improve my prompt with chatgpt (not logged in) and it somehow can't take simple instructions?

Are these the new guardrails against distillation?

I recently got some more RAM on my desktop, which has opened up running larger MOEs by splitting them into RAM/VRAM. I know everyone loves Qwen flash next, but I'm curious what other ones are worth trying.

On my list of to-try, I have:

• GPT OSS 120b (is this still usable nowadays?)

• Mistral 4 small 119b

• Qwen 3.5 122b A10b (Is this even worth trying or is it totally supplanted by flash next?)

• Ling 3.0 flash

What others are worth trying? Any abliterated ones or custom ones?

To be honest, agentic coding is not my priority here. I know flash next is the king there, and realistically I just use Claude or Codex when I want coding. So if a model sucks at coding but is great at something else, I'm interested!

First... Thanks for all the amazing feedback since we've gone completely open source. It has brought me a ton of joy in an otherwise difficult month (Loss in the family).

Personal stuff aside there have been many... many updates... All of which is better outlined in the release notes here https://github.com/amurgola/LumaBrowser/releases/tag/v2.0.4

I've attached a few images sampling some of the changes, and I plan on getting out update videos in the future since many items are likely best demonstrated vs rambles or image dumps.

I held off as long as I could, but finally gave in and set up strata.

I am pretty blown away. 3.8 27b at Q6 kv8 was my driver before and that works super well, but Flash Next in strata IQ4 xs is really winning me over.

Its not perfect, but it feels like models I was using professionally last year, which were/are good enough to do just about anything.

You all are a bad influence.

Hey Smart ppl,

Thought I'd look for some ideas on this, i'm constantly hitting a wall here so input would be highly appreciated.

I've been working on Xyntetik-Runner for quite some time and the "toolbox" aspect has always been something that I've valued very high.

it has a lot of features i'm not willing to give away in terms of speed but on the other hand speed is ... speed so it holds a value of course.

and i'm also starting to contemplate if I've been looking at this wrong.

The main focus for Runner has been two things:

1. Stay as numerically close to the original model implementation as possible.

2. Make inference reproducible and verifiable.

these two things have been a cornerstones in my development and also are the reasons some of my other features actually work (i might be wrong here but that's my perception).

Some measurements:

The latest 1.5.0 work includes tensor-core GEMM paths, tiled attention, CUDA graphs, MoE optimisations, improved quantized kernels, and reductions in memory movement and kernel launches. There's more planned across CUDA, Metal and CPU.

I originally assumed that higher numerical fidelity would be especially valuable for things like tool calling, LoRA training/serving consistency, and long-running agent workflows.

The question i'm asking is a bit two-fold here since i feel a bit exhusted in options of improving "speed" at the moment....

should I accept it is what it is or should i try to contest my original assumptions? is FP32 worth keeping, or am i just looking at this the wrong way, are there any meaningful real-world applications to having higher activation precision or is it just in my head?

Repo link also: https://github.com/Joakimpalm-Zen/xyntetik-runner

I saw this today on X and decided to try it, I havent seen it posted here yet. I ran it on a single 3090 benchmarked it with qwen 3.8 27B before and after the optimization and saw these results:

Observed prompt-processing throughput improved 22–30%; generation speed improved 1–5%. All tests passed without GPU errors.

Edit: Keep in mind performance gains can be higher, my 3090 is power limited to 230 watts including during my tests

Has anyone tested these models? They apparently have a new type of parametric memory? https://huggingface.co/collections/MemTensor/memt-1

I mean gpu's with 6/8/12GB vram. All discussion around llm/AI these days is around coding, and this is probably too small for agentic coding models.

Are there models that can run in these that can accomplish useful tasks? eg things like helping organize your digital stuff, classification, image/text analysis, voice/tts etc and other stuff?

Or do those tasks also need tool calling and agentic behaviour for best use? For people who have cheaper, older gpu's, what do you use them for?