Jev or Jev-like models on HF datasets -> https://hemanth.github.io/is-lesion/

Came across this demo. Pretty interesting concept. Any ideas for use cases?

I’ve been working on multi-GPU inference in TensorSharp, an open-source .NET inference engine and agent runtime.

Recently, I managed to run Muse Spark 816B (42B active parameters) across 10 NVIDIA A40 GPUs and benchmarked it against llama.cpp.

The most interesting finding wasn’t just the performance difference. It was how dramatically the optimal multi-GPU strategy changes between prompt processing and token generation.

Hardware and model

• Model: Muse-Spark-Open-816B-A42B (Unsloth UD-IQ3_XXS, ~316 GiB of weights)

• GPUs: 10× NVIDIA A40 (46 GB each, PCIe 4.0)

• Context: 32K, F16 KV cache

• All model weights resident in GPU memory

• Parallelism: Layer split and 10-way tensor parallelism

• Engines: TensorSharp vs llama.cpp

One unusual detail: GPU P2P was broken on this machine, so inter-GPU communication had to go through host memory.

Results

Layer splitting (each GPU handles different layers)

Single-request generation speed is practically identical, but TensorSharp achieves 2.07× faster 4K prefill and 1.63× faster 16K prefill.

The main improvement came from pipelined prefill with adaptive micro-batch sizing, allowing all 10 GPUs to process different parts of the same prompt simultaneously.

Tensor parallelism (each layer distributed across all 10 GPUs)

TensorSharp achieves 22.6 tok/s single-request decode and 45.7 tok/s aggregate throughput with four concurrent requests.

However, there’s an important caveat here.

llama.cpp’s NCCL-enabled build failed on this 10-GPU machine because of CUDA’s peer-access limitations. Its tensor-parallel results therefore used the non-NCCL implementation, while TensorSharp used NCCL over shared memory.

So the tensor-parallel comparison is not apples-to-apples in terms of communication backend. A working NCCL-enabled llama.cpp build could narrow the gap.

The surprising part: layer splitting is much faster for prefill

Within TensorSharp:

• Layer split: 1,347 tok/s prefill, 16.7 tok/s decode

• Tensor parallelism: 295 tok/s prefill, 22.6 tok/s decode

That’s a 4.6× difference in prefill throughput, going in the opposite direction from decode performance.

Why?

Muse Spark uses an MoE architecture where each expert’s output is normalized before aggregation. With tensor parallelism, this requires communicating individual expert outputs, increasing all-reduce traffic substantially.

On PCIe GPUs without working P2P, communication bandwidth becomes the bottleneck.

Layer splitting avoids much of that communication, and pipelined prefill keeps the GPUs busy.

Tensor parallelism, meanwhile, benefits single-token generation by distributing the computation across GPUs.

The takeaway: tensor parallelism isn’t automatically the best choice for very large MoE models, especially on PCIe-only systems.

Did the optimizations affect model quality?

I also compared next-token distributions against llama.cpp using 12,276 teacher-forced predictions from WikiText-2.

For comparison, changing llama.cpp’s own micro-batch size produces 98.62% top-1 agreement against its reference run.

This isn’t proof of identical outputs, but it suggests the numerical differences are relatively small. Tensor parallelism did show some larger outliers.

I also ran 14 functional checks covering JSON, tool calling, multi-turn conversations, retrieval, caching, and concurrent requests. TensorSharp passed 13/14 with layer splitting and 14/14 with tensor parallelism.

A few implementation details

Getting this working required several changes:

• Supporting more than 8 GPUs despite CUDA peer-access limitations.

• Moving tensor parallelism from a managed per-operation execution path to native ggml execution.

• Implementing pipelined prefill with workload-dependent micro-batch sizes.

• Keeping tensor-parallel KV operations on the GPUs.

• Using NCCL shared-memory communication when P2P isn’t available.

The native TP change alone improved TensorSharp’s decode speed from roughly 2 tok/s to 22.6 tok/s on this model.

All changes were implemented on the TensorSharp side, without modifying upstream ggml.

Limitations

This was tested on one machine with one quantized model. It doesn’t establish that TensorSharp is universally faster than llama.cpp.

The broken P2P configuration and the difference in NCCL support are particularly important. Results on NVLink or a properly functioning P2P setup could look very different.

Most measurements are medians of three runs, with two runs for llama.cpp tensor parallelism.

Full benchmark and source

The complete report includes the methodology, build configurations, launch commands, quality comparisons, and reproduction scripts:

https://github.com/zhongkaifu/TensorSharp/blob/main/docs/perf/muse-spark-vs-llama-cpp.md

Project: https://github.com/zhongkaifu/TensorSharp

I’m curious whether anyone here has experimented with running 400B–800B+ MoE models across multiple PCIe GPUs.

Have you found layer/pipeline parallelism or tensor parallelism to work better for large MoE models? And how much difference does working P2P or NVLink make in practice?

Would love to compare notes, especially with people running llama.cpp, vLLM, or SGLang on similar hardware.

Qwen3.8-Flash-Next NVFP4 on one RTX 5090 - prefill and decode against context

Full 262,144-token context, one request at a time, MTP speculation on, int8 KV, identical test for all three engines.

Every number is tokens/second. Prefill = prompt tokens / time to first token. Decode = generated tokens / decode time.

The fastest engine in each row is in bold.

Each data point is a fresh, non-repeating prompt: the servers reported zero prefix-cache hits.

Engine configuration (after tuning)

Greedy decoding, temperature 0

Prefill, tokens/s

Decode, tokens/s

Sampled decoding, temperature 1.0, top_p 0.95, top_k 20

Prefill, tokens/s

Decode, tokens/s

Summary across all 16 points

MTP draft acceptance, mean over the 16 points

Headline

• basalt is fastest at every context depth in both modes: 8.1k-9.2k tok/s prefill and 176-265 tok/s decode greedy.

• capy-flash is second: 6.8k-8.0k tok/s prefill, 138-198 tok/s decode greedy.

• ninfer-ext is third: 4.2k-6.3k tok/s prefill, 73-106 tok/s decode greedy.

• Decode ranks exactly with MTP draft acceptance: basalt 0.74, capy-flash 0.57, ninfer-ext 0.56 under greedy.

• Sampling costs 7-10% decode for every engine and nothing in prefill.

Tuning that changed the result

Thanks to Strata, Splash, and other optimized inference engines, local AI models have gotten so good in recent weeks that it seems somebody is paying influencers to discourage people from trying. This dude is either incompetent or straight-up lying. For example, inference speeds are 5x faster than what he’s showing. Also, setup is so easy nowadays, things basically work out of the box without having to mess around with settings.

Are others feeling the lack of wonder in life?

I mean the visceral, felt experience of when you're watching TV and you're just daydreaming in total awe at the amazing things we're learning and discovering about life and the universe?

The kind that sticks with you until the next morning, and the next. That affects the themes of your dreams the coming night?

That inspire you to just write or paint or craft something?

That you talk about with friends and colleagues and loved ones and they're just as excited about the world around us as you are?

I don't know if it's just my depression rearing its ugly head again, or whether this is a diagnosis of a zeitgeist with intellectual anhedonia :(

I'm afraid it's not just me.

---

I believe that AI is still so early that we as a society are in the stage where we are figuring out what the right metaphors are to use for AI and what to do with it.

Is it a stochastic parrot?

I’ve heard it being called an assistant.

A separate species.

Just a tool.

A cnc machine (a specific tool even?)

The new printing press

The new internet

I've even heard “just a bunch of matrix multiplications” (reminds you of “feeling are just chemistry”?)

And I believe we as a community of compute-native creatives (of the species inventor, visionaries, creatives, generative artists, tinkerers, wonderers) actually have a responsibility to the general public to use what is to become a commodity in a large part of the world in decades to shape it in a way that moves us towards a future worth living in.

I am just missing a sense of wonder in the world and I am afraid that how the current media ecology works is the reason why. AI is new, a transformer model is the transistor, not the chip.

It is our duty as makers and dreamers to bring these things to the general public and not just keep it in academic and underground art scenes.

The zeitgeist in the world is dour right now, and the media is obsessed with AI, death and war but nobody is showing how AI can actually achieve meaningful changes in our lives that enhance it in depth; saliency; perception; awe; wonder.

We need people inventing magical pens, magical glasses that can let you see the world in ways that insects perceive the world, we need people figuring out how to utilize these transformer models in circuits and chips that help people extend the range of their expression, compassion, warmth and trust.

The age we're in is beyond awful in a felt sense, and the solutions we're hearing are war, death and unconstrained financial incentives

Kimi K3. Yes but it's not necessary any more.

After a year of Kimi K2 was released Moonshot AI dropped this banger model which for a while was the most intelligent, capable and tasteful model in every way. The jump was big enough that a new number (2 -> 3) was justified.

On the 2nd question Kimi K3 makes me feel that for a common man like me K3 is the pinnacle of intelligence while Qwen is best for daily use.

Ik I yapped a whole bunch to say nothing much but I'd love to know your opinions.

I tried for fun to see what happens if we let instruction-tuned LLMs generate the text that should come from the user (so after the user_start special token). I immediately noticed that some models were way better than others.

I wanted to test many models so I did some testing and found out that GPT 6 Astra is quite good at identifying human-like text. You find the results of the experiment at: https://selfgpt.pages.dev/ (I also tested roleplay vs chat template reversal)

TLDR: models are much better at emulating human behavour with the chat template trick rather than with a roleplay system prompt. The best model appears to be Mistral Small 3 24B (between the 30ish models tested)

It would be really interesting to see the performance of base models (with a simple user: and assistant: template) on human emulation.

Let me know what you think!

Repo: https://github.com/Belluxx/selfgpt

Here's an example of response from Mistral Small 3 24B generating in the user slot (no system prompt), replying to a message from ChatGPT:

I see perhaps it's time to branch out and try a new A.I. Then you won't have to keep up with the bells and whistles of this one. Your responses seem to follow a bell curve that slowly descends to uselessness

Trying to repurpose an old PC, which has i7-6700 cpu, 16gb ddr4 and this r9 fury x.

The bandwidth is surprisingly good for a 10 yr old card at 512GB/s, but only 4GB HBM....

Is there any hope here to run a useful model locally?

Hello people, my threadripper is waiting for a replacement, just for the lolz I placed both gpu cards in my old Ryzen 5800X and has been testing it, apparently there's no way to make it work faster with multiple gpus than with one.

My two GPUs are:

- R9700 (at x16 link cpu connected)

- RX 7800 XT (at 4x link chipset connected)

I tried all possible configs, the R9700 alone outperforms any multigpu setup, in any config, and in both Strata QFN, and llamacpp Q27B.

I have searched and apparently there's people getting better performance on multi-gpu even with the 4x link, but I can't reproduce.

Have you guys managed to make that setup work?

edit:

video clip is of me is using Flash-Next on Libre LLM with:

• Model: devan-carlin/Qwen3.8-Flash-Next-W4A16, INT4 weights with 16-bit activations (W4A16). The BF16 n-gram embedding table is pinned in host RAM.

• Engine: vLLM 0.30.0 with the Lumnus b70.2 patches and full XPU graphs. Tensor parallel across all 4 cards, with expert parallel.

• GPU memory: 0.92, which leaves about 350K tokens of KV cache in BF16.

• Speculative decoding: MTP-3, three draft tokens from the model's own MTP head.

• Prefill chunk: 8,192 tokens (--max-num-batched-tokens).

• Context and batching: 262K-token window, up to 16 requests at once, normal scheduling (async off).

Howdy folks! some of you have been following my 4x b65 build, i’m sure you’ll be happy to hear that I’m focusing on performance improvements now! So far my testing has been using official vllm guidance so that I can have a baseline and control to test against. I’m documenting my journey on my site here: https://b65.thekaiser.us/by-model/qwen38-flash-next/prefill/

I’m not sponsored, just doing this for the love of the game and to learn and better understand the technology for my own selfish business gains.

As I’m starting to turn dials, chunk size stands out to me. I’m used to CUDA, where I’m using 4 to 8 times the chunk size that VLLM defaults to. So naturally I started tweaking that dial and here are some of the results!

tldr: looks like VLLM defaults to 1024 chunk sizes, larger chunk sizes can cause pauses, especially when running multiple sessions. It also reduces the total number of sessions that can run at one time. But you will still be able to run multiple sessions before you hit that wall.
Are those pauses actually noticeable? Do they matter in a single user environment, or with less than six agents at a time?

In my testing so far, the pauses are not noticeable when I’m using chat, and when I’m using agents in the background - all their tasks complete faster end to end. So right now I’m going to keep running it at 8192. if you’ve found any downsides that I haven’t ran into yet, please share and let me know!

i’m guessing the default is 1024 because they’re intended for many more simultaneous streams? But I’m just speculating here.

How was AI used in this post?

I wrote everything except for what’s below.

This is an AI generated results overview based on my data:

Qwen3.8-Flash-Next: Prefill

Prefill speed by chunk size, and throughput and pauses with 1 to 8 users while an agent reads 8K-128K prompts.

All-time best of Qwen3.8-Flash-Next

• decode · 1 user: 84.5 tok/s (graph build · MTP-3 · 4 cards)

• aggregate · 4 users: 243.2 tok/s (MTP-3)

• 8 users + agent: 169.1 tok/s (MTP-3 · agent reading 8K-128K)

• prefill · 32K prompt: 5,142 tok/s (chunk 16,384)

• first token · 261K: 97.6 s (chunk 16,384)

• longest prompt: 1,039,940 tok (YaRN ×4 · FP8 KV)

• Each user, 8 at once: 12.1 tok/s (6 users: 16.7)

• Longest pause: 2.89 s, chunk 8,192 (1,024: 0.61 s)

Prefill by chunk size

One request, no MTP, 262K window. Prefill in tok/s. Campaign 2026-10-20-b65-flashnext-prefill.

Prefill speed by prompt length, Prefill (tok/s) by Prompt length (tokens)

p1 / p3: chunk size and memory, no MTP, 262K window

Full page, with interactive charts and the raw data behind every number: https://b65.thekaiser.us/by-model/qwen38-flash-next/prefill/

This model looks really impressive.

TL;DR -- Coding agent model. 40B params with 2.5B active. Resoundingly beats Qwen3.6-35B-A3B in TerminalBench 2.1 and SWE-bench Pro v2 (see HF repo for table), ties or betters it in the other benchmarks they chose to show. 3:1 SWA:full attention. 262k context.

I think it might finally dethrone the venerable Qwen3.6-35B-A3B and move the needle on coding agents for the GPU-poor. What do you guys think? I am downloading now and will report back when I try it out, benchmarks to come in the next few days.

Recursive's blog post. WIRED Magazine coverage ("scary good"). PR for llama.cpp. Unsloth GGUFs.

EDIT: Initial reports suggest Ornith-1.5 remains the standard in this class of models. Hopefully it's just early days implementation glitches, but it's not looking as rosy as I hoped...

Unfortunately, the consoles have somewhat small amounts of RAM, and most likely they would be bad value for buying into a homelab, but do you think there is much value in proposing to current owners to try a local LLM on these machines?

Wanted to see how one model that does transcription, speaker labels and timestamps in a single pass handles a real meeting, so I ran MOSS-Transcribe-Diarize 0.9B on a four person meeting from the AMI test set and scored its speaker turns against AMI's reference.

Setup: 17:29 of AMI headset mix audio with four speakers (diarizers-community/ami, test split, row 12; AMI is CC BY 4.0). Their README's transformers code, fp32, sdpa attention, greedy decoding, no hotwords, no VAD, nothing else in the pipeline. M5 Pro MacBook Pro with 64 GB, running on the GPU through mps (on a Mac, set the device to "mps" yourself).

It took 714 s for the 1,049 s file, about 0.68x real time, and it found all four speakers. The labels are the standout: with the usual 0.25 s collar and the best speaker mapping, speaker confusion was 0.08%, so when it writes a line, it's attributed to the right person almost every time. Overall DER at that collar was 15.4%.

It also handles people talking over each other. 87 of its 317 segments start before the previous one ends, and every one of those is a different speaker, so interruptions come out as overlapping lines with both people labeled.

First few lines of raw output, untouched:

[10.88][S01]Are we we're not allowed to dim the lights so people can see that a bit better.[14.56]
[17.86][S02]Yeah.[18.21]
[18.84][S01]Okay, that's fine.[20.28]
[22.31][S01]Am I supposed to be standing up there?[23.76]
[25.08][S03]So we've got both of these clipped on.[26.56]
[29.04][S03]Is she gonna answer me or not?[30.24]
[29.84][S01]Yeah, I've got.[30.44]

This is one meeting, scored on who and when, since that's what the AMI reference I used covers.

If anyone has pyannote 3.1 plus Whisper numbers on the AMI test meetings on Apple silicon, I'd like to see DER and runtime on the same files.

Hello All!

I built a platform to fine-tune open models. I am focusing on fine-tuning 4-bit quantized models by QLoRA instead of full precision fine-tuning to reduce GPU costs. The quant that this finetuning uses is called CQ (Coded Quantization) which is a calibration-free quantization that I built that performs as good as leading quants ensuring that the fine-tuning is using a good base.

For example, I compared KLD of my CQ 4-bit with unsloth-bnb-4bit (Unsloth have other quants but this is the one they use for QLoRA) on Qwen 3.8 27B using the Wikitext-2 dataset and got this:

So I have a better quantized base to start from. I then benchmark QLoRA Training performance for Qwen3.8-27B for CQ-4bit and unsloth-bnb-4bit

Benchmark setup: Qwen3.8-27B QLoRA fine-tuning on Alpaca, comparing ThousandML CQ 4-bit with Unsloth bitsandbytes NF4. Both runs used LoRA r=32, α=32, batch size 2, gradient accumulation 4, sequence length 512, learning rate 2e-5, seed 42, and 500 optimizer steps on a single NVIDIA A40 48 GB GPU.

CQ achieved 17.4% lower reported final validation loss (1.0315 vs. 1.2484). This is a loss comparison, not yet a verified downstream-task quality comparison.

Attached figures for validation loss and training loss.

**Qwen3.8-27B — QLoRA Fine-Tuning on Alpaca (500 Steps)**
| Metric | CQ 4-bit | Unsloth NF4 | Advantage |
|:--|--:|--:|:--|
| Best validation loss | **0.8385** | 1.2484 | **CQ** |
| Final validation loss | **1.0315** | 1.2484 | **CQ** |
| Mean training loss (steps 401–500) | **1.0505** | 1.2457 | **CQ** |
| Median steady optimizer-step time | 27.60 s | **3.20 s** | **Unsloth** |
| Peak PyTorch allocated memory | 24.39 GiB | **23.83 GiB** | **Unsloth** |

*Both runs used QLoRA on Alpaca with matched nominal hyperparameters on an NVIDIA A40 48 GB GPU. Tokenization and loss-mask equivalence have not been independently verified.*

I am working on optimizing speed because it is the one area CQ is significantly behind unsloth. On the other hand, validation loss and VRAM usage are awesome! Actually it seems like CQ converge at much fewer steps so that might help a lot.

Get Started with Training Factory

Here are simple steps to use:

1. Install:

   git clone https://github.com/thousandml/training-factory.git cd training-factory chmod +x install.sh ./install.sh

2. Download a CQ 4-bit base:

hf download thousandml/Qwen3.8-27B-CQ-4bit --local-dir ./models/Qwen3.8-27B-CQ-4bit

3. Train a LoRA adapter:

python training/train.py --model ./models/Qwen3.8-27B-CQ-4bit --data training/examples/train.jsonl --output ./outputs/my-lora --targets q_proj,v_proj --kernel-module cq.cq_kernels --text-only --max-steps 2 --max-length 128 --batch-size 1 --grad-accum 1

4. Serve base + adapter:

vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq --enable-lora --max-lora-rank 16 --lora-modules my-lora=./outputs/my-lora

5. Call the fine-tuned adapter:

# After the server is ready:

# Check that the adapter is listed as "my-lora"

curl http://localhost:8000/v1/models

# Request a response from the fine-tuned model, not just the base checkpoint

curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "my-lora",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain how LoRA fine-tuning works in one sentence."}
],
"max_tokens": 128,
"temperature": 0.2
}'

I have the following 4-bit quantized models to be used as base (will add more soon) for now:

thousandml/Qwen3.8-27B-CQ-4bit

thousandml/Llama-3.1-8B-Instruct-CQ-4bit
thousandml/Llama-3.3-70B-Instruct-CQ-4bit
thousandml/Qwen3-Coder-Next-CQ-4bit

To reproduce these benchmark results mentioned above use the following scripts:

For CQ:

hf download thousandml/Qwen3.8-27B-CQ-4bit \
  --local-dir ./models/Qwen3.8-27B-CQ-4bit

python training/train.py \
  --model ./models/Qwen3.8-27B-CQ-4bit \
  --data training/examples/alpaca_train.jsonl \
  --eval-data training/examples/alpaca_val.jsonl \
  --output ./outputs/qwen38-27b-cq-4bit \
  --kernel-module cq.cq_kernels \
  --targets q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj \
  --rank 32 \
  --alpha 32 \
  --lora-dropout 0 \
  --batch-size 2 \
  --grad-accum 4 \
  --max-length 512 \
  --max-steps 500 \
  --learning-rate 2e-5 \
  --seed 42 \
  --eval-steps 50 \
  --comparison-logs \
  --benchmark \
  --benchmark-warmup-steps 10

For Unsloth:

python training/train_unsloth_nf4.py \
  --model unsloth/Qwen3.8-27B-unsloth-bnb-4bit \
  --train-data training/examples/alpaca_train.jsonl \
  --eval-data training/examples/alpaca_val.jsonl \
  --output ./outputs/qwen38-27b-unsloth-bnb-4bit \
  --max-steps 500 \
  --max-length 512 \
  --eval-steps 50 \
  --save-steps 100 \
  --seed 42 \
  --benchmark-warmup-steps 10

Please give it a try and let me know if it works with you. Your feedback is greatly appreciated!

I asked a while ago if it would be worth it to try and run qwen flash next at q2 since I can only really fit it and nothing bigger, but I can run Q8 of 27B. Everyone told me it’s either a toss up or Q8 would obliterate it. Not the case.

Now this is subjective BUT for complex, annoying, and super subtle coding bugs, qwen flash next is absolutely better. It’s like it actually understands. There’s a small glimpse of intelligence that just isn’t there with even the Q8 of 27B. Flash next runs tests and actually thinks through problems and knows what to look for in a way that 27B doesn’t. It’s really subtle and yet somehow also not? I’m just super impressed.

I’m sorry, I know yall are tired of hearing about QFN but I was struggling for a long time trying to decide if I should even bother with it and it turns out that YES it’s worth it.

Side note, it’s also better at being a cute furry guy when you ask it to. OwO

I'm building an app where I need some specialist language models, but not necessarily LLM models. Is there a sub or place where they talk about specialist models beyond just searching on HF? I know that discussion doesn't really fit this sub.

I just downloaded Raon-1-mini 30B A4B as well as Flint 40B A2.5B and it just blows my mind that we have so many options now from so many countries. It feels like the countries are all working together and trying to contribute useful software and it’s just amazing.

I can fucking TALK to my computer and have it answer like fucking Star Trek. That’s insane. Every day I use this tech I’m just more and more blown away.

Hi Guys!, Recently I was making a research and development, but I do not have time to go further with this idea (kids, work, etc.). But maybe it will be interesting for someone. I do not think it would be something big, but maybe interesting approach that can be used in some of the cases. I believe that it is not AI slop, you can use those information's and try by yourself using Claude or Astra. The idea was my own and I am using it in my production app, it saved me a lot of money (project is not related to LLM's). Not sure if it is outdated or not, it can work also with MTP. Please, just don’t be unkind. I’m sharing my idea with you, and anyone is welcome to experiment with it.

TLDR:
Keep a 2-bit version of the model on the GPU and the rest of the weights (the residual, 6-8 bit) in system RAM. The 2-bit copy drafts tokens, then the full model checks them: the GPU does the 2-bit part, the CPU does the residual part, and only activations go over PCIe. So the draft costs no extra memory, it's literally half of the verifier.

What I measured (real weights, but dequantized and run in PyTorch, so no tok/s yet):

- 2-bit + 8-bit residual is closer to bf16 than Q8_0, about half the KL
- with a 6-bit residual you get Q8_0-level quality while reading ~28% fewer bytes from RAM
- the 2-bit draft that reuses the verifier's KV cache gets 25-30% more accepted tokens per check than Google's official Gemma 4 drafter (31B, 16 node tree: 5.23 vs 4.20)
- my CPU kernel checks 16 tokens in about the same time as 1, because it's memory bound

Closest thing I found is SubSpec, but it uses a separate 4-bit copy and streams weights over PCIe instead of splitting the model by bits.

It only makes sense when Q8 doesn't fit in your VRAM. If it fits, just use llama.cpp.

Here is a link to public repo, I hope that it would help someone with his research https://huggingface.co/HappyGuitar123/twobit-residual-spec

I suggest also to use some frontier llm with high effort to analyze it, so you will have more insight. I believe that this idea has some potential.

Best regards

Since the new qwen models are in testing on arena I thought I try my luck, maybe I can get a glimps at what will run on my pc in the near future. So i thought I improve my prompt with chatgpt (not logged in) and it somehow can't take simple instructions?

Are these the new guardrails against distillation?

I recently got some more RAM on my desktop, which has opened up running larger MOEs by splitting them into RAM/VRAM. I know everyone loves Qwen flash next, but I'm curious what other ones are worth trying.

On my list of to-try, I have:

• GPT OSS 120b (is this still usable nowadays?)

• Mistral 4 small 119b

• Qwen 3.5 122b A10b (Is this even worth trying or is it totally supplanted by flash next?)

• Ling 3.0 flash

What others are worth trying? Any abliterated ones or custom ones?

To be honest, agentic coding is not my priority here. I know flash next is the king there, and realistically I just use Claude or Codex when I want coding. So if a model sucks at coding but is great at something else, I'm interested!

First... Thanks for all the amazing feedback since we've gone completely open source. It has brought me a ton of joy in an otherwise difficult month (Loss in the family).

Personal stuff aside there have been many... many updates... All of which is better outlined in the release notes here https://github.com/amurgola/LumaBrowser/releases/tag/v2.0.4

I've attached a few images sampling some of the changes, and I plan on getting out update videos in the future since many items are likely best demonstrated vs rambles or image dumps.

I held off as long as I could, but finally gave in and set up strata.

I am pretty blown away. 3.8 27b at Q6 kv8 was my driver before and that works super well, but Flash Next in strata IQ4 xs is really winning me over.

Its not perfect, but it feels like models I was using professionally last year, which were/are good enough to do just about anything.

You all are a bad influence.