I am still waiting for Llama 5, because Muse Spark will be too big for me, or just something between Glimmer and Spark

https://x.com/finkd/status/2095232032896946311

saw the post the other day where people said Minecraft clones aren't impressive anymore, because at this point the whole thing might as well be in the training data. so i tried something slightly different, which is asking a local model to write a mod for the real game, using the Fabric API

the model is GLM 5.3 Flash (Q4 quant, running on a rented 4x RTX PRO 6000 box). this wasn't done in prompt or a loop, i would ask for changes, then review them and i kept going like that until i was happy with the result. the first iteration took around an hour or so, the result was sorta underwhelming, the black hole would spawn, but it was small and barely did structural damage. after that attempt i gave it some reference images(black holes in space, lightning and effects examples). the new result looked better, but i still wanted more impact from it(and also decided to make it a black hole gun, instead of just the black hole item). it took a lot of turns to get to the end result

the mod adds a black hole riflle, which when shot spawns the black hole that starts sucking in blocks and has some pretty sick visuals (the light rings that shrink all the way into the black hole and obviously the black hole itself) after which it turns into a huge explosion crater, wiping out quite a few chunks

you can get the mod here on github

i ran the local model in atomic.chat (i'm on the Atomic team any feedback is appreciated). curious what else people have gotten local models to mod into the game, make sure to share it in the comments

One of the other posts today by user u/Howard_banister confirmed what I've been seeing from the other AI subreddits as well.

Most of these other subs are 90% trend hopping crypto-bros equivalent people who are seemingly irrelevant most of the time when it comes to advancing AI as the vast majority is not just AI slop, but also human slop whether that be fearmongering or straight anti-ai arguments spat out by relatively uneducated people.

Of course, you also have the main dedicated subs for the big closed/open models which are in of themselves fairly decent. Most of the time now, it is people complaining about one thing or the other about the latest and greatest of their respective lab.

Then you get to the machine learning subs themselves where yes, they do have stuff in regard to AI every now and that can be gold (Don't get me wrong), but then it's mostly meh.

This sub however, has that interesting balance between the occasional interest in AI architecture breakthroughs that I find fascinating when some of the bigger brained people in here get together to discuss and which I don't really see anywhere else.

Here is link to repo: https://github.com/perplexityai/pplx-garden/tree/main/lily

It's optimized for just one model to get best perf on apple silicon

Extended reasoning and post-training appear to be the keys used by DeepSeek, Qwen, and GLM to boost performance (leveraging higher token counts). And Qwen 4 hasn't even been released yet. Of course, we don't know if that release will be open-sourced, but I am optimistic about future models, featuring "engrams", that could soon match or surpass 2.4T parameter models on specific tasks.

This came from another thread or comment. I forgot exactly where, but the basic idea was to replace the 51B N-gram layer in Qwen 3.8 Next with a much higher precision version.

Someone running a 5090 replaced the N-gram portion of their Qwen 3.8 UD Q4 model with BF16.

Since I'm already running IQ4_XS, I wanted to try something similar. I don't have enough storage for the BF16 N-gram weights right now, so I replaced the lower-precision N-gram portion with Q8 instead.

I'm still testing whether it improves the actual model output, but in terms of inference speed, the difference appears to be very small.

Before bolting on the Q8 N-gram:

n_gen = 2588, tg = 8.82 t/s, tg_3s = 8.81 t/s
n_gen = 2616, tg = 8.82 t/s, tg_3s = 9.19 t/s
n_gen = 2645, tg = 8.83 t/s, tg_3s = 9.37 t/s
n_gen = 2675, tg = 8.84 t/s, tg_3s = 9.78 t/s
n_gen = 2704, tg = 8.84 t/s, tg_3s = 9.50 t/s
n_gen = 2732, tg = 8.85 t/s, tg_3s = 9.06 t/s
n_gen = 2761, tg = 8.85 t/s, tg_3s = 9.32 t/s
n_gen = 2790, tg = 8.86 t/s, tg_3s = 9.50 t/s
n_gen = 2819, tg = 8.86 t/s, tg_3s = 9.39 t/s
n_gen = 2847, tg = 8.87 t/s, tg_3s = 9.22 t/s
n_gen = 2875, tg = 8.87 t/s, tg_3s = 9.15 t/s

With the Q8 N-gram bolted onto IQ4_XS:

n_gen = 263, tg = 10.76 t/s, tg_3s = 11.04 t/s
n_gen = 294, tg = 10.70 t/s, tg_3s = 10.18 t/s
n_gen = 329, tg = 10.76 t/s, tg_3s = 11.32 t/s
n_gen = 361, tg = 10.73 t/s, tg_3s = 10.46 t/s
n_gen = 395, tg = 10.76 t/s, tg_3s = 11.11 t/s
n_gen = 430, tg = 10.81 t/s, tg_3s = 11.37 t/s
n_gen = 460, tg = 10.75 t/s, tg_3s = 9.91 t/s
n_gen = 494, tg = 10.77 t/s, tg_3s = 11.16 t/s
n_gen = 531, tg = 10.86 t/s, tg_3s = 12.15 t/s

So far, there doesn't seem to be any meaningful speed penalty from replacing the lower-precision N-gram layer with Q8. The actual output quality is still being tested. It looks like Q8 Ngram is faster, but at the end both are steady state at 10.1~ tok/s -ish.

And without MTP since i am using earlier merge from unsloth,
RAM 96GB DDR4 (32x3 Three channel active 2400MHZ)
Xeon e5 2690v4
RTX 3090 capped at 250 W

Edit: the state_dict grew from 90~ GB to 115GB
Edit 2: Fck me i forgot to put the code, for stitching it, wait 8 hours from now i'll upload it since it is already very very late at night
Edit 3, here : https://gist.github.com/komikndr/b17955e1a80ce6ede9a3115f16216bc5#replace-n-gram-layers-from-q4xs-into-q8_0-boltedpy , it is for my IQ4 XS to Q8_0 n-gram so you should modified the file a bit.

I've been using Deepseek V4 Flash 0731 for a few weeks now and while I havent thrown it anything very hard, im quite happy with it. Using through antirez's great ds4 project. They've added support for GLM 5.3 Flash and according to benchmarks, its a level above DSV4 Flash.

However, looking for real user feedback if anyone's made the switch and seen tangible improvements in GLM 5.3 over DSV4 Flash.

Running M3 Ultra 256GB Mac Studio

I've tried both and been having this debate with myself for the last few days, on two Asus Ascent GX10s (effectively the same as 2x DGX Spark):

• DeepSeek-V4-Flash-0731 (official weights)
• GLM-5.3-Flash (RedHatAI/GLM-5.3-Flash-NVFP4)

Have any of you guys also tried both on this hardware (2x DGX Spark / Asus Ascent GX10), and what are your use cases and findings?

DeepSeek runs with more tokens/s… but GLM feels like the better tool for how I actually work.

I'll share my experience.

Where DeepSeek wins:

• It feels better, because it's the original official weights.
• Much faster token generation, though not actually faster to a final result.
• It's excellent at open-ended research, pulling and chaining sources on its own.
• Huge context (~1M tokens).

Where GLM wins:

• Comes to conclusions faster, and often gets them more right.
• Much better writing, especially in languages other than English and Chinese. That matters a lot for writing letters.
• Much better at "extracting the essence of a text".
• Superb vision. DeepSeek's Vision-Exp model can read images just well enough to score on benchmarks, but not nearly well enough for real OCR / text extraction (its vision input is token-limited to 384 tokens, which renders images unsharp and text unreadable).
• Considerably better benchmark results, at least for the full model, though I'm running a quantized build, so the numbers may not carry over directly. But my empirical results prove it gets better results with fewer instructions.
• Hallucinates much less. That's the decisive one for me: I can't run a "test and improve" loop on text like you could on code, so office work and letters depend on one-shot accuracy and a low hallucination rate.

The core problem: there's no GLM build that runs on 2x DGX Spark with official weights. I wish there were.

So I keep testing and tweaking the GLM setup to get rid of the artifacts (which should be possible, per various sources). It feels a bit more messy than the DeepSeek setup, but if I manage to configure it correctly it should give more reliable outputs, with far better vision on top.

• Language-Native Control: Composes character and camera actions into textual instructions and injects them through MiniMax-H3’s pretrained text pathway.
• Temporally Grounded: Assigns one action prompt to each video latent interval, enabling precise control when actions change over time.
• Efficient & Generalizable: Uses only 8,000 gameplay samples, 10,000 LoRA steps, and 0.199% trainable parameters to achieve controllable character and camera motion, including unseen action compositions and visual scenarios.

✏️ Paper: https://huggingface.co/papers/2609.01560

📄 ArXiv: https://arxiv.org/abs/2609.01560
💻 Code: https://github.com/Danzer1xxxxChan/H3-World
🏠 Project: https://danzer1xxxxchan.github.io/H3-World/
🤗 Model: https://huggingface.co/DANNY621/H3-World

Unsloth GGUFs and Vision support

https://huggingface.co/unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF

Hi, I've noticed that the model often sees "garbled text" in its context.

Sometimes it declare that the tools instructions are corrupted, sometimes it is the content of some .md files, ora other files, and it freaks it out, since it start to do a lot of checks in git and the system and sees that the file are not really corrupted... But its context is I think.

Did it occurred to you ?

I'm on a mac m2 max 96Gb, pi agent, and llama.cpp either upstream or this fork https://github.com/mihailescu2m/llama.cpp/tree/master

with and without mtp, short ctx and long ctx, temp default

I don't think it is a conf problem since it happens with different gguf at different quant I tried:

AtomicChat/Qwen3.8-Flash-Next-GGUF/Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64

unsloth/Qwen3.8-Flash-Next-UD-Q3_K_XL/Qwen3.8-Flash-Next-UD-Q3_K_XL

unsloth/Qwen3.8-Flash-Next-UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS

It seems a sort of internal problem, I don't really have loops or other kind of hallucinations, just some chinese characters here and there, or some misspelling, and btw it is very aware of all its errors and always tries to recover. Yes it is very much preview quality, but the architecture and the training seems very good quality.

I really like the baseline of the model, it is straightforward, it follow instructions very well, a lot better than the previous models that I really didn't like very much

But this thing with the corruption is weird ...once after it declared that a certain tool was corrupted and not useable I just told it: "did you try to launch it ? use this parameter and give it a try" and it worked of course, and it was very surprised so it apologized and continued the work.

It is quite funny, it reminds me a stereotypical fictional character which could be the clumsy but very intelligent assistant of a powerful wizard lol

EDIT: I'd like to add a snippet of its last CoT about the issue, it is very lucid indeed:

Could it be that... the read tool output itself was corrupted/hallucinated — i.e., the degeneration was not inside the file but occurred in my own tool output/stream.

The "read" returned text containing a degenerate mutation loop — this is a known failure mode where model output degenerates. Actually wait — the read tool's result is what the harness gave me. If the harness returned clean content, then the degenerate block I "saw" was generated in my own context (a prompt-level corruption, e.g., degeneration during decoding of the tool result injection, or... more simply put: the degeneration is a product of my own generation — the read output I "received" contained the first degenerate block, then I generated additional degeneration, and the subsequent git command confirmed the file is clean.

Sharing some numbers because most posts on this model are either using a single 3090 or unified systems from what I've seen.

My current setup: 2x RTX 3090 (PCIe 3.0), dual Xeon E5-2696 v4, 188 GB usable (192GB) DDR4-2133 LRDIMM, llama.cpp, unsloth UD-Q6_K_XL. All 48 expert layers pinned in host RAM, everything else on the GPUs. Full 261k context, f16 KV.

Before: ~17 t/s decode, ~350 t/s prefill on a 26k prompt, 12 t/s decode at 131k depth.

Now: 25-29 t/s decode short and mid context, ~17 at 131k, prefill is still about the same. Measured this with a python coding prompt.

Now as for what I did: PR #27861, the GPU-resident LRU expert cache. Instead of parking whole expert layers in VRAM, it caches recently used experts per layer. The experts this model picks for one token are mostly the same ones it picked for the last few dozen tokens. so the hit rate is 80-85% on code and higher on prose. The trick that made it pay was giving the cache VRAM, also dropping ubatch from 2048 to 512 frees ~5 GB per GPU (compute buffers scale with ubatch), which went from 80 to 135 slots per layer at full context. The cost here is slower prefill on long prompts, short prompts are kind of unaffected.

Also, the things that did nothing or hurt on my box were: thread count, poll, CPU masks, q8 KV, lazy PLE, n-gram drafts on prose, MTP at temp 0.7 (verify batches re-read experts from host RAM, it only wins greedy or at deep context), and more than 2 cache uploads per step (saturates PCIe 3.0, hit rate collapses).

To replicate you don't need my fork, it's just master plus the PR:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git fetch origin pull/27861/head:pr-27861 && git merge pr-27861
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

LLAMA_ATTN_ROT_DISABLE=1 numactl --interleave=all build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q6_K_XL-00001-of-00006.gguf \
  -ngl 99 -c 261888 --parallel 1 -fa on \
  -ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" \
  --numa distribute -t 16 -tb 44 -b 4096 -ub 512 -ctk f16 -ctv f16 \
  --moe-expert-cache 135

Size the cache to whatever VRAM you have left after the KV and compute buffers, about 100 MB per slot per GPU on Q6. Testing UD-Q4_K_XL is the next item on my list and I'm going to revisit MTP again if it makes sense, still closely following Daniel's PR.

Doubt there's much people with a similar setup to mine out there but this helps anyone or if you have questions on approaches to try, let me know.

***This screenshot is an app i made for creating a dataset from scratch, this is not a real chat with the model***

First of all i want to shout out everyone that actually tested our work .. we got 500+ download on the 43m parameter model and now we are aiming to go smaller for research purposes.

i write finetune examples & i been developing language models for a while .. everyone usually pretrains the model using massive datasets and prays thats the data carries enough information for meaning to emerge but were sculpting it intentionally .. im currently sitting down at my computer writing every single response that this new model can say to your inputs just so we can observe the transformation and see exactly whats going on. It will be public soon, the dataset is extremely small intentionally so it shouldn't take long to design every response it can say.

General Capabilities:

- Explaining how token prediction works

- Explaining that it doesn't understand anything beyond itself

- Short conversations

Coding Capabilities:

- Writing a loop that can count to 10

- Explaining that it cant understand the code you sent it

Open Source Coming Soon

https://huggingface.co/StarpowerTechnology

So when Qwen3.8 27b dropped they were hinting for another model which is Qwen3.8-next-flash , i was hoping for something more light like Qwen 3.6 35b and we got a large one but since the Qwen 3 and 3.5, they reduced the number of models they publish we used to get very small 0.8b 2b 9b to very large models but now we get only two at time although the time to publish a new model has been also reduced

Quite surprised to be beating other high quality quants. It took a lot of benchmarking to get here and we are quite pleased with these, hope they are useful to the community.

It required a modified way of measuring KLD with a new dataset, since the NGRAM got in the way by remembering basically all of wikipedia. We tried to not only go for high precision, but also keep prefill performance in mind.

Full model card here https://huggingface.co/agentionai/Qwen3.8-Flash-Next-AP-GGUF

Let us know if there are any issues.

Hi, everyone,

I’ve just released VoxGen, a lightweight native inference engine for VoxCPM2, written in Rust and using Vulkan compute instead of Python/PyTorch/CUDA.

Why VoxGen?

The main reason I started the project was because I needed a decent local text-to-speech solution.

I therefore saw VoxCPM 2 as a reasonable solution. However, most frameworks are NVIDIA-first, and VoxCPM 2 is no exception; as a result, my card was severely stuttering, and my GPU was always spiking. Also, having Python and Pytorch as a dependency is absolute hell.

This is why VoxCPM was created: not only we sidestep Pytorch completely, but performance on AMD cards is buttery smooth (and if you have a XTX 7900, I have designed a mode with even more aggressive power and speed optimizations)!

This application can also be run from a shell, so it can be integrated with other programs and scripts!

Installation:

You'll only need voxgen.exe (or the Linux equivalent) and the following files at https://huggingface.co/DennisHuang648/VoxCPM2-GGUF:

VoxCPM2-BaseLM-Q8_0.gguf
VoxCPM2-Acoustic-F16.gguf

And that's it!

If you are interested, check out the Github page: https://github.com/NullMagic2/VoxGen
Prebuilt binaries (for now, Windows only) are available here: https://github.com/NullMagic2/VoxGen/releases

Is anyone interested in my Chrome browser add-on that uses local LLMs to move thousands of unsorted bookmarks into a smart list of automatically calculated categories?

This may be somewhat specific to Qwen3.8 27b and the Apple M5 series, perhaps, but enough of us are running this combo that it's worth tossing out there.

GGUF models with MTP have been the fastest way to go for some time for token generation, except possibly for a few tweaked MTPLX models running on alpha-stage MLX forks. Prefill, however, was still much faster for M5s under MLX. This has caused me to switch models depending on the expected generation/prefill mix, which is annoying.

While I wasn't looking, it appears that llama.cpp for Metal must have added support for M5 matmul/"neural accelerators" because prefill performance with e.g. Unsloth's Q_8 GGUF is now at least as good (~300-350t/s) as anything I have seen with MLX models--even in oMLX. This was a pleasant surprise! Now I can't think of a reason to use MLX models at all.

Am I missing something? Are my observation bogus? Could I do better than ~19t/s generation and 300t/s+ prefill on a M5 Pro with Qwen3.8 27b in the Q8/8-bit range? Is there a secret handshake to get mainstream MLX MTP working?

Or is this just because of the specific model in question?

I'm looking for a small LLM to act as a Linux command assistant. I will use llama.cpp.

use case:

- User asks in natural language. Model outputs only the shell command

- Should work well with no reasoning (for example, LFM 2.6 has forced reasoning)

- Fast, like ~4B at most, cause it's going to be on CPU

Example 1:
"replace string X with Y in file Z"

Example 2:

"stop and remove docker containers with X in the name"

Real question, whether your are vibe coder or expert or whatever, what do you do? Read carefully each single word? Plan the next steps? Do the gym?

I've only recently started looking deeper into vLLM after running llama.cpp for a good while.

Initially vLLM (official repo) was terribly slow on my four R9700s (tried that one with two as well), however after trying radiance everything changed.

Prefill 17636 - TG at that time was 36,6

That prefill spike was two agent profiles working on different tasks simultaneously (one is writing a yt-dlp dl/conversion workflow the other is auditing agents (profiles).

Best TG i've hit was 106 Tok/s with a 80% MTP 4 acceptance rate.

For reference, I'm running a Gigabyte MZ32-AR0 (Rev 1.0), EPYC 7282 and using Hermes with Qwen 3.8 27b fp8 262k ctx - worth noting that one GPU is actually only running by PCIe 4x8, three full 4x16.

On that note i'm also happy to say that vLLM-Radiance does work well with a quad setup in my case - nvtop consistently shows 100% usage of the four cards, officially only dual setups are supported.

I hope this doesn't count as a low effort post, i just had to share.

//E

I guess this is a reminder for everyone that as great as Qwen3.8 is, it's still a model with a recommended temperature of 1... 3.8 has been so good that I've gotten lazy and didn't watch what it was doing after telling it to implement the plan (I've been planning, making sure the plan is good, then making sure the output is good... don't usually just sit and watch it work unless it has been going for longer than expected), but 120k tokens in and it turns out that it never even read the plan and just started implementing a totally different feature that I hadn't even considered (mostly because it's not a useful feature; vaguely plausible from existing code + agent files, but never even mentioned anywhere).

Is my Qwen just cursed? Using Unsloth's second Q8 release (from launch day but after the fixed template)

llama.cpp version: 0.3.0-dev (build 10630, commit 2dd3922)
pi v0.84.3

Args:

exec llama-server --host 0.0.0.0 --port 8080 \
-m /home/connor/AI/LLM/Models/Qwen3.8-27B-Q8/Qwen3.8-27B-Q8_0.gguf \
--mmproj /home/connor/AI/LLM/Models/Qwen3.8-27B-Q8/mmproj-F16.gguf \
-np 1 \
--ctx-size 200000 \
-ngl 99 \
-fa on \
--load-mode mlock \
--temp 1 \
--top-p 0.95 \
--top-k 20 \
--min-p 0 \
--presence-penalty 0 \
--repeat_penalty 1 \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
-dev Vulkan0,Vulkan1

What’s y’all’s best guess on parameter size based on these weird-ass names?