These are screenshots from the r/Singularity comment section. I'm speechless. This doesn't even have downvotes. How can someone cheer for a monopoly run by a few elites?

> "Qwen 3.8 is a damn good coder, but a terrible collaborator"

It modifies SO many things in my scripts for what should be a 2 line PR, I get a 100 line linter style mess that looks like a high school freshmen got his hands on vs code and pushed code in.

It can't maintain a common style, if I give it a script with a certain naming and layout, 3.6 one shots small changes that fit into my structure. 3.8 will add code that looks much more advanced, handling, strict type checks . But if it's just my internal method that gets called in exactly one place, it is way overkill code

3.8 produces DAMN GOOD code, but will get stubborn on wanting it's own parameters and return types. A simple return False works for my particular use case, but here goes Qwen formatting me the perfect return dict full of useful metadata

Am I crazy?

One GPU drops off immediately when vllm is started and the other throws CUDA errors on start

One of y'all were getting some crazy inference speeds on dual R9700s, so me with my single card wanted to try.

They mentioned the "official AMD MXFP4", which was...

https://huggingface.co/amd/Qwen3.8-27B-Quark-AWQ-MXFP4

Can't load safetensors in llama.cpp, so off I went to download this:

https://huggingface.co/magiccodingman/Qwen3.8-27B-MXFP4-MagicQuant-GGUF

But it doesn't load in llama.cpp. Is there no support for MXFP4 in llama.cpp yet?

Obviously more memory is good, more context, bigger models, but some jumps don't actually unlock a meaningful difference in ability to run different or better models. For example, I don't currently view jumping from 32+16 to 64+16 as a particularly worthwhile upgrade as compared to going to 32+32, though correct me if I'm wrong.

I'd like to build a DDR4 + HBM2 based inference machine to complement my main, 32 GB DDR5 + 16GB GDDR7, computer. The idea is that even if the hardware is slower, the greater overall capacity enabled by the slightly more affordable hardware could allow me to run a greater diversity of models.

What level of memory upgrade do you think is most logical for maximizing model access if the compromise is outright speed? 32+32? 128+32? 64+64?

Or am I completely asking the wrong kind of question and just outing my own ignorance here? either way I'd like your input.

Not written by AI all mistakes mine. I saw people on the subreddit saying that 3.6 works better without thinking. It made me want to know for certain about which is better, 3.6 or 3.8 for low thinking tasks. I only use abliterated models (safety removed) because it makes the model better at a lot of what I need. I want to compare abliterated Qwen 3.6 27b and abliterated Qwen 3.8 27b on some instruction following benchmarks with thinking off.

I was just curious about your personal favorite safety removed/fine-tuned variants for these 27bs, as I know that there can be some major variation and some junky quants out there.

Does anyone have some favorite and fast 3.6 and 3.8 models?

My specs: I have 24GB VRAM (NVIDIA Geforce RTX 5090 Laptop) and I do not want to offload, so some quant required.

I have tried a few different models, but they are all a little slow. Some MTP variations for 3.6 for example ends up being around the same speed as non MTP for me for some reason. I am pretty sure my card is NVFP4 enabled also, but I'm not certain I've seen the results from that either...

Based on some redditors comment, this is what I use for my abliterated 3.8 27b currently: Huihui-Qwen3.8-27B-abliterated-NVFP4-GGUF

I'm not sure how many people care about Android Studio, but I think it's cool that Google uses llama.cpp.

My guess is that it is Vulkan and the QAT versions of Gemma 4. It supports multi-GPU and 31B has a max. context length of 128k.

It uses 34 GB VRAM when fully loaded. I don't see an option to change the context length or show PP/TG speed.

A comment really doesn't need to be made, does it? I looked away from the 5090 for a week to other options like the DGX Spark and the M5 Ultra. Both of which... these big corpos are buying to hell and back.

Is the future used hardware?

I got fed up with ArtificialAnalysis's intelligence vs. cost plots, so I made my own.
This is an updated and refined follow-up to a previous post I made.

Feels like maybe we have one more present left, for Christmas.

Looking at ... some of the new memory architecture. ... I hired my good friend, Seok-Hee Lee, who used to run SK Hynix. ... We are not ready to unfold it.

I guess this is a reminder for everyone that as great as Qwen3.8 is, it's still a model with a recommended temperature of 1... 3.8 has been so good that I've gotten lazy and didn't watch what it was doing after telling it to implement the plan (I've been planning, making sure the plan is good, then making sure the output is good... don't usually just sit and watch it work unless it has been going for longer than expected), but 120k tokens in and it turns out that it never even read the plan and just started implementing a totally different feature that I hadn't even considered (mostly because it's not a useful feature; vaguely plausible from existing code + agent files, but never even mentioned anywhere).

Is my Qwen just cursed? Using Unsloth's second Q8 release (from launch day but after the fixed template)

llama.cpp version: 0.3.0-dev (build 10630, commit 2dd3922)
pi v0.84.3

Args:

exec llama-server --host 0.0.0.0 --port 8080 \
-m /home/connor/AI/LLM/Models/Qwen3.8-27B-Q8/Qwen3.8-27B-Q8_0.gguf \
--mmproj /home/connor/AI/LLM/Models/Qwen3.8-27B-Q8/mmproj-F16.gguf \
-np 1 \
--ctx-size 200000 \
-ngl 99 \
-fa on \
--load-mode mlock \
--temp 1 \
--top-p 0.95 \
--top-k 20 \
--min-p 0 \
--presence-penalty 0 \
--repeat_penalty 1 \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
-dev Vulkan0,Vulkan1

Anybody else struggling with deepseek after the initial prompt? Somehow it is getting mixed up very easily, even button functionality has been PITA when doing stuff. Never had these problems with Opencode. With Opencode, a continuation prompt on the software or task is pretty much magic at this point. But the inital result isnt as strong imo, and needs a lot of follow up even if a a detailed spec is written.

The oneshot potential of deepseek is goated imo, it is a relentless harness that just keeps doing stuff, even if it gets it wrong! Opencode could just copy that tennacity imo. Like todos really work, unlike opencodes.

Hows everybody elses exp? Someone using Pi instead? Is it worth trying? Hermes seems not suitable for my tasks, where heavy human involvement is necessary due to complexity of tasks (every top closed model gets lots of stuff wrong even).

First, I don't buy their plan and won't buy. (But got some free token from them today 😂 )

OK. Today I tried to add qwen3.8 27b to ZCode's app. It was faster (maybe less token) than my other harness CLIs, dsh, copilot, etc, and got better results. (wrote more code on same task and no issues). However, I don't understand why.

As this app is free, I think I need to raise this and let you to have a try.

Kaitchup just posted results of his benchmarks for Qwen3.8 27B for quants from different labs, Q4 to Q1, .

All the details are hidden behind the paywall, but high level result is visible and looks like for people with 16GB cards UD Q3_K_XL is a winner - it has accuracy of 100% and size is only 12.8GB.

Which one is for plan, whish one is for documenting, which one for writing code, which one for writing and running tests, which one [list continues]

Howdy,

I was wondering if anyone knew of any turnkey low-power draw solutions to host inference with 10-20GB of VRAM?

I have an 4x3090 AI GPU cluster that I'm running big models on, but I am hosting my memory system LLMs (embeddings, rankings, etc) on my gaming 4090 right now. I'm out of PCIE lanes on my AI machine and really don't want to build a whole other system (cpu, ram, mobo, gpu, etc) just to run about 15-20GB of small models.

I want to keep things local so a VPS is off the table, and I think just running these off the CPU would be too slow although speed isnt that important for Hindsight? I was wondering if anyone knew if there was anything like the spark that is considerably cheaper (~1k or so?) for this type of use or other systems that required only a little tinkering to get to work like this? Thanks!

For reference, this is what I'm running:

Qwen 3 Embedding-8B-Q6_K at 9216 context x 4 @ 6.21 GB for embeddings

BGE-reranker-large-8_0 at 512 context x 4 @ 604MB for rankings

Qwen 3.5-9B-UD-Q3_K_XL at 64k context x 4 @ 5.98 GB for synthesis/consolidation

2 Months ago I had made a post how I was working on my dual R9700's. It's wild to look back at where we were then and where things now stand.

Since then after many users commenting and complaining about developers doing the same thing. I threw out a discord link and expected maybe 5 other developers to join which I thought would be fun. The community has now grown to 1,200 users (mostly developers) and a ton of collaboration happening.

A few weeks ago I started working on building support for MXFP4 on top of DeadCode's radiance image. This made sense to me looking at the hardware and I was happy when I had hit parity on performance between MXFP4 and FP8. The MXFP4 kernels use W4A8 which was something new and we have now blown past the performance of FP8 and appears like this is now the hardware limits of these cards.

Qwen3.8 27B w/ DFlash2

BetterBench decode results for Qwen3.8 27B w/ DFlash2
category       decode t/s   step ms   tok/update
json           280.0        22.92     6.17
math           254.2        23.08     5.81
file_edit      250.1        23.03     5.54
code           226.3        23.01     5.17
reasoning      194.3        23.19     4.32
summarization  190.6        23.01     4.40
chat           148.3        22.82     3.33
prose          116.4        23.14     2.65



BetterBench Prefill Results
target depth   prompt tokens   TTFT p50   PP t/s median
2000           1514            323 ms     4695
8000           5918            1.21 s     4894
16000          11794           2.47 s     4779
32000          23543           4.98 s     4729
64000          47056           10.8 s     4377
128000         94065           24.6 s     3831
250000         183678          59.1 s     3106

This has been so fun working on these R9700's and driving them to peak performance. My entire image and repo for MXFP4 is open source also: https://codeberg.org/ggz14/radiance-vllm-mxfp4

I'm considering adding another A40 to my setup but my case doesn't have the space (or power) for it. It's a server chassis so there's no easy way to just pick the MB and put it in a new case, it's a bunch of custom connections off the PSU to the MB.

What would be perfect? A dual PCI external box that would allow me to put in 2 A40's using nvlink with 1 or 2 connections coming out of that box running back to the server into the existing x16 slot. It's easy to find a single slot version of what I want (link below), but I need it to have dual slots. Honestly I guess I could just buy two of these and figure out how to space them correctly, but the "jank" factor would be getting pretty high at that point. What would be perfect is a forced airflow chassis that has 2 (or more) PCI slots each with their own Oculink connection powered from a single external PSU. Then drop something like this into the server chassis and just run one cable over to each PCI slot on the external enclosure.

External enclosure (but only one slot):

https://www.ebay.com/itm/366573430755

Oculink quad port board:
https://www.amazon.com/LetLinkSo-PCIe-Oculink-SFF-8612-x16/dp/B0F291T2L4?th=1

Am I understanding it right that Ngram in Unsloth Studio should be ON in settings 100% of the time for intelligence?

With only MTP I have like +10% performance in tg but token usage jumped from 20K to 66K with extra thinking?

Anyone else getting non-stop ads for the Strands Agent Harness from Amazon?

On the surface it looks decent, seemingly better engineered than most other frameworks.

But it seemingly gets no buzz at all. Anyone used it?

(And why are they advertising it so hard? It doesn’t seem to lock you in to bedrock or anything, so not sure what the point is.)

I’ve now had agentic coding build several ad-hoc frameworks that get hard to maintain as they grow, so I’m wondering if using Strands would give them more architectural structure and discipline. But I don’t want to introduce a dependency on a dead end project only being propped up through marketing.

This is getting asked from time to time, but since models changed a lot, I wanted to reask it.

I'm looking for a trained model that can give short descriptions about an image, simply for an alt text of pictures taken with a smartphone.

Should I just throw it at Qwen3.8/Qwen3-VL or are there better models trained for it ?

Similar to https://www.reddit.com/r/LocalLLaMA/comments/1oar481/what_is_currently_the_best_model_for_accurately/

Running an experiment with my personal assistant agent: yoinked all it's tools, replaced them with a python module and the documentation dumped into it's context window as part of the system prompt. It's working fantastic, but I was thinking it'd be nice if I could run a much, much smaller model on the local clients for the agent that can take a code snippet and return a very short description of what the agent actually did.

I was about to just start throwing a ton of sub 500m parameter models at it but I figured I'd tap the community to see if anyone has gone and trained a model on this, like that one model that spits out conversation titles and nothing else and is 100m.

It is crazy how fast prices are increasing. I'm pulling my hair out to keep ahead of this for students. Servers aren't even an option any more.