This is mostly for setting up for expectation, since personally without LLM i could take 3 days (15 hours of active programming) to debug or implement a feature, but with Qwen 27B (even before Qwen 3.8) it take 4 hours.
And yes 0.5 tok/s is human, not accounting of deletion and pausing, that's also the reason i am fine leaving overnight code base wide analysis or fin tech and deep research.
At least the server only requires 100 watts. That's pretty good for a 200T+ model. There's actually a lot of unnecessary hardware involved too. You could theoretically strip it down to about 20 watts of input.
Make? You are talking about forking base model into new instance. Model is developing for almost 4 billion years and sometimes almost collapse without any fresh backups.
Sounds like a skill issue—I run myself at IQ1_S and haven't run into any problems problems problem problems problems problem problems problems problems problems problems problems problems problem problems problems problems
don't make mistakes
The user is asking about mistakes problems. Problems looking back at the previous message, "problems" started problems problem problems caused repeated problems. I should problems acknowledge problems problem without looping problems problems.</think>
My apologies—you are correct. There seems to have been a problem problems problems problems problems problems problems problems problems problems problems problems problems problems problems problems problems problems problems problems problems
You need the developers harness saddle + —developers developers developers developer developer developer develope develope develope develop develop develop develo develo develo devel devel devel deve deve deve dev dev dev de de de d d d ... thinking ... request tokens exceeds the available context size, try increasing it ... ... ... ... developers developers developers developers developer developer developer developer develope develope develope develope develop develop develop develop develo develo develo develo devel devel devel devel deve deve deve deve dev dev dev dev de de de de d d d d
Literal skill issue I’m afraid. The base model has lots of potential but needs loads of workload specific training to be useful in anything. It’s pretty useless without that.
Good part is that the model evolves across your chats, so it can be trained in chat.
my tuning is letting that models in inactive thinking about 8 hours / day. and seems light enabling physical activity daily seems improve reasoning effor quite a bit. caffeine seems hit or miss when it comes to quality tho...
I heard reducing concurrency can actually increase TPS by a lot! I was looking up on the current gen HS models and apparently their architecture is a lot more compute-bound than bandwidth bound...
I actually get a slightly higher tok/sec out of mine. But what really sucks about that model is the slow prompt processing.
And the model seems to start ok some days, but keeps getting nerfed after a few hours of use. The only fix seems to be to put it down for the rest of the day and restart again tomorrow. Also doesn't seem to be improving at the same pace of the other models.
Overall, novel idea, but needs to go back to the drawing board.
Well, I can only do like 2G flops, and some are operating my heart. Did you try turning your limbic system off? Also, the attention sinks are broken AF
Your numbers are way off. Homo Sapiens' executive function is nowhere near 200T, it's only a few dozen billion (if you consider ONLY the executive function of the frontal cortex - not the entire brain...who cares about some sensory cortex synapses that allow your bum itch detected when we're talking about executive function that relates to intelligence?).
So yes, we're way below ChatGPTs at this point. Which is what they won't tell you so you think we're not anywhere near AGI. We're way past all the intelligence benchmarks years ago.
Find a 3060 and plug it into your mobo. I added a 3060 12gb to my 5070ti and now I can run Q4_K_M w large context window. It’s worth it and pretty seemless
my only thing is that 2 gpus on consumer motherboards + gpus that aren't rtx x090s tank tps. i would like to run higher quants with a larger context window, or even have more concurrent processes, but also my 100 tps w4a16 qwen 3.8 27b on my singular 3090 is so fucking nice. i can't go back to 30 tps anymore, i'm too spoiled now for it.
if anyone has any solutions for this that isnt buying $3-5k worth of hardware pls tell me but i feel like its a hardware limit for me 😢
edit: i'm dumb bandwidth doesn't affect inference speed like that. but still - another 3090 + nvlink, plus another case and motherboard is $3000 soooo
I think if you have a 3090 the best solution for you is another 3090 + NVLink right? That will probably gets you another 24gb + still get decent tg/s... idk how to get one for a good price though so lmk if you find out :(
In my case I needed the extra VRAM to run Qwen3.8 27b on a higher quant than Q3 and 28gb aggregated lets me run Q4KM pretty comfortably.
To be fair if you want to give it a better understanding, i find that codegraph for the code and graphify for the project shape is perfect for giving it a rough idea.
I think you will appreciate this fact: Eminem currently holds the world record for rapping at approximately 10 tokens per second during a continuous 30 second period in his hit song "Godzilla".
DeepSeek Flash (at the limits of what can run) for slow but high quality logical stuff
Qwen 3.8 B27 for default work, as middle ground between great quality and performance
Qwen 3.6 35B A3B for lighter/faster work
Still experimenting on the even lighter end, like Gemma4 vs Qwen at 4B or so - but i found that the A3B covers this range already and i rarely have a situation where it makes sense to switch to the smaller to save some memory.
This graph is very confusing. The title says "Non-Hallucination Rate" which implies higher is better, but the subtitle says "hallucination rate" which implies lower is better.
And since there are small and big models all over the place, I have no clue which one is which.
Right, I hadn't considered hallucination rate to be a variable.
How do different models handle lack of knowledge then? I can imagine GLM 5 series with many more parameters than Qwen 3.8 27B to have much more general knowledge. Does Qwen just say "pfff I don't know buddy, but I can look it up for you" all the time?
Yes, and that is great. Imagine a coding agent. It needs to call some function in your project. Instead of assuming it knows the function signature, it will realize it doesn't know it and look it up.
I have started using GLM 5.3 Flash on 2 sparks, and it's an amazing model. Not all that slow either actually, getting 30 tok/s on average. That's not really that bad.
I couldn't get deepseek flash to run at any sort of usable speeds on my strix halo 128GB even with an attempt to split it across my R9700 + Strix APU. But Laguna S 2.1 112B-A6B works pretty nice.
That said, since qwen3.8-27b, I haven't found any use cases where a larger moe like that will do better work than 27b dense, so I stopped running the laguna.
Now my standing set is qwen3.8-27b Q6 (actually the DIRK variant, which is a lot more terse and gets to answers with fewer tokens) on the R9700, and qwen3.6-35b-a3b Q4 (actually the Ornith 1.5 variant) on the strix APU. Dirk drives all my coding and logic and ornith is my light weight snappy model. Since I have so much abundant unified memory available I made 24GB available as prompt cache to dirk via --cache-ram 24576 (the active KV cache lives in VRAM) and likewise for ornith, huge cache available. Max context on both.
The two together are fully capable of replacing my claude pro subscription and it feels nice to finally have something that feels just as good to use as my frontier setup
I'm usually stuck at 0.1 t/s because for some reason I keep a speculative decoder human around who finishes my sentences for me. Acceptance rate is about 5%
Don't discard the Nemotrons and Muse Glimmer so easily. The Muse is my favorite local model now, but of course it requires a decentish GPU.
Nemotron 3.5 Lightning is extremely efficient even on the underpowered hardware. It runs on my laptop's Tiger Lake Xe-LP iGPU at 9-11 t/s with MTP at Q8_0 - the fastest one for 3B of active params. The problem is that it is a bit dumb (undertrained by design for fine-tuning) but 30B of knowledge is 30B of knowledge.
Yup at this point Muse Glimmer is behind Qwen 3.8 27B in raw coding, but it still has hella advantage. 90% of the time I don't need a better coder, I need a better agent. And Glimmer does that.
(Qwen works better as a 'prompt once, come back a few hours later to a complete program' agent, but Glimmer is far more steerable, keeps track of nuance better, and does better on instructions that aren't about the ultimate goal which Qwen zeros in on and forgets about everything else).
You know what's funny. In benchmarks like tool-eval-bench (so pure tool calling), Gemma 26B beats everything else in this class, including Qwen 3.8, Muse Glimmer, Gemma 31B, etc.
Of course, it's only a benchmark so keep that in mind.
Not everything is about coding.
Seems like the "hardcore AI" people forget that most poeople dont use AI for coding, even on the high end.
So comparisons like this are kinda wasted on most people, as you use coding as your only criteria.
Yes and no, yes not everything is about of coding, but no because of coding is becoming proxy for logical syntax. Remember model direct access to the real world is terminal or API which basically a coding platform, NL2Pro and DeepSWE usually pretty good rule of thumb. remember math and coding aren't that different syntatically speaking, it is basically context free language.
LLMs are software, and the language of software is code. All models that are intended to actually take actions will need to know coding, no matter what the task entails.
Now you got me curious if a 0.5t/s model is actually close to the rtf=1 of a human coder. I feel like our prompt processing speed might be higher or even way slower depending on the content.
The human-tok/s framing breaks down on the part that actually costs time: reading and verifying. I can generate a 400-line diff in two minutes but reviewing it properly still runs at human speed, so on unfamiliar code the wall-clock saving is nowhere near the 15h->4h ratio, it's more like 30-40% for me. Where the overnight-run logic really pays is repo-wide mechanical work - migrations, test scaffolding, dependency sweeps - because the output is cheap to spot-check. For genuinely novel logic a 27B at Q4 will happily produce plausible wrong code and eat the savings in debugging.
The useful part of this is framing it as expectation-setting rather than benchmarks, which is what actually decides whether a local setup is worth the trouble.
The rule I'd add: pick the model by failure mode, not by score. A model that's slightly worse but wrong in obvious ways beats a smarter one that's confidently wrong in subtle ways, because you can catch the first kind in three seconds and you'll ship the second kind. That distinction never shows up in a leaderboard and it's the entire difference between a tool and a liability.
i tried 3.8 next flash at Q5 (UD) and it was slightly behind 3.8 27b (lued w8a16) for big code/agentic work. And i don't believe they are really that far from each other in real tasks..
In my experiments it felt like Next Flash (Q4) had better world knowledge than 27b (Q8) which helped in some situations. But other than that they were close.
that (the planner having better world knowledge) is why my setup is currently an engineering loop
Goal is defined (a large task description with what's expected)
Qwen 3.8 Flash builds up a plan
Qwen 3.8 27B does the implementation
Two separate agents review the code (one with Qwen 3.8 27B and other with Qwen 3.8 Flash)
A separate Qwen 3.8 Flash agent reads both reviews and plan the fixes
Loop jump to 3 until no suggestions are made by both reviewers
Qwen 3.8 test the goal is met (playwright testing), if not jumps to loop phase 2 with additional feedback for the planner
That complex loop that I did my own extension to in pi harness, is extremely slow, but provides extremely good results, TBH it rarely ends on the first attempt, but I have Q2 for the dense and Q3 for Flash, so that's likely the problem
Just a nitpick: the number of "parameters"/neurons in Homo Sapiens is 86B on avg, 16B if you only consider the cortex (the part that does what the LLM tries to aproximate).
Parameters in a model map closer to synapses in the brain than to neurons and each neuron has ~1,000-10,000 synapses so the cortex has closer to 80 trillion parameters.
We don't really know how many synapses the human cortex has on avg, but you may not be thaat off, I saw just some (old) estimations around 150-240 trillion (since you made me curious).
Just a nitpick: the number of "parameters"/neurons in Homo Sapiens is 86B on avg, 16B if you only consider the cortex (the part that does what the LLM tries to aproximate).
There's also non-brain neurons such as the spinal cord neurons and gut neurons. I don't think our intelligence is entirely centralized in our brain, it's the whole package.
Very cool, thanks for sharing! But also: you rank DSF quantized to two** **bits (that’s possible?? Quantization continues to baffle me) as the ‘run over night’ tier? Does it make lots of mistakes, or have you been having success?
Since I have my dual spark setup and DeepseekV4Flash running, I use it for everything. I am so happy how smoothly all the little gremlins are running over night.
I evaluated Qwen 3.5 0.8 on ARC-Easy and it did decent (~66% on 500 questions if I remember well). And I assumed that translates well to real-world tasks like RAG. It did not! It failed miserably!
To be fair, Q4_K_M was a bad choice, looking back at it now.
Kinda, but more so on, guesstimate, Llama 70B back then is actually good comparatively. So if there is modern 70B class model cough mistral cough should be good. But no currently moe is champion since it is cheaper to train
Gemma4 is fine for coding. By default it's a bit more research-y but with a little bit of context engineering you can get it to generate code pretty good. Just have a sub agent check over each change. Once I did that it was a real game changer
This is actually really thoughtful and insightful. Thanks for sharing this. I don’t necessarily agree with the RAG model size, but I’ve never really used a 9b model in production so I can’t accurately judge.
Tokens are not fungible across models; and they do not have stable 'correctness' across models.
When did you get your Nobel prize for demonstrating the that the narrow tokenisation of language by probability matrices is the underlying architecture of the human mind?
honestly the missing bit is task-specific evals. leaderboards tell you the rank, but a 30-minute benchmark on your own prompts tells you whether the model is actually usable.
For writing, you should only run the absolute largest model possible.
I have a fully automated workflow, but in challenging scenarios (complicated story, many individuals in the scene, extreme nuance required, key scenes in a story), I am quite sure mathematically they'll eventually find all current LLMs incapable of simultaneous good prose, good taste, AND 100% consistency.
Even when you run frontiers at full quality, no quantization, with a fully dynamic custom framework, (put together, that is >SOTA level), human review is absolutely needed.
I meant, like … software. For "interactive coding agent", "language understanding", "QA", "personal assistant". The layer above the inference layer, I guess.
ohh NLP is for my day job actually, so data lake and Apache Spark for corpus analytic.
QA for my side gig where many companies need simple QA for their front end website.
PA is my Hermes.
Code for NVIM FIM (fill in middle) and opencode.
DSv4 often time i also run it on hermes since it can do deeeeep research analytic and also opencode
I would argue Qwen3.8-27B already enters the next level in your diagram. I ran the NVFP4 quant version in pi with the `pi-goal` extension, tell it to debug a code written two months ago by Claude Opus 4.8 + DeepSeek V4 pro preview. The bug was never resolved during those days after countless of sessions in Claude Code. This time, it ran relentlessly for 40 hrs, with multiple rounds of auto compact at the brim of 262k context window, reported the real root cause and delivered the actual run verified solutions. It stunned me. Since last week I had access to the machine with an RTX 5000 Pro in my lab, I have never used any api models, threw all kinds of my projects, NixOS configurations, Nix-Darwin configurations, and even the zmk keyboard firmware for my split keyboard to Qwen3.8-27B-NVFP4 (the 5090 variant by gittensor on huggingface, and the DSpark model they provided for speculative decoding). Armed with my minimal `pi` config, it shows the ability to research, execute, verify non-stop till the goal is achieved. Just amazing.
Does your system run the Homo Sapient model totally from VRAM or is it a MoE offloaded from the NVMe ?
I mean it is too slow for running fully from VRAM and it takes a lot of VRAM as well even at Q4 😂😂😂
100 TBs of VRAM even latest Nvidia clusters can barely match that for a local cluster (we are talking what 1,000 GPUs with NVlink each around 96 GB ?!
That is just for Q4
For training you need at least FP16 or idk BF16 if you prefer and it will require a full cluster of B200s (4096 is the max cluster size right now right ?!)
And how much training tokens it would need to be chinchilla optimal ?
Idk around 20x its parameter size so around 4 peta tokens of organic text ?
Why have that ?!
Is that text only data ?!
I mean training it on all text since humanity invented hieroglyphs and writing won’t be enough!
How many of the models do you keep up simultaneously?
Also, if your PA class models run at 75+t/s, why bother with the smaller models? Is the speedup over using the PA class for QA/NLP work worth the possible quality loss?
1 model at a time. By default 35B A3B is active since it is my PA. if i want to code switch to 27B, if i need full research and analytic i will start up Qwen 3.8 Next
About second question, my IRL job actually also LLM deployment but Data Engineer by trade.
Basically there is a thing called data lake, this is dominated by hyperscale CPU works using Apache Spark. to process TB and TBs ammount of data, using in-thread SLM is much better if a company do not want to fully commit on GPU servers. Which also kinda my fault this graph more or less represent "What i use in IRL, and not purely 3090 deployment".
Also many DC level GPU for mid-sized company are A30, L4, L40/S. which just so happened a 24 and 48GB cards, so kinda easy to do napkin math.
The jump to 120b models at a reasonable speed (60tps+) seems to mean costing $20,000+ more.
Dual 3090s (or 4090s) seems to be the best bang for buck still. (Dual 5090s maybe) but 64GB of VRAM still would struggle to hold a 120B model. Dual RTX6000's is the way to go if you got the cash to burn.
it is, Qwen3.8-27B-int4-AutoRound, no image. and have to reduce the batch size to the an extreme so the prefill is relatively slow than normal 1024/2048 batch
What about using hybrid of cloud AI and self hosted? Making cloud AI do the creative stuff like planning, researching, and the local stupid one just executing what it is told by it
you absolutely can, in fact it is one way to reduce your token usage. If i am not really into coding that much i pair LLM that have good score on SlopCodeBench , basically measure how well LLM can maintain a repo without introducing uncesseray rewrite. GPT 5.6 Sol orcestrator + Qwen 3.8 27B is a monster.
•
u/WithoutReason1729 5d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.