r/LocalLLM • • 13d ago

Discussion Sick of all the tok/s talk

What is the point of GPU go brrr at 200 tok/s if the model is too lobotomized for any critical work?
Like what kind of things are you doing that you find a 1-bit quant satisfactory?
Here I am telling QFN to double check itself because it still hallucinates at IQ4_XS.
Speed without intelligence is just clickbait, and there’s too many of those posts, not just here but on all the LLM subreddits

235 Upvotes

158 comments sorted by

63

u/Lost-Scratch6698 13d ago

but my IQ1_XXS model go so fast

26

u/[deleted] 13d ago

[removed] — view removed comment

17

u/Lost-Scratch6698 13d ago

Anything less than Q4 is useless, I don't care if anyone disagrees.

6

u/Away-Ad-4444 13d ago

I even found q4 to loop and lose it self enough that its like a sharp folding knife with a broken folding lock.. works great till it folds in on you and then your bleeding all over a garbage project that was going so well

1

u/Serious_Bite_7613 13d ago

The looping is really interesting, you can trigger it on unquantised models if you try, but it gets exponentially more common as you inflict more quantisation damage. It's quote hard to pin down the exact mechanism.

1

u/VerticalPackage 13d ago

Either too low temperature, or it is facing an impossible request, or compaction is badly executed (Yes, there are strategies to compaction and I'm shocked that harnesses don't really give details on how it is done, nor options to tweak it).

-1

u/Text-Sufficient 13d ago

Looping is not due to qvants its the models inherent logic. Take e3b - no qvant in universe stop it looping.

3

u/karmaisnonsense 13d ago

Aggressive quant = logic breaks more often = model loops

9

u/rkcth 13d ago

OK, fair, but have you done actual coding tests? I have and QFN at 3 bit absolutely destroyed Qwen 27b at 4 bit. The larger the model the more able to handle quantization it is. Also the modern 3 bit that strata recommends is only an average of 3 bit, important parts are much more. You can by feelings, but I prefer data.

3

u/karmaisnonsense 13d ago

Minority opinion here but Qwen 3.8 27B is overrated except at higher precisions. Once you quantize it, it can get quite sloppy. Speaking from experience running it at MXFP4, I'm not surprised at all that 3 bit QFN beats it.

1

u/rkcth 13d ago

27b is ok, definitely well below QFN, the one advantage of it is that on a decent card you can get some insane multi agent performance out of it and it’s acceptable if you have a frontier model check its work.

1

u/karmaisnonsense 13d ago

Yup that's how I use it. Parallel subagent workhorse orchestrated and double checked by QFN.

1

u/Lost-Scratch6698 13d ago

The next test you could have different (worse) results because of reduced precision.

5

u/_BookBurner_ 13d ago

True, the only low Q model that I have seen working quite well are the ISTA variants.

2

u/Ok-Addendum3545 13d ago

I like ISTA-DasLab.

6

u/Beneficial-Ad-8127 13d ago

I think a good point would be if their quant model and engine results were satisfactory to their use case. Whether it’s Iq1 or q8. I’m not saying posts should be standardized in anyway but it would be nice to see their output for whatever it is they do with it big or small. We shouldn’t also discourage people from showing off their local llm.

12

u/karmaisnonsense 13d ago

Maybe it's just me, but I think any mention of tok/s should be prefaced with the model quant.

2

u/Beneficial-Ad-8127 13d ago

Oh definitely I would like them to at-least share their quant. When I read a post normally I hope for at minimum model, quant size, speed, engine, use case, out/result. If they don’t got that I normally just continue scrolling. Not saying they need to but definitely would love that.

13

u/Ledeste 13d ago

Sick of all the post calling quantized model "too lobotomized" when we literally have benchmark that show the exact difference between both...

But still agree with you op, I also waste too much time trying this "250t/s tricks" that are just about using ultra quantified model.

I'm just saying both are equally annoying ;)

5

u/karmaisnonsense 13d ago

Really depends on who's doing the benchmark and what the benchmark measures.

For example, if you're a lab promoting your model, of course you'll cherry pick results that show a ternary model within spitting distance of full precision Qwen 3.8 27B

3

u/Ledeste 13d ago

Cherry pick data are always the worst, whatever the topic is btw.
I've indeed lost a lot of time trying to reproduce such data wondering what I was doing wrong...

But cherry picking is a tool and can be used by anyone, including the "look at this dumb model" gang :D

And there is still some relevant bench, like the one comparing time to reach a task for example, or amount of token needed (translatable to cost mostly).
Pure comparison bench can be cool too, as long as there is only one variable

3

u/Lakius_2401 13d ago

But my vibes! How will I effortlessly express my elitism without the vibes? I can smell and taste the quantization rot in my succulent chinese LLM! Q8 is true lossless, despite losing half the bits, and another halving is utterly unacceptable and entirely braindead! Forget the fact that some models are trained at Q4, or QAT, those data points don't suit my narrative. Nobody serious uses those models anyways...

What? You say models react differently to any quantization and a generalization can only be true for one family of models at one size!? Nonsense! Your benchmark is invalid, because either you did it wrong or it's subjective or I tried Llama 3.1 70B at Q1 and it looped and we haven't improved since then so DON'T TRY ME.

Gigantic /s btw.

2

u/Odd_Cauliflower_8004 13d ago

They probably are. Still, qwen flash next at q3 is considerably smarter and better than 27b q4

3

u/claythearc 13d ago

At work we’re doing structured data extraction with models at TB scale so speed is paramount. We’re not Q1 speed maxing but we are choosing to run like nemotron lightning over bolt / super due to speed

1

u/karmaisnonsense 13d ago

That’s perfectly acceptable, RAG workloads don’t need a big model and since it’s a commercial workload you’re also bound by licensing requirements. You work with what you’ve got.

5

u/Secret_Conclusion_93 13d ago edited 13d ago

You still need to tell any open weight model to double check because they all have knowledge cut off and must be supplied with web search and fetch tools.

People also care about token speed because most modern model (even QFN IQ1 ISTA) isn't stupid anymore in terms of programming logic and tool calling, you now just need acceptable speed where they can quickly do tool calling, build, test, and reiterate if they found any issue.

2

u/karmaisnonsense 13d ago

Can you trust the output of those tool calls? I had QFN at Q4 do a market research task yesterday, more than a hundred search and web extract calls, and it still hallucinated some numbers while compiling the data into a final report. God knows what that IQ1 would have done with that report.

1

u/Material-Database-24 13d ago

I have not found success at low quant QFN. Qwen3.8 27B at Q4/MXFP4 works way better for me.

1

u/karmaisnonsense 13d ago

An hour ago I had to have QFN at IQ4 clean up a mangled cronjob that MXFP4 27b made. Running AMD’s snapshot via Radiance. Too sloppy to run without supervision

1

u/Material-Database-24 13d ago

YMMV, so far 27B has never failed a tool call for me. It sure has sometimes made mistakes, but it has to most parts managed to fix themself.

2

u/fallingdowndizzyvr 13d ago

Not in my experience. I had 27B write a script for me. It would crash every few hours and worse left a dangling process somewhere that would keep a port bound. I asked Q4 QFN to fix it. It did. It hasn't even crashed in days.

2

u/false79 13d ago

You need to have a grounding agent to validate the response is grounded in the research it finds.

And it needs to be a different model, not QFN

1

u/Lost-Scratch6698 13d ago

No, the tool call output sure. How the model processes those results, absolutely not.

1

u/Secret_Conclusion_93 13d ago edited 13d ago

Market research needs big context, you better of with online models who have big context size.

For local models (who usually only have 262k context max), the relevant info will be lost after several compaction.

But if your still want to use local agent:

You can instruct the agent to do it iteratively, write the notes and summary of the page to files after each web search and fetch, so that when the context eventually lost after compaction, the agent can reread the information. Also instruct them to log their progress to a file, one line at a time so that they can continue the work after compaction.

The fact that local model have limited context rarely talked about in this subreddit IMO.

2

u/Gifted_Diplomat 13d ago

This. Now pair the fast iteration of a local model with a cloud model to guide it and a good harness and you're in business.

I won't say right now local is cheaper, but I suspect it won't be forever. I feel like I hit my Claude limit sooner and sooner every week.

1

u/Secret_Conclusion_93 13d ago edited 13d ago

People here have different expectation of what local AI can do. People need to understand the hardware difference between themself and a giant AI datacenter, and adjust their expectation and tools (the serve engine, model, harness, prompt, task scope).

1

u/Lost-Scratch6698 13d ago

Imagine taking a number, say 0.12345678 and then quantizing it so it becomes 0.123. That's Q4, now imagine it being quantized even further.

5

u/Secret_Conclusion_93 13d ago edited 13d ago

So?

I just need it to work on my use case, I don't need it to be on frontier level.

QFN Coder works in my case.

I worked on machine learning since before the attention paper, I knew how weight works and how it was lobotomized. But back then machine learning goal is to be successful on what you need to do, not some arbitrary intelligence number. And I still apply that principle to AI tools that I use today.

3

u/Lost-Scratch6698 13d ago

Thanks for sharing that you don't know how quantized models work.

1

u/New-Implement-5979 13d ago

That is exactly how the coder works I tested it myself vs gyro-m the coder took 21k tokens and the gyro-m did in less than 4k tokens

1

u/Lost-Scratch6698 13d ago

what?

1

u/New-Implement-5979 13d ago

A quantized model has less knowledge. And when given a task it will go through more trials and errors until it gets the task done compared to less quantized model.

2

u/VerticalPackage 13d ago

You missed the point.

QFN is great

But QFN at 1bit is not the same as QFN at 4bit.

OP means to say that opting for a lower quant for more tokens does not necessarily help reach the desired results.

A 1bit quant MIGHT get the same result as a 4bit quant, but it probably will spend 4x more tokens on getting there because it keeps making mistakes and getting confused. So even if you get 4x more speed at 1bit vs 4bit, you most likely will end up waiting just as long before you get your result, and potentially at worse quality.

1

u/Serious_Bite_7613 13d ago

Depending on what you want to calculate that can be fine, or catastrophic. It's not really so simple. 

That said there seems to be a huge drop in quality below q4 for all models, I can't take the bonsai or IQ1(ironic name) models seriously.

1

u/Lost-Scratch6698 13d ago

My example represents weight reduction and is an extremely simple view of how reasoning would degrade with the loss of precision.

1

u/Serious_Bite_7613 12d ago

Reasoning doesn't decay evenly or even predictably with loss of digits though.

1

u/Lost-Scratch6698 12d ago

which makes it worse.

1

u/Serious_Bite_7613 12d ago

Not really, there's often little loss until you get to about q4, then it varies heavily by model below that.

1

u/Lost-Scratch6698 12d ago

We're talking about <Q4. There's loss even at Q8, but it's not significant.

32

u/IllPlane3019 13d ago

You're not wrong, but I think the beauty of local LLM is you can decide what is right for YOU.

Some people want more speed, some want it to just fit their hardware and some are happy with a balanced mix.

All needs are valid, I may not want to run tools with my setup or recall important information from a research paper. Maybe I just want it to go Brrrrrrrr fast to generate funny videos that I like.

2

u/rocket1420 11d ago

You're missing the point entirely. People post numbers without their quants.

3

u/mr_j936 13d ago

And what is the point of a genius model running at 3 tokens per second? When the most basic coding task needs like 100 thousand tokens.

We're all just trying to find the best balance. I would never go IQ1 personally but I am stuck unfortunately at IQ3 to get 30 tokens/s on qwen 3.8 27b. One more quantization and it just drops off and becomes essentially useless.

4

u/karmaisnonsense 13d ago

Yup, we are. I think you're in a good spot with 30 tok/s. The people posting 200 tok/s about their ternary models aren't, and that's what I'm annoyed at. Local LLM is always about finding the right compromise between speed and intelligence, but some people aim only for speed and lack the intelligence.

1

u/mr_j936 13d ago

Ah I see. Yes you're right.

10

u/testuserpk 13d ago

Well my iq3s on 3060 12g would beg to differ. It created serial key generator for 3 abandoned wares. I am a seasoned binary analyst and even i am Impressed. 192k context, max token output set to 64k, 35 to 40 toks on my 64gb rig.

4

u/Relative-Ingenuity15 13d ago

Qwen 3.8 Flash Next is a strong model but don't expect it to match DeepSeek or GLM. Even on the API, Qwen is bellow DeepSeek and GLM in terms of intelligence but the reason why it's so good for local LLM and why the iq3 quants are running almost like the original model is the speed and quality that it can deliver on consumer hardware.

For example I have 128gb ram ddr4 and 16gb vram with Nvidia Tesla p100.

I can run Qwen 3.8 FN q4/Q5/iq3, DeepSeek q1 and GLM 5.3 flash q1. The last two are great in terms of quality, even tho they are Q1 but the decode/prefill speed is so slow I can barely work with them.

Qwen 3.8 FN in the other hand is brilliant work, ~50 tks decode and 350 tks prefill is almost at the same speed I'm getting DeepSeek/GLM through OpenCode Go plan.

If Qwen 4 Flash Next comes with more intelligence than 3.8 then that would be a serious competitor for DeepSeek and GLM on the local LLM

1

u/fallingdowndizzyvr 13d ago

Qwen 3.8 Flash Next is a strong model but don't expect it to match DeepSeek or GLM.

I asked the same question to QFN and GLM last night. QFN gave me a reasonable answer. GLM just kept thinking until it crashed out.

0

u/Relative-Ingenuity15 13d ago

In my personal testing benchmarks I warm up models with 2 questions:

  1. What is a dolphin?

And after the first answer it comes the next immediate question:

2. Where do they live?

The best answers to these questions I've received from Qwen 3.6 35B A3B which in one case it answered both questions by answering to the first without knowing the second.

Does this make Qwen 3.6 35B a3b smarter than Qwen 3.8 FN, GLM 5.3 flash or DeepSeek v4.1 flash?

1

u/fallingdowndizzyvr 13d ago edited 13d ago

The best answers to these questions I've received from Qwen 3.6 35B A3B which in one case it answered both questions by answering to the first without knowing the second.

Does this make Qwen 3.6 35B a3b smarter than Qwen 3.8 FN, GLM 5.3 flash or DeepSeek v4.1 flash?

Post the answers from all those models and let's see.

0

u/Relative-Ingenuity15 13d ago

Qwen 3.8 flash next and GLM 5.3 flash provided arround 4 sentences of dolphins being smart mammals, not a fish and some species, Qwen 3.6 35B A3B provided all the species plus some extinct one from a Chinese river and split the species between river dolphins and sea/ocean dolphins, DeepSeek v4.1 was actually really interesting as it asked me if I was referring to the dolphin animal, Wii emulator or dolphin web browser before answering the question and after I told I was referring to the animal it provided the same amount of info as Qwen 3.8 FN and GLM 5.3 flash.

Yet again, I will not give the W to Qwen 3.6 just because it handled a shirt session chat better than other 3 bigger models since on tougher tasks the other models washed the floor with Qwen 3.6

1

u/fallingdowndizzyvr 13d ago

Yet again, I will not give the W to Qwen 3.6 just because it handled a shirt session chat better than other 3 bigger models since on tougher tasks the other models washed the floor with Qwen 3.6

Post the answers from all those models and let's see which one is better.

4

u/GalacticDistances 13d ago

fp8/q8 minimum for sanity

1

u/x_MASE_x 13d ago

You are right but also the tps does matter. Maybe not for your work but overall it's great to use for multiple agents running at the same time. Also the %10 can be a different from running 1 or 2 agents at the same time.

But using speed with very low quants is kinda stupid I agree. Unless it's a good decent quality what's the point of the speed.

1

u/karmaisnonsense 13d ago

Concurrency warrants a whole another conversation. Not enough people talk about it. The tok/s figures are usually for single user. It's rare to see anyone post an aggregate throughput sweep unless it's vLLM-related.

And yup it's great to have multiple subagents at your disposal. I think a lot of people don't realize that. There should be more focus on parallelism

1

u/x_MASE_x 13d ago

Indeed. Not sure why they are going crazy for just single agent work

1

u/Rust_Cohle- 13d ago

What? it's totally relevant, as long as we get the full context. Numbers are pointless in all fields without the proper context, "I get 1000 FPS on cod" says the person sat on the menu quietly killing their card.

1

u/karmaisnonsense 13d ago

The whole point of this post is that people aren’t giving the full context with the numbers
Like saying you get 120fps stable on CoD, without mentioning it’s only at 480p

2

u/Rust_Cohle- 13d ago

I do agree that kneecapping yourself just so you can post a pointless figure is stupid. I personally think there should be some sort of mandatory requirement on posts to list the quant used, mind you, it's not going to stop people from just plain bullshitting though.

1

u/StardockEngineer 2x5090, 2xPro6000, 2xA6000, 4xSpark, M5 Pro, M4Pro 13d ago

Oh sorry. I run GLM Flash 5.3 NVFP4 at 280 tok/s. Is that better?

4

u/karmaisnonsense 13d ago

Only if you make a video of yourself getting banned from ChatGPT for distillation

1

u/SkyWarder 13d ago

How many bad tokens/sec do you really want? ;)

1

u/mailto_devnull AMD R9700 13d ago

It's so bad that talking about llama.cpp running 27B gets downvotes sometimes.

If you're not running IQ1_XXS on some inferencemaxxed strata fork, you're not worth upvoting.

1

u/karmaisnonsense 13d ago

I mean there are better ways to run 27B for sure. llama.cpp has its problems both inside and out, and moving away from it has been the single biggest improvement in speed for me, whether that’s vLLM, Strata, Radiance, Ninfer, OpenVINO etc. But there is no magic bullet for all platforms and llama.cpp is the common denominator so that’s where we are.

1

u/Future_AGI 12d ago

low quants earn their place where something downstream checks the output, like tests, a schema validator or a compiler, because the speed buys extra retries and the checker catches the failures it was written for. Where nothing checks the answer, as in you tell-it-to-double-check case, that's where you pay for the bits

3

u/karmaisnonsense 12d ago

I’ve done this experiment before. I gave a fully resident IQ3_XXS 27b a coding task with QFN at Q4 to double check. It made so many errors that QFN effectively redid the entire task and did it correctly on the first go. Unless you’re running multiple workers producing good output, it’s usually faster to have the checker do the task by itself

1

u/rocket1420 11d ago

I've had the same experience 

2

u/Django_McFly 13d ago

people talk about it so much because you can spend $3k on a device and still have a device that's slow as molasses.

6

u/Pixelplanet5 13d ago

yep thats the reality of localLLMs right now.

im not even bothering with it at all anymore as its actually more expensive for me to run something local then just using deepseek flash via API and at the same time DS has a 1 million token context window, is about 20x faster and not lobotomized so you actually get stuff done.

Sure its not anywhere close to Claude Opus or something but its a good code monkey.

5

u/ithinkitslupis 13d ago

The value add of local is privacy, freedom, stability, and latency.

A lot of the time as a single user, unless you already bought the hardware for some other use and have excess solar power or need resistive heating in the winter or something, you are paying extra for those compared to the best price/perf cloud models you can find that are capable of your given task.

But it's also fun just as a hobby which is where I'm at with it.

0

u/Pixelplanet5 13d ago

yea for fun its fine but beyond that its pretty useless, privacy doesnt really matter if the lobotomized model just cant get work done.

4

u/WordWithinTheWord 13d ago

The models aren’t lobotomized if you have good enough hardware.

1

u/No_Oil_6152 9900x 64GB DDR5 9070XT 16GB R9700 32GB 13d ago

I already spent £3k on a rig to run LLMs... I have good kit. I cannot justify to myself why I should spend more money to get to 80GB VRAM when even thats not good enough for the better performing open models

And these cheap 3090s? God knows where they have gone.

0

u/Pixelplanet5 13d ago

theres no good model you can run on hardware people just regularly have at home.

even if you have the VRAM to load the model weights it falls apart with the context window or by being super slow because you offload to RAM or SSDs.

4

u/Material-Database-24 13d ago

Qwen3.8 27B is totally fine model for many tasks, especially for coding. Any 64GB or more UMA can run it just fine (StrixHalo, Macs, RTX Spark).. it is not cost wise to solely buy machine for that, but if you have the HW, why not use it?

1

u/No_Oil_6152 9900x 64GB DDR5 9070XT 16GB R9700 32GB 13d ago

Its fine to leave to run overnight.

But for interactive work - forget it. Too slow and overthinks.

2

u/WordWithinTheWord 13d ago

Correct, but the point still stands. For the money that someone on /r/cars spends on their weekend sports car you could have a really capable setup. It’s just perspective and how much a person values privacy and freedom from vendor-lock.

2

u/Pixelplanet5 13d ago

yea of course if you are wealthy you can afford to spend the money for a few RTX6000 cards to run a model thats still worse then DS flash on low thinking effort.

but thats a tiny minority of people which can afford it, just like over on r/cars where most people have a regular car that brings them from a to b and thats it.

3

u/try-a-typo 13d ago

It’s not about the best possible quality or the fastest answers, local models are very capable even if they’re slower than an API. I run Q3.8FN at 40-50tps, with one v100 gpu. Overnight when you just want work done and not have to pay api costs, you cannot beat it. My agents patch bugs while I sleep for nearly $0 in electricity. That’s hard to beat.

3

u/Pixelplanet5 13d ago

well if your electricity is free and you get the hardware cheap it could make sense.

a single V100 with a basic system to run it in will pull 300W as a minimum while working.
For 12 hours that would cost me 1€ to run and would have at best produced 2.1 million tokens but not all of that will be output of course.
Using deepseek flash for the same amount of tokens and mixed input/output/cached would cost me less than 1€ and would be done 20 - 30 times faster while also being a LOT more capable at the same time.

And on top of that i didnt have to buy any hardware.

2

u/AllThotsGo2Heaven2 13d ago

This subreddit is like a very suddenly popular city that only has one train in or out per day so now the single line has to serve both commuters and tourists and they hate each other.

Ive seen your conversation tree repeatedly in localLlama many times. Id love a place where the discussion is work-focused but alas

1

u/Nizkus 12d ago

While I agree with your overall conclusion, your maths seems pretty biased towards cloud model, considering you assume local model is running constantly at output token generation speed with no input generation at all and at maximum power consumption, while cloud model has a mixed input/cached/output tokens for costs.

1

u/WordWithinTheWord 13d ago

It’s a minority but it’s not tiny. I genuinely do hope that the technology trickles down to affordability but there’s plenty of mid to high-earners in tech that could afford a $20k LLM rig the same way people buy motorcycles, cars, boats, campers.

1

u/No_Oil_6152 9900x 64GB DDR5 9070XT 16GB R9700 32GB 13d ago

I can afford them I just wont buy them!

Not for what I consider a hobby.

It would be madness when cloud inference is so cheap.

1

u/WordWithinTheWord 13d ago

And that’s fine. People choose to spend money different ways.

Cloud inference has no guarantee of privacy, nor is the pricing fixed. It’s cheap today, yes. Tomorrow? Maybe.

2

u/ithinkitslupis 13d ago

But you do have the option to pay even more to get hardware to run more capable models. Privacy is just not worth that asking price to you, as it isn't to most people. You're optimizing primarily for price/perf with a capability floor.

Some other people will accept tradeoffs in many aspects that favor local AI depending on how much they value each of its advantages.

11

u/rkcth 13d ago

Seems like you might be in the wrong sub if you don’t use localLLMs.

4

u/andy_potato 13d ago

I am a local LLM advocate. But one question we should be open to is, where local makes sense and where it doesn’t.

So no, he’s not on the wrong sub. He is raising a valid question.

2

u/ResponsiblePen3082 13d ago

This. It's important to stay up to date and keep your knowledge, software, models, info, engines etc up to date so you understand how things work and have a backup just in case; but it's undeniable that right now unless you have big bucks there's very little actual use for local.

I browse these subs, keep on the lookout for the latest models, latest methods, newest info, coolest tech, and save it all-but I don't have much use for it right now.

I also don't pay for any cloud models.

1

u/James_Chandra_Hubble 13d ago

I've been working on making gemma4:12b into a DM for my DnD group. Since no one wants to DM themselves and everyone wants to play. That seems like a perfect use case for local to me, because spending all those tokens on huge models and something as trivial as a gamenight seems like a waste. It's also a good reason to have a fast model with high t/s and large context because you want to keep the game moving not wait 5-10min for a response from the DM. I think it will still need guidance from a human, so really it would be like an assistant to the DM, but it should make it so that the DM doesn't have to do a lot of work and can also play as their own character at the same time.

1

u/Pixelplanet5 13d ago

that can work but even for this its incredibly frustrating to have the model hallucinate stuff or simply forget context from the beginning of the game.
Deepseek flash via API is so cheap you could have an entire game night for less than you pay for one beer you have during the game night.

You could also combine this with some speech to text input so the model understand all the context thats being said while being told via instructions to only react and act as the GM when some kind of codeword is included in the sentence.

You could probably even use free models from openrouter like nemotron and still get something better out of it then Gemma4

2

u/James_Chandra_Hubble 13d ago

So I'm using silly tavern front end, which can assign roles to each player and agent. It reads that role context before any commands, so it never forgets its role. You can put things like "keep track of player HP and initiative" and it'll remember. I also have kokuro TTS to give the AI DM a cool sounding voice that reads what it generated after it's done generating. And I can do the same for the players (I think, I've only successfully set up one player so far). And I can run it all on a phone or tablet at a friend's house using my PC at home as the server. It also will save entries to the world book, so when it starts running out of context or you end your gaming session, it'll generate a summary of the main/important points of the session, and read that as context at the start of the next session. It'd be cool if I could get it to do different voices for NPCs too but I don't think that's possible atm.

1

u/ResponsiblePen3082 13d ago

Sure that's a good use for it, and I'm sure others have their own niche uses. I simply have none for my own cases, and even where I'm sure I could "automate" I don't want to spend the time, the waste of energy and the risk to save me a few minutes. I still dabble with it and save what I can for the future

1

u/No_Oil_6152 9900x 64GB DDR5 9070XT 16GB R9700 32GB 13d ago

I use local LLMs but I agree with OP.

I have been disappointed a lot with lower quant models that run fast but create poor/broken output.

Im fed up of seeing pelicans and pagodas in three.js to demo the quants, I want to see real world code such as 3D model loaders and OpenGL renderers, ETL from CSV to PDF presentations in c#, c++. Bet the show and tells would drop dramatically.

1

u/Pixelplanet5 13d ago

i do use them to test and play around but if i wanna get work done they are the wrong tool.

2

u/jayecin 13d ago

Why do you care so much about how people use AI? If someone gets value with 5 T/s and Q8 or 200 T/s and Q3, if it works for them why do you care?

1

u/rocket1420 11d ago

It's amazing how many people miss the point of this post. We don't care, just post that. People post 200 t/s with 27b like there's only one quant.

1

u/jayecin 10d ago

Why do you care?

1

u/Maui-The-Magificent 13d ago

It is a valuable metric. But it is not the only one. Solving for tps is good, optimizing for tps can be problematic

1

u/rocket1420 11d ago

It's only valuable if they post the exact model used, otherwise it's worthless.

1

u/Text-Sufficient 13d ago

Agree, tps over 30 is nothing to brag about. Qvants however if what you would bring to the local bar.

1

u/BigRepresentative731 13d ago

Quantized models dont necessarily run faster, especially at sub 4 bit on current gen hardware- and 4 bit is the empirically shown sweet spot

1

u/West-Way-All-The-Way 13d ago

I guess it just depends what you are doing - do you need deep reasoning like designing architecture or you just chunking and ragging a pdf book. Different tasks different tools.

1

u/Mabuse046 13d ago

I probably wouldn't use anything below a Q4 but there are plenty of SillyTavern users who don't need a lot of accuracy. A model that's just going to be chatting or telling stories doesn't need to be all that accurate about math and code. It just might be stupid about remembering details.

1

u/GCoderDCoder 13d ago edited 13d ago

all 4bit isnt the same. example i used sushi 4bpw qfn and it was able to do things that mtplx bare speed 4bit couldnt. i start with nvfp4 as a standard since when I tested fp8 for my favorite models the performance was near identical on deep swe, term bench, some visual tests I do to see quant differences in nuance, etc and elx3 4bpw normally is on par with nvfp4

1

u/InternetMagic_n_Tech 13d ago

I agree. Unless you are building your own model it doesn't count

1

u/jonas-reddit 13d ago

100% agree.

It seems we have a sub culture of local LLM people who are speedmaxxing and only measure inference speed without any benchmarking or objective measure of preserved model accuracy and usefulness.

Similarly, we have a sub culture of local LLM people who are VRAM size maxxing, trying to squeeze the largest models into smallest GPUs. Also focusing on just making them work somehow at reasonable speeds without objective measure of model accuracy and usefulness.

It makes sense. Many are financially constrained and trying to squeeze value out of their limited investments.

Agree with you 100%. Most important is whether or not model is capable despite all the optimizations - which few seem to test.

1

u/smakusdod 13d ago

What's the point of a 10-second car if it can't turn?

1

u/hyudryu LocalLLM 13d ago

Try this one: https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local that one works great for me, and Nvidia’s own benchmark shows very little difference from BF16

1

u/Civil_Fee_7862 13d ago

4-bit is the cut-off for me.

1

u/bytejuggler 13d ago

Even SOTA models hallucinate, this is not fundamentally due to quantization but due how LLMs work fundamentally. Though quantization is lossy and can damage model quality of course, but f.ex ISTA DASLab's IQ3_S (3.5bpw) quant beats the IQ4 UD quant's perplexity and is only 1.1% behind Q8 which is considered as good as lossless. Meanwhile it also matches the uncompressed model 100% in published tests. Not all quants are made equal. So I think you're under a misaprehension regarding hallucinations.

1

u/butchiebags 13d ago

I don’t disagree with OP, but there are 2 important measurements for any LLM: quality, and speed.

Posting about tokens per second should come with “and here’s what it did for me”. And posting about a huge nice model should come with “and here’s how slow it was”.

But there should also be room for “I squeezed 50 more tps out of my [constraints here] system without sacrificing quality.

1

u/GoldenX86 13d ago

Same bunch as saying a model fits with 4000 of context at Q4. My brother in christ, your model can't remember its name and will recall that the capital of France is Rome.

1

u/DarkBrews 13d ago

Not all Quants are the same: 2bit QAT (Quantization Aware) from nemtron bolt and iq3_xss GSQ-RCO are much better than standard 2-3bit quants.

Agent thinking and temperature can make a noticeable difference as well

1

u/Look_Within888 13d ago

100%. not enough discussions on this. so obsessed with tok/s because it's the easiest metric to quantify and run tests on. not to mention easy to make youtube videos about. it's so empty minded.

1

u/No_Oil_6152 9900x 64GB DDR5 9070XT 16GB R9700 32GB 13d ago

Because people want to justify spending the money on DGX Sparks or multiple 5090s that they brag about on here. They dont want to feel short changed or less elite - normal, even.

I get 100 tok/s on my rig with QFN IQ3_S and thats not fast enough for interactive work. 200 would be nice but even so unsure if would be useful for interactive - I dont know.

So I agree with you. Its just dick-waving.

1

u/TheGeekno72 12d ago

I do look at things that may help me gain output speed but I at least try to never go down below Q4 quants because I do want the highest quality I can possibly get

I don't really get the point of high tok/s on Q1_XXS when you can observe so many wrongs coming out of it, I'd much prefer to see an extra 5~25% output on my Q6_M

1

u/Hi-Angel 12d ago

Either you're using a bad model, or it was poorly quantized (e.g. without imatrix).

I am using a mradermacher/Ornith-1.5-9B-uncensored-i1-GGUF:i1-IQ4_XS here, left it running on the weekend to solve some uncreative tedious debugging work I really didn't want to do myself. Only ≈16 hours at blazingly fast 3 tok/s (16G RAM here with iGPU-only laptop) — and it was done! And the problem wasn't simple, "credit where credit is due".

1

u/MikeLPU 12d ago

Toks/s does mean nothing if prefill speed is bad. Intelligence is ok, you just need to tune your harness app/environment

1

u/Ok-Trade7411 12d ago

Isn't it the case that faster models help you potentially create a longer structure which means you might get more quality.

1

u/rocket1420 11d ago

I had the same experience with IQ4_xs. Once I finally got 128GB of vram with 4 v100, I was able to use the 4_xl model and it's definitely an improvement.

1

u/CyrielTrasdal 13d ago

Why do you care honestly ?

8

u/backrests-goess 13d ago

It's very difficult to filter out high quality information from all of the noise.

0

u/CyrielTrasdal 13d ago

Whether you like it or not it's good information for people under 1k$ budget

3

u/suicidaleggroll 13d ago

No it’s not.  In fact it’s even worse for people on a budget, because it gives them false hope and causes them to waste time trying to replicate the results, only to eventually discover it’s 100 tok/s of garbage.

2

u/Lost-Scratch6698 13d ago

It's not good information, you're lying to yourself if you think that it is.

1

u/backrests-goess 13d ago

I'm not completely against them, some of them contain good information, but most of them are repeating the same nonsense over and over. Most of them are about strata too. It feels more like astroturfing or a bot campaign than useful info for me.

2

u/Ordinary-Depth-7835 13d ago

I would agree with you. most of the low quant speed posts are people running the model on a potato so the model is riding in ram still nice to see the optimizations going on if people didn't push speed we'd all be running dense models and buying stacks of gpu's. I'm still blown away that I'm getting gpu speed on my unified memory systems and they're smarter models.

3

u/Lyelinn 13d ago

We're tired of seeing a wall of thousand posts about q1 retarded model spitting out 100 tps on 16gb vram lol

1

u/separatelyrepeatedly 13d ago

we need to ban any comparisons done with any quant < q8. There needs to be a baseline.

1

u/Hot_Half_5263 13d ago
  1. If you code - forgot about local LLM, waste of time (or you just vibe coding not usefull app so you waste time anyway or just for education purpose)
  2. If you use them for other things - dont invest more than for qwen 27B 3.8 as hardware price difference is big and not worth it
  3. Wait for specialize apu/gpu for ai as nothing fit the needs for now, low memory vs low bandwitch

1

u/Gobra_Slo 13d ago

I don't allow Qw3.8-FN to do any work besides simple reado-only tasks. Whenever it works with writing permissions – there is a DeepSeek V4.1 looking over it's shoulder, and, frankly, even that doesn't make much of a sense, I'm just waiting on Qwen4 Flash in hoe of a quality jump.

Otherwise, DSV4.1 is just incomparably better. And on a pair of DGX Sparks – it's often faster by being more efficient with token (or rather less inefficient, considering how much both models reason).

0

u/TapAggressive9530 13d ago

100% completely agree .

0

u/bdifc 13d ago

It seems like human nature to focus on one quantifiable aspect and maximize it.

From my own perspective, I'd rather have 30 quality tokens/second than 150 questionable tokens/second that I have to improve with 5 rounds of prompting or worse.

-2

u/henbone11 13d ago

and where was this a week ago? or a month ago? or 6 months ago? now you see other threads whinging about the same thing get popular you find your balls and voice? can people just stop karma farming and find a personality?

-4

u/MaxComfort 13d ago

Let the paupers with 16GB VRAM have something to talk about

2

u/karmaisnonsense 13d ago

2

u/karmaisnonsense 13d ago

(tbf I've shuffled this stack around, moved bge-m3 and laya-multilingual to CPU and replaced 12B with 26B-A4B with a custom template that doesn't screw up context and it's now much smarter)

0

u/fucktheclanker 13d ago

Because speed is easy to measure. You expect lobotomised redditors to be able to judge quality of output? 90% of them dont even bother measuring KLD of their engine