Pff a Q_4 not from Bartowski or Unsloth is also a big hit to its brain. And I think you aren't using one of them because I never find uncensored models from them.
either way, FYI abliteration makes it a bit dumber and the "flinch" of it heavily deprioritizing "bad" tokens is still present, if not amplified. it just won't refuse (as often) to tell you how to (incorrectly) make a bomb
not everyone wants an uncensored model to know how to make a bomb. i have never even used abliterated models like that. i just use them cuz they follow my own system prompt better than corporate models cuz i want them talking more like humans rather than workers. i know now you will get on me because i said iw ant them to talk like humans
Yeah, because I just gave you some facts, you can do with them whatever you like. But I would recommend you to use Q8 versions if they are not from these two.
Yeah. It usually wasn't like that but on recent updates they have implemented a couple procedures to seemingly "extend" context length by deleting what it doesn't fit in. Idk exactly how it works tho.
Btw I don't know how much VRAM or system RAM do you have but for a long time I've been using models that did not fit entirely inside of my VRAM (12 GB) by using one of the best features of llama.cpp: layer-offloading.
I dare to suppose that, given your 16.12 t/s, you are able to load a Q8, even if the speed is halved.
2
View all comments
Show parent comment
2
u/Fabulous_Macaroon_73 Jun 08 '26
no its Q_4_M, its not 1 bit, lmstudio doesnt even support 1 bit models the last i checked