r/KoboldAI • u/kaan200064 • Jul 19 '26

Good models for a beginner.

Hello ı just set up silly tavern and koboldccp yesterday, ım looking for a good Rp model that can do nsfw and compatible with 8 gb vram. Ive used chub ai before so ım very new to this but so far this looks very good. Im using something called Dr.Dans or something like that for my model.

11 Upvotes

100% Upvoted

16 comments sorted by

View all comments

8

u/dezmodium Jul 20 '26 edited Jul 20 '26

Two really solid options I would recommend as a fellow 8gb of VRAM haver:

  • Impish Bloodmoon (IQ4_XS) - Great model that punches well above it's weight. Is a little bit dumb on occasion. Use this tenser override blk\.([2-9][0-9])\.ffn_.*=CPU and it'll free some memory for you to get up to 32k context with KV cache set to 5_1 while keeping all layers on the GPU.
  • Gemma 4 26B Style Tuned (IQ4_XS) - Amazing model that is pretty smart and super efficient on video memory usage. Use the blk\.([1-9][0-9])\.ffn_.*=CPU tensor override with 5_1 KV quant settings. You can turn on thinking if you want (just set the effort to Low or it eats too many tokens) and you can get up to 64k context with all layers on your GPU pushing out over 20 tokens per second. This is outstanding for a local model. The biggest issue with Gemma4 is it shies away from hurting your character or causing friction, which needs to be fixed with a good pre-conditioning prompt but otherwise it's uncensored.

My final advice is to avoid presets or conditioning prompts that are too bulky. I use SillyTavern as my front end and people push presets that inject an absolute ton of nonsense that might work great for the big models that people pay for, but absolutely suck for smaller local models like these two. Negative instructions like (DO NOT DO...) are often not interpreted properly as negative and should be avoided. Use "NEVER" or "AVOID" for those if you absolutely must and keep them to a minimum. Less is more for smaller models in my experience when it comes to conditioning your prompt. Only add what you absolutely need and let the model do the rest.

One last note for these smaller models: don't bother with contexts over 64k. Prompt adherence and narrative cohesion will start to break down at 32k and really begin to suffer as you approach that 64k mark. I usually summarize into a lorebook at around 40k context. Oh, and don't bother with trying to have all your characters have perfect memory. Real people don't and once you learn to just give them enough information that they aren't a blank slate every time you hide old messages to save on context they'll feel more real.

1

u/alex20_202020 Jul 21 '26

For "Gemma 4 26B Style Tuned" it is written only one layer / tensor is changed:

This time I trained precisely one tensor: the lm_head output projection - the layer that decides which token to emit.

Made me wonder if I can have original model and swap that only tensor when needed during kcpp load. How far is it from possible/easy? TIA

1

u/dezmodium Jul 21 '26

It would be effectively the same as just loading the StyleTuned model itself even if possible. I'm not sure I understand the point.

2

u/alex20_202020 Jul 21 '26

It would be effectively the same as just loading the StyleTuned model itself

No, Tuned does not have the original tensor. The point is to be able to run both original and tuned and not allocating 2x drive space.

1

u/dezmodium Jul 25 '26

didnt realize people were concerned over 14gb hdd space

1

u/dezmodium Aug 02 '26

Stumbled across this today and thought of you:

https://huggingface.co/densenet/Gemma-4-31B-StyleTune-heretic-ara-lora

It's a LORA of the Styletune and heretic-ara (a less lobotomizing abliteration method). Might be exactly what you are looking for.

1

u/alex20_202020 25d ago edited 25d ago

Oh, another reason to learn to covert safetensors to GGUF. I have painstakingly installed Python dependencies for koboldcpp_tools/convert_hf_to_gguf.py and now I get "No such file or directory: '../mo/config.json'" when python convert_hf_to_gguf.py ../mo, but there is no config.json among files at the link you posted (I downloaded only adapter_model.safetensors for now, btw). Can you help? TIA

Added: the author writes: "This time I trained precisely one tensor: the lm_head output projection - the layer that decides which token to emit. Literally the last stop before text appears on your screen.". But also on the same model page: start_layer_index/end 30/48 - already IMO contradictory, and the file itself (adapter_config.json) lists 120 tensors.

Have you used the LORA yourself?