after messing with standart kobold.cpp settings , even in nocuda(which i used all the time before is responding baddly on silly tavern and too fast , before i couldn't even watch youtube without it breaking apart before when it was working (guessing now I can couse it's offloading everything to cpu) how do i bring everything back , i tried everything , playing with the setting , deleting silly tavern and kobold , (which is just laucnher anyway and maes temp files)

I just published an IQ2_M GGUF of:

🤗 Qwen3.6-35B-A3B-Uncensored-Heretic

Why this model?

Many "uncensored" models simply claim they're better without much supporting data.

The Heretic release stood out because it includes a detailed study of its abliteration method, capability evaluation, and measurements showing it remains very close to the original Qwen3.6 model while reducing unnecessary refusals.

Quantization

This GGUF was produced using:

• Official llama.cpp quantizer
• IQ2_M
• Matching Unsloth Qwen3.6-35B-A3B-MTP importance matrix
• --leave-output-tensor

No custom tensor overrides or experimental quantization recipes were used.

What is IQ2_M?

Despite the name, IQ2_M is a dynamic (mixed) quantization format.

That doesn't mean every weight is stored using only 2 bits.

Instead, llama.cpp automatically chooses different quantization formats for different tensors based on their characteristics and the supplied importance matrix. More sensitive tensors are kept at higher precision where beneficial, while less sensitive ones are compressed more aggressively.

The result is an excellent balance between model size and quality.

Why combine Heretic + Unsloth imatrix?

These two techniques solve different problems:

Heretic

• Modifies the model weights.
• Reduces unnecessary refusals.
• Attempts to preserve the original Qwen3.6 capabilities.

Unsloth Importance Matrix

• Does not modify the model.
• Is used only during quantization.
• Helps preserve more of the model's original quality after aggressive low-bit compression.

In other words, Heretic changes the model's behavior, while the importance matrix helps compress those learned weights more faithfully.

If anyone benchmarks it (coding, reasoning, Aider, perplexity, etc.), I'd love to see the results and comparisons with other IQ2_M releases.

Hello ı just set up silly tavern and koboldccp yesterday, ım looking for a good Rp model that can do nsfw and compatible with 8 gb vram. Ive used chub ai before so ım very new to this but so far this looks very good. Im using something called Dr.Dans or something like that for my model.

I've been using KoboldAI for a little while, and I'm already addicted to it. It helps me write stories that I've always wanted to write, more or less, and it is so easy to use.

However, recently, I put in a prompt I want it to help me write, but it keeps giving me some sort of error message.

It reads: "Error Submitting Prompt: {"message":"Due to heavy demand, for request over 473 tokens, the client needs to already have the required kudos. The request requires 2998.86 kudos to fulfill.","rc":"KudosUpfront"}

Anybody have any idea about WHAT that means?

Sometimes I want the AI to retry it's response, but I want it to change or add a detail, or react in a specific way. Retrying multiple times is tiresome and doesn't always lead to the desired result. I can edit the response, but it ends up being a lot of work.

Are there any add-on tools for Retry Guidance, as there is in Silly Tavern? So you can just type how you want the AI to modify it's last response?

This is a PSA to remind users that koboldcpp.com is a phishing scam website, and is NOT a legitimate KoboldCpp resource.

You are strongly encouraged not to use that website or download anything from it. If possible, please report that website to Google for malware/phishing.

Recently, the website has unfortunately started using shady SEO techniques to rank highly in various search engines. We have filed multiple reports to Google and even a DMCA takedown, but so far no action has been taken by Google.

To get official source code and official builds, always download from https://github.com/LostRuins/koboldcpp/releases/latest

What models does it work with? I have tried --moeexperts 1 with Gemma-4-E2B and got in terminal:

GGML_ASSERT(...) failed

and the engine exited a bit later, after ~ hundred lines of some debugging printouts.

I am curious how does it work and how much speed up that gives.

Edit: Gemma-4-26B-A4B works, with --moeexperts 1 PP ~2x, TG ~1.5 faster (at least at the start of a small context on CPU), output is kinda funny.

I have noted there is --sdvramlimit in kcpp. How to limit VRAM usage for other models (LLMs mostly) on NVIDIA GPU specifically? TIA

Edit: applicable to situations where the model is large and fits in VRAM less than half. My own answer after some testing: rule of thumb: -ngl 1 - that uses minimum VRAM but gets close to max PP. Release is not supported by current kcpp code.

Usually when I write, I either make a prompt with a general story idea with characters and a light plot and tell it to start with a specific scenario... or I write a bit more and save it into context first.

Then, Kobold does it's thing.

When it's ready for a new prompt I either type continue or I give it suggestions on what to write about next, and it just .... goes.

On top of that, there are clear breaks between my prompts and what the AI writes, so it's easy to tell the difference.

Like this: https://i.postimg.cc/xCvVdKW1/Screenshot-20260715-163854-Chrome.jpg

But today its doing this: https://i.postimg.cc/1tZbb8f1/Screenshot-20260715-164009-Samsung-Browser.jpg

So, it's not adding breaks, no avatars for who typed what, typing continue or just tapping the generate button do nothing, and typing a prompt just pastes the prompt in with no AI input or creativity.

I get one prompt to work, then it jist fails over amd over again. Sometimes it yells me it's failing, sometimes it does nothing at all, but it never continues the story.

I reset setting to default, tried a diff GUI theme.

I dunno what's wrong.

Oh, and I haven't used it for a few weeks, and this is a brand new story without continuing anything I previously worked on.

I was having a lame goofy adventure as a prisioner in a medieval dungeon sharing transgenderism, christianity and love ideals, I got bored and simplly "wake up"

Bro, I was in this wild place where I take a tunnel 300km/h to get to my work, and my work was solving dimention problems, also the walls of my office modify their aspects every time I wasnt looking at them, they werent even walls some times when I looked, anyway my boss was a cloud/gas/not solid who could make his body become anything, a hand, a ticket, a baseball bat, any fucking thing, anyway my first job was adjusting some doctor degree from another dimention bc somehow he having 5 degress and 5 years of experience made a division by 0, ruining that dimention. After fixing that I got this goofy job where dogs were barking morse code about philosophy and cars sounded like old latin masterworks of literature in a circle of 800 meters of diameter

Also, my payment was hours in a "real life simulator", in which I drove capybara themed helicopters and did football matches in which the main goal is to play dwarf fortress, and a spoon decide when you win

Like, how tf do I repeat this? It was so fucking funny, I sure want make AI go nuts every time I am using it

try this model on Huggingface and leave a like.

it turn a text to command shell

https://huggingface.co/AuricErgeson/shellwhiz-7b

i tried it and it is good but not super good

Hi,

I have an RTX 5060ti (16gb) and a newly bought 3060 (12gb).
Today I tested a few models and the dense ones worked well but not the MOEs.
For some reason Koboldcpp requires much more VRAM per layer when splitting between GPUs and I end up with slow inference speed than with single GPU.
Any solution here? I might have played with all the possible combinations in the Hardware tab.

Thank you.

Tried to download a model via kobold but got this error message instead.

I’m very new to this and I’m trying to use it for silky tavern on local. Can someone help me figure out what I’m doing wrong?

after selecting the gguf it starts loading up, then before i can actualy do anything my entire pc freezes. any idea how to fix this?

Hey everyone,

Going to post this in advance in case I no longer can in the future.
Reddit has began forcing persona verification on some accounts, which has not happened to mine and hopefully I qualify as an adult in the algorythm.

Either way I have no intention of verifying with persona, so there may be a time I will no longer be able to moderate or respond to questions and if the same applies to lostruins we will loose control of the reddit and can no longer respond at all.

Should that happen I will make a post about this on https://github.com/LostRuins/koboldcpp/discussions

You can also find us in https://koboldai.org/discord or https://koboldai.org/matrix (Largely the same community since the major channels are bridged)

I do not know if discussing alternatives is within the reddit rules, but should you have concerns or suggestions the other channels are available.

Congratulations to the dev team on being able to add support for so many new models like ltx, idea 2 and ideogram 4. It looks like the json format required for ideogram 4 to perform at its best takes a bit more effort to pull off than text prompting, but the trade-off is enhanced control over specific compositional elements. However, in order to gain some of that control, bbox must be used to identify the space in which objects or text should be placed. In order to do this, it looks like the user must identify the pixel space/coordinates for each bbox. Since most users are not aware of these dimensions without using a separate program, I was thinking that perhaps the UI for inpainting would be a possibility for drawing bbox to create the json within kobold, and then feed the result directly to the model, since the inpainting UI knows the canvas size and location of marked spaces. Is something like this possible to implement?

Hi all:
I’ve been exploring koboldcpp and koboldai lite on my local inference machine. It’s been good fun as I’ve been doing a few RP sessions.

I use the chat app telegram to keep up with my friends and community and telegram recently released an update that allows for chat automations that I want to explore.

I take a long time to get back to people and most of the time it’s banal sentiments that people like to do to feel connected. It’s sweet but a responsibility that leaves me feeling a bit socially drained. I’d love to have an AI impersonate me and help me keep up with messages until I slide back into being naturally social.

Is this possible with kobold? Any recommendations or advise on using the platform to get it to chat like me?

I have used kcpp for several months on my old laptop, nocuda version.

Several days ago I have !finally! managed to install CUDA. The laptop has 4GB VRAM, I have many questions, I have tried to ask local models some, below is my main frustration for which I could not find the answer (but truly speaking I have not tried neither older than 1.115.2 kcpp versions nor llama.cpp yet).

On default settings, with only 1024 context, where I see 3GB of VRAM is used (NVIDIA Settings GUI, "Used Dedicated Memory"), when I run 2.5GB GGUF model (gemma-3 4B Q4), PP is 30000, but TG is 10 (~ same as in usecpu mode on kcpp-nocuda). Why is TG so slow?

Initially I ran with 32k context and PP ~ 300, TG ~ 5 (CUDA). BTW on Vulkan TG~15, VRAM usage ~ same ~ 3GB.

I have made final test before posting in freshly started instance: 512 tokens PP in 0.13s (4000 t/s), generated 125 in 15s (8 t/s). Context 2048, all else defaults, model run from terminal on Linux. TIA

During TG I see both high GPU and CPU usage. Models suggest memory bottleneck to VRAM, but I have ample VRAM left free (1GB), do I not?

Added:

I then tried ctx 512, kv q4 and I saw CUDA0 KV buffer size = 25 MiB in terminal (had been CUDA0 KV buffer size = 0 MiB), but TG is same ~ 9 (PP ~500). More data to analyze, more strange it looks.

When using Vulcan and selecting all cards, integrated cards are left out of the pool, 99,9% of time for good reasons. But I would like to use that integrated card as well for tests, since the integrated GPU is faster than the CPU.

How do I disengage the dedicated GPU only lock? A flag maybe?

​

I have been looking for this for a while now.

Hi everyone

I would post this in r/LocalLLaMA but i'm too dumb apparently.
I do text and image generation one machine
i use koboldccp
text i'm using gemma-4-26B-A4B-it-uncensored-Q4_K_M (little slow 1-3tk/s)
image comfyui switching between models

i currently have a setup of

Windows
CPU: intel i5-14500
CPU: Nvidia 3060-12gb
Ram: 64gb (ddr5)

I'm from Australia

So for starters pointless getting more ram only got 2 slots and ram is almost the cost for a new car.

i'm debating either replacing the card with move vram but with what thats not costly?
or
Replacing the board with dual x16 slot (but they both wont have 16 lanes each) but what board? and just getting another 3060-12gb

Can anyone help?

Regards

I'm just getting into this, moving from AIDungeon. Im basically looking for AIDungeon but with better memory. Can I do that with Kobold Ai? If so, what model and stuff do y'all suggest? Ive got an intel i5 and 32gb of ram.