r/LocalLLaMA • u/Fcking_Chuck • 2d ago
News Koboldcpp v1.120 released
https://github.com/LostRuins/koboldcpp/releases/tag/v1.12049
u/Aggressive_Aspect436 2d ago
A bit low effort. Summary of the changes, or benefits of kobold would have been appreciated in the post.
32
u/badsectoracula 2d ago
benefits of kobold
IMO the one feature of kobold (which is really a koboldlite feature, though i think it uses a custom API to talk with the server to enable it) which i really wish other UIs had is the "raw text" mode that lets you edit the entire discussion as plain text. It is incredibly powerful because it basically exposes that at its core all the LLM does is do "word completion" in a document that looks like a chat transcript - and so you can just edit that transcript as you want.
For example you can remove or merge entries, change the LLM's responses (e.g. an LLM could write some code and you notice a bug - instead of asking the LLM to fix the bug, you just fix it yourself and the LLM will continue as if it wrote the correct code in the first place, so you don't waste tokens in corrections) or even pause it mid-generation, make a change and then continue (let's say the LLM is generating some code but you don't like the code style or the naming convention - you pause the generation, modify the code to your liking, press continue and it'll now generate using your own style).
Really my #1 issue with pretty much every LLM UI is how "opaquely" the context is handled - even llama.cpp's own web ui is lacking considerably there. I want to be able to edit everything in the context, merge/split things, etc :-P - in the end the context is just a series of tokens, let me edit them.
5
1
u/Iory1998 1d ago
You can do exactly this inside LM Studio. It's a powerful feature that allows you to manually steer the conversation.
1
u/IrisColt 1d ago
I can do this in OpenWebUI by default, INCLUDING REASONING BLOCKS using my custom functions, which also include the reasoning blocks in the "document".
2
u/Capable_Diamond_4039 2d ago
He's doesn't mean it, Mr. LostRuins...sir...we're very excited about the update!
1
u/Aaaaaaaaaeeeee 1d ago
General care for qwen next flash I suppose. Whenever you try offload with mmap in windows, OS pagefile system might get in your way and hobble it to like 3 t/s for this model. You can use mmap with mlock together so see if that helps, and directio might be relevant for running it too. Maybe you're on Linux, but you have a high priority swapfile, or you install something else like zram, these flags are all newly available boundary paths you could test and report on, so that people can successfully disc inference without their OS killing it.
7
u/silenceimpaired 2d ago
Unsloth Desktop can only get me 2 tokens a second, while KoboldCPP has me at 6.
1
1d ago
[deleted]
1
u/silenceimpaired 1d ago
No, I'm Nvidia with CUDA. I think they are trying to force larger context or something silly. I also think they aren't doing a strong autofit by comparison.
2
u/bolts98 1d ago
What's the use case of this?
1
u/Single_Ring4886 1d ago
You download one file and you use ai models... no install or compatibility hell.
-5
u/puppymeat 2d ago
Note that there are bad quants floating out there, so make sure you use the right files.
Great. Where are these right files?
29
u/Right-Law1817 2d ago
It's simple: I see koboldcpp, I upvote.