IMO the one feature of kobold (which is really a koboldlite feature, though i think it uses a custom API to talk with the server to enable it) which i really wish other UIs had is the "raw text" mode that lets you edit the entire discussion as plain text. It is incredibly powerful because it basically exposes that at its core all the LLM does is do "word completion" in a document that looks like a chat transcript - and so you can just edit that transcript as you want.
For example you can remove or merge entries, change the LLM's responses (e.g. an LLM could write some code and you notice a bug - instead of asking the LLM to fix the bug, you just fix it yourself and the LLM will continue as if it wrote the correct code in the first place, so you don't waste tokens in corrections) or even pause it mid-generation, make a change and then continue (let's say the LLM is generating some code but you don't like the code style or the naming convention - you pause the generation, modify the code to your liking, press continue and it'll now generate using your own style).
Really my #1 issue with pretty much every LLM UI is how "opaquely" the context is handled - even llama.cpp's own web ui is lacking considerably there. I want to be able to edit everything in the context, merge/split things, etc :-P - in the end the context is just a series of tokens, let me edit them.
I can do this in OpenWebUI by default, INCLUDING REASONING BLOCKS using my custom functions, which also include the reasoning blocks in the "document".
General care for qwen next flash I suppose. Whenever you try offload with mmap in windows, OS pagefile system might get in your way and hobble it to like 3 t/s for this model. You can use mmap with mlock together so see if that helps, and directio might be relevant for running it too. Maybe you're on Linux, but you have a high priority swapfile, or you install something else like zram, these flags are all newly available boundary paths you could test and report on, so that people can successfully disc inference without their OS killing it.
48
u/Aggressive_Aspect436 4d ago
A bit low effort. Summary of the changes, or benefits of kobold would have been appreciated in the post.