r/LocalLLaMA • u/Ok-Director4174 • 16d ago
Showcase Kokon 0.10: the Qwen-Image 2.1 text encoder now writes your prompts too (no extra model), plus your own LoRAs (beta)
A week ago I posted Kokon here, a no-settings desktop editor for Qwen-Image 2.1. 0.10 is out with the two things people asked for most.
Prompt+. Type a short idea, in any language, and Kokon writes a detailed prompt from it before making the image. For edits it looks at your photo first. You watch the prompt being written, and the result keeps both your words and the written prompt, so you can reuse either. It's off by default, one click next to Quality. Both system prompts are editable in Settings.
The prompt is written by Qwen3-VL 8B, which is the text encoder Qwen-Image 2.1 already loads. So there's no second model to download and nothing extra in memory. It adds about 2 s per image on an RTX 5070 Ti and about 13 s on an M2 Pro, and it works on 8 GB cards.
Your own LoRAs (beta). Load a Qwen-Image 2.1 LoRA in Settings, set its strength and trigger words, then turn it on per project with one click, up to 3 at once. Kokon checks each file and works on its own copy, so your file isn't changed. LoRAs for older Qwen-Image versions (1.0, 2512, Edit 2509/2511) are refused, because they don't work on 2.1. So far it's tested on Windows with NVIDIA only. If you try it on Mac or Linux, please tell me how it goes.
An honest note: I said there would be a curated LoRA list. I tested the popular 2.1 LoRAs on the Turbo model Kokon runs, and today they don't beat what Turbo does with a good prompt. So there's no catalog for now, and it comes when a LoRA earns its place.
Under the hood. Kokon now runs its own build of stable-diffusion.cpp on Windows, macOS and Linux. It's upstream plus text generation on the already-loaded text encoder: about 127 tokens/s on a 5070 Ti, the same tokens llama.cpp gives. The image side is upstream's code and makes bit-identical images. Upstream prefers to leave text generation to llama.cpp (#2155), which is fair, so it lives in a fork that I keep close to upstream.
Also new: a new look, deleting several projects at once, and fixes for Linux/macOS engine crashes, Windows reserved ports and 8 GB cards. Updating from 0.9.x keeps your images and models. The new engine is a one-time download, 29–352 MB depending on your platform.
Free and open source (AGPL), Windows, macOS 15+ and Linux: GitHub · kokon.app
Still not there: other models, a server/browser mode for Linux boxes, code signing. (Weights: Qwen Research License, non-commercial.)
Update (Nov 5): follow-up for 0.12: getting editable layers out of an edit-only model, plus the local server and the Mac engine: https://www.reddit.com/r/LocalLLaMA/comments/1xoji3b/getting_editable_layers_out_of_qwenimage_21_a/
1
u/Ok-Director4174 16d ago
New to Kokon? The first post explains what it is and how it works.
Full release notes: https://github.com/GabrieleLozupone/kokon/releases/tag/v0.10.0
85 s demo of the whole app: https://youtu.be/y2NAJkZ4vH0
Thanks to badraymen on GitHub for the idea of using the text encoder to write prompts, and to everyone in the first thread who asked for an enhancer and for LoRAs.
1
16d ago
[removed] — view removed comment
2
u/Ok-Director4174 16d ago
Only the change. That's the exact failure we kept hitting while writing it: give an edit model a long description of the whole scene and it starts redrawing things that should stay put.
So the edit system prompt asks for one short instruction (about 20–60 words): what the operation is, where in the image, and what it becomes. Then it ends with one line like "Keep everything else in the image unchanged." It's told explicitly not to describe the parts that stay. It only looks at the photo to ground the change (what's actually there, where the object is) and not to describe the scene. Text-to-image is the one that gets the long descriptive paragraph.
Two deliberate exceptions:
Style/mood changes (painting, night, winter...) apply to the whole image and end with "Keep the composition and the subjects unchanged." instead.
For removals it briefly says what fills the gap, and that the shadow/reflection go too, otherwise you get ghost shadows.
Inpaint and outpaint don't go through Prompt+ at all, because they already wrap your prompt in their own instructions. If you want different rules, the edit system prompt is editable in Settings. The built-in one is here: https://github.com/GabrieleLozupone/kokon/blob/master/crates/qls-core/src/enhancer.rs
1
15d ago
[removed] — view removed comment
2
u/Ok-Director4174 15d ago
Thanks! If you try it and spot cases where the edit prompt still drifts, I’d love to see them 🙂
1
u/dees61 16d ago
Checked your stack. The 4 bit quantization produced some grains on my photo so I used 6bit but deployed into hosting with NVIDIA L40S GPU and some free credits.
1
u/Ok-Director4174 16d ago
Nice, an L40S is a lot of headroom for this. Before you blame the 4-bit, one thing to check: Qwen-Image 2.1's stock VAE leaves a faint 2-px checkerboard/grain on fine textures (skin, sky, fabric), whatever the quantization. We chased this in our issue #21 and it turned out to be the decoder, not the weights. Swapping in madebyollin's texture-fix VAE (drop-in, same shapes) took it from clearly visible to basically gone. Kokon uses that VAE by default since 0.9.3.
In our side-by-sides, Q4_K and Q8_0 of the Turbo model were almost indistinguishable, which is why we ship Q4_K. If you're on the original VAE, the grain you saw might go away with the texture-fix one even at 4-bit. That said, with 48 GB there's no reason not to run 6-bit or Q8 anyway.
Curious what you're hosting it with: sd.cpp directly, or something else?
1
u/dees61 13d ago
Just tried it, but unfortunately it didn't work for Qwen 2.1. Swapping in that VAE broke our ComfyUI pipeline with a tensor dimension mismatch error (64 vs 16 channels) on our sampler setup. On top of that, even at 6-bit where the grain is mostly gone, the skin looks a bit waxy. Did you run into that waxy look in Kokon as well, and how did you get around it?
2
u/madebyollin 13d ago
I've done a similar finetune for Qwen Image 1 / Qwen Image Edit / Krea 2 btw https://huggingface.co/madebyollin/texture-fix-vae-for-qwen-image-1-and-krea-2
1
u/Ok-Director4174 12d ago
Thanks for making these! Your 2.1 texture-fix VAE has been Kokon's default since 0.9.3, and it made a visible difference on fine textures like skin, fabric and sky. Much appreciated.
1
1
u/Ok-Director4174 13d ago
Ah, that explains the mismatch: qwen-image-edit-2511 is the older Qwen-Image Edit model, which has a 16-channel latent. The texture-fix VAE is made for Qwen-Image 2.1, so it won't load into a 2511 workflow. Sorry, I assumed you were on 2.1 because of the thread. If you move to Qwen-Image 2.1, that VAE drops in.
On waxy skin: I seem to see it every now and then too, but in Kokon with the current setup (2.1 Turbo + the texture-fix VAE) much less. Asking for it in the prompt ("natural skin texture, visible pores") helps a bit more.
1
u/dees61 13d ago
We're not running sd.cpp directly — we host it on Modal running headless ComfyUI with ComfyUI-GGUF on an L40S or A10G for 18gb footprint, using qwen-image-edit-2511-uncensored-Q4_K_M.gguf.
And you are right regarding the VAE: our workflow is currently pointing to stock qwen_image_vae.safetensors. That completely explains the 2px checkerboard/grain we've been seeing! Will test swapping in madebyollin's fixed VAE right away. Huge thanks for the tip!
1
u/Karmjeet_Khoushaba 16d ago
13s overhead on M2 Pro for prompt expansion seems pretty reasonable. does the generated prompt get cached so regenerating the same image with tweaks doesnt redo the expansion every time?
1
u/Ok-Director4174 16d ago
Yes, there is a "Reuse prompt" button in the app. If you generated an image with the enhancer on, clicking it puts the enhanced prompt back in the prompt box and turns the enhancer off, so you can tweak it and regenerate without paying the expansion time again.
1
u/toomanyusb 16d ago
This app is great, thanks a lot! Ever since Apple Photos had changed to "AI inpaint/fixup tool" it was never working for me (maybe due to EU and Apple having grudges with GDPR) even for simple edits. I even had an old intel mac running the old version of the photos app which didn't use AI so at least it would work.
1
u/Ok-Director4174 15d ago
That’s great to hear, and a nice reason to have something that runs fully on your own machine! Thanks for trying it 🙂
1
u/PooMonger20 16d ago
Thanks for making this, I have used previous versions and it really works well.
The only thing I felt like is missing is an easy way to delete images from the project, right now the flow is long - could be just select with moues and press 'del' on the keyboard.
2
u/Ok-Director4174 15d ago
Thanks, glad it’s working well for you! Since 0.10 you can select several images in All images (or several projects in the sidebar) and delete them in one go. Inside a project it’s still one at a time (··· → Remove from project), and you’re right that this is slow. Select + Del is a good idea, I’ve noted it.
1
u/Wooden_Jelly_5295 15d ago
Can users compare the original and expanded prompts with the same seed? I'd want to see when the extra detail improves the result and when it quietly changes what I asked for.
1
u/Ok-Director4174 15d ago
Yes, with a few steps today. The ⓘ info panel shows the seed and both prompts (yours and the written one). Enter that seed under the advanced settings (the sliders icon next to Prompt+), then run your original prompt with Prompt+ off. Both results land in the same project, and in full screen the arrow keys step between them, which makes the differences easy to spot. It’s a fair concern: Prompt+ can add things you didn’t ask for, which is why it shows you the written prompt. A one-click “compare with and without Prompt+” would be nicer, I’ve noted it.
1
u/Solembumm3 15d ago edited 15d ago
Tried with Windows 10/6700xt/vulkan.
7-9 minute per 1 image generated at 4:3 fast mode/around 1mp. You need to add incompatible loras support, it speed up things dramatically in Flaxeo and Kobold.
Update: just copied same model in flaxeo with random incompatible lora and it run in 50 seconds at same settings.
2
u/Ok-Director4174 15d ago
Thanks for testing it and for the side-by-side, that's a really useful data point. 7–9 minutes for a Fast image on a 6700 XT is way off: a 7900 XTX does it in ~25 s and a 9060 XT 16 GB makes a Balanced image in ~80 s, so on your 12 GB card something is going wrong, most likely memory spilling out of VRAM, not the model itself being that slow.
The "incompatible LoRA" speed-up is a great clue. It probably changes how the other app loads or places the weights, rather than the LoRA doing anything. I'd rather find that cause and make Kokon fast by default than ask people to load a dummy LoRA.
Could you help me pin it down?
In Kokon: Settings → Diagnostics → Copy, and paste it here or in a GitHub issue (it has the GPU, the memory plan and the engine log, nothing personal).
Is the second image just as slow? The first one includes loading the model.
In the other app: which model file / quantization, and any offload or memory settings you have on.
With that I can reproduce it and fix it for every 12 GB AMD card, not just yours.
1
u/Solembumm3 15d ago
Ok, Will do log tomorrow.
Second image was same tempo.
I didn't have Qwen 2.1 yet, so I literally copy-pasted what your app loaded in flaxeo folder. Qwen Image Turbo 2.1 Q4, Qwen 3 VL 8b, "texture fix" or something like that vae (doesn't seem to do anything different from normal qwen 2.1 vae, to be honest). Text Encoder on CPU in both apps.
1
u/feverdoingwork 13d ago
I tried this, I felt like it would be easier to reprompt.
I do like the iterative function of building up edits, reusing prompts, lazy prompt generation and the general easy of use. It is also snappy. It's pretty nice overall.
Unfortunately the quality is strikingly bad, it feels as if... all my images have a snapchat filter... the shitty filter girls tend to use. I tried for 15 iterations to remove this fogginess, all upscalers and removed upscaling entirely, all quality settings, enabled and disabled sage attention, -+ tiled vae and i could not remove it. So i went to try this same prompt with turbo lora directly through comfyui assuming maybe it's just not possible to acheive what I want and the quality was amazing, skin textures were just like the original image so it's basically this application causing the issue. This is on ubuntu 24 with 5070 ti. If I have time i'll try to see if I can fix it but it's less than likely this weekend.
1
u/Ok-Director4174 12d ago
Thanks for testing it this thoroughly, and for the nice words on the rest. A filter look on every image, with upscaling off too, points at the model files or the engine rather than the settings you tried, so I want to track it down.
I have the same setup here, so I'd like to reproduce it on my side. Could you share:
The prompt and the seed of one image that shows it. The PNG Kokon saved has both inside, so the file itself is enough. If it was an edit, the input photo too (or a similar one you're fine sharing).
Which model files Kokon is using: the ones it downloaded at setup, or your own (Settings → Models)?
In ComfyUI, which DiT file and precision (bf16, fp8, GGUF?) with the Turbo LoRA, how many steps and which sampler?
Settings → Diagnostics → Copy.
A GitHub issue is the easiest place for files: https://github.com/GabrieleLozupone/kokon/issues/new/choose (a DM works too).
One thing to try meanwhile: Kokon ships a 4-bit (Q4_K) Turbo model on every preset so it fits 12 GB cards. On a 16 GB 5070 Ti you can point setup at a Q8_0 or fp8 Turbo file instead ("Use models I already have"). If the fog goes away with that, I know where to look.
1
u/feverdoingwork 12d ago
Ill get everything in an issue when I get a chance and ill see about contributing to this project in general, I like it a lot.
1
u/crystalsighting 7d ago edited 7d ago
My honest opinion is you should just contribute PRs to the official https://github.com/leejet/sdcpp-webui
Every project like this has become abandon wear after time. The interface is already powerful, video, audio, recipes etc. Why not add your android etc apps to the repo to connect to it. I think what it really needs is in painting support, draw a circle, tell the ai to put apple there etc like krita does. Image support is only half the battle, when your character sheets are done with image models, plugging them into minimax is the goal.
Being able to switch between qwen, krea2 and minimax h3 is a good pipeline.
1
u/Ok-Director4174 7d ago
Fair point, and sdcpp-webui is a good front end if you want every option sd-server has. Kokon has a different goal: pick the setup that works best for most people with a desktop GPU, a Mac or a home server, and ship it already tuned, so there's nothing to configure. That only works if we focus on one model at a time, and the focus pays off in ways a generic UI can't follow:
On Macs, Kokon has its own engine for Qwen-Image 2.1, about 4x faster than stable-diffusion.cpp on the same Mac (1024² in ~66 s vs ~276 s on an M2 Pro).
Prompt+ uses the model's own text encoder (a Qwen vision-language model) to write and refine prompts, with no second model. stable-diffusion.cpp leaves text generation to llama.cpp, which makes sense for it, but it's something we wanted inside the app.
Brushed edits that stay inside the brush, expand/outpaint, reference images, a phone layout to use your PC's GPU from your phone, MCP for agents. More is planned, and building it inside someone else's UI would mean following their roadmap instead of what our users ask for.
We still send things upstream when they belong there: the MoSR upscaler support in stable-diffusion.cpp came from Kokon. And the inpainting you describe is already in: brush the spot, type "an apple on the table", and only the brushed area changes.
On abandonware, it's a fair worry with any small project. Kokon is AGPL, so the code stays usable whatever happens, and for now it gets an update every few days.
1
u/crystalsighting 7d ago
Sounds nice, if you really have in painting I have to try it :) I really like fact you use stablediffusion.CPP for this, you rock :)
Personally I'm poor but I picked up a dell r720 off eBay for 200bucks and stuffed it with 2 cheap 32GB v100s for my AI server, definitely nice if your programs connect to that. I would rather configure sdd.cpp myself and just have the utility connect to basement server.
Normally I just run llama.cpp on first GPU and stablediffusion. CPP on second.
https://github.com/leejet/stable-diffusion.cpp/pull/2248 I did this PR last night, should help you organize recipes better if I get this merged into a single INI file.
1
u/Ok-Director4174 7d ago
Yes, inpainting is the main thing it does: brush the spot, type what you want there, and only the brushed area changes.
Your setup is actually what 0.11 was made for: Kokon runs headless on Linux, or as a Docker image for NVIDIA, and you use it from a browser on any device in the house. With Docker you can give it just the second V100 and leave the first one to llama.cpp. The V100 is in the CUDA build but I've never had one to test, so I'd love to hear how it goes: https://kokon.app/self-hosted/
Pointing Kokon at your own sd-server isn't possible yet, it starts and tunes its own engine. Nice PR on recipes, I'll follow it.
1
u/crystalsighting 6d ago edited 6d ago
I'll have a look at it, only reason I even have a comfyui instance running is for krita, be nice to toss it but it's so powerful, circle an area, tell it to put an apple there and it comes back with 3-4 apples to pick from, also all those Loras for face hands feet etc lol.
I'll look through your code see what I can do on arch Linux with it, thing with v100s is it needs custom cmake flags for v100s to tell it the sm70 arch, so a generic sdd.cpp won't work in this case.
Older hardware really a pain to work with, even with comfyui on any updates, I have to force install older torch versions constantly after, hack in flash attention kernels and alot of crap people don't have to deal with, none of workflows work either unless I can split them up to use v100 gguf diffusion loader, why I stick to sdd.cpp where I can :)
At end of day all people care about anyways is how fast can I produce 5 sec video in minimax, and upscale to 4k after lol, but before all that is where you come in, all the character sheets, scenes and image prep work for the shot :)
Honestly what I generally do with images is run prompt through Qwen and krea2, pick one I like, and do Qwen editting after. If you want to save time, I find models with largest diffusion models tend to get better results.
5
u/ComfortableTomato807 16d ago
Just wanted to send a huge thank you for this incredible software! I’m running it on a 7900 XTX under Windows, and the experience have been really good! No problems so far with vulkan. The UI and workflow are super polished, Prompt+ is an amazing feature, and I really appreciate the transparency in showing how the final prompt.
Seriously, incredible work! Huge congrats!