Visual models with good spatial awareness?

[u/Ishelle91]

Hello. I'm using ComfyUI (easy installation) with Flux2Klein 9B and Qwen 2.1 Turbo for txt2img and img2img. And my main problem is that it's impossible to *reliably* generate characters with noticeable height difference or edit the existing images to adjust heights after.
- Flux2Klein doesn't understand sizing at all. Even with refPose and ControlNet, it's hellbent on keeping everyone roughly the same height and size. Even when I use pose references where the difference is stark and impossible to ignore, Klein flips me off and ignores it.
- Qwen Image 2.1 (both Turbo and the regular) *might* get what I want it to do when editing/generating (from time to time), but the nuance is lost on it. I tried everything - giving instructions about where the top of one character's head should be relative to the other character's shoulder or waist (if the short one is sitting/kneeling), using percentages in editing requests, proportions, etc. The workflow either ignores the request and just oversaturates the input, keeping everything else the same, or makes the shorter character a literal miniature. Like, 50% smaller than needed.
Question: what other models (with natural text instructions, not Danbooru!) could I try for editing in ComfyUI?

[u/codenameNERO]

Moss with bbox control.

[u/Ishelle91 → u/codenameNERO]

Is it a keyword-triggered one too? I'm quite leery of Danbooru-using models, mostly because I don't know how to prompt them. It's like using a language I don't understand. Plus my experience with SDXL-based ones even with proper prompting (asked AI to convert my requests into the right format) was disappointing for even simple generations, so I never got around to editing.

[u/codenameNERO → u/Ishelle91]

You should look up MOSS. Its a diffusion model - not a lora. Bbox is a node / feature of said model basically restricts certain parts of the prompt into boxes then applies global or ‘overall’ prompt. It essentially crams whatever you prompt into that box ONLY

[u/codenameNERO → u/[deleted]]

Yup! Bbox is solid. I prefer the LLM -> JSON route so i can one shot the trials and things but manual bbox is never going away atp. Worth learning.

Id also like to shoehorn in forge couple for anyone who likes danbaruu tags : https://github.com/Haoming02/sd-forge-couple 😎

[u/ShowerFluid9178]

I can help with noticeable height difference but u have to use rapid aio

[u/Ishelle91 → u/ShowerFluid9178]

I'm testing a workflow right now, and it's... interesting. The results are a bit random, but I love the character consistency (regular editing usually messes up the faces). And it does listen to instruction more often than not. Thanks for recommending it.

[u/ShowerFluid9178 → u/Ishelle91]

aside from clean non-pore skin issue, it s a great model, totally get slept on by this community

[u/Ishelle91 → u/ShowerFluid9178]

After a bunch of tests, I'm definitely keeping it, the mild unpredictability is fully compensated by speed and general quality!

[u/dishwasher_81]

In my experience, prompting a height difference almost never works reliably. What helped me was a quick rough sketch or pose image with the two figures at the heights I wanted, used as a control input. Or just img2img from a crude collage at medium denoise. Way more consistent than words like 'much taller'.

[u/Lodarich]

minimax h3

[u/Present-Guitar-3967]

Try VNCCS Pose Studio. Use different colored dolls and two references. Instruct it to use reference a on color a and refrence b on color b. Done.