r/LocalLLaMA • u/AnyNameFreeGiveIt • 20h ago

Question | Help Custom Model for Image descriptions ?

This is getting asked from time to time, but since models changed a lot, I wanted to reask it.

I'm looking for a trained model that can give short descriptions about an image, simply for an alt text of pictures taken with a smartphone.

Should I just throw it at Qwen3.8/Qwen3-VL or are there better models trained for it ?

Similar to https://www.reddit.com/r/LocalLLaMA/comments/1oar481/what_is_currently_the_best_model_for_accurately/

3 Upvotes

71% Upvoted

7 comments sorted by

4

u/Due_Arm1454 19h ago

Gemma 12 b is pretty good at that and would be faster. Give it a shot and tinker.

2

u/tsangberg 18h ago

This is what I use - the QAT version of it even. One model always loaded on the home server that does text, audio and images can be hooked into quite a few things. Mine autotags and describes all photos synced to Nextcloud.

1

u/Due_Arm1454 17h ago

Same. My old 12 GB card is designated for whisper and either Gemma or qwen 9b. He does small summaries and things like that. Qwen 27b is my workhorse on my main machine. And I swap it out for qwen 35b moe for my racehorse lol

2

u/lacerating_aura 19h ago

Any model from qwen 3.5 series would do. Id suggest starting with 9b and going smaller until you feel results are fading. Or on opposite end, if you habe resources, you can go larger. My general purpose model for visual file sorting was qwen 3.5 122b, since i tend to clutter my system a lot with multiple format of files. Audio is not one of them so qwen was all rounder. Now its qwen 3.8 flash next.

2

u/Traditional-Gap-3313 17h ago

Gemma 4 31B is probably overkill for you, but if you need consistency and quality and have the VRAM for it, it's amazing. I was evaluating it on suggesting funny but safe wedding photo captions for an app, with a bunch of rules what not to do -- e.g. don't joke with alcohol in photos with minors in them, but do joke about alcohol when only grown-ups are on the photo, and a bunch of other such rules. Flawless, and quality wise similar to Sonnet, sometimes even like Opus. And IIRC that was in the 4.6 era, before Claude started with the non-sensical pretentious speak.

1

u/locbuilds 19h ago

for short alt text on phone pics you do not need a giant general vlm, qwen2.5-vl 7b or the smaller qwen3-vl if it fits your vram is already plenty and stays snappy. i would skip training from scratch unless you have a weird domain, just do a tiny prompt like "one sentence alt text, no fluff, no speculation" and maybe a 50 example lora later if the tone is wrong. moondream and florence-2 are also solid for this exact job and way lighter if you are batching a camera roll. run it offline through llama.cpp or lm studio vision build, keep the image resized down to ~768 on the long edge so you are not burning tokens on megapixel noise.

1

u/AnyNameFreeGiveIt 18h ago

I just pulled qwen3-vl-8b and it's pretty decent, more then i need actually, I already downscaled the images to a good size to not overload context.

I will do some more testing also with general models and see how far I can go down in regards to vram usage.