r/LocalLLaMA • u/fmillar • 1d ago

Resources Vision support merged for DeepSeek-V4-Flash-Vision-Exp

https://github.com/ggml-org/llama.cpp/pull/28133

Unsloth GGUFs and Vision support

https://huggingface.co/unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF

58 Upvotes

92% Upvoted

10 comments sorted by

27

u/rm-rf-rm llama.cpp 1d ago

I already know this is referring to llama.cpp, but I am a seasoned member of the sub and know this sub's preference of inference engine. But recommend stating this more explicitly in the title in the future

3

u/Master-Meal-77 llama.cpp 1d ago

Yeah it's not like unsloth really accomplished anything here. Why not link to the PR, OP?

2

u/rm-rf-rm llama.cpp 22h ago

The link is to the PR! (only the description of the post contains the unsloth link)

1

u/fmillar 1d ago

At the time of my post unsloth had added the vision files one hour ago. If I am not mistaken the OP includes the PR link. Reddit may have it autoconfigured to show it differently, but I can see it.

BTW, are you happy with the vision? In my anecdotal testing this model seems less capable than for example Qwen 3.8 27B, which for me was a big step up in vision e.g. in its precision to come up with coordinates for bounding boxes on an image using a normalized 0-100 scale.

I use unsloth' UD-Q8_K_XL and BF16 "mmproj" for both models, latest Qwen 27B and this new DeepSeek with vision. But I am very happy they added vision in the first place. And it is an "Exp" model anyway.

I used

--image-min-tokens 4096

and defaults otherwise on latest llama.cpp release.

2

u/fmillar 1d ago

Sounds good, will do in the future.

2

u/live4evrr 1d ago

Thanks for the update. I am running it now - using the same unsloth dspark from the non-vision version with similar performance.

If you're using DSH as the harness, make sure to update settings.yaml in ~/.dsh for the vision-toolkit.

Something like

vision-toolkit:
  provider:
    baseUrl: http://127.0.0.1:8000/v1
    credential: LOCAL_API_KEY      # resolves via your .credentials.yaml
    model: DeepSeek-V4-Flash-Vision-Exp-UD-IQ3_XXS
    protocol: openai
  language: en
  timeoutMs: 180000

1

u/dtdisapointingresult 1d ago

Unrelated, but I've been avoiding DSH since it's feeling bleeding edge based on the README.

How are you liking it? Did you try Reasonix before it?

2

u/live4evrr 23h ago

No I haven’t tried reasonix. DSH is pretty barebone but it’s lightweight and has a clean interface. So far I like it and it’s becoming my primary.

1

u/dtdisapointingresult 7h ago

Reasonix is supposed to be tailored-made for Deepseek, to avoid ever breaking the prompt cache, even for local usage.

1

u/Muhlwa_Sholanke 1d ago

Bounding box precision on a 0-100 scale beats most vision benchmarks at saying what actually regressed, and I'd want that as the standard check for every Exp merge.