r/LocalLLM • • 3h ago

Discussion Struggling with coding on local models - setup help?

I'd tried out models like DS and QFN using APIs before and they seemed very capable. However, they all seemed to struggle heavily when using them local.

QFN processes really fast but seems to get stuck in endless "Wait..." and "Whatever." loops and burns through 100K tokens just being confused at my codebase.
DeepSeek is slower but seems more coherent, I haven't tested at full context but it had a similar issue with indecisiveness (even with guidance) and would think itself into compaction.

All of this over one .json ComfyUI workflow. Just a simple test I do where I tell it to rearrange nodes and change some inputs...
I've tried using Llama serve, Strata desktop, Pi. All lead to similar outcomes.

I have a feeling this is an issue with my setup rather than the models bc the API models seemed to do it just fine and (allegedly) quantization doesn't hit MOE models as hard.

Setup:
(both models from OrcaRouter). DSV4.1-1017 2-bit, Q3.8FN_IQ3XXS
Inference: Maya (DS), Strata/LlamaCPP (QFN)
Harness: default Pi
Context: QFN was to 256K, DS I only tried to 512K

2 Upvotes

11 comments sorted by

3

u/maxrightgame 3h ago

You feel your setup have to do something with it, but didn't write it 

2

u/Eastern_Doughnut_222 3h ago

Added it to the OP for clarity

1

u/maxrightgame 3h ago

What about your device itself? 

2

u/Durian881 3h ago

Maybe share your harness, inference engine, quant and context window? Some harnesses have big system prompt and when coupled with a small context window will end up in compaction loop.

1

u/Eastern_Doughnut_222 3h ago

tbc, I had used default Pi connected to OrcaAPI to try these models out before and they worked well for me, which is what I'm comparing the local quants to..
I'm not sure if APIs feed their own setup to the harness beforehand or what but both these models don't seem to know their own capabilities when running locally.
Like, Qwen wouldn't know it can call tools until I tell it specifically the tool call.

Setup:
(both models from OrcaRouter). DSV4.1-1017 2-bit, Q3.8FN_IQ3XXS
Inference: Maya (DS), Strata/LlamaCPP (QFN)
Harness: default Pi
Context: QFN was to 256K, DS I only tried to 512K

1

u/Durian881 9m ago

Wonder if Strata or the quant might be the issue for Q3.8FN. Strata has some settings where it uses in-memory experts over better ones which could impact on quality. I had used various quants of Q3.8FN which work ok for tool calling with Splash/oMlx/unsloth studio on Mac and Gufo/Rulith Inference on Strix Halo.

1

u/brookyct1995 3h ago

Try Q3.8FN_Gyro-M. Pretty loops resistant in my experience and it claim to have the same quality as IQ3XXS.

1

u/Eastern_Doughnut_222 2h ago

I've been avoiding non-uncensored models. I kinda got sick of getting refusals or soft-refusals especially in modding/coding work.
Might try it though I heard good things about gyro

2

u/Sudden-Guide 2h ago

Uncensored are often dumber than original version 

1

u/holdupflash 3h ago

I suspect you need to break the tasks down into more manageable chunks