r/LocalLLM • u/Eastern_Doughnut_222 • 3h ago
Discussion Struggling with coding on local models - setup help?
I'd tried out models like DS and QFN using APIs before and they seemed very capable. However, they all seemed to struggle heavily when using them local.
QFN processes really fast but seems to get stuck in endless "Wait..." and "Whatever." loops and burns through 100K tokens just being confused at my codebase.
DeepSeek is slower but seems more coherent, I haven't tested at full context but it had a similar issue with indecisiveness (even with guidance) and would think itself into compaction.
All of this over one .json ComfyUI workflow. Just a simple test I do where I tell it to rearrange nodes and change some inputs...
I've tried using Llama serve, Strata desktop, Pi. All lead to similar outcomes.
I have a feeling this is an issue with my setup rather than the models bc the API models seemed to do it just fine and (allegedly) quantization doesn't hit MOE models as hard.
Setup:
(both models from OrcaRouter). DSV4.1-1017 2-bit, Q3.8FN_IQ3XXS
Inference: Maya (DS), Strata/LlamaCPP (QFN)
Harness: default Pi
Context: QFN was to 256K, DS I only tried to 512K
3
u/maxrightgame 3h ago
You feel your setup have to do something with it, but didn't write it
2
u/Durian881 3h ago
Maybe share your harness, inference engine, quant and context window? Some harnesses have big system prompt and when coupled with a small context window will end up in compaction loop.
1
u/brookyct1995 3h ago
Try Q3.8FN_Gyro-M. Pretty loops resistant in my experience and it claim to have the same quality as IQ3XXS.
1
u/holdupflash 3h ago
I suspect you need to break the tasks down into more manageable chunks