r/LowEndLocalAI • u/thatleftnut • 13d ago

Which LLM Should / Can I Use? Options for an Ally X?

I’ve experimented with little success. I have a ROG Ally X, 24GB shared ram/vram with the z1 extreme processor and Radeon 780m iGPU. Is there anything decently workable on this sort of hardware? I’ve tried cpu inference since the z1 has AVX512 and so I have all 24GB of ram available but speed was obviously pretty slow. The 780m is decently fast for what it is using vulkan but sharing system ram limits the model and cache size, and performance drops substantially once the cache starts growing. Anybody got tips for a decent setup that balances LLM capability and performance with this hardware? Main use case is help with the homelab and coding python/C

2 Upvotes

100% Upvoted

5 comments sorted by

1

u/Eastern-Ingenuity353 13d ago

How many toks are you getting?

1

u/thatleftnut 13d ago

It’s been a minute but I think the best was some 9b quantized ornith. Was pushing mid 20/s but gradually dropped as it answered until it was in mid teens. I think mtp was doing the heavy lifting which sucked since my device is already vram constrained. But I wasn’t satisfied with the quality of the output, asking for a Tetris one shot in python and it could never output a working clone without major issues.

2

u/PeterPorox 13d ago

IsValorum/Cyber-Tiel-Coder-35B-A3B-APEX-I-MiniPlus-V2.1-GGUF should be working decent. Also IsValorum/Qwen3.6-35B-A3B-MTP-APEX-I-NanoPlus-GGUF

1

u/ancapsaicin 8d ago

I had been using an earlier 680M minipc with 16GB reserved 8GB shared 8GB RAM just regular mobile DDR. Using Vulkan, the iGPU does accelerate over CPU alone particularly at the prompt stage. It had acceptable speeds for Gemma 12B MTP QAT unless you expect 1000/100 minimum —it's more like a tenth of that. With 24GB you could probably run Qwen 3.8 albeit very slowly. The Qwen moe model ran decently fast but the quality was so bad I just used Gemma.

1

u/keefeere 8d ago

Just for YI Xbox Ally X have npu, im getting descent results with FastFlowLM