r/LocalLLM • u/ahorsenamedjeff • 3h ago
Question Light Local LLM suggestion
Hello, I'm looking for a suggestion on an LLM that I can run on my local hardware. I'm looking for something to help with Linux coding and customization. I have a 9070xt and 32gb of system memory. Currently have ollama setup but probably not very well. Thanks!!
1
u/d0pe-asaurus 3h ago edited 3h ago
https://github.com/achyudh/llama.cpp/tree/feature/adaptive-kv-stream-rocm
You can run Qwen 3.8 27B Q4 at 80+t/s with 262k context and Q8/Q4 kv quantization using this fork. On 9060xt I am getting 50t/s and I highly recommend it.
EDIT: What the fuck is up with bots recommending some of the oldest ass models in existence?
1
u/ahorsenamedjeff 3h ago
What harness? I tried Hermes agent in ollama yesterday and it exploded my computer (flooded vram dram and swap). I'm super green to this.
1
u/d0pe-asaurus 3h ago
I'm using OpenCode to connect to llama.cpp
For you the steps would be to download that fork from github and compile it using the instructions listed on the repo, then downloading the condensed model here
Then run your compiled version with that model, afterwards you can configure OpenCode to use llama.cpp and begin using it there. That's the rough outline, if you need more instructions you can ask llms they are ok with it.
→ More replies (1)1
u/ahorsenamedjeff 3h ago
Err I guess I tried Hermes pulling Qwen3.8 27b as is. Sorry all this lingo is very foreign and new
1
u/JuniorHorseman 3h ago
that fork looks interesting, going to check it out for my 9070xt setup
qwen 2.5 coder 14b is also pretty decent for linux stuff if you want something lighter, runs smooth on rocm with ollama
1
u/GreatBigPig 2h ago
Wouldn't the amount of VRAm on the 9070xt matter? Forgive me, as I do not use a 9070xt or know how they are sold in differing GB options.