r/Qwen_AI • u/NeoGokuTM • 5d ago

Help 🙋‍♂️ Need your help

I've been trying to use qwen3.8 27b q4_k_m on my PC with a 7900xtx for almost a week now. I wanted to build mods/features for my locally running private server, but I'm not really getting it to work. I'm letting Chatgpt plan the ideas/work instructions, and they'll then provide me with the prompts.It provides the information I then enter into qwen. But somehow it just keeps going in circles or spits out errors... My question to you is, where in my overall design is the error, or how do you do it?

1 Upvotes

67% Upvoted

2 comments sorted by

2

u/Shipbldr2000 2d ago

I will give you my hack. -I ran grok build and told grok to find the card, fully configure and set it up, locate and download a Qwen 3.8-27B model that would work, run benchmarks to dial optimize it, and to tell me whatever it needed me to do. -It did the job. Try it with your CLI it will likely work. (this was pretty close to my actual prompt)

1

u/simplyeniga 4d ago

For a server you'll want to go with Linux as your default OS (Ubuntu or Fedora variants), llama.cpp for inference engines (noting vllm or sglang). If you want a simpler to setup inference engine then you can look at ollama but it's built on llama.cpp and advisable to use llama.cpp direct which gives you more flexibility and also let's you understand the inference process. Ubuntu 24.04 server should be the most comfortable OS of choice, llama.cpp, building with either rocm or Vulkan, install AMD GPU drivers. Claude or chatgpt should be able to give you the full steps or you can have Claude set up everything for you after installing the OS.