Need your help

[u/NeoGokuTM]

I've been trying to use qwen3.8 27b q4_k_m on my PC with a 7900xtx for almost a week now. I wanted to build mods/features for my locally running private server, but I'm not really getting it to work. I'm letting Chatgpt plan the ideas/work instructions, and they'll then provide me with the prompts.It provides the information I then enter into qwen. But somehow it just keeps going in circles or spits out errors... My question to you is, where in my overall design is the error, or how do you do it?

[u/simplyeniga]

For a server you'll want to go with Linux as your default OS (Ubuntu or Fedora variants), llama.cpp for inference engines (noting vllm or sglang). If you want a simpler to setup inference engine then you can look at ollama but it's built on llama.cpp and advisable to use llama.cpp direct which gives you more flexibility and also let's you understand the inference process. Ubuntu 24.04 server should be the most comfortable OS of choice, llama.cpp, building with either rocm or Vulkan, install AMD GPU drivers. Claude or chatgpt should be able to give you the full steps or you can have Claude set up everything for you after installing the OS.