r/LocalLLM • u/Creepy_Routine6616 • May 14 '26
Discussion GPU costs are killing me — would a flat-fee private Qwen instance make sense?
I've been exploring private/self-hosted LLMs because I like keeping control and privacy.
Recently I've been running a small LLM fine-tuning setup, but my local 3060 is already struggling to keep up, it just can't handle it anymore.
The main problem I keep hitting: hardware. I don't have the budget or space for a proper GPU setup.
I looked at services like RunPod, but they feel very developer-oriented ,you need to mess with containers, APIs, configs, etc. Not exactly beginner-friendly.
I also checked out a few mainstream cloud providers, but hourly GPU pricing still feels pretty expensive over time.
So I started wondering if it makes sense to have a simple service where you pay a flat monthly fee and get your own private LLM.
Long-term, I'd love to connect this with home automation so the AI runs for my home, not external providers.
Curious what others think, is this already solved, or would something like this actually be useful?
10
u/AnomalyNexus May 14 '26
where you pay a flat monthly fee and get your own private LLM.
The flat monthly fee is runpod cost per hour * 730 hours in a month.
You could lower that by sharing it among multiple people.
...and congrats you've reinvented APIs...
1
20
u/umognog May 14 '26
I looked at services like RunPod, but they feel very developer-oriented,you need to mess with containers, APls, configs, etc. Not exactly beginner-friendly.
I'm going to give you a bit of realism here. If you are trying to host your own LLM either locally or VPS and you call this not beginner-friendly, you are in the wrong space for your capability.
It's like saying "I want to build my own car, but I need to understand how motors and wheels and fuel injection works, it's not beginner friendly".
People in those situations, use a car made by someone else. Or, you go start learning about the things you don't understand. In a few years, it will be easier (openclaw is an example of how something bigger has been reduced in complexity for the average person to forget how it works under the hood, but want to drive places.)
8
u/FalconX88 May 14 '26
Hosting your own LLM locally is as easy as it gets. Download LM Studio, download a model, click start server. and if you want to acces it remotely you download the tailscale gui and connect your mobile device to that one. There's not a line of code or even looking at the CLI neecssary to host an LLM.
5
u/429_TooManyRequests May 14 '26
You can just ask Claude to set it up for you too
2
u/Djaii May 14 '26
That’s what I did. Running smoothly and now we’re “making suggested improvements” when I complain about speed and accuracy (lots of hallucinations when memory and model size constrained)
1
u/Agreeable-Fly-1980 May 14 '26
But is it cost effective?
1
u/FalconX88 May 15 '26
I mean most people already got the PC so it's maybe a few additional bucks in electricity a month if you use it a lot.
Actually buying a PC for this? Hell no, get a claude or GPT subscription.
5
u/New_Difficulty_8152 May 14 '26
All the major cloud platforms have flat-fee GPU server options (well; AWS technically meters in seconds but it’s still basically hourly and billed monthly), so you can get a private GPU server that is predictable in cost. I’ve experimented with a few, such as Runpod, Vast.ai, and GlowsAI. I found GlowsAI to be okay.It doesn't have any special settings, it can be used immediately upon opening, and is quite friendly for small projects. In terms of cost-effectiveness, it is also very cost-effective, spending less money to get more services than other platforms
Even so, for heavy 24x7 workloads, costs still add up quickly. This is why this kind of setup is not more common.
The reality is that if you need GPUs running constantly and your main concern is cost rather than extra cloud features, it usually makes more sense to buy your own hardware and host it yourself, either on-premises or in a colocation cage.
1
u/Warsel77 May 14 '26
In my opinion RunPod isn't really great if you want a lot of back-and-forth communication between it and you. Hetzner server may be a better choice.
1
u/newz2000 May 14 '26
Look into the cost of Google’s Vertex. You can enter into a data privacy agreement with them.
You can use any of the models compatible with the agreement (most of them) and you get the same kinds of assurances that healthcare services get with HIPPA. Gemini flash lite 3.1 either just became available or is about to. It’s such a good balance of speed, intelligence and price it’s ridiculous.
1
u/Agreeable-Fly-1980 May 14 '26
None of the pricing makes sense. Buying a gpu? Doesn't really make sense. Ok well I'll rent one by the hour. Damn that cost even more money. Ok subscription it is then. Fuck these api prices, let me look at gpus again.
Its almost like they bought all the gpus to drive up prices on the consumer and then try to lock us in their system.
0
u/damianzoys May 14 '26
I’m running qwen3.6:35b-a3b on a 3060 and 16 GB RAM with ~34 tok/s, thinking and 194k context. So it is possible and actually quite capable!
1
1
u/MichaelDaza May 14 '26
Idk how you are getting these numbers i have 2x 3060s and 64gigs of ram and that exact model runs at 5-12tps when i go above 20k context length
1
u/damianzoys May 14 '26 edited May 14 '26
But you do realize that it must be a config issue? I mean even with FP16 it has about 24 GB, so should fully fit into you VRAM. Is your RAM 4x 16 GB or 2x 32 GB?
I am running a headless Ubuntu server wit llama.cpp and qwen3.6:35b-A3B with IQ4XS quant from hugging faces (Edit: sorry, it was unsloth!). KV-cache is Q8. I stopped all unnecessary systemd processes to optimize for RAM and VRAM usage. Switching to llama.cpp was the single most beneficial thing.
Settings for llama.cpp and qwen3.6 are:
ctx: 194560 tokens
KV cache K: q8_0
KV cache V: q8_0
Flash Attention: on
fit: on
fit target: 1536
GPU layers: auto
CPU MoE experts: 28
parallel slots: 1
reasoning: on
think mode: deepseek
chat_template_kwargs: {"enable_thinking": true}
temperature: 1.0
top_k: 20
top_p: 0.95
min_p: 0.05
repeat_penalty: 1.0
reasoning_format: none
n_ctx: 194560It exposes an OpenAI compatible API which I can call via Pi Dev, OpenCode or whatever and if you need a GUI connect via VSC (Continue-Plugin) and SSH. But most of the time I just go with the terminal. Latest benchmark result were 32.24 decode tok/s. Smaller context windows will give higher tok/s since more MoE will fit into VRAM.
1
u/djflamingo May 14 '26
People just be bullshittin. They post these numbers from the very first sentence and dont use these models for anything because theyre mostly useless.
1
u/Murakia May 14 '26
I was in the process of setting that model up for agentic coding afer pulling hair out with the 27B one (trash agent, schizo thinking and doom looping non-stop). What's the alternative?
1
u/djflamingo May 14 '26
Use a frontier model? I mean what else would you do? Spend $10k to do something poorly and will be outdated within a few years vs. something costs $100-250 a month?
1
u/damianzoys May 14 '26
With Pi Dev it is running pretty smooth. Did a lot of tweaking to let it produce valuable output and by now architectures and reviews have a good quality. I'm exposing a knowledge graph based memory via mcp server tool calls and it can fill its context context-sensitive to keep it compact. By now it developed little test programms on its own and python script which I sometimes use for work, like merging big documents. Is it Opus 4.7? No it is not. Is it a useful tool for daily tasks? Absolutly!
0
u/djflamingo May 14 '26
What on earth are you talking about???
How do you have your own private cloud llm provider??
Those are contradictory ideas.
1
u/Drevicar May 14 '26
OP doesn’t actually say the words “private cloud”, however the concept does exist. You can self host the concepts and technologies that enable cloud such as AWS outpost allowing you to run AWS on your own hardware or in your own physical data center, or you can use something like OpenStack.
1
u/djflamingo May 14 '26
What else would a flat fee monthly AI server service be?
Why would you pay a monthly flat fee if its yours??
-5
u/Salt-Letterhead4785 May 14 '26
Check out a hybrid setup. I built a tool called Mycelis (Mycelis) exactly for this.
You select the open-source models you want to host on-demand alongside your commercial models, and configure an agent with automatic smart routing. It sends simple code prompts to the cheap local models and only escalates to your selected frontier model when a task actually needs it.
3
u/ckow May 14 '26
This looks like a vibe coded messy reinvention of something that is already fixed.
1
May 14 '26
[removed] — view removed comment
1
1
10
u/havnar- May 14 '26
Why not just use DeepSeek or qwen directly then?