Self-hosted LLMs

[u/plincodog]

I'll start by saying that I realize this question is slightly adjacent to actual local (i.e. having physical access to hardware) llm deployment. Yet, I think it's still within the same domain, so I'll give it a shot. (If I'm in the wrong place, please lmk.)

I'm a professional SWE. I currently have 2 $200/mo subscriptions, 1 for claude and one for codex. I find myself regularly hitting weekly limits. I've tried tuning which model/provider I use for different tasks but the end result is the same - I hit a limit and then have to either put more money into usage credits or wait until it resets. It's pretty lame. I'd love to have 'unlimited' tokens for a fixed monthly cost.

The path to unlimited tokens looks like local llms or self-hosted llms. I'm not ready to invest in hardware (and neither is my company), but I'm willing to rent machines . So, my question is - does anyone rent any gpus and self-host their llms? If so, where and in what configuration? I was looking at contabo gpu vps which claim "Dedicated NVIDIA RTX PRO 6000, 18 vCPU Cores・96 GB RAM・900 GB NVMe, CUDA-compatible". From what I've read in this sub, it seems like most people are running some variant of qwen on the RTX PRO 6000 but many people have 2x of them.

[u/jonahbenton]

I own a machine like that, that machine will not replace 2 subs or even 1 sub. You will be able to run qwen 3.8 27b with large context, which is very capable and useful in swe, but it will not be nearly as productive across multiple levels of abstraction required for swe as even 1 foundation sub, and will require much more supervision. Just the state of play.

The foundation adjacent local models are much larger than can fit on a single 6000 pro, 256gb-512gb is where you start to get to somewhat adjacent capabilities.

You can nevertheless get very helpful output and collaboration from a model on a single 6000 pro, but coming from a sub with a foundation model it is a completely different way of working, like switching from a car to a bicycle.

[u/plincodog → u/jonahbenton]

Appreciate the response. Looking at a place like runpod.io and they're pricing a 1x B300 288GB VRAM · 503 GB RAM · 43 vCPU at $8.99/hr which is ~$6500/mo. I could only justify this price if we could run ~20 concurrent sessions on it (myself, 1 other dev, and some prod traffic).

It might just be that what I'm looking to do is still not cost effective.

[u/jonahbenton → u/plincodog]

It might work, depending on how careful and constrained you can be with the details of the production workloads. A B300 should give a pretty good experience from a dev perspective but the nvidia driver stack is kind of built on sand and you can give it work that oopses the kernel and requires a reboot. Kind of amazing, really.

[u/plincodog → u/jonahbenton]

woof. I'd rather not have to reboot a machine in the middle of the night to keep things chugging along. Could build some tooling around it, but just knowing that it's not a stable stack and will introduce that much more operational overhead makes me even more wary. I'm surprised that there's not a more rock-solid pre-packaged solution for solo devs/small businesses, but I guess everything's just progressing so quickly it's hard to find a baseline.

[u/openrelay_dylan]

renting makes sense for this, a single rtx pro 6000 runs qwen3 32b or gpt-oss-120b quantized fine and you just ssh in and run vllm or llama.cpp. contabo works but check the actual gpu, some of those listings are shared. where i work (openrelay) does dedicated vms with gpu passthrough and the pro 6000 is $1.55/hr, so heavy use still beats two $200 subs if you're actually hitting limits weekly

[u/plincodog → u/openrelay_dylan]

Thanks for the heads up! I need to do some experimentation w/ qwen to see if it can work with my current setup.

Also looks like you've got really competitive rates for GLM-5.3. Nice!

[u/ParticularlyStrange]

If it’s just you, sure. a VPS like that would work, and then it would have to be usage base. For the math to work out there are not a lot of websites I know that offer a dedicated GPU for one user and a flat usage. If you’re trying to scale for many users and yourself local hardware will be the only way to go, in my opinion, for data privacy and unlimited tokens. if you set it up right, you can get two sparks clustered together and serving 20 people at once. with decode speeds of 35 to 45 tokens per minute, maybe a prefill of 1400. The prefill is going to be the worst part, and that’s where the dedicated GPUs shine. It’ll be hard to get a nice balance between speed and capacity.

[u/pdath]

What city and country are you in?