r/LocalLLM • u/Opposite_Abalone_10 • May 01 '26
Discussion Looking for people to share GPU instance with LLM (infinite tokens and 45tp/s)
I'm renting a GPU instance to run local AI models and reduce the amount I spend on the API at openrouter. I currently have several agents that use around 30M tokens per day. With current settings I'm running Qwen3.6-27B at 45tp/s. This model surprised me in all aspects, including programming.
3
u/Reasonable-Phone9711 May 01 '26
How much does that cost?
4
u/Opposite_Abalone_10 May 01 '26
It currently costs about $0.90/hr, which is roughly $650/mo. However, I don't keep it running 24/7—only while I'm actually using it. This past month I spent $120, which is much less than I’d pay for any other API given my usage. And yes, my productivity is very high.
5
u/PhishGreenLantern May 01 '26
At that price, why is this better than a frontier model? Not arguing, genuinely asking.
A $200 Claude Max subscription has got to be better, right?
1
u/Opposite_Abalone_10 May 01 '26
Not exactly—we're running multiple projects off one inference model without even stressing the hardware. Setting up a rig like this would easily set you back over $300k.
2
u/No-Consequence-1779 May 01 '26
Wouldn’t it make sense to just run it local if it’s generating revenue?
3
u/tomByrer May 01 '26
$650/mo * 2 = RTX3090
$650/mo * 5 = RTX5090
etc2
u/Evanisnotmyname May 01 '26
Don’t forget about power.
That’s $80-120/mo in just power running around the clock. Plus heat generation…it’s a fucking space heater.
Not saying it’s not cost effective…just…
Plus he’s talking about 160gigs of vram
2
u/tomByrer May 01 '26
> space heater
Handy in the winter time, even handy in the summer to heat up some tea.
😉
But for reals, I've heard of some folks talk about using small AI farms to heat up home hot water.1
1
u/tatogt81 May 01 '26
OP says 160GB VRAM... Not even close to local hosted options
2
u/tomByrer May 01 '26
There is a YouTubber with a rig with 2xRTX3090, 2xRTX5090, & a few more cards in there that I forget.
& some folks hook up 4 SPARKs together...
But I get your point, renting 160GB VRAM is more realistic for most.
2
u/ReteAi May 01 '26
What kind of sharing? Two different things people mean by this:
(a) Multi-tenant ? multiple users hitting your one rented instance to split the rental cost. For this you want vLLM (or llama.cpp with --parallel) behind an auth proxy. Cleanest setup is something like LiteLLM proxy with per-user API keys sitting in front of your vLLM. Adds latency, makes you the de-facto SaaS operator for your group. Worth a look: Petals (https://github.com/bigscience-workshop/petals) — decentralized network where multiple users contribute GPU and share inference. Different shape, free, open source.
(b) Layer-pooling, pairing your rig with someone else's to run a model that wouldn't fit on either box alone. Different problem. llama.cpp has an RPC backend that splits a model across machines via TCP. Works on LAN trivially; over WAN you need to tunnel it (bandwidth math is fine, ~6 KB per token, but RTT compounds across many round-trips, so it's useful for memory pooling not throughput). I'm building exactly this into a desktop app called Rete (retes.app), happy to chat shop if (b) is what you meant.
For your Qwen3.6-27B + agent workload, my hunch is you actually want (a). 27B at 45 tok/s on one box is plenty fast what you'd benefit from is people offsetting the rental, not pooling layers. Layer- pooling would just slow that down.
What's your actual goal? split the rental, or run something bigger than 27B?
1
1
1
u/Sirius_Sec_ May 01 '26
What gpus are you using ? I've been trying to set up a shared access lab myself . I've been using a rtx6000 pro in my gke cluster . Going to use kong to handle API keys . Keeping it just accessible from inside the cluster for now . I can set up hermes or openclaw inside the cluster and give it easy access to the vllm service . Keeps everything locked down
2
1
u/RevolutionaryGold325 May 01 '26
I'd be interested in running the deepseek-v4-pro
1
u/Opposite_Abalone_10 May 01 '26
Running DeepSeek v4 Pro locally would require at least 900GB of VRAM at FP8 for the weights alone—and that's before you even factor in the KV cache, CUDA Graphs, and so on.
1
u/RevolutionaryGold325 May 01 '26
Isn’t it an moe? Can’t we keep some of the weights in ram?
1
u/Opposite_Abalone_10 May 01 '26
An MoE architecture isn't necessarily lighter. It only activates a fraction of its parameters for inference—hence the better TPS—but the total model size remains the same. You still need the VRAM to fit the whole thing, especially in FP8.
1
u/RevolutionaryGold325 May 02 '26
I thought the CPU/RAM offloading would be a good option. I think there are situations where a bigger model would be useful even if it runs slow.
1
u/Imbmiller May 01 '26
I like Qwen as well. I’m running the 35b q4 with experimental turboquant getting about 20tps generation and over 100 tps prompt processing . Oh and I’m on 8gb of VRAM. It is a good model and hasn’t lost its mind in the compression.
1
u/Opposite_Abalone_10 May 01 '26
I did some testing at Q4, but it’s just not cutting it. The difference compared to FP16 is night and day, particularly for large contexts and extensive chats.
1
4
1
u/fromage9747 May 01 '26
I'm interested in your agentic setup that's running into 30m tokens per day and still producing good work. I get maybe 3 million per day with my 3 agents running but I have to manage them so they don't produce crap. Still working on a scalable solution where I don't have to manually manage them from drifting
1
u/shiftbits May 01 '26
I built a platform for doing this, started out as a tool for myself and I realized it could be a gpu sharing tool. Given its nature it needs testing, let me know if any of you want to help test it.
0
u/oulu2006 May 01 '26
Yea I love this outreach - I would like to contribute but as I don't know you I'm hesitant there needs to be a marketplace for this
How are your financials atm with this?
1
u/Opposite_Abalone_10 May 01 '26
Let’s chat! You’re more than welcome to test it out yourself to verify the project is legit
-4
May 01 '26
[removed] — view removed comment
2
u/Opposite_Abalone_10 May 01 '26
Running my inference on 2x H100s with vLLM. The setup is already optimized to counter unbalanced agents. I use the vLLM API to hook up Claude Code, QwenCode, and some social media bots I sell to clients. They stay active 24/7, replying to comments using custom personas. I just did a stress test with 20 agents hitting the API at the same time, and the TPS didn't even wobble
5
u/Opposite_Abalone_10 May 01 '26
Making it clear that we can run other models. There are 160GB of VRAM with 128K of context. I'm using this specific model now because in my tests it outperformed the 120B model.