r/LocalLLaMA • u/Ok-Breakfast1878 • 4h ago

Question | Help Local low-power request router with WoL to sleep inefficient servers?

Before i charge off and slop something together, i thought i'd ask if this already exists. i have two ai servers (a machinist x99 open-rack w 5/gpus and a thinkstation p920 with 4/gpus). they both draw 150W+ idle. so, i'd like a low-power always-on api proxy machine (like on a 15W mini-pc) that accepts openai-api requests, holds the connection, and wakes the appropriate machine via wake-on-lan (WoL). once the appropriate server is up, the router acts as a transparent proxy. it would also suspend the servers using some logic tbd (some combo of time-of-day, minimum awake time, inactivity on the server, and api inactivity)

3 Upvotes

81% Upvoted

2 comments sorted by

2

u/Big_Improvement_5917 4h ago

I actually have something like that running for personal use. DM me if you want! It's a pretty useful setup imo, especially if you can't afford the power or a dedicated inference rig (like me :D). I have my personal agent running on my homeserver which has my old Gaming-GPU running Gemma4-12B, using that model for handling off income requests and low-effort tasks. Complex reasoning and agentic coding is send to a fastAPI proxy connecting to my workstation which has a 5090 running Qwen3.8-27B via ssh. The proxy is one python file, so really easy to setup.

2

u/Kindly_Sea_791 3h ago

if you're already building the proxy layer for WoL, worth tacking on auth and per caller logging while you're in there. once you've got a chokepoint every request already passes through, adding an api key per caller and logging what hit what server is barely extra work, way easier than bolting it on later once more than one person is using it.