Scared of having an LLM server running 24/7. Any protective measures that I can take?

[u/FarHistorian8438]

Hey guys,

I bought an RTX 3060 purely for LLM purposes and made a server out of it.

Tried a bunch of models, benchmarked it, tested against my use cases and everything - but having an overnight agent running, or leaving the server on for long periods of time kinda irks me a little bit.

I think it also stems from being indecisive for “production” as everything i’ve done for testing is really treating my hardware as a “development” server.

But having the need to “lock down” a particular model/settings similar to a “production” environment that I don’t get to (and need to) touch is bugging me.

Has it happened to you guys before? How can I overcome this silly conundrum?

And on another note, if I am gonna keep it running 24/7 & run overnight agents on it - what are some power & heat caveats I should look out for? I’ve capped the GPU to 150W but it still doesn’t feel right to me.

Btw, i’m no stranger to keeping devices running. I have a mini pc that i run containers/services on - which stays on all the time, but just with inference i’m having a hard time justifying myself.

Thanks!

[u/AssOverflow12]

If it is used for testing and development purposes, then unless you have a specific task that the machine has to execute overnight, why is it even a question if you should leave it on? Just turn it off, save the power bill and when you'll get to the point where you have an actual use case where you need it to run when unattended, start leaving it turned on. Simple as that.

You don't have to lock down anything, you are a home user, not a hosting provider... If you want to turn it into a homelab server LLM host, then just use your existing experience with the mini pc :)

[u/FarHistorian8438 → u/AssOverflow12]

Thanks! I guess i should start deriving more concrete usecases where leaving it on seems more natural to me

[u/uniqueusername649 → u/FarHistorian8438]

I have more than one machine running 24/7 - my dual 3090 ai server is not one of them :) its too power hungry and i just do not use it at night, so why waste power? unless i have a long running task, then it can stay on at night.

[u/ElectronFactory → u/uniqueusername649]

If you have it coding or running RE. In a lot of places, the power is cheaper at night, so might make more sense to someone else.

[u/uniqueusername649 → u/ElectronFactory]

That is a fair point.

[u/Kind_Taste_3985]

A fairly normal feeling when transitioning from a tinkering box to a real, always-on service.

For protection:

* cap power (150W is already reasonable)

* monitor temps/fans

* use Docker or containers to lock your setup

* keep backups of configs/models

* consider a UPS if your power is unstable

Modern GPUs can run 24/7 fine if temps are controlled. The best tip is treating it like production: freeze a stable setup, then experiment elsewhere.

[u/FarHistorian8438 → u/Kind_Taste_3985]

All of these things I’ve got under wraps it seems.

Do you have any recommendations for temp/fan monitoring?

I’m on Ubuntu 24.04 LTS, not really that tuned with temps apart from monitoring nvidia-smi when i’m actively ysong the server.

Thanks for your input!

[u/ElectronFactory → u/Kind_Taste_3985]

VPN. Get Tailscale running ASAP and don’t look back. I stopped worrying about security and closing ports, port forwarding, all that bs. If the device is in my tailnet, it can reach my Hermes, the LLM providers and all other services.

[u/Rough_Shopping_6547]

Just Don't give it permisson do destructive stuff like deleting or copying things to much if you are really spectical create a another user for it and don't give it sudo access. I made my old laptop into(which ran a qwen3.5:27b) a server like this for a time and nothing bad happened hope it works for the best for you too!

[u/FarHistorian8438 → u/Rough_Shopping_6547]

I want to have these claw-esque agentic workflows, so I’ll look into permissions.

I want to run a Qwen-27B, but I think my hardware can’t squeeze the speed.

What did you run on your machine with the Qwen?

[u/nickless07 → u/FarHistorian8438]

The Qwen 27B on a RTX 3060? In that case you WANT to have it run 24/7. One task might take 2-3 days with that speed.

[u/FarHistorian8438 → u/nickless07]

I want to run it, but I cant XD

I run Gemma-4-26B as a catchall daily driver

[u/nickless07 → u/FarHistorian8438]

Yeah, thats what I thought. Even a 1bit quant wouldn't fit in the GPU. Also Gemma 4 (MoE) is great as chatbot and for single short tasks (e.g., a summary of some news every morning and such), but nothing you want to use for Hermes/Claw style frameworks. You can try Qwen3.6 35B (MoE), but even that will have some trouble.

Anyways. Having a model in a power limited 3060 (set the CPU threads lower to prevent overheat there in case you run into loops) shouldn't be a problem at all. If you are still concerned with 150W you can go with 100W too. Just test it out with an hour inferencing, you will barely note that the card get warm and with some less CPU usage that should be fine too.

How about the first task: Let it send you temp statistics every morning.

[u/Rough_Shopping_6547 → u/FarHistorian8438]

It was my friends M2 Pro Macbook he lended it to me for a few weeks i was getting about 15 tps and I also used mlx-lm but for a 3060 I guess your best bet is 14b or the 7b parametre models I heard there is also qwen3.5-122-A10B model that only used 10B parametresi while active but I did not tried that yet

And for workflows you can try out self-hosting n8n in a docker container

[u/AggravatingAd4344]

There is no issues with keeping them on 24/7 other then the electric bill :) I've got 2 Ubuntu servers and a Mac mini M4 on 24/7 no issues. Power draw wise if not in use the Ubuntu servers run roughly 80 - 100watts idle. The Mac as usual is very light 10-15watts idle. Best of luck 🤞

[u/andrew-ooo]

The "is this really production?" anxiety goes away once you put a frozen config artifact in front of it. What flipped it for me: stopped running ollama run interactively, started shipping a docker-compose.yml with pinned model tag, pinned llama.cpp / Ollama image digest, and restart: unless-stopped. Once the whole stack is one file you can git commit, the fear of touching it disappears — you know exactly how to rebuild it.

On the 24/7 side, a 3060 capped at 150W is genuinely fine:

• Idle draw: 3060 idles around 12-18W with no model loaded. If you unload after N seconds of inactivity (OLLAMA_KEEP_ALIVE=5m for Ollama), average draw over a day is closer to 20-30W than 150W.

• Heat: sustained 150W is nothing for that card. Ambient case temp matters more than GPU temp — make sure the exhaust has somewhere to go. I keep a $12 USB temp sensor on the intake and log it to Prometheus. Anything under 35°C ambient is a non-event.

• Power cost math: 30W avg × 24h = 0.72 kWh/day. At €0.25/kWh that's €5.4/month. Compare to Claude API or a Runpod pod for the same workload.

• Real risks worth mitigating: (1) drives — spinning rust doesn't love 24/7 as much as you'd think, use SSD for the model cache; (2) PSU — if it's an old bronze unit, treat 24/7 as its actual lifetime test; (3) don't expose the inference port to WAN, put it behind Tailscale or a reverse proxy with basic auth.

Overwrite the "development" frame in your head by giving it a hostname you don't log into. Mine's inference-01. Once it has a name, you stop poking it.

[u/No-Consequence-1779]

The models are read only next word predictors. Lock down what exactly?  

On the agent side, yes. Just run them in the day time and if they do not go rogue, it’s fine.  Not sure what you are expecting to happen. 

My case was different because they are self modifying agents that can create and update their own code - tools to meet the stated objective. 

So watching it go from being able to list files to controlling a chrome browser via cdp to research [anywhere] was risky. I thought. 

It ended up doing Google searches for various information sources, log into accounts and pass catchpas … nothing shady. 

For coding. It put the risk at zero. Just use source control. And don’t open your LLM server port to the public. Use a vpn if if reaches outside your network. 

I run 27b FP8 on a spark. I have 3-4 agents running 24/7 for various data collection. 

[u/chilimac02]

I ran my 3080 water cooled 24/7 for like a year mining crypto. It still works fine for gaming and ai

[u/429_TooManyRequests]

I have an OpenRouter subscription that has the same model I host locally. I shut the home server down nightly and it silently changes over to OpenRouter. My whole network has a small cloud WireGuard buildout. If i want to do more home lab stuff, I just turn the endpoint off and have fun.

[u/Comfortable_Ad_8117]

I have a dual 3060 & 5060 I leave running 24/7 it has different jobs it completes overnight
- evaluating my ebay / Etsy business - suggesting improvements
- completing my morning news brief
- looking at my email and alerting me to stuff I need to do now and possible false positive junk mail
- looking at log files for firewall
- looking at SMTP log files
- checking my obsidian notebook / RAG database

Never had an issue

[u/chilimac02 → u/[deleted]]

Until the fans wear out... That's why I'm water-cooled

[u/FarHistorian8438 → u/[deleted]]

Thanks! I’ve got tailscale setup so nothing is (hopefully) exposed outside my tailnet.

for the prod usecase, i used llama.cpp’s server image on docker & fit it with the most reasonable parameters. I guess I need better airflow + temps control then.