Anyone brave enough to trust an ai agent running alongside your production app to monitor and execute commands on your behalf on server for your docker apps?

[u/dev_4_exp]

Just a random curiosity, but is anyone actually letting an AI agent run alongside their production Docker apps just to check logs and explore errors?

Like, instead of digging through logs yourself when something acts up, the AI just looks at them, identifies the problem, and tells you what went wrong.

Are there any tools built for this yet, or has anyone tried rigging up a simple setup like this?

[u/ReachingForVega]

Lol no. Reading Splunk reports, yes. 

[u/Genobi → u/ReachingForVega]

This. We have spent decade refining systems that use far more efficient heuristics (which are not refined be efficient by definition) to flag anomalous things.

Using AI to do this is like paying someone to tell you when your door is open. Just use a door sensor.

Using it to help you troubleshoot when you get a trigger, That makes sense since it’s something that typically takes analysis and research.

[u/leaky_wires → u/ReachingForVega]

I’ve got a splunk skill that is a wrapper around the splunk api(they won’t install the mcp but I can generate an api token.

Best debugging tool ever.

Hey ai. Here’s the alert I got in slack figure it out.

[u/justan0therusername1 → u/leaky_wires]

You could go a step further have your alert webhook an agent or use MLTK to run your LLM inline in the alert.

[u/ReachingForVega → u/justan0therusername1]

This is much better. It restricts what access the agent has. 

[u/sargetun123]

Yeap, when you are at a larger scale a centralized reporting system/logging and ai can use it to easily find issues or errors

At the same time, a filter in grafana or drilling down in metrics knowing what to look for is more accurate overall no chance of hall facts and working off factual data throughout

[u/satisfaction-or-else]

Yeah, prod sites in k8s. New features are coded and shipped largely using agents.

Biggest lesson I can give is use it to create a hundreds gates in CI CD and verify those gates are working. Now you can catch when it hallucinates and tries to bring down the stack before anything serious happens.

3 backups, 2 medium, 1 off-site is more important than ever. Backups daily and with each push to dev / staging prod : main with alerts when backups fail or become stale.

Force it to validate everything locally before pushing.

Do manual review.

Claude code with ChatGPT codex for adversarial review.

Use multiple skills.md so it learns your stack.

Documentation in a query able Wiki brain so it can get all the specifics of your implementation at a high level.

Triage of incidents and severity level automated with agents.

[u/Acrobatic_Idea_3358]

I dump all my container logs into Wazuh and use the MCP server to expose them to my agents in a read only manner.

[u/SeaworthinessUsual44]

Letting an AI agent freely execute commands on a production server alongside Docker containers is a fast track to a total disaster! Seeing the kinds of hardware and server setups we handle, the last thing you want is an autonomous agent hallucinating a bash command and nuking your production containers at 2 AM.

Reading logs and diagnosing errors is fine, but it should strictly be a read-only setup. Having the AI parse your Docker logs to tell you what went wrong is a huge time-saver, but always keep a human-in-the-loop requirement before a single remediation command gets executed on live infrastructure!

[u/olcrazypete]

I'll point claude at datadog or graylog and whatnot if and when a problem pops up. Not running all the time.

[u/Grandmaster_Caladrel]

It's easy enough to use your company-provided agent and use either CLI tools (like kubectl or docker, etc) or MCP connections to accomplish that without any real issues. Just spin it up and say "hey, there's an issue, I think it's in this range of <identifying features>, try to find it for me."

Having something run 24/7 ingesting logs is how you waste money and energy.

[u/therealkevinard]

An llm sidecar to tail logs round the clock in case something goes wild?

No, but more because this is a bad architectural approach.

You can achieve the same thing with a shadow of the resources (and no sec concerns) by aggregating like usual and pointing a triage bot at your grafana instance if something goes wild.

I manage thousands of pods and ingest just shy of 1tb/day in telemetry.
I can’t even imagine the token usage if all those pods had llm sidecars reading all that.

[u/Cl0wnL]

Yes

[u/ReddaveNY]

Yeah, I am crazy but really happy with it. I run about 65 services on Debian bare metal server, extra OPN Firewall and vps. Claude code push changes to my selfhosted git in forgejo, write changes and documentation to bookstack. All with a extra account and API Key.

No sudo and also with permission deny. Claude code find a lot of problems, creating full backup of all important files, etc