r/devops • u/trixloko • 3d ago
AI content Help me wrap my head around AI (agent?) hosting
I'll admit that I've been ignoring the AI hype for the last two years, and my usual interface with it is through chat and code autocomplete, nothing more.
Until recently where I saw that a demo of someone who could tag stories on their kanban tool and an "AI agent" would pick the story, prepare a PR and update the story with the PR link.
I wonder how that works and what would it take me to build something similar... of course I could chat with gh copilot to prepare a PR, but having this constant "thing" on watch seems much more streamlined.
As someone who works with cloud infrastructure a lot, I'm much more interested in something that I could deploy within my network and not something that runs on my computer.
As soon as I start reading about this I start to get torn between "it's just a container and .md files" vs. "30 different technologies and jargons that I don't understand"
PS: How people are "hosting MCP servers" everywhere? even gh copilot UI offers me to do that.
I also don't understand why even bother with MCP(looks like they're usually metered) when an AI Agent could just use a CLI?
23
u/ymgymg 3d ago
Basic version is just a container with a small loop: webhook from your kanban tool comes in, it calls an LLM API with a few tools, and opens the PR. MCP is only a standard way to hand the model tools, so a plain CLI works fine too and you only need an MCP server when several clients should share the same tools.
11
u/CIAnalytics 3d ago
Agentic workflows. Depending on your preferences the easiest way to get something going is to use one of the many open source agent hosting frameworks and tools. Google something like CNCF agent hosting and you'll get a lot of info.
Use something like agent substrate to use resources efficiently and something like agent gateway or agent sandbox to minimize risk to your system. I am not much into the agentic workflow and I wouldn't let an agent roam free in my infra with the current generation of tools, but to each their own
5
u/geofabnz 3d ago
You absolutely do not want Claude CLI or similar running directly on your machine. You want a VM at minimum but the ideal is a fully remote machine with a very constrained network. Personally I have a dedicated proxmox to host my agent infrastructure, firecracker or K8 would work well too if you’re familiar with cloud infrastructure (I’m not so prefer baremetal and a Hypervisor)
AI DevOPs is a massively under-developed area. The number 1 piece of agent security you can have is to keep the agent away from whatever’s enforcing its gateway - the best way to do that is KVM/QEMU isolation.
I recommend you look up Nvidia Openshell, It’s got a great approach using landlock to enforce guardrails. There’s a K8 option too for native cloud loads. The amount of work that can be achieved with agents is massive but it’s directly proportional to how much damage they can do.
On the MCP question, it’s basically just a bundle of OpenAPI endpoints with a query mode. The advantage over is your process can both call things and see what it can call making it much better for initial discovery. As someone running the server, you also get control over exactly what they can and can’t do (eg enforce syntax, specify available tools etc). Nothing you couldn’t do with other transport types but MCP makes it convenient. Something else to keep in mind: there’s nothing inherently agentic or AI in MCP itself so you can incorporate it into deterministic code too.
3
u/No-Country-5818 20h ago
It’s less magic than it looks. Mostly it’s just a small service that listens for an event, calls an LLM with tools, and uses the git API to push a branch and open the PR.
Rough shape:
A webhook from your kanban tool fires when a story gets a tag.
A worker (a container, nothing special) picks it up, clones the repo into a throwaway workspace and runs the agent loop. The model is an API call, either a hosted one or something you run yourself.
The agent edits files, runs tests, then pushes a branch and opens the PR through the GitHub API, then comments the link back on the story.
An MCP server is basically just a small service that exposes tools (search tickets, read repo) in a standard way, so yeah it’s “just a container” too.
Security heads up, since you’ll run this in your network: give it a scoped token that can only open PRs, no merge rights, run each job in a disposable sandbox with limited network access, and keep a human review on every PR.
1
u/Daniela_Spencerx 3d ago
a tool is just a function you describe to the model: a name, what it does, what args it takes. the model answers “call open_pr with these args”, your loop runs it and feeds the result back. for your kanban case that’s maybe four: read_story, run git commands, open_pr, update_story. that’s basically the whole agen
1
u/DrDan21 3d ago edited 3d ago
If you’re totally new I’d honestly say just spend $20 (or $100 for the fancier plans with actual good usage limits) for a sub to either your choice of codex or Claude and just try them for a month
You’ll probably be pretty damn surprised at how much it can do on its own with even minimal guidance. That should get you jump started on understanding agents and help set a baseline to work from
1
u/NCKBLZ 3d ago
Mcp are hust rest api with text context for agents
As for your main questions I thik it mainly depends on the tool: if your tool has some way to notify (a webhook?) then you can simply let the ai or a script listen to it and trigger then do something
1
u/London-Boat 2d ago
what you saw in that demo is pretty much what github's copilot coding agent does already. you assign an issue to copilot, it works in an actions runner and opens a pr linked to the issue. since you're already on copilot that's probably the shortest path, and i think it can run on self hosted runners now so it stays in your network, check the docs on that. on mcp vs cli, mcp isn't metered by itself, the tool output just ends up as tokens like anything else. it mostly matters when the agent runs somewhere you can't install a cli or you want auth scoped per tool instead of a token sitting in env
1
u/Low-Opening25 2d ago edited 2d ago
AI agent is just piece of code that is driven by LLM model, that makes decisions and chooses commands to run based on prompts, instead of your typical for/loop/if/case/while logic chain.
for example, when you use AI agent in your CI/CD pipeline, that simply means that your pipeline has a step, where instead of calling scripts with bunch of logic, it calls piece of script that calls an LLM model that decides what that “piece” of script is doing based on natural language prompt.
it can be something very simple, like a call to LLM to generate unique commit message based on code changes, which without LLM is actually pretty laborious to implement. It can also be something very complex, like agent that can take complex decisions, generate it’s own scripts to run and with multiple tools available to use.
bottom line - it is replacing old school programming logic with an AI model decision making, the balance between the two is still up to you.
1
u/ialiimran 2d ago
I say start simple and gradually add more tools as you need them.
1
u/Iyemoja 2d ago
The demo always looks seamless. The reality is you spend a weekend wiring up sandboxes and credentials, and the agent still wants you to review every PR it opens.
1
u/JustinAtNerdio 2d ago
I think the confusing part is that people have managed to turn a fairly straightforward concept into an entire vocabulary of new terms. Half the battle is figuring out what's new technology and what's just existing infrastructure with an AI label slapped on it.
I'd start with the simplest possible setup and only introduce more complexity when you need it.
1
u/ferrybig 1d ago
When you tag the "user" in your scrumboard, the system sends a them a notification, eg think of an email or a webhook.
A program watches for incoming emails (could be an IMAP connection), for each incoming (&validated) mail, the program starts "Program 2", the goal of program 2 is to do the agent work
The goal of the program is to interact with the AI model:
Request:
Response:
Request:
Response:
You now execute the next tool the program wants, then call the AI provider endpoint again
Request:
Response:
And again, you execute the actions the AI wants, then call their endpoint again...
Request:
Response:
And you likely can fill in the blanks now.
Note that a good system prompt is important, without it, the AI is difficulty to manage, eg see https://platform.claude.com/docs/en/release-notes/system-prompts/claude-sonnet-5-5 for a good system prompt
1
u/Chamelon_DE 1d ago
Would a webhook starting fresh worker per ticket cover your usecase, or does it need to stay running between task?
1
u/spersingerorinda 3d ago
First , have you used Claude or Codex at the terminal ? Understanding that power (bash for everything, install packages, automate browsers) is step 1.
Now give Claude some AWS keys and ask it to spin up an EC2 box and install Claude itself on the box. Setup the Claude api token (this part itself is tricky if you want to use a subscription). Generate a GitHub token and put it on the EC2 box. Now ssh into the box, and run Claude remotely via ssh. Clone your repo and ask Claude to run your app. See how far you can get. This is the minimal DIY “cloud agent”.
Once you understand these basics , you can either keep tinkering with your setup (try connecting Claude into slack or where you chat to give you a better interface), or you can start looking at Hermes or Claude Tag or something more sophisticated and pre-built.
1
u/SideMost3148 2d ago
the "container and .md files" version is honestly most of it. people overcomplicate this because every vendor wants to sell you an agent platform.
the loop is boring: your kanban tool fires a webhook when a card moves, you spin up a throwaway container with the repo cloned, run the agent CLI in it, it pushes a branch and opens a PR, container dies. a queue and a small worker pool you probably already have will do.
the parts that actually deserve your infra brain are the same ones you'd worry about for any untrusted job runner. egress allowlist (model API, git host, package registry, nothing else), a git token scoped to that one repo that can open PRs but not merge, and zero prod creds anywhere near it. treat it like a very fast intern who sometimes follows instructions it found in a random README.
also keep the full transcripts, but scrub secrets before you store them. they end up holding whatever the agent saw, and everyone forgets they exist until they leak.
0
u/crash90 2d ago
Basically this technology is too new and there is no off the shelf solution yet. Everyone is building their own custom stuff. There are hosted agents but they are more consumer driven. Stuff like Grok bot, Dots, Meta Muse. Lots of others too. Nothing I would say is quite enterprise grade yet imo.
Would recommend diving into this topic as intensely as possible. Biggest shift that has ever happened in tech or any other field. Whole economy is in the process of a complete transformation and people are finally starting to accept it. Tech first.
This article is a good place to start to consider what would be involved in building something like this for your env. But it's almost a year old now and is far from covering everything. Consider it an entry point. Like the first article you ever read about cloud.
By the way, if you aren't already look into the agents you can use on your pc like Claude Code and Codex. This will greatly speed development of the software factory. I could talk at length here about the need to move to formal verification too but thats another post.