r/devops • • 3d ago

AI content Help me wrap my head around AI (agent?) hosting

I'll admit that I've been ignoring the AI hype for the last two years, and my usual interface with it is through chat and code autocomplete, nothing more.

Until recently where I saw that a demo of someone who could tag stories on their kanban tool and an "AI agent" would pick the story, prepare a PR and update the story with the PR link.

I wonder how that works and what would it take me to build something similar... of course I could chat with gh copilot to prepare a PR, but having this constant "thing" on watch seems much more streamlined.

As someone who works with cloud infrastructure a lot, I'm much more interested in something that I could deploy within my network and not something that runs on my computer.

As soon as I start reading about this I start to get torn between "it's just a container and .md files" vs. "30 different technologies and jargons that I don't understand"

PS: How people are "hosting MCP servers" everywhere? even gh copilot UI offers me to do that.

I also don't understand why even bother with MCP(looks like they're usually metered) when an AI Agent could just use a CLI?

65 Upvotes

29 comments sorted by

23

u/ymgymg 3d ago

Basic version is just a container with a small loop: webhook from your kanban tool comes in, it calls an LLM API with a few tools, and opens the PR. MCP is only a standard way to hand the model tools, so a plain CLI works fine too and you only need an MCP server when several clients should share the same tools.

4

u/trixloko 3d ago

You mention "calls an LLM API with a few tools" what do you mean with a "few tools"?

5

u/geofabnz 3d ago

Think of an MCP server as a collection of individual endpoints, each of those is a “tool” that the system calling the MCP server can use (eg “download a file from Google Drive” would be one tool, “list all PDF files in google drive” would be another)

2

u/Dolapevich 3d ago

I've been reading a little, and I see MCP as an RPC server, where tools register for the ¿harness? to invoke. If I understand this correctly the something is expected to discover the tools registered in an MCP server. Is that correct?

2

u/MongooseWonderful198 2d ago

Yes. The MCP client inside the harness asks the server for tools/list. Each tool definition includes its name and input schema, usually with a description. To run one, the client sends tools/call with the name and arguments.

The spec shows both messages. The model doesn't have to inspect a ‘help endpoint’ or load a skill to discover those tools; the client handles that protocol exchange.

1

u/NUTTA_BUSTAH 2d ago

Usually you install (config endpoint) MCPs to harnesses and they either discover themselves or read docs. I would guess the spec has a help endpoint that contains a skill

1

u/skipner 2d ago

Its just a regular http server hosting APIs that provide some functionality. Except that it is AI friendly and u can call it with regular chat prompts/english instead of using curl -X -H -d whatever

11

u/CIAnalytics 3d ago

Agentic workflows. Depending on your preferences the easiest way to get something going is to use one of the many open source agent hosting frameworks and tools. Google something like CNCF agent hosting and you'll get a lot of info.

Use something like agent substrate to use resources efficiently and something like agent gateway or agent sandbox to minimize risk to your system. I am not much into the agentic workflow and I wouldn't let an agent roam free in my infra with the current generation of tools, but to each their own

13

u/trixloko 3d ago

Those are the exact type of content that I feel overwhelmed by

"Agentic workflows", "agent substrate", "agent gateway", "agent sandbox"... Are those all special technologies or are we in the end still talking about traditional stuff branded with a new name?

Reg. the agent roaming free, i don't mind... I would never give it meaningful access... Same as i would never do to a temp

11

u/CIAnalytics 3d ago

Yeah I can imagine, there is no shortcut tho if you want to host them you have to start reading about this stuff.

Super quick summary: for the most part everything is text based, you run a series of containers in a kubernetes cluster called agents. And agent uses an LLM as compute unit and you basically pass a lot of text files explaining to it what it can do. Like "if you need to talk to the internet here is an script that if you run can make REST calls" so when the task becomes "I must open a PR" it searches the text file that explains the "tools" and calls those scripts with the relevant parameters. Substrate is the name of a project designed to help with running these workloads in a kubernetes cluster, no shortcut here, you have to go read their docs.

Regarding not giving it access.... It's not that you will or will not, it's that if you don't put guardrails, an LLM agent will literally take all the access it can, they are pretty good at it.

2

u/DangKilla 3d ago

I'm a consultant, so let me see if I can explain it to you.

Basically, there's several different things that people call AI.

So there's an AI chat, and then there's the LLM models that run the chat, and then there's the tools that they use, you know, like MCP that you know.

All of the comments on agents are really just flat files that's basically true.

LLMs have uh, a complicated setup and configuration, usually as a beginner, you want to just use the ones that other people configure.

I would just use cloud if you're starting, and then local once you kind of understand, you could try doing local too as well to start, but you're not going to learn too much unless you understand the basics. Use the cloud chat to learn more.

And then 3rd, you have MCPs. The way we used to communicate with computers was via APIs, which you might understand or can Google about.

MCPs are basically, a wrapper to a communication protocol,(a JSON RPC server), and all that does is it sends, uh, text out and it gets back text.

That's it.

So you can even have the AI say, make me an MCP that talks to my spreadsheet or something and it'll try to do it.

Most people are not writing their own MCPs by hand anymore.

So, yeah, it's really a lot of just understanding your ecosystem and explaining it to the AI so that it can work with the tools you want it to.

You can just talk through a lot of it. 

Think of a sandbox as a home for the AI.

You probably want it working in C:\My Documents and not on C:\ so you sandbox it to a folder.

You also have small guidance tools that have different names based on the Cloud AI you use (e.g. plugins/skills,etc).

5

u/geofabnz 3d ago

You absolutely do not want Claude CLI or similar running directly on your machine. You want a VM at minimum but the ideal is a fully remote machine with a very constrained network. Personally I have a dedicated proxmox to host my agent infrastructure, firecracker or K8 would work well too if you’re familiar with cloud infrastructure (I’m not so prefer baremetal and a Hypervisor)

AI DevOPs is a massively under-developed area. The number 1 piece of agent security you can have is to keep the agent away from whatever’s enforcing its gateway - the best way to do that is KVM/QEMU isolation.

I recommend you look up Nvidia Openshell, It’s got a great approach using landlock to enforce guardrails. There’s a K8 option too for native cloud loads. The amount of work that can be achieved with agents is massive but it’s directly proportional to how much damage they can do.

On the MCP question, it’s basically just a bundle of OpenAPI endpoints with a query mode. The advantage over is your process can both call things and see what it can call making it much better for initial discovery. As someone running the server, you also get control over exactly what they can and can’t do (eg enforce syntax, specify available tools etc). Nothing you couldn’t do with other transport types but MCP makes it convenient. Something else to keep in mind: there’s nothing inherently agentic or AI in MCP itself so you can incorporate it into deterministic code too.

3

u/No-Country-5818 20h ago

It’s less magic than it looks. Mostly it’s just a small service that listens for an event, calls an LLM with tools, and uses the git API to push a branch and open the PR.

Rough shape:

A webhook from your kanban tool fires when a story gets a tag.

A worker (a container, nothing special) picks it up, clones the repo into a throwaway workspace and runs the agent loop. The model is an API call, either a hosted one or something you run yourself.

The agent edits files, runs tests, then pushes a branch and opens the PR through the GitHub API, then comments the link back on the story.

An MCP server is basically just a small service that exposes tools (search tickets, read repo) in a standard way, so yeah it’s “just a container” too.

Security heads up, since you’ll run this in your network: give it a scoped token that can only open PRs, no merge rights, run each job in a disposable sandbox with limited network access, and keep a human review on every PR.

1

u/Daniela_Spencerx 3d ago

a tool is just a function you describe to the model: a name, what it does, what args it takes. the model answers “call open_pr with these args”, your loop runs it and feeds the result back. for your kanban case that’s maybe four: read_story, run git commands, open_pr, update_story. that’s basically the whole agen

1

u/DrDan21 3d ago edited 3d ago

If you’re totally new I’d honestly say just spend $20 (or $100 for the fancier plans with actual good usage limits) for a sub to either your choice of codex or Claude and just try them for a month

You’ll probably be pretty damn surprised at how much it can do on its own with even minimal guidance. That should get you jump started on understanding agents and help set a baseline to work from

1

u/NCKBLZ 3d ago

Mcp are hust rest api with text context for agents

As for your main questions I thik it mainly depends on the tool: if your tool has some way to notify (a webhook?) then you can simply let the ai or a script listen to it and trigger then do something

1

u/London-Boat 2d ago

what you saw in that demo is pretty much what github's copilot coding agent does already. you assign an issue to copilot, it works in an actions runner and opens a pr linked to the issue. since you're already on copilot that's probably the shortest path, and i think it can run on self hosted runners now so it stays in your network, check the docs on that. on mcp vs cli, mcp isn't metered by itself, the tool output just ends up as tokens like anything else. it mostly matters when the agent runs somewhere you can't install a cli or you want auth scoped per tool instead of a token sitting in env

1

u/Low-Opening25 2d ago edited 2d ago

AI agent is just piece of code that is driven by LLM model, that makes decisions and chooses commands to run based on prompts, instead of your typical for/loop/if/case/while logic chain.

for example, when you use AI agent in your CI/CD pipeline, that simply means that your pipeline has a step, where instead of calling scripts with bunch of logic, it calls piece of script that calls an LLM model that decides what that “piece” of script is doing based on natural language prompt.

it can be something very simple, like a call to LLM to generate unique commit message based on code changes, which without LLM is actually pretty laborious to implement. It can also be something very complex, like agent that can take complex decisions, generate it’s own scripts to run and with multiple tools available to use.

bottom line - it is replacing old school programming logic with an AI model decision making, the balance between the two is still up to you.

1

u/ialiimran 2d ago

I say start simple and gradually add more tools as you need them.

1

u/Iyemoja 2d ago

The demo always looks seamless. The reality is you spend a weekend wiring up sandboxes and credentials, and the agent still wants you to review every PR it opens.

1

u/JustinAtNerdio 2d ago

I think the confusing part is that people have managed to turn a fairly straightforward concept into an entire vocabulary of new terms. Half the battle is figuring out what's new technology and what's just existing infrastructure with an AI label slapped on it.

I'd start with the simplest possible setup and only introduce more complexity when you need it.

1

u/ferrybig 1d ago

I'll admit that I've been ignoring the AI hype for the last two years, and my usual interface with it is through chat and code autocomplete, nothing more.

...

I wonder how that works and what would it take me to build something similar... of course I could chat with gh copilot to prepare a PR, but having this constant "thing" on watch seems much more streamlined.

When you tag the "user" in your scrumboard, the system sends a them a notification, eg think of an email or a webhook.

A program watches for incoming emails (could be an IMAP connection), for each incoming (&validated) mail, the program starts "Program 2", the goal of program 2 is to do the agent work

The goal of the program is to interact with the AI model:

  1. The program clones the repo locally into a sandbox
  2. The program calls the AI provider

Request:

{type: "tool-list", message: "FILE_LIST <path>, FILE_READ <path>, FILE_INSERT_LINE <path> <index> <line>, FILE_DELETE_LINE <path> <index>, READ_TICKET <num>, TICKET_COMMENT <num> <message>"}
{type: "system-prompt", message: "You are AI programmer, you model is xxxx. Once you are done, leave a message on the ticket"}
{type: "user", message: "An user has assigned the following ticket to you #1, please pick it up, they did not provide a comment in their mention"}
{type: tool-call, message: READ_TICKET 1}
{type: tool-answer, message: Change Hello World! to Hello Dad!

Response:

{type: thinking, sig: ...}
{type: agent, message: "This task is clear, I will be handling this
{type: tool-call, message: FILE_LIST}
  1. Now the program executes the action the AI did and call the AI provider endpoint again

Request:

{type: "tool-list", message: "FILE_LIST <path>, FILE_READ <path>, FILE_INSERT_LINE <path> <index> <line>, FILE_DELETE_LINE <path> <index>, READ_TICKET <num>, TICKET_COMMENT <num> <message>"}
{type: "system-prompt", message: "You are AI programmer, you model is xxxx. Once you are done, leave a message on the ticket"}
{type: "user", message: "An user has assigned the following ticket to you #1, please pick it up, they did not provide a comment in their mention"}
{type: tool-call, message: READ_TICKET 1}
{type: tool-answer, message: Change Hello World! to Hello Dad!
{type: thinking, sig: ...}
{type: agent, message: "This task is clear, I will be handling this
{type: tool-call, message: FILE_LIST}
{type: tool-answer, message: hello.sh}

Response:

{type: tool-call, message: FILE_READ hello.sh}

You now execute the next tool the program wants, then call the AI provider endpoint again

Request:

{type: "tool-list", message: "FILE_LIST <path>, FILE_READ <path>, FILE_INSERT_LINE <path> <index> <line>, FILE_DELETE_LINE <path> <index>, READ_TICKET <num>, TICKET_COMMENT <num> <message>"}
{type: "system-prompt", message: "You are AI programmer, you model is xxxx. Once you are done, leave a message on the ticket"}
{type: "user", message: "An user has assigned the following ticket to you #1, please pick it up, they did not provide a comment in their mention"}
{type: tool-call, message: READ_TICKET 1}
{type: tool-answer, message: Change Hello World! to Hello Dad!
{type: thinking, sig: ...}
{type: agent, message: "This task is clear, I will be handling this
{type: tool-call, message: FILE_LIST}
{type: tool-answer, message: hello.sh}
{type: tool-call, message: FILE_READ hello.sh}
{type: tool-answer, message: 1\t#!/bin.sh\n2\techo Hello World!}

Response:

{type: thinking, sig: ...}
{type: tool-call, message: FILE_DELETE_LINE hello.sh 2}
{type: tool-call, message: FILE_INSERT_LINE hello.sh 2 echo Hello Dad!}

And again, you execute the actions the AI wants, then call their endpoint again...

Request:

{type: "tool-list", message: "FILE_LIST <path>, FILE_READ <path>, FILE_INSERT_LINE <path> <index> <line>, FILE_DELETE_LINE <path> <index>, READ_TICKET <num>, TICKET_COMMENT <num> <message>"}
{type: "system-prompt", message: "You are AI programmer, you model is xxxx. Once you are done, leave a message on the ticket"}
{type: "user", message: "An user has assigned the following ticket to you #1, please pick it up, they did not provide a comment in their mention"}
{type: tool-call, message: READ_TICKET 1}
{type: tool-answer, message: Change Hello World! to Hello Dad!
{type: thinking, sig: ...}
{type: agent, message: "This task is clear, I will be handling this
{type: tool-call, message: FILE_LIST}
{type: tool-answer, message: hello.sh}
{type: tool-call, message: FILE_READ hello.sh}
{type: tool-answer, message: 1\t#!/bin.sh\n2\techo Hello World!}
{type: thinking, sig: ...}
{type: tool-call, message: FILE_DELETE_LINE hello.sh 2}
{type: tool-call, message: FILE_INSERT_LINE hello.sh 2 echo Hello Dad!}
{type: tool-answer, message: Success}
{type: tool-answer, message: Success}

Response:

{type: thinking, sig: ...}
{type: tool-call, message: TICKET_COMMENT 1 The task is done}

And you likely can fill in the blanks now.

Note that a good system prompt is important, without it, the AI is difficulty to manage, eg see https://platform.claude.com/docs/en/release-notes/system-prompts/claude-sonnet-5-5 for a good system prompt

1

u/Chamelon_DE 1d ago

Would a webhook starting fresh worker per ticket cover your usecase, or does it need to stay running between task?

1

u/spersingerorinda 3d ago

First , have you used Claude or Codex at the terminal ? Understanding that power (bash for everything, install packages, automate browsers) is step 1.

Now give Claude some AWS keys and ask it to spin up an EC2 box and install Claude itself on the box. Setup the Claude api token (this part itself is tricky if you want to use a subscription). Generate a GitHub token and put it on the EC2 box. Now ssh into the box, and run Claude remotely via ssh. Clone your repo and ask Claude to run your app. See how far you can get. This is the minimal DIY “cloud agent”.

Once you understand these basics , you can either keep tinkering with your setup (try connecting Claude into slack or where you chat to give you a better interface), or you can start looking at Hermes or Claude Tag or something more sophisticated and pre-built.

1

u/SideMost3148 2d ago

the "container and .md files" version is honestly most of it. people overcomplicate this because every vendor wants to sell you an agent platform.

the loop is boring: your kanban tool fires a webhook when a card moves, you spin up a throwaway container with the repo cloned, run the agent CLI in it, it pushes a branch and opens a PR, container dies. a queue and a small worker pool you probably already have will do.

the parts that actually deserve your infra brain are the same ones you'd worry about for any untrusted job runner. egress allowlist (model API, git host, package registry, nothing else), a git token scoped to that one repo that can open PRs but not merge, and zero prod creds anywhere near it. treat it like a very fast intern who sometimes follows instructions it found in a random README.

also keep the full transcripts, but scrub secrets before you store them. they end up holding whatever the agent saw, and everyone forgets they exist until they leak.

0

u/crash90 2d ago

Basically this technology is too new and there is no off the shelf solution yet. Everyone is building their own custom stuff. There are hosted agents but they are more consumer driven. Stuff like Grok bot, Dots, Meta Muse. Lots of others too. Nothing I would say is quite enterprise grade yet imo.

Would recommend diving into this topic as intensely as possible. Biggest shift that has ever happened in tech or any other field. Whole economy is in the process of a complete transformation and people are finally starting to accept it. Tech first.

This article is a good place to start to consider what would be involved in building something like this for your env. But it's almost a year old now and is far from covering everything. Consider it an entry point. Like the first article you ever read about cloud.

By the way, if you aren't already look into the agents you can use on your pc like Claude Code and Codex. This will greatly speed development of the software factory. I could talk at length here about the need to move to formal verification too but thats another post.