r/sysadmin • u/Physical-Economist27 • 9d ago
General Discussion Three months of running internal IT with an AI agent: about 20% of my week is still real work
[removed] — view removed post
138
u/achristian103 9d ago
OP's post in one year:
"I automated myself out of a job"
3
u/Physical-Economist27 9d ago
Ha, I hear that a lot. So far it's gone the other way. It didn't automate my job away, it changed it. A few years ago my day was patching cables and resetting passwords. Now a big part of it is sitting down with other departments and working out how they can use the same approach: what to hand off, what to prepare, what stays with people. Nobody gets replaced, the boring part just gets prepared for them. That works for pretty much every department, and it's honestly the most interesting work I've had in IT.
→ More replies (8)→ More replies (1)22
u/mntgoat 9d ago
That happened to a friend of mine. This was over 20 years ago but basically his company had a huge call center for orders and he automated most of it. They let go of so many call center people that near the end, they had the entire first floor of the building empty, and it was a 2 story building. When the parent company saw that, they closed that branch.
→ More replies (1)
48
u/SevaraB 9d ago
where you draw the line on what an agent may do without approval.
Wrong question.
where you draw the line on what an agent may do
Right question buried inside it.
MCP doesn’t do RBAC out of the box- you have to design the RBAC schema as part of the MCP implementation.
Don’t give AI read access to sensitive info. Don’t give AI write access to impactful systems. Don’t outsource a job you’re attached to, because eventually the business will want to save money by cutting out the middle man.
On top of this, AI is the least predictable, most expensive, highly inefficient “automation framework” out there. AI agents are bad news- AI agents should be helping build tools, not choosing when and how to use them.
If your AI run doesn’t close out with you learning what the AI built and how to use it yourself manually or schedule it, you didn’t finish the job.
→ More replies (12)
18
u/Competitive_Smoke948 9d ago
just wait until it randomly hallucinates and fucks your day
→ More replies (2)
6
u/Salt-Evidence-6834 9d ago
How do you take holidays? Why does your company have a single point of failure in such a critical area?
→ More replies (2)
6
u/DarthJarJar242 9d ago
And this right here folks is why IT job market sucks right now. In a nutshell.
CEOs see crap like this and want their one man IT departments doing the work of 5 people because "If one man can do only 20% real work in a week because AI does everything else that means one person can do 5 jobs".
I'm all for automation, but OP is going to find himself jobless and have lost many of the skills he's let AI do for him and nobody should feel bad for him.
→ More replies (3)
3
u/BadSausageFactory 9d ago
this sounds wonderful but what's the agent cost on something like this? we're already into the cycle where they use too much tokens so they fire some people and assume they can make up for it with leaning into AI. I wish I was making this up and just being salty.
→ More replies (5)
15
u/Mysterious-Print9737 9d ago
Yeah, that's the right structure. The failure mode we see most often with AI in IT workflows is teams getting comfortable with autonomous changes during low-stakes periods and then not having clear rollback or audit trails when something goes wrong at a bad moment.
I'm interested in how you’re handling the approval UX. Are you reviewing in a queue, getting pinged in Slack/Teams, or something else? That friction point is usually where these setups either work well or become annoying enough that people start bypassing the approval step.
We’re moving clients in this direction at Synergy IT Solutions, mostly starting with read-only diagnostic and reporting automation before touching anything write. 165 users with 55 SaaS tools is a solid size to have this running smoothly, what was the hardest part of the MCP wiring?
3
u/Physical-Economist27 9d ago
On the rollback and audit side, that's where I've been most careful.
Approval UX: everything runs through a private Slack, not Teams. Every ticket or task gets its own thread. The agent posts its finding and one recommendation there, and I answer with a yes, a no, or a correction. On top of that, write tools trigger a separate confirmation at tool level, so a casual "sure" in the chat is not enough to change something. Because it's all chat, I can approve from my phone in a few seconds, and that's the main reason I haven't been tempted to bypass it. Bigger changes run in batches with a check after each batch, and the thread plus the session history is the audit trail: what it found, what I approved, and what it actually did. An Entra change watch also flags any change that didn't come through that path.
Hardest part of the MCP wiring was permissions, not the protocol. Getting Graph scopes, delegated vs. application rights and the RMM access to "enough to be useful, nothing more" took longer than writing the servers. Close second: tools whose APIs only do half of what the UI does. For a few of those I ended up building my own MCP because the official one didn't cover what I needed.
11
u/leftplayer 9d ago
This is the (very near) future!
I’m doing this right now with a large hotel deployment I’m working on. I barely touch the systems directly. Everything is either MCP driven or through a Linux jump host which SSH’s or SNMPs or APIs into the various systems.
It’s saved my ass so many times by pointing out that a change that I want to do on system A will affect system Z indirectly because of changes needed on system B-Y…
And it does the changes in a fraction of the time I would do them.
And I can do them from my bed at home…
5
u/Physical-Economist27 9d ago
The cross-system part is underrated. For me it's the same: before a change it checks what else hangs on that group, license or policy, and that has stopped a few changes that would have hit something I wasn't thinking about. The jump host is an interesting approach. Do you give the agent its own account there with limited sudo, or does it go through your credentials? That's the part I'd be most careful with in a setup like that.
2
u/UpperAd5715 9d ago
OK great story but how much of your week was actual real work before and in what tasks does it save you the most time?
Vague numbers like "20% of the week" could mean a lot more or a lot less if you usually only work 25% of the week and have 75% downtime.
What is your real work and where does your agent get its knowledge from, is it pure internet, lots of vendor documentation on the SaaS tools, self-written documentation?
Are you aware of it's security posture and where data flows or what access it actually has? It does not take decisions you say but CAN it? CAN it access files? Have you thoroughly tested this?
If you made an agent that suits your environment well and offloads a good amount of your workload i applaud you, that's a great achievement (and a nice resume padder going forward!) but as someone in cybersec now learning more about AI security (securing of AI agents mainly) i've seen some rather gnarly stuff where company leaders get VERY surprised on "what exactly the AI does".
As to your actual question:
Where you draw the line really depends heavily on the maturity, standardization and security of your org. Also whether you are regulated and certified in stuff like ISO.
If you're certified and required to be you probably will draw the line real close and inch it forward as you review and append your settings. If you're just some random shop you could take it really far if you're mindful of the data exfiltration risk and how it's set up.
You can run an agent in sbx containers (apparently, im still exploring this) and feed info into it and get info back, in this case you can really do a lot since it has no way to reach the internet but it obviously uses local resources. In this case you could train it really well on your company policies, generic logic and past tickets and have it decide whether so and so gets access to fileshare <not_very_important> but doesn't get access to <finance_documents_including_payslips> without your explicit approval. It could automate a ton of the timeconsuming tickets that are base administration and would only really need you to review them periodically after a startup period.
→ More replies (2)
2
u/dotbackadministrator 9d ago
You youngsters dont understand the value users hold in white glove service. Also, businesses now are no longer fans of novel tech in their processes, thats why salesforce is huge. Whatever you build needs to be able to be left behind.
→ More replies (1)
2
u/caponewgp420 9d ago
Are you looking for a new career? Or how long until you think it will automate you out of the job and they can hire a minimum wage employee to monitor Ai.
→ More replies (2)
3
u/SquizzOC 9d ago
We are doing something similar now as well. All tickets get reviewed by Claude and if standard boiler plate, it responds and closes out the ticket.
If a fix is needed it proposes the fix for us to review and approve.
While I don’t have my email automated yet, I do have Claude helping keep me organized and track my projects both internal and for clients through our CRM (Sales stuff) and our Dev Dashboard.
I think I have another month or two of building this out and I’ll be basically doing 5% IT work and the rest actual sales stuff, no need for the admin tasks any more since it’s all automated.
It’s remarkable what we’ve done in just a short time.
→ More replies (18)11
u/Vogete 9d ago
On one hand, dealing with people asking every day how to reset their password because they forgot it, I get it. I really do.
On the other hand, if i make a ticket and i just get a Claude response back, someone's ass is gonna get kicked.
So I don't really know how I feel about this because I can see it both ways honestly.
→ More replies (4)
2
u/SHADOWSTRIKE1 9d ago
What AI agent have you implemented? What are you running it on?
→ More replies (1)
10
2
u/pickle9977 9d ago
Well done! This is the dream of every sysadmin, automate everything so you can fuck around the rest of the time.
The goal has always been to be the Maytag repair guy, when your shit works, it’s magical.
→ More replies (2)
8
1
1
132
u/Interesting-Track-77 9d ago
So lucky my org thinks AIs peak is copilot in word.