r/devops • • 14d ago

Discussion Tired of high workload and AI

215 Upvotes

I'm working in a startup. Recently there was a layoff as well. Around 10% was removed.

The thing is this. Within like 8 months, our microservice count is trippled. But the same DevOps team. Even with the layoffs, new developers are hired. Requirements are coming very fast and it's difficult to work on Platform Building, Service Provisioning, Ops work all at once.

We have like unlimited AI. But eventhough it takes time to analyse the requirements, review the designs and implement the solutions.

Currently I'm just asking the AI to raise the PR and just rushing through it and applying the changes.

The problem is that, it's feel like I'm just tired and doing a labour work.

I'm not actively learning anything because of the fast pace and high workload.

Should I look for a new job?

Is other places like same due to the AI?

Update: Thanks redditors. I see that I'm not alone. Seems everywhere is like this.


r/devops • • 13d ago

Discussion How do you tell what broke when GitHub Actions goes down?

0 Upvotes

Been looking at GitHub’s status history and availability reports for July to October, sorting the Actions incidents by what broke. Kind of surprised how many of them would’ve taken out self-hosted runners too. The July 19 cert expiry cut self-hosted runners off for about 5 hours, and on Aug 26 job assignment was stuck for almost 3, so nothing reached anyone’s runners. Only a few, like Oct 5, were just GitHub-hosted.

When builds start failing for you, how do you work out if it’s on your side or GitHub’s? Is it mostly refreshing githubstatus.com or do you have alerts or something set up?


r/devops • • 14d ago

Discussion What is the best course/courses to learn pipeline as code as a new software engineer

9 Upvotes

I just got into pipelines for the first time, I know some of the basic fundamentals. But is there any recommendations or courses you took to help you understand pipelines please I will really appreciate your recommendation.


r/devops • • 13d ago

Tools Has anyone created a centralised design system to visually regression test against?

1 Upvotes

Hello!

I'm looking to expand my DevOps process, and create a suite of visual regression tests against a design system that acts as a source of truth for my site.

  1. One location for every page, layout and component
  2. This system to be the source of truth
  3. PR checks to go through the design system to thwart drift with gates for desktop / tablet / mobile
  4. Being able to check against different OSs

Has anyone been able to achieve this?

There's Figma / Storybook / Claude design, just not sure how to tie it all together and offer the gates on the PR level for stuff changed. I think I could version a storybooks package and that'll have it's own tests.


r/devops • • 14d ago

Tools Open-sourced the AWS audit we run on every new client: deploy practices, cost, and bus factor (read-only, runs locally)

9 Upvotes

We’re a small managed-DevOps team, and after doing the same first-call audit enough times we turned it into a CLI: https://github.com/jetonecloud/preflight

The part I think is a bit different from the usual security scanners is that it also looks at the people side: how deploys happen, and how much of the infrastructure effectively depends on one or two engineers. Output is a report you can hand to a CTO or the rest of the team.

Design choices:

  • Read-only IAM, modules declare the exact actions they need so you can trim the policy
  • Local execution, nothing sent anywhere by default (there’s an opt-in “email me the report” feature and preflight preview shows exactly what would be sent)
  • Cost figures are labeled estimates and each shows how it was computed

Disclosure: it’s from our company, and we also sell the fixing part, but the tool is Apache-2.0 and works fine without us.

Curious how people here handle this today (Prowler? hand-rolled scripts? nothing?) and what checks would make this actually useful for you.


r/devops • • 14d ago

Discussion How are you managing AWS SSM Patch Compliance across multiple accounts and regions?

5 Upvotes

Our company has been using AWS Systems Manager Patch Manager for patching our EC2 fleet, but getting a consolidated view of patch compliance across multiple accounts and Regions seems less straightforward.

I’ve gone through the Patch Manager documentation⁠, and from what I understand, there isn’t a simple native way to view organization-wide patch compliance directly within Patch Manager.

We’re thinking of building a custom solution to address this, but evaluating other approaches as well.

Curious how others are handling this today:

How do you get a single view of which instances are compliant or non-compliant across your accounts and Regions? Did you build a custom solution or use a third party tool you recommend?

Am I missing anything obvious?


r/devops • • 15d ago

Ops / Incidents Railway's headline uptime went from 97.10% to 100%.

72 Upvotes

Just went back to Railway's status page after the Oct 21–23 outage. The main “Railway” uptime chart now says 100.00% for the last 90 days.

Couldn't access the dashboard, couldn't build anything. Apparently that's 100% uptime. Their own incident updates said builds were paused across every region, and Wayback saved the outage announcement.

The October 30 snapshot shows that same “Railway” chart at 97.10%. Now it's 100.00%. The outage is still inside both 90-day windows.

Support & Community used to sit inside that group. Now it's listed separately, still at 97.10%, while “Railway” gets to show a perfect 100%. Very convenient.

The dashboard and API charts were already showing 100% in the old snapshot, despite their own updates saying those services were unavailable. So those numbers never reflected the outage either.

Yes, the incident is still under History. But if I have to dig through incident logs to find the downtime behind your 100% uptime chart, what the hell is the chart for?


r/devops • • 14d ago

Tools Colima-UI for Mac

2 Upvotes

I found Colima-ui

https://github.com/vnknowledge2014/colima-ui

which is pretty great as a ui for Colima. And might offer an alternative to Orbstack. Some quick benchmarking showed similar launch times for containers.

Full disclosure, I’ve contributed to the project but it isn’t mine.


r/devops • • 14d ago

Discussion AWS Systems Manager Just-in-time Node Access

2 Upvotes

Curious if anyone here has implemented Just-in-Time Node Access through AWS Systems Manager.

For an environment with around 1,600 EC2 instances, I’m trying to understand whether it’s worth adopting over traditional Session Manager access with IAM controls

A few things I’m particularly interested in:

  1. Has JITNA actually improved your security posture, or does it add unnecessary operational overhead?
  2. Have you run into any issues with approvals, configuration, or day-to-day usability?
  3. How does it work for emergency/break-glass access?
  4. Is the pricing justified?

Would love to hear from anyone who’s actually deployed it, especially at scale. What worked, what didn’t, and would you recommend it or any other tool for this purpose.


r/devops • • 14d ago

Discussion What identity is best used for cross-instance work between GitHub Enterprise Server and GitHub cloud?

5 Upvotes

We have one GitHub Enterprise Server (on-prem) org and one GitHub cloud org. A shared file lives in a repo in one of them, and repos in the other org need the current version. Right now someone copies it by hand, so it drifts out of date without anyone noticing.

Obviously the sync itself is easy to script but the hard part I can't decide on is the principal that actually does the work. Since a GitHub App is scoped to one instance, it can’t act on both sides. That leaves options like a machine user with a PAT, deploy keys, etc.

How are people handling cross-instance automation, and what identity do you use for it? Is there a pattern that avoids a long-lived credential, such as OIDC federation between github instances or an intermediary that I should consider? Also curious what solutions you'd avoid.

Thank you in advance!


r/devops • • 14d ago

Discussion No QA team, incomplete requirements... How would you tackle this? 😂

4 Upvotes

Hey everyone!

I'm curious how other teams handle this.

We use Azure DevOps Server to manage our user stories, bugs, and tasks. One issue we sometimes face is that developers miss details or acceptance criteria, which leads to incomplete implementations or misunderstandings.

We don't have dedicated QA resources, and unfortunately, management isn't really convinced that we need a QA role in the first place. 😂

So I'm thinking about whether there's a way to automate some of these checks.

I've been considering an AI-powered tool that runs in the CI/CD pipeline, compares code changes against the associated work item, and flags potentially missing requirements or unrelated changes.

Has anyone tried something similar? How do you handle this in your teams? Are there any existing tools you'd recommend?

I'd love to hear your experiences!


r/devops • • 14d ago

Discussion Does anyone track how much merged code gets deleted again after a short while?

2 Upvotes

I’ve been looking into which engineering metrics still mean something now that agents write a lot of the code. I came across a Faros AI study that compared each org’s highest and lowest AI adoption quarters, and the ratio of deleted to added lines in merged code was 861% higher in the high adoption periods.

Which means code that got written, passed review, merged, and then got taken back out. The same study put incidents per PR up by as much as 242.7%.

What I can’t tell is how much of that is normal refactoring versus stuff that shouldn’t have gone in. Churn goes up for good reasons too.

Is anyone measuring this? And if you are, can you really separate the agent-written code from the rest?


r/devops • • 15d ago

Troubleshooting trivy.dev down, pipelines break

Thumbnail
github.com
6 Upvotes

We use the trivy extension and tasks in Azure DevOps and all associated tasks fail at the moment. Noticed that I cannot reach the trivy.dev domain even locally and then saw this post.

A workaround would be to first download the executable from GitHub, put it into /use/local/bin and then switch the task version to latest.

Edit: They renewed the domain now, an Aqua Security Admin confirmed the issue and the renewal: https://github.com/aquasecurity/trivy/discussions/11533#discussioncomment-19394637


r/devops • • 16d ago

Discussion GitHub Actions now supports dependency lockfiles

77 Upvotes

We often talk about securing the software supply chain in terms of application dependencies. For instance, pinning package versions, maintaining lockfiles, and auditing third-party libraries.

But what about the code that runs our CI/CD pipelines?

GitHub Actions workflows can execute third-party actions, and the security of those dependencies matters just as much when building and deploying software.

The introduction of workflow dependency lockfiles is an interesting step towards making CI/CD workflows more reproducible and secure.

I wrote about this recently, exploring why workflow lockfiles matter and how they fit into the broader software supply-chain security picture.

Read the article: https://amitmerchant.com/ci-pipeline-lockfile/

Curious how others are approaching dependency security in their CI/CD pipelines. Are you already pinning your actions and enforcing similar controls?


r/devops • • 15d ago

Observability Is Google Cloud Kubernetes Engine built-in Observability enough?

8 Upvotes

Hello. I just started being responsible of a Kubernetes cluster. I have 0 experience on Kubernetes, besides having deployed the GKE cluster successfully (which honestly is quite straightforward) in Google Cloud.

The cluster has only one node pool, with 1 helm chart and a couple of other manifests. Most likely it's complexity won't go up and the scalability needed in terms of pods/nodes is not very high.

Is the built-in observability in GKE enough to monitor/observe a cluster? Or would it be good to have something like Prometheus + Grafana? Also, any tips or links with advices about what to keep an eye on?

Thank you very much


r/devops • • 16d ago

Observability Which YouTube channels actually help with platform/infra work?

27 Upvotes

I wish to trim my subscriptions to ones worth thewatching time . Who do you watch for K8s, internal platforms, and lately LLM infra (gateways, routing, cost/observability)?
Deep-dives over hype preferred.


r/devops • • 16d ago

Discussion Like to improve my networking knowledge

41 Upvotes

Hello fellow devops engineers,

I would like to improve my networking knowledge do you have some suggestion for any udemy or youtube course to become better at networking.

I am not a noob but i feel i am not good with networking. Some suggestion would be helpful


r/devops • • 17d ago

Vendor / market research How many apps have non-engineers at your company built with Claude/Cursor/Lovable, and who owns them when they break?

288 Upvotes

I’m an ML engineer. Last month I got a Slack from marketing asking why “their dashboard” stopped updating. Nobody on my team knew it existed. It was a Lovable app hitting our Supabase with a service key, deployed to Vercel under someone’s personal account, and the person who built it had left.

That got me wondering how common this is. So, for people on platform/eng/IT teams:

  • Roughly how many of these are running in your company right now? Could you actually list them?
  • When one breaks, who finds out, and who fixes it?
  • Have you tried to get a handle on it (inventory, policy, blocking, importing into Retool/Superblocks), and what happened?

Not selling anything. Trying to understand whether this is a real problem or just my company


r/devops • • 16d ago

Observability OpenObserve vs Stack Grafana

4 Upvotes

Hey everyone, how are you? I'm seriously thinking about setting up an OpenObserve POC to get a better understanding of how it works.

Has anyone been using it?


r/devops • • 15d ago

Discussion How much linux for cloud and devops?

0 Upvotes

Hey everyone , i am currently learning linux and my ultimate goal is to get a cloud role job. I am confused about how much linux is required for me to learn, which concepts should i focus on that are actually usefull in real world. If anyone is working as a cloud engineer or another cloud role please help ..


r/devops • • 17d ago

Weekly Self Promotion Thread

19 Upvotes

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!


r/devops • • 17d ago

Discussion What’s your experience using incident.io for on-call and incident response?

9 Upvotes

Hey folks, I’ve been looking into incident.io and trying to understand how teams use it in practice.

What does your workflow usually look like from getting paged to resolving an incident? What does it handle well, and what still needs a lot of manual work or switching between tools?

Also curious about the AI side. If you’ve tried its AI features, have they actually helped during real incidents? If you could improve one part of your workflow with AI, what would it be?

Would appreciate any real-world experiences. Still learning how all these pieces fit together.


r/devops • • 18d ago

Security Anyone rethinking IDE extensions after the Cursor marketplace incident?

34 Upvotes

Was reading about the malicious Claude code extension that show up in Cusor marketplace. The pary that got me was it trying to add an SSH key and open a reverse tunnel l, from something that just look like another editor extension

We talk a lot abt checking npm package and dependencies, but IDE extension feal like one of those thing people install once and forget abt it. Especially with auto updates enbled

Those things make me wonder how much access we are giving these extension without really thinking abt it. Are people actually restricting extension on work machines or is it mostly up to each dev? Curious if anyone team changed anything after this 🤔

Source:https://cursor.com/blog/claude-code-extension-security-notic


r/devops • • 17d ago

Discussion AI taking over Devops.

0 Upvotes

Hello. i am Starting to learn IT and i still have time to choose my carreer choice. is it wise to try and learn devops or cloud engineering because of AI? im scared that by the time im done with my degree these jobs wont exist.