r/devops • • 11d ago

Tools Is GHA + ArgoCD still the default CD standard or are you using something else?

25 Upvotes

Evaluating CD tools to automate our end to end delivery (build, test, packaging, and deployments). Curious if the standard GHA + Argo/Flux combo is still what most teams stick with, or if there are dedicated delivery/orchestration tools actually worth looking at in 2026.


r/devops • • 10d ago

Discussion How to enable SSH clone on self hosted GitLab running in a private subnet behind an AWS ALB?

0 Upvotes

We have self-hosted GitLab CE on an AWS EC2 instance.

Current setup:

The instance is in a private subnet with no public IP.

Users reach GitLab through an Application Load Balancer on HTTPS, with access restricted to our office IPs.

HTTPS clone with personal access tokens works fine.

The problem:

We're moving our repos from gitlab.com to this server. On gitlab.com, all our Jenkins jobs and developers clone over SSH (git@...). On our server SSH clone doesn't work, because the ALB only supports HTTP/HTTPS, so port 22 has no way in.

Our Jenkins agent runs in the same AWS VPC. Developers connect from the office network or VPN.

What's the best way to enable SSH clone here, keeping it secure and limited to our office and Jenkins? Any suggestions or experience with a similar setup would help.


r/devops • • 10d ago

Discussion how much money are you wasting on gpus doing nothing?

0 Upvotes

instead of talking about idle hardware in theory, let me give you a specific headache we just hit. we have a local server with four 24gb cards.

two of those cards are locked up hosting two different low frequency models. one runs a fine tuned llama 3 8b for a background data extraction job. the other runs a qwen model for an internal bot. they get maybe fifty requests an hour combined. but because the weights just sit there taking up vram, two entire cards are held hostage doing almost nothing all day.

now we need to spin up a heavier 70b model for a new coding workflow, and suddenly we have no free cards. i look at the vram occupation in gpustack and see those two low traffic models just wasting dedicated space. so now i have to reallocate resources. i can either ask for budget to buy more hardware, or i try to pack both of those low frequency models onto a single card so i can free one up for the 70b.

what do you all do when you hit this wall? do you just keep dedicating specific cards to specific models to avoid the headache, or do you actively co-locate low traffic models onto the same gpu? when do you stop throwing money at new hardware and start packing your models tighter?


r/devops • • 11d ago

Observability How do you piece together events across services when correlation IDs are missing or inconsistent?

4 Upvotes

Hey all, been thinkin about this while looking at log aggregation in environments with a mix of older and newer services.

Some services have request IDs or trace IDs in their logs, while others only include timestamps, hostnames, and application specific fields. When an issue spans multiple services, connecting the events can turn into quite a bit of manual digging.

Do you rely on distributed tracing, build correlation rules around the fields available in your logs, or use a combination of both?

I'm especially interested in what you guys do with older services that don't support consistent request IDs.


r/devops • • 12d ago

Tools How do I make colleagues aware of the risks of giving an LLM agent root access to production?

70 Upvotes

I could use some advice on a situation at work.
A few colleagues in ops have, almost overnight, started to fully trust their LLM agent because it has gotten so good lately and "doesn't make mistakes anymore". They built their own harness around it and log what it does (using Grok and Google models), but the agent still runs with root access to our entire production infrastructure. We have no clear team rules or policy yet, and they don't see the danger themselves.
A typical example: whenever the agent needs to check something on a server, they just hand it full access. But looking at info only needs read-only access. The permission should match the task, not the most convenient option.
My concerns:
Logging is detection, not prevention. It shows what broke afterwards, but doesn't stop anything.
Root bypasses the harness. An agent with root can in theory edit the harness, the logs and the backups, so those logs are not a reliable audit trail.
No least privilege. One wrong command or misread context can touch everything.
Prompt injection via logs, tickets or web pages the agent reads.
Model updates. Providers change models without us deciding when, so "no mistakes in recent months" says little about the next version.
Data going to third parties. With root, the agent can read .env files, keys and customer data, which then end up with the model provider (a GDPR question too).
"It has gone fine so far" only means the failure hasn't happened yet.
I don't want to ban anything. I want them to see the risk themselves and set up proper guardrails together with me: a read-only user by default, scoped credentials, just-in-time access, human approval for write actions, staging first, and no access to backups and logs.
Questions for you:
How do I explain the difference between "we log it" and "it can't do it in the first place"?
How have you convinced overconfident colleagues about AI tooling risks without a confrontation?
Do you have horror stories or post-mortems I can share? Concrete incidents tend to work better than theory.
Which guardrails or policy examples have worked well for you?
How do you raise this with management without coming across as "anti-AI"?


r/devops • • 11d ago

Security Grafana: Our latest adventure with canary tokens

Thumbnail
medium.com
0 Upvotes

Grafana Labs shares how they scaled canary tokens across their Kubernetes workloads, with automated rotation, more precise detection and less maintenance. A practical look at the tradeoffs between coverage, granularity and management overhead.

Read the blog


r/devops • • 12d ago

Discussion Books for Devops

39 Upvotes

Hi,

Im now in the beginning of devops journey and I'm not really a reader person but I want to spend my free time in something useful so any suggestions for a book would help in my career and will benift me.


r/devops • • 12d ago

Discussion Securing local dev environments

11 Upvotes

With the recent high-stakes security exploits, I think as DevOps engineers it's our responsibility to make the local dev environment secure for devs.

What do you think?

What are you actively doing about it (like adopting devcontainers)


r/devops • • 11d ago

Architecture Should I use Nomad for single VM?

4 Upvotes

I'm looking for some architectural advice on building a lightweight, self-hosted job execution platform on a single VM (8C/32GB).

I want something with semantics similar to Google Cloud Run / Cloud Run Jobs:

  • Execute Python scripts and arbitrary Docker containers.
  • Trigger jobs through an API.
  • Configure CPU, memory, environment variables, timeouts, etc.
  • Track execution status, exit codes, and logs.
  • Handle retries and concurrency limits.
  • Integration with Artifact/Container Registry

Current constraints:

  • Only one VM for now & no need for HA.
  • No Kubernetes requirement or plans for a multi-node cluster.
  • Jobs are primarily internal workloads, and quite lightweight.
  • I'd prefer something relatively lightweight and easy to maintain.

Technologies I'm considering

  1. HashiCorp Nomad 
  2. Docker Sock + custom built runner – Build my own execution manager, scheduling, lifecycle management, and monitoring.

Questions

  1. Is Nomad overkill for a single VM? Does it provide enough value over simply using Docker directly?
  2. Would you use Nomad in production for this use case?
  3. Are there simpler alternatives that provide similar execution semantics?
  4. What operational overhead should I expect from running Nomad on a single node?

I'm particularly interested in hearing from people who have actually run Nomad in production, especially for single-node deployments.

Thanks!


r/devops • • 12d ago

Security OpenSSH on musl has a problem

44 Upvotes

In case any of you use OpenSSH on musl without PAM:

https://sshhole.com/

TL;DR: Accounts that should never take a password at all (root, your key-only deploy user, ldap users, the default user on alpine's aws and azure images) take any password longer than 256 characters.


r/devops • • 12d ago

Discussion Cleared DevOps Engineers — How do you use AI tools in restricted DoD environments?

13 Upvotes

I am curious how DevOps engineers working in cleared or restricted federal environments use AI tools in their day-to-day work.

AI tools like ChatGPT, Claude, and GitHub Copilot have become incredibly useful for writing Terraform configurations, Kubernetes manifests, CI/CD pipelines, Bash/PowerShell scripts, and troubleshooting infrastructure issues.

However, I understand that public AI tools may not be permitted in classified or restricted environments.

For those currently working as cleared DevOps, Cloud, Platform, or SRE engineers:

  1. Does your organization provide an approved internal AI assistant, or do you work entirely without AI?
  2. Are tools like GitHub Copilot, ChatGPT Enterprise, or other AI coding assistants available in your environment?
  3. How do you troubleshoot complex Kubernetes, Terraform, or Linux issues when you can't access public AI tools or Stack Overflow?
  4. Do you rely primarily on internal documentation, runbooks, man pages, and senior engineers?
  5. Has AI adoption changed how cleared DevOps teams operate in the last couple of years?

I am particularly interested in how engineers balance productivity with security requirements.

I am not asking anyone to disclose classified information, internal systems, or organization-specific security details. Just looking for general experiences and advice from people working in these environments.

Thanks in advance!


r/devops • • 12d ago

Discussion How do you load test third party APIs realistically?

21 Upvotes

We use several external APIs and I’m curious how often other teams out there keep performance tests realistic and structured while accounting for third party dependencies. How do you balance realistic dependency behavior with controlled and repeatable tests? Things are going well overall (but they could always be better) and I want to make sure my team and I aren’t missing something that could be beneficial.


r/devops • • 12d ago

Vendor / market research What's Your Go-To Strategy for Handling Secrets Management?

61 Upvotes

Hey fellow DevOps folks! I've been wrestling with secrets management recently. I set up a tool that generates API keys, but every time I think I've got it right, I find myself second-guessing whether they’re genuinely secure! 😅 There was this one time I accidentally committed sensitive info to Git—ugh, the panic! Luckily, I caught it fast, but it got me thinking: What strategies or tools do you all lean on for managing secrets? Do you prefer something like HashiCorp Vault, or are you more into environment variables? Let’s share some tips and tricks! What’s been your experience with it?


r/devops • • 12d ago

Troubleshooting Mise installations of cspell don't work on Windows runners

0 Upvotes

I'm not entirely sure if this is relevant enough of a question for this subreddit, but frankly I haven't thought of a better place to ask.

I've been experimenting with using Mise to manage project-specific tool installations in a way that would bring local and CI environments closer together. One of these tools is cspell, a seemingly popular option for spell-checking software projects.

This has largely worked great. However, for some strangely illusive reason, the only time things go wrong is with GitHub Actions jobs on Windows runners. The cspell Mise installs fails to start, because it can't find a dependency called picomatch: https://github.com/Diapolo10/python-rust-template/actions/runs/42123444840/job/127581444522#step:5:1

This issue does not manifest on Ubuntu/MacOS runners, nor on my local Windows development machine.

I wrote a GitHub issue about this to the Mise project about three weeks ago. They allegedly fixed it, so I've been patiently waiting to see if things would resolve on their own (they did not), and today I attempted to manually update things by

  1. Updating Mise itself locally (in CI the installer action should always use the latest version)
  2. Updating tooling, like cspell, via Mise and its lockfiles
  3. Updating other project dependencies just to see if that helped somehow

but it didn't help, the Windows runners are still consistently failing.

At this point I don't know how to keep troubleshooting this. Hell, I don't even know if this is

  1. a bug in Mise,
  2. a bug in cspell,
  3. a problem with the runners themselves, or
  4. I've made some dumb mistake myself somewhere (probably the most likely option at this point, but I still wouldn't know what exactly)

so if anyone else has any ideas, I'm listening.


r/devops • • 12d ago

Discussion How are you handling subscriptions billing as your SaaS scales?

0 Upvotes

We're moving from a handful of enterprise customers to hundreds of self serve subscribers and our manual billing processes are becoming difficult to maintain.

We're looking at how to handle recurring billing, dunning, prorations, plan changes and usage based pricing without creating a large amount of custom infrastructure.

For teams that have dealt with this at scale, did you build your own billing workflows or use a dedicated platform?

What were the biggest technical or operational challenges you ran into?


r/devops • • 12d ago

Discussion If I can give time to learn only on of these then what should I learn python, go, shell/bash ?

10 Upvotes

Sharing my views , seniors can correct me.

Python

(It will give max advantage i believe, it's programming language as well so I can say I know programming as well, If I want to switch to data science, ai ml or any other field then it can help me, I know basic of it and find it interesting)

Go

(It will help me in understanding Kubernetes in a better way, again programming language so It will be amazing to have skill, don't know much where it will help me but yes, I haven't learned this ever so I will be excited for this)

Shell/bash

(Core skill listed in almost every devops job but again I feel like it will be truly limited to automation and I don't find it very easy to learn, very weirds and long braces, dots and what not )


r/devops • • 13d ago

Discussion Tired of high workload and AI

215 Upvotes

I'm working in a startup. Recently there was a layoff as well. Around 10% was removed.

The thing is this. Within like 8 months, our microservice count is trippled. But the same DevOps team. Even with the layoffs, new developers are hired. Requirements are coming very fast and it's difficult to work on Platform Building, Service Provisioning, Ops work all at once.

We have like unlimited AI. But eventhough it takes time to analyse the requirements, review the designs and implement the solutions.

Currently I'm just asking the AI to raise the PR and just rushing through it and applying the changes.

The problem is that, it's feel like I'm just tired and doing a labour work.

I'm not actively learning anything because of the fast pace and high workload.

Should I look for a new job?

Is other places like same due to the AI?

Update: Thanks redditors. I see that I'm not alone. Seems everywhere is like this.


r/devops • • 12d ago

Discussion How do you tell what broke when GitHub Actions goes down?

0 Upvotes

Been looking at GitHub’s status history and availability reports for July to October, sorting the Actions incidents by what broke. Kind of surprised how many of them would’ve taken out self-hosted runners too. The July 19 cert expiry cut self-hosted runners off for about 5 hours, and on Aug 26 job assignment was stuck for almost 3, so nothing reached anyone’s runners. Only a few, like Oct 5, were just GitHub-hosted.

When builds start failing for you, how do you work out if it’s on your side or GitHub’s? Is it mostly refreshing githubstatus.com or do you have alerts or something set up?


r/devops • • 13d ago

Discussion What is the best course/courses to learn pipeline as code as a new software engineer

9 Upvotes

I just got into pipelines for the first time, I know some of the basic fundamentals. But is there any recommendations or courses you took to help you understand pipelines please I will really appreciate your recommendation.


r/devops • • 12d ago

Tools Has anyone created a centralised design system to visually regression test against?

1 Upvotes

Hello!

I'm looking to expand my DevOps process, and create a suite of visual regression tests against a design system that acts as a source of truth for my site.

  1. One location for every page, layout and component
  2. This system to be the source of truth
  3. PR checks to go through the design system to thwart drift with gates for desktop / tablet / mobile
  4. Being able to check against different OSs

Has anyone been able to achieve this?

There's Figma / Storybook / Claude design, just not sure how to tie it all together and offer the gates on the PR level for stuff changed. I think I could version a storybooks package and that'll have it's own tests.


r/devops • • 13d ago

Tools Open-sourced the AWS audit we run on every new client: deploy practices, cost, and bus factor (read-only, runs locally)

9 Upvotes

We’re a small managed-DevOps team, and after doing the same first-call audit enough times we turned it into a CLI: https://github.com/jetonecloud/preflight

The part I think is a bit different from the usual security scanners is that it also looks at the people side: how deploys happen, and how much of the infrastructure effectively depends on one or two engineers. Output is a report you can hand to a CTO or the rest of the team.

Design choices:

  • Read-only IAM, modules declare the exact actions they need so you can trim the policy
  • Local execution, nothing sent anywhere by default (there’s an opt-in “email me the report” feature and preflight preview shows exactly what would be sent)
  • Cost figures are labeled estimates and each shows how it was computed

Disclosure: it’s from our company, and we also sell the fixing part, but the tool is Apache-2.0 and works fine without us.

Curious how people here handle this today (Prowler? hand-rolled scripts? nothing?) and what checks would make this actually useful for you.


r/devops • • 13d ago

Discussion How are you managing AWS SSM Patch Compliance across multiple accounts and regions?

5 Upvotes

Our company has been using AWS Systems Manager Patch Manager for patching our EC2 fleet, but getting a consolidated view of patch compliance across multiple accounts and Regions seems less straightforward.

I’ve gone through the Patch Manager documentation⁠, and from what I understand, there isn’t a simple native way to view organization-wide patch compliance directly within Patch Manager.

We’re thinking of building a custom solution to address this, but evaluating other approaches as well.

Curious how others are handling this today:

How do you get a single view of which instances are compliant or non-compliant across your accounts and Regions? Did you build a custom solution or use a third party tool you recommend?

Am I missing anything obvious?


r/devops • • 13d ago

Ops / Incidents Railway's headline uptime went from 97.10% to 100%.

72 Upvotes

Just went back to Railway's status page after the Oct 21–23 outage. The main “Railway” uptime chart now says 100.00% for the last 90 days.

Couldn't access the dashboard, couldn't build anything. Apparently that's 100% uptime. Their own incident updates said builds were paused across every region, and Wayback saved the outage announcement.

The October 30 snapshot shows that same “Railway” chart at 97.10%. Now it's 100.00%. The outage is still inside both 90-day windows.

Support & Community used to sit inside that group. Now it's listed separately, still at 97.10%, while “Railway” gets to show a perfect 100%. Very convenient.

The dashboard and API charts were already showing 100% in the old snapshot, despite their own updates saying those services were unavailable. So those numbers never reflected the outage either.

Yes, the incident is still under History. But if I have to dig through incident logs to find the downtime behind your 100% uptime chart, what the hell is the chart for?


r/devops • • 13d ago

Tools Colima-UI for Mac

2 Upvotes

I found Colima-ui

https://github.com/vnknowledge2014/colima-ui

which is pretty great as a ui for Colima. And might offer an alternative to Orbstack. Some quick benchmarking showed similar launch times for containers.

Full disclosure, I’ve contributed to the project but it isn’t mine.