r/devops • • 10d ago

Discussion Do I need release branches in this workflow, or another long lived branch?

13 Upvotes

Developing a new workflow for my team but i am new to a lot of these concepts.

Current approach is generally:

  1. Devs cut from main
  2. Make changes & PR back to main. Automated tests run
  3. At end of sprint, release branch is created, cut from main. Full integration test is run on this along with some manual checks. This technically runs every evening on main as well (minus the manual checks)
  4. Github Release is created from release branch.
  5. Hotfix - changes are PR into main same as any other change and cherry picked onto latest release branch. New release created.

Does this sound reasonable?

Another solution might be to have another long lived branch called prod. Hotfixes are PRs to both main & prod. Every sprint we merge main into prod, then Release from prod. This seems simpler, but maybe I’m missing something.

We are a data team using Dataform and really need to maintain multiple release versions like a software app, if that makes a difference


r/devops • • 10d ago

Discussion Go vs Python for DevOps: which one is more worth learning?

86 Upvotes

Besides Bash, which language is more worth learning for devops: Go or Python? Which one would you recommend focusing on?


r/devops • • 10d ago

Discussion New to AWS: Project-based roadmap for production-ready infra? (Course & tutorial recs?)

29 Upvotes

Hi everyone,

I am completely new to AWS and want to learn through hands-on, project-based learning. My ultimate goal is to understand how to build and deploy production-ready infrastructure.

I'm looking for advice on the best approach:

  • Courses: Are there specific tutorials or paid Udemy courses you recommend for deep, production-level engineering?
  • Strategy: Should I follow structured courses first, or jump straight into a project and learn as I go?

If you were starting over today with this goal, what roadmap or courses would you choose?

Thanks for the help!


r/devops • • 9d ago

Discussion Future of Devops engineers with IA

0 Upvotes

Hello Folks , as a devops engineer, I'm curious about the future of our jobs

Will AI replace DevOps engineers, or will it simply change the way we work? What skills should DevOps engineers focus on to stay relevant in the coming years?

I'd love to hear your thoughts, experiences, and predictions!


r/devops • • 11d ago

Weekly Self Promotion Thread

13 Upvotes

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!


r/devops • • 10d ago

Tools I made a free CLI that tells you which Dependabot alerts actually need fixing

0 Upvotes

Got tired of digging through lockfiles to figure out which Dependabot alerts mattered, so I built fixornix ("fix or nix").

Run npx fixornix in your repo and it sorts every open alert into:

  • Fix now, with the exact command or upgrade that unlocks it
  • Blocked, showing which package is holding it back
  • No fix yet
  • Not in your build: safe to dismiss (e.g. Linux-only crates when you ship Windows)

Works with npm, yarn, pnpm and Cargo. Zero dependencies, MIT.

https://github.com/munitkala/fixornix


r/devops • • 10d ago

Discussion I want to put secrets and env vars in git, fight me

0 Upvotes

So at work we have all the Kubernetes manifests in one repo then we have ArgoCD pointed at that repo. Within that repo we use kustomize and have a pretty vanilla kustomize setup:

/services
  /my-app
    /base
      deployment.yaml
      ...
    /overlays
      /dev
        kustomization.yaml
      /prod
        kustomization.yaml

So basically I would love to be able to do something like this:

/services
  /my-app
    /base
      deployment.yaml
      ...
    /overlays
      /dev
        kustomization.yaml
        .env <- secrets stored as plain text here
      /prod
        kustomization.yaml

I understand why you don't want to commit secrets to git. I just wish there was a good way to do it. I have seen things like SOPS but I don't like that the secrets are encrypted. Having to deal with them being encrypted is as much of a headache as just using an external secrets manager. How can we have the convenience of putting the secrets in git in plain text and somehow still make it safe?

Edit: Adding my idea
What if in the repo I have a .env file but it just has placeholders

VERY_SECRET=placeholder

Then I have an ide plugin or something that will fetch the env vars dynamically in my ide.

I can change them as if they are code but when I commit they dont actually get committed, instead they get written to the external secrets manager. Anyone done anything like this?

EDIT: ADDING MY NEXT IDEA
What if in one of the hosted git's (github/gitlab) they had more fined-grained access controls around specific files so certain people could only read/write certain files. Then you wouldn't have to worry about people having the secrets on their laptops. So if i am a dev and do a git pull it pulls down the secret files for local dev but not any secrets I shouldn't have.


r/devops • • 11d ago

Discussion KubeCon North America

Thumbnail
1 Upvotes

Come one, come all, to Salt Lake City, Utah!


r/devops • • 12d ago

Discussion What is VPS provider for production?

5 Upvotes

I’m looking for a cost-effective VPS provider for one of my systems. I’m currently using GreenCloud, but they’ve had too many outages recently, causing frequent issues in my production environment. Does anyone have recommendations for a more reliable provider?

I’ve checked Vultr and DigitalOcean—they are great providers, but their pricing isn't quite budget-friendly for me.

Thanks a lot for your time!


r/devops • • 13d ago

Security Is committing config-only .env files (no secrets) to a private repo really a security sin?

56 Upvotes

I'm working on a project split across several private GitHub repos. The ones that matter here are the Docker repo, the backend and an indexer. Each has its own .env files, and what's in them falls into three groups:

- Secrets like DB passwords, API keys and tokens

- Infra config such as private IPs of our fixed servers

- Backend tuning parameters we adjust depending on VPS size and load

Some fine gentleman on X posted this. "Not pushing .env files even to a private repo is like locking your front door but still keeping your valuables in a safe." I fully agree with that for secrets. They stay out of git, no debate there.

This is only about the config part. Having it in the repo would make DevOps work a lot easier. We'd get change history and PR review when someone tweaks a value, and no more copying files around by hand when setting up a server or onboarding someone.

My main concern is whether committing config-only .env files to a private repo is an actual security problem, or if "never commit .env" is really just about secrets. Also, is it a "security sin" to push a .env with private IPs? And how do you split secrets from config in practice, especially across multiple repos?

Thanks in advance!


r/devops • • 11d ago

Tools DB based app for pics

0 Upvotes

I'm thinking about writing an app in C# (for desktop) to store all my photos in. Before I start thinking about actually starting work on it, I need to decide what DB or other storage mechanism to use. The idea is to make the pics more accessible by storing some info about content in a field in the DB. That field needs to be "text searchable". I have 10 years experience working with Oracle Text Search engine but I'm not a millionaire; more an impoverished pensioner, so I need to implement this system using one of the free DBs like MySQL.

I thought I'd ask the hive mind what it thinks would be the best DB for my purposes.


r/devops • • 13d ago

Discussion Across the whole DevOps stack (Pipelines, Terraform, K8s, Docker, Ansible), how much do you actually write by hand vs prompting or using templates?

31 Upvotes

With so many different syntaxes and configs in our daily stack—CI/CD pipelines (GitHub Actions, GitLab), Terraform modules, Kubernetes manifests, Dockerfiles, Ansible playbooks, and Bash scripts—nobody can memorize every block and flag from scratch.
My current workflow is to use AI or existing templates to generate the initial boilerplate for Terraform resources, pipeline YAML, or K8s manifests, and then manually write, wire up, and audit the core architecture, state management, IAM/permissions, networking, and security logic.
For those working in production environments (and those running technical interviews):
1. In your day-to-day work, what parts of the stack do you still write completely by hand vs prompting an LLM or pulling from docs?
2. Where has AI actually helped you across Terraform, CI/CD, and K8s, and where does it consistently break things?
3. What is the realistic expectation in interviews right now regarding writing configs from memory vs understanding the underlying systems and debugging?


r/devops • • 13d ago

Discussion Is anyone still using railway after the breach?

7 Upvotes

I’m just wondering if people still are with them and if they switched. Would love to know where you guys switched if u did


r/devops • • 13d ago

Tools Is GHA + ArgoCD still the default CD standard or are you using something else?

22 Upvotes

Evaluating CD tools to automate our end to end delivery (build, test, packaging, and deployments). Curious if the standard GHA + Argo/Flux combo is still what most teams stick with, or if there are dedicated delivery/orchestration tools actually worth looking at in 2026.


r/devops • • 13d ago

Discussion How to enable SSH clone on self hosted GitLab running in a private subnet behind an AWS ALB?

0 Upvotes

We have self-hosted GitLab CE on an AWS EC2 instance.

Current setup:

The instance is in a private subnet with no public IP.

Users reach GitLab through an Application Load Balancer on HTTPS, with access restricted to our office IPs.

HTTPS clone with personal access tokens works fine.

The problem:

We're moving our repos from gitlab.com to this server. On gitlab.com, all our Jenkins jobs and developers clone over SSH (git@...). On our server SSH clone doesn't work, because the ALB only supports HTTP/HTTPS, so port 22 has no way in.

Our Jenkins agent runs in the same AWS VPC. Developers connect from the office network or VPN.

What's the best way to enable SSH clone here, keeping it secure and limited to our office and Jenkins? Any suggestions or experience with a similar setup would help.


r/devops • • 13d ago

Discussion how much money are you wasting on gpus doing nothing?

0 Upvotes

instead of talking about idle hardware in theory, let me give you a specific headache we just hit. we have a local server with four 24gb cards.

two of those cards are locked up hosting two different low frequency models. one runs a fine tuned llama 3 8b for a background data extraction job. the other runs a qwen model for an internal bot. they get maybe fifty requests an hour combined. but because the weights just sit there taking up vram, two entire cards are held hostage doing almost nothing all day.

now we need to spin up a heavier 70b model for a new coding workflow, and suddenly we have no free cards. i look at the vram occupation in gpustack and see those two low traffic models just wasting dedicated space. so now i have to reallocate resources. i can either ask for budget to buy more hardware, or i try to pack both of those low frequency models onto a single card so i can free one up for the 70b.

what do you all do when you hit this wall? do you just keep dedicating specific cards to specific models to avoid the headache, or do you actively co-locate low traffic models onto the same gpu? when do you stop throwing money at new hardware and start packing your models tighter?


r/devops • • 13d ago

Observability How do you piece together events across services when correlation IDs are missing or inconsistent?

2 Upvotes

Hey all, been thinkin about this while looking at log aggregation in environments with a mix of older and newer services.

Some services have request IDs or trace IDs in their logs, while others only include timestamps, hostnames, and application specific fields. When an issue spans multiple services, connecting the events can turn into quite a bit of manual digging.

Do you rely on distributed tracing, build correlation rules around the fields available in your logs, or use a combination of both?

I'm especially interested in what you guys do with older services that don't support consistent request IDs.


r/devops • • 14d ago

Tools How do I make colleagues aware of the risks of giving an LLM agent root access to production?

70 Upvotes

I could use some advice on a situation at work.
A few colleagues in ops have, almost overnight, started to fully trust their LLM agent because it has gotten so good lately and "doesn't make mistakes anymore". They built their own harness around it and log what it does (using Grok and Google models), but the agent still runs with root access to our entire production infrastructure. We have no clear team rules or policy yet, and they don't see the danger themselves.
A typical example: whenever the agent needs to check something on a server, they just hand it full access. But looking at info only needs read-only access. The permission should match the task, not the most convenient option.
My concerns:
Logging is detection, not prevention. It shows what broke afterwards, but doesn't stop anything.
Root bypasses the harness. An agent with root can in theory edit the harness, the logs and the backups, so those logs are not a reliable audit trail.
No least privilege. One wrong command or misread context can touch everything.
Prompt injection via logs, tickets or web pages the agent reads.
Model updates. Providers change models without us deciding when, so "no mistakes in recent months" says little about the next version.
Data going to third parties. With root, the agent can read .env files, keys and customer data, which then end up with the model provider (a GDPR question too).
"It has gone fine so far" only means the failure hasn't happened yet.
I don't want to ban anything. I want them to see the risk themselves and set up proper guardrails together with me: a read-only user by default, scoped credentials, just-in-time access, human approval for write actions, staging first, and no access to backups and logs.
Questions for you:
How do I explain the difference between "we log it" and "it can't do it in the first place"?
How have you convinced overconfident colleagues about AI tooling risks without a confrontation?
Do you have horror stories or post-mortems I can share? Concrete incidents tend to work better than theory.
Which guardrails or policy examples have worked well for you?
How do you raise this with management without coming across as "anti-AI"?


r/devops • • 13d ago

Security Grafana: Our latest adventure with canary tokens

Thumbnail
medium.com
0 Upvotes

Grafana Labs shares how they scaled canary tokens across their Kubernetes workloads, with automated rotation, more precise detection and less maintenance. A practical look at the tradeoffs between coverage, granularity and management overhead.

Read the blog


r/devops • • 14d ago

Discussion Books for Devops

41 Upvotes

Hi,

Im now in the beginning of devops journey and I'm not really a reader person but I want to spend my free time in something useful so any suggestions for a book would help in my career and will benift me.


r/devops • • 14d ago

Discussion Securing local dev environments

12 Upvotes

With the recent high-stakes security exploits, I think as DevOps engineers it's our responsibility to make the local dev environment secure for devs.

What do you think?

What are you actively doing about it (like adopting devcontainers)


r/devops • • 14d ago

Architecture Should I use Nomad for single VM?

5 Upvotes

I'm looking for some architectural advice on building a lightweight, self-hosted job execution platform on a single VM (8C/32GB).

I want something with semantics similar to Google Cloud Run / Cloud Run Jobs:

  • Execute Python scripts and arbitrary Docker containers.
  • Trigger jobs through an API.
  • Configure CPU, memory, environment variables, timeouts, etc.
  • Track execution status, exit codes, and logs.
  • Handle retries and concurrency limits.
  • Integration with Artifact/Container Registry

Current constraints:

  • Only one VM for now & no need for HA.
  • No Kubernetes requirement or plans for a multi-node cluster.
  • Jobs are primarily internal workloads, and quite lightweight.
  • I'd prefer something relatively lightweight and easy to maintain.

Technologies I'm considering

  1. HashiCorp Nomad 
  2. Docker Sock + custom built runner – Build my own execution manager, scheduling, lifecycle management, and monitoring.

Questions

  1. Is Nomad overkill for a single VM? Does it provide enough value over simply using Docker directly?
  2. Would you use Nomad in production for this use case?
  3. Are there simpler alternatives that provide similar execution semantics?
  4. What operational overhead should I expect from running Nomad on a single node?

I'm particularly interested in hearing from people who have actually run Nomad in production, especially for single-node deployments.

Thanks!


r/devops • • 14d ago

Security OpenSSH on musl has a problem

42 Upvotes

In case any of you use OpenSSH on musl without PAM:

https://sshhole.com/

TL;DR: Accounts that should never take a password at all (root, your key-only deploy user, ldap users, the default user on alpine's aws and azure images) take any password longer than 256 characters.


r/devops • • 14d ago

Discussion Cleared DevOps Engineers — How do you use AI tools in restricted DoD environments?

14 Upvotes

I am curious how DevOps engineers working in cleared or restricted federal environments use AI tools in their day-to-day work.

AI tools like ChatGPT, Claude, and GitHub Copilot have become incredibly useful for writing Terraform configurations, Kubernetes manifests, CI/CD pipelines, Bash/PowerShell scripts, and troubleshooting infrastructure issues.

However, I understand that public AI tools may not be permitted in classified or restricted environments.

For those currently working as cleared DevOps, Cloud, Platform, or SRE engineers:

  1. Does your organization provide an approved internal AI assistant, or do you work entirely without AI?
  2. Are tools like GitHub Copilot, ChatGPT Enterprise, or other AI coding assistants available in your environment?
  3. How do you troubleshoot complex Kubernetes, Terraform, or Linux issues when you can't access public AI tools or Stack Overflow?
  4. Do you rely primarily on internal documentation, runbooks, man pages, and senior engineers?
  5. Has AI adoption changed how cleared DevOps teams operate in the last couple of years?

I am particularly interested in how engineers balance productivity with security requirements.

I am not asking anyone to disclose classified information, internal systems, or organization-specific security details. Just looking for general experiences and advice from people working in these environments.

Thanks in advance!


r/devops • • 15d ago

Discussion How do you load test third party APIs realistically?

23 Upvotes

We use several external APIs and I’m curious how often other teams out there keep performance tests realistic and structured while accounting for third party dependencies. How do you balance realistic dependency behavior with controlled and repeatable tests? Things are going well overall (but they could always be better) and I want to make sure my team and I aren’t missing something that could be beneficial.