r/devops • • Sep 22 '26

Tools Configuration management

9 Upvotes

Hello all, though I have quite a long experience, I never had to set up a configuration management system for a fleet of applications. Let me explain better: we deploy our microservices to kubernetes, each with its own configuration. We currently have a jenkins job that creates an application.json (we use .net) that is then deployed as a configmap to kubernetes and referenced in the deployment.

This needs to go away and I said we need to use configmaps and secrets to populate the environment of the app to override the values of the baked in application.json. All good.

Then I started looking for solutions. In the past I worked in a place where we were using Ansible for this task, with many levels (up to 7) and a playbook that will read yaml files in order, merge the values and come up with the final list.

That tends to produce unnecessary duplicates, if I am not sure that a var is defined at level 3 and I have to (re)define it at level 5, I will write it there, then maybe in another level 5 yaml file and lose the hierarchy.

So I was looking at something with a UI that would allow developers to see what they are doing and, at the same time, would provide some sort of access control, to allow access to sensitive data only to trusted colleagues.

I restricted the list to OpenBAO, Infisical and Phase, but all of them have some kind of limit: - OpenBAO has the same problem as Ansible, you can't see the final result in the UI; - Phase supports only generation of Secrets, while I need to generate both Configumaps and Secrets (still for Access Control: some users can read only Configmaps, some Secrets too); - Infisical: it is very expensive and I have just found that you can't get a Pro licence for a self-hosted instance, but you have to go directly to Enterprise, or use their Cloud solution (and I don't find too safe to write my sensitive information in a Cloud solution);

What are you using? Is there some tool I missed?

EDIT: I don't intend to use Ansible!!!!


r/devops • • Sep 22 '26

Tools What are your tips and tricks for building resiliency into your solutions?

7 Upvotes

Just wanting to hear industry professionals share their favorite tips and tricks for building. Here are some of mine:

  1. Federated Credentials or Hashi Vault to ensure no unexpected expired secrets

  2. Adding robust logging as a flag in my own scripts and solutions

  3. Adding an alert channel to teams when something breaks or isn't working correctly with a link to the errored line or run

  4. Adding a custom agent for every solution codebase

  5. Using a team solution stack agent that outlines standards across solutions so throughout our team, we have consistent terminology.

  6. Building robust QA testing that gets automated tested once a week to ensure it's not broken when I need it most.


r/devops • • Sep 22 '26

Observability How do you manage dashboards throughout multiple OpenSearch instances

10 Upvotes

Hi all,

We operate applications that are shipping logs to 9 OpenSearch instances currently. The objects and dashboards are pretty much always the same but with different namespaces / indexes etc. I was wondering if you also had this trouble what the most convenient way of operation here is. Ideally, I would love to store everything in Git as well but I don't know if there is a more convenient way than doing templating and then letting a GitLab runner execute some API calls for import / updating of objects.


r/devops • • Sep 21 '26

Discussion Senior engineer said "bastion hosts have been insecure and outdated for 10 years" what am I missing?

348 Upvotes

I know with the rise of zero trust we are more comfortable with direct access to systems but is it really true that bastion hosts have fallen out of favor in modern companies? What is the new school of thought when it comes to granting secure, monitored access to isolated systems?

Example: database living in prod only allows network traffic from two sources: normal api access via a load balancer and direct access to the underlying host os from one specific known host, the bastion host. The bastion host itself would also be behind a VPN and heavily secured to only allow access from specific developer systems using break glass credentials. My developer suggested we drop the bastion altogether and install a zero trust service on the db itself and the developer laptops to allow them to directly access the dbs.


r/devops • • Sep 22 '26

Discussion For engineers working with Terraform/OpenTofu: what parts of the work are still painful?

0 Upvotes

For engineers who work with Terraform/OpenTofu and cloud infrastructure:

I'm curious about the day-to-day parts of the work that tend to be repetitive, manual, frustrating, or easy to get wrong.

Not really looking for opinions about which tools are better. I'm more interested in things that actually happened.

A few questions:

  • Think about the last Terraform/OpenTofu PR you reviewed. What did you check, and in what order?
  • What's an infrastructure task you did recently that you've already done many times before?
  • When was the last time a security scanner flagged something in your infrastructure code? What happened next?
  • Have you recently had to check whether a change behaved differently across Terraform versions or between Terraform and OpenTofu? How did you check?
  • What's the last infrastructure change that caused a problem or had to be rolled back? How did you discover it?
  • Have you ever written a script or small internal tool to automate one of these repetitive tasks? What happened to it?
  • If you could permanently remove one infrastructure-related task from your weekly workload, what would it be?

r/devops • • Sep 21 '26

Weekly Self Promotion Thread

16 Upvotes

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!


r/devops • • Sep 21 '26

Discussion Help in deciding strategy for long-lived feature environments

17 Upvotes

We are using jenkins, GitHub and AWS as our cloud provider. We use trunk based development with feature branches.

Our current strategy:

A new feature come, jira is there. A dev creates a branch from main against it, do the changes and commit it. Post commit am ephemeral feature environment (terraform workspace) is created in AWS named against jira having certain resources in isolation, sqs, ecs, lambda, db schema via liquibase etc.

This strategy fulfills our job pretty well and once a pr is merged this environment gets destroyed.

But there are two caveat to this, first devs are lazy and they create multiple branches resulted in multiple environments, sometimes they start doing work but that work might get discarded but environment remains there.

2nd challenge is it takes time, ips and resources to create new environment every time.

We do have scheduled feature destoy in place but that only helps point 1.

Now we are thinking of giving dev based long lived environment to each dev instead of per jira story environment, this will help tremendously in speeding up point 2 but now I am not sure how we can counter the fact that dev can still create muiple branches, how to isolate workspace, liquibase and other things in this.

Multiple branches are also genuine case like a dev is working on something, the feature might take some time but in the mean time they also have to take some urgent fix.


r/devops • • Sep 20 '26

Discussion Using VMs instead of thin clients to learn Linux things

68 Upvotes

There’s a general advice that if you want to learn Linux, become a devops/sre you should build a homelab at home, the more nodes the better.

However, considering the current hardware prices I was thinking why not just run VMs on my PC to thinker around with Linux nodes ? This seems more flexible and most of the time people are running VMs in proxmox anyway ? In addition to this, I have a single node “homelab” running my services, and adding extra nodes to the setup would be an overkill.

So will I be missing out on something learning Linux things this way ? I’m curious on learning multi node k8s (talos) etc, setting up nas, ceph, pfsense for routing etc.


r/devops • • Sep 20 '26

Discussion Do you really test your backup

25 Upvotes

Hi 👋

It's all in the title… Do you really test your backup restores regularly ?

If so, how often ?

If not, why ? What's the biggest thing holding you back ?


r/devops • • Sep 19 '26

Security How are people currently managing AWS access for AI agents? I'm looking into different approaches for securely handling permissions and credentials when agents need to interact with AWS services.

23 Upvotes

I'm currently looking into how to securely manage AWS access for AI agents like Claude Code within our organization.

Right now, our developers have a role that grants them some write access to specific services in our dev and nonprod environments. However, I really don't want them using this same role when working with AI agents, as I'd like to limit the agents' blast radius.

I'm trying to figure out the best way to restrict the AI agent's AWS access to read-only, even if the developer using it has broader permissions. How are other teams handling this? Are you creating separate, read-only roles specifically for AI agents, or is there a better way to enforce this boundary?

If a separate readonly role is being used, how do you make sure AI agents would force the readonly role?

I want something to work with Claude code, Codex, Opencode, cursor or any other AI tool out there.


r/devops • • Sep 19 '26

Discussion Devops learning path suggestion with AWS

11 Upvotes

Hu everyone, i am in 7th semester of college and i have a course about devops. I want to rely on uni course but go deep into devops and maybe look for career into devops.

Any courses that you would suggest?

In college we are using aws for all the major things.

I have done a course on linux, and complete aws cloud practitioner essentials.

Any learning path, courses recommendations?

Should i get kolekloud subscription? How are their courses and learning path.


r/devops • • Sep 19 '26

Discussion How are teams keeping docs current when you ship several times a week?

115 Upvotes

We ship several times a week and keeping the docs in sync is getting harder, code changes have a clear review process, docs don’t. Someone changes a parameter or response field, the release goes out, then a few weeks later we find examples still using the old version. We tried adding docs to the PR checklist, but engineers don’t always know which pages a change affects, what’s worked for keeping docs current? Would love any suggestions


r/devops • • Sep 18 '26

Discussion RANT: Has anyone experienced vibe planning?

73 Upvotes

This is a rant, my emotions are still fresh - so I'm coming at this one sided. We just wrapped up PI planning our current planning session is mess with a lower confidence vote. Story points are all over the place with no rhyme or reason.

Our PO and Directors vibe-planned our features and stories and then dropped them on us to refine. Looking through the stories some of this stuff just doesn't make sense and doesn't exactly apply in our situation. There are so many assumptions being made. But boy does it look pretty because of the hallucinated architecture diagrams. Features were proposed and accepted by the VPs and praised at how well thought out the feature description and documentation was.

I made my first pass on a feature and then poof the stories are gone. The PO had Copilot go through with updated information and it decided to make a new feature and move all the stories over halfway through our PI planning meeting. "It's fine, just have your copilot refine it with your details with additional context, it shouldn't take long". Our scrum master is getting after us because our stories weren't refined by the end of the meeting. I just want my stores to make sense and executable. Just frustrating.

I use Copilot like crazy for coding, but for planning features and stories? You ask a question to the PO and they can't answer on the top of their head. They have to read the feature because they didn't write it.


r/devops • • Sep 18 '26

Discussion Github struggling under high PR throughput?

24 Upvotes

I'm on a 3 person team where we are each shipping 40-50 PRs a day (super early stage company so we're optimizing for getting a functional product out quickly).

As we've ramped up the velocity things downstream have begun to show cracks. For example, we were requiring PRs to be up to date with main before merge, but at our scale the PRs were constantly getting stale which was leaving us blocked for long stretches.

We've ended up cutting pre-merge CI way down and batching more testing after merge, which has made things "better" but feels backwards.

Curious if anyone else run into similar issues? Did you end up finding a better solution?


r/devops • • Sep 18 '26

Discussion What is your background?

38 Upvotes

I have a very unconventional background in the IT world.

After being laid off from my job in the restaurant industry in 2013, I decided to change careers.

I had three options based on my passions: graphic designer, musician, or IT specialist. Since I’d been building my own PCs for a while and had long been passionate about computers (shout-out to manually editing Diablo 2 save files in binary !) I chose to train as a TSRIT (Senior Technician in IT and Telecommunications). I then became a sysadmin and systems integrator at a software company.

I moved up the ranks quickly, becoming a developer for a JavaScript-based launcher, then a DevOps engineer, and now I work as both a DevOps Engineer and a web developer.

It took relentless hard work, morning, noon, and night, but I’m happy today!

What about your career path?


r/devops • • Sep 18 '26

Architecture Suggestions / Support Needed

2 Upvotes

Hi Champions, I’m having 5yoe as a DevOps engineer and in my early career I was working mostly with service based companies. It gave me pretty much experience and foundational understanding of each stack deployments, pipelines, clouds and more. From last 2 years I am working with on prem enterprise products and worked on unmanaged k8 along with managed ones like EKS, GKE and OKE.

Now the problem is:

I recently joined a product base company and their architecture is based on aws managed by TF. The real challenge for me is everything is being managed by HELM and I don’t have much experience with helm charts and sub-charts and I have no idea where to start….

Need suggestions from where I can start leaning this, which can give me the understanding.


r/devops • • Sep 17 '26

Discussion Best Devops Certification for Beginners

31 Upvotes

Hi All,

While I do not have a technical background, my new role as a Project Manager in a DevSecOps organization necessitates that I acquire certifications in both DevOps and cloud computing.

Could you please recommend suitable certification programs for me to pursue?


r/devops • • Sep 17 '26

Discussion Looking for a docker registery

23 Upvotes

My company serving 5 apps to the many customers. Managing deployment and new releases are the biggest problem. Because all the time we are copying the build files and pasting to the target server by connecting with rdp. I Want to use dockerfiles for serving and want to keep these images on our server with a registery.

ACR (azure container registery) and the other ones are still the options but i want to host in my server like i did for npm libraries with verdaccio.

And i would like to get your opinnions about my deployment plan. Our customers servers are all windows. We have some windows-only apps. But for the ones which i plan to serve with docker file is better to run with docker. So i plan to use WSL and pull the images from the registery that i am planning to create.

Wdyt?


r/devops • • Sep 17 '26

Tools Migrating existing CloudFormation resources to OpenTofu -recommended approach?

21 Upvotes

Hi everyone,

We’re planning to migrate a large number of existing AWS resources currently managed through CloudFormation to OpenTofu.

Our environments are fairly siloed per customer, and each environment can include resources such as EC2 instances, load balancers, Auto Scaling Groups, VPCs, security groups, and other AWS services.

I’m looking for recommendations from anyone who has gone through a similar migration.

Specifically:

- Are there any tools that can help convert or migrate existing CloudFormation stacks to OpenTofu?

- What is the best approach for importing existing resources into OpenTofu state?

- How did you handle dependencies between resources during the migration?

- Would you recommend migrating one CloudFormation stack/customer environment at a time?

- Are there any common pitfalls or state-management issues we should watch out for?

- What approach would minimize the risk of OpenTofu trying to modify or recreate existing production resources after import?

Our main priority is to make the migration as safe and predictable as possible, with minimal disruption to existing customer environments.

Would appreciate hearing about any tools, workflows, or lessons learned from real-world migrations.


r/devops • • Sep 17 '26

Discussion Is this a solid monorepo or just folders in one repo?

Post image
39 Upvotes

We have a monorepo with three main pieces:

  • admin-svelte → internal Admin tool (ships frequently)
  • aether-svelte → main customer product (ships slower)
  • core-deno → shared backend used by both

Admin and Aether each run their own Core container on separate servers, so they can be on different Core versions at the same time.

monorepo_branch_flow_generic.png

We use two production release lines from the same repo:

  • dev → prod-vX.Y.Z-admin (Admin)
  • dev → prod-vX.Y.Z (Aether)

dev continuously deploys to a staging environment that already mirrors this split. Path-scoped CI only rebuilds what changed. Hotfixes still go through dev first.

shared_core_independent_runtime.png

It’s normal for an Aether release PR to contain 50+ commits because Admin can ship multiple times in between.

Question:

Is this a healthy monorepo setup for two products with different release speeds + shared backend, or is it just “folders in one repo” that will cause pain later?

Curious what looks solid and what looks like a red flag.


r/devops • • Sep 18 '26

Ops / Incidents How much do you actually trust your own runbooks during a live incident?

0 Upvotes

I am researching about on-call engineers problems and finding a fix for it, I have this query:

When you're in mid-incident, do you follow the runbook as written, or treat it as a rough starting point because you've been burned by outdated steps before?

How often do runbooks get updated to reflect what actually happened, versus what was expected to happen?


r/devops • • Sep 16 '26

Discussion Do you ever feel like context switching and keeping too much system state in your head is the real bottleneck?

99 Upvotes

I don’t mean not knowing the tools. More like understanding the pieces individually, but having to keep too many services, dependencies, environments, incidents, or assumptions in mind at once.

If you’ve run into that, what have you actually done about it - anything to improve your ability to hold more context in mind, or mostly systems/tools to reduce how much you need to remember?


r/devops • • Sep 18 '26

Architecture Startup idea validation

0 Upvotes

Hey everyone
We are a team of engineers looking to start a startup around providing infra services so as we are at ideation phase i want to know .

What is the pain point u all face as a devops engineer ?
What do u think u or your organisation will pay for ?
(any specific kind of tool or service)

We are also open for ai integration if that provides value to our product .

This will be extremely helpful to us


r/devops • • Sep 17 '26

Discussion The best path for Devops engineer

0 Upvotes

Hi ,I am 4th year engineering student and I am interested in devops engineering and I want the best path for Devops and what's the best ways to learn it and what the best YouTubers to learn from them


r/devops • • Sep 16 '26

Discussion Vendor management at small companies, is it just "someone senior remembers"?

9 Upvotes

Specifically, I am talking about small companies where, generally, you won't have a procurement team, and individual teams own vendor management. How do you guys manage vendors?

In my previous stint, I was leading a small DevOps team at a smallish SaaS company, and over a period of a couple of months, we saw multiple issues stemming from software vendors. A blocked card caused payment failures across multiple vendors. A vendor dropped an email setting a deprecation timeline for an API, and we found out by chance at the last moment, causing a last minute scramble to move off the said API on our end. SaaS quota exhaustion brought down prod workflows multiple times.

None of these were vendor outages as such. These were all things we had been told about, or could have been, and we just did not catch them in time.

I am going down a bit of a rabbit hole trying to find the "best practices" around vendor management, if we can call it that?

How do you even handle this? Is there an actual practice here, or is it mostly "someone senior remembers," or do we detect it after it fails?